Re: Reading kafka and save to parquet problem

I have ever tried to use readStream and writeStream, but it throw "Uri without authority: hdfs:/data/_spark_metadata" exception, which is not seen in normal read mode. The parquet path I specified is hdfs:///data Regard, Junfeng Chen On Thu, Mar 8, 2018 at 9:38 AM, naresh Goud wrote: > change

Re: Reading kafka and save to parquet problem

change it to readStream instead of read as below val df = spark .readStream .format("kafka") .option("kafka.bootstrap.servers", "host1:port1,host2:port2") .option("subscribe", "topic1") .load() Check is this helpful https://github.com/ndulam/KafkaSparkStreams/blob/master/SampleStreamApp

Reading kafka and save to parquet problem

I am struggling in trying to read data in kafka and save them to parquet file on hdfs by using spark streaming according to this post https://stackoverflow.com/questions/45827664/read-from-kafka-and-write-to-hdfs-in-parquet My code is similar to following val df = spark .read .format("kafka"