Hi I have Spark Streaming code which streams from Kafka topic it used to work
fine but suddenly it started throwing the following exception
Exception in thread "main" org.apache.spark.SparkException:
org.apache.spark.SparkException: Couldn't find leader offsets for Set()
at
org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413)
at
org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413)
at scala.util.Either.fold(Either.scala:97)
at
org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:412)
at
org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:528)
at
org.apache.spark.streaming.kafka.KafkaUtils.createDirectStream(KafkaUtils.scala)
My Spark Streaming client code is very simple I just create one receiver
using the following code and trying to print messages it consumed
JavaPairInputDStream<String, String> messages =
KafkaUtils.createDirectStream(jssc,
String.class,
String.class,
StringDecoder.class,
StringDecoder.class,
kafkaParams,
topicSet);
Kafka param is only one I specify kafka.ofset.reset=largest. Kafka topic has
data I can see data using other Kafka consumers but above Spark Streaming
code throws exception saying leader offset not found. I tried both smallest
and largest offset. I wonder what happened this code used to work earlier. I
am using Spark-Streaming 1.3.1 as it was working in this version I tried in
1.4.1 and same exception. Please guide. I am new to Spark thanks in advance.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Kafka-could-not-find-leader-offset-for-Set-tp24066.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
