There is a known issue that Kafka cannot return leader if there is not data in the topic. I think it was raised in another thread in this forum. Is that the issue?
On Wed, Jul 29, 2015 at 10:38 AM, unk1102 <umesh.ka...@gmail.com> wrote: > Hi I have Spark Streaming code which streams from Kafka topic it used to > work > fine but suddenly it started throwing the following exception > > Exception in thread "main" org.apache.spark.SparkException: > org.apache.spark.SparkException: Couldn't find leader offsets for Set() > at > > org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413) > at > > org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413) > at scala.util.Either.fold(Either.scala:97) > at > > org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:412) > at > > org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:528) > at > > org.apache.spark.streaming.kafka.KafkaUtils.createDirectStream(KafkaUtils.scala) > My Spark Streaming client code is very simple I just create one receiver > using the following code and trying to print messages it consumed > > JavaPairInputDStream<String, String> messages = > KafkaUtils.createDirectStream(jssc, > String.class, > String.class, > StringDecoder.class, > StringDecoder.class, > kafkaParams, > topicSet); > > Kafka param is only one I specify kafka.ofset.reset=largest. Kafka topic > has > data I can see data using other Kafka consumers but above Spark Streaming > code throws exception saying leader offset not found. I tried both smallest > and largest offset. I wonder what happened this code used to work earlier. > I > am using Spark-Streaming 1.3.1 as it was working in this version I tried in > 1.4.1 and same exception. Please guide. I am new to Spark thanks in > advance. > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Kafka-could-not-find-leader-offset-for-Set-tp24066.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
