I have run into similar excpetions ERROR DirectKafkaInputDStream: ArrayBuffer(java.net.SocketTimeoutException, org.apache.spark.SparkException: Couldn't find leader offsets for Set([AdServe,1]))
and the issue has happened on Kafka Side, where my broker offsets go out of sync, or do not return leader for this particular partition ./bin/kafka-run-class.sh kafka.tools.GetOffsetShell --topic AdServe --broker-list BROKER_IP --time -1 this shall return u valid offsets for all your kafka partitons On Thu, Jul 30, 2015 at 7:58 PM, Umesh Kacha <umesh.ka...@gmail.com> wrote: > Hi Cody sorry my bad you were right there was a typo in topicSet. When I > corrected typo in topicSet it started working. Thanks a lot. > > Regards > > On Thu, Jul 30, 2015 at 7:43 PM, Cody Koeninger <c...@koeninger.org> > wrote: > >> Can you post the code including the values of kafkaParams and topicSet, >> ideally the relevant output of kafka-topics.sh --describe as well >> >> On Wed, Jul 29, 2015 at 11:39 PM, Umesh Kacha <umesh.ka...@gmail.com> >> wrote: >> >>> Hi thanks for the response. Like I already mentioned in the question >>> kafka topic is valid and it has data I can see data in it using another >>> kafka consumer. >>> On Jul 30, 2015 7:31 AM, "Cody Koeninger" <c...@koeninger.org> wrote: >>> >>>> The last time someone brought this up on the mailing list, the issue >>>> actually was that the topic(s) didn't exist in Kafka at the time the spark >>>> job was running. >>>> >>>> >>>> >>>> >>>> >>>> On Wed, Jul 29, 2015 at 6:17 PM, Tathagata Das <t...@databricks.com> >>>> wrote: >>>> >>>>> There is a known issue that Kafka cannot return leader if there is not >>>>> data in the topic. I think it was raised in another thread in this forum. >>>>> Is that the issue? >>>>> >>>>> On Wed, Jul 29, 2015 at 10:38 AM, unk1102 <umesh.ka...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi I have Spark Streaming code which streams from Kafka topic it used >>>>>> to work >>>>>> fine but suddenly it started throwing the following exception >>>>>> >>>>>> Exception in thread "main" org.apache.spark.SparkException: >>>>>> org.apache.spark.SparkException: Couldn't find leader offsets for >>>>>> Set() >>>>>> at >>>>>> >>>>>> org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413) >>>>>> at >>>>>> >>>>>> org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413) >>>>>> at scala.util.Either.fold(Either.scala:97) >>>>>> at >>>>>> >>>>>> org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:412) >>>>>> at >>>>>> >>>>>> org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:528) >>>>>> at >>>>>> >>>>>> org.apache.spark.streaming.kafka.KafkaUtils.createDirectStream(KafkaUtils.scala) >>>>>> My Spark Streaming client code is very simple I just create one >>>>>> receiver >>>>>> using the following code and trying to print messages it consumed >>>>>> >>>>>> JavaPairInputDStream<String, String> messages = >>>>>> KafkaUtils.createDirectStream(jssc, >>>>>> String.class, >>>>>> String.class, >>>>>> StringDecoder.class, >>>>>> StringDecoder.class, >>>>>> kafkaParams, >>>>>> topicSet); >>>>>> >>>>>> Kafka param is only one I specify kafka.ofset.reset=largest. Kafka >>>>>> topic has >>>>>> data I can see data using other Kafka consumers but above Spark >>>>>> Streaming >>>>>> code throws exception saying leader offset not found. I tried both >>>>>> smallest >>>>>> and largest offset. I wonder what happened this code used to work >>>>>> earlier. I >>>>>> am using Spark-Streaming 1.3.1 as it was working in this version I >>>>>> tried in >>>>>> 1.4.1 and same exception. Please guide. I am new to Spark thanks in >>>>>> advance. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> View this message in context: >>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Kafka-could-not-find-leader-offset-for-Set-tp24066.html >>>>>> Sent from the Apache Spark User List mailing list archive at >>>>>> Nabble.com. >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>>>> For additional commands, e-mail: user-h...@spark.apache.org >>>>>> >>>>>> >>>>> >>>> >> >
