I mean to show the Spark Kafka Direct consumers in Kafka Stream UI. Usually we create a consumer and the consumer gets shown in the Kafka Stream UI. How do I log the offsets in the Spark Job?
On Mon, Nov 23, 2015 at 6:11 PM, Cody Koeninger <c...@koeninger.org> wrote: > What exactly do you mean by kafka consumer reporting? > > I'd log the offsets in your spark job and try running > > kafka-simple-consumer-shell.sh --partition $yourbadpartition > --print-offsets > > at the same time your spark job is running > > On Mon, Nov 23, 2015 at 7:37 PM, swetha <swethakasire...@gmail.com> wrote: > >> Hi, >> >> We see a bunch of issues like the following in Our Spark Kafka Direct. Any >> idea as to how make Kafka Direct Consumers show up in Kafka Consumer >> reporting to debug this issue? >> >> >> Job aborted due to stage failure: Task 47 in stage 336.0 failed 4 times, >> most recent failure: Lost task 47.3 in stage 336.0 (TID 5283, >> 10.227.64.52): >> java.lang.AssertionError: assertion failed: Ran out of messages before >> reaching ending offset 225474235 for topic hubble_stream partition 55 >> start >> 225467496. This should not happen, and indicates that messages may have >> been >> lost >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-have-Kafka-Direct-Consumers-show-up-in-Kafka-Consumer-reporting-tp25457.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >
