What exactly do you mean by kafka consumer reporting? I'd log the offsets in your spark job and try running
kafka-simple-consumer-shell.sh --partition $yourbadpartition --print-offsets at the same time your spark job is running On Mon, Nov 23, 2015 at 7:37 PM, swetha <swethakasire...@gmail.com> wrote: > Hi, > > We see a bunch of issues like the following in Our Spark Kafka Direct. Any > idea as to how make Kafka Direct Consumers show up in Kafka Consumer > reporting to debug this issue? > > > Job aborted due to stage failure: Task 47 in stage 336.0 failed 4 times, > most recent failure: Lost task 47.3 in stage 336.0 (TID 5283, > 10.227.64.52): > java.lang.AssertionError: assertion failed: Ran out of messages before > reaching ending offset 225474235 for topic hubble_stream partition 55 start > 225467496. This should not happen, and indicates that messages may have > been > lost > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/How-to-have-Kafka-Direct-Consumers-show-up-in-Kafka-Consumer-reporting-tp25457.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
