That means the consumer on the executor tried to seek to the specified offset, but the message that was returned did not have a matching offset. If the executor can't get the messages the driver told it to get, something's generally wrong.
What happens when you try to consume the particular failing offset from another (e.g. commandline) consumer? Is the topic in question compacted? On Tue, Jan 16, 2018 at 11:10 PM, Justin Miller <justin.mil...@protectwise.com> wrote: > Greetings all, > > I’ve recently started hitting on the following error in Spark Streaming in > Kafka. Adjusting maxRetries and spark.streaming.kafka.consumer.poll.ms even > to five minutes doesn’t seem to be helping. The problem only manifested in > the last few days, restarting with a new consumer group seems to remedy the > issue for a few hours (< retention, which is 12 hours). > > Error: > Caused by: java.lang.AssertionError: assertion failed: Got wrong record for > spark-executor-<consumergrouphere> <topichere> 76 even after seeking to > offset 1759148155 > at scala.Predef$.assert(Predef.scala:170) > at > org.apache.spark.streaming.kafka010.CachedKafkaConsumer.get(CachedKafkaConsumer.scala:85) > at > org.apache.spark.streaming.kafka010.KafkaRDD$KafkaRDDIterator.next(KafkaRDD.scala:223) > at > org.apache.spark.streaming.kafka010.KafkaRDD$KafkaRDDIterator.next(KafkaRDD.scala:189) > at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) > > I guess my questions are, why is that assertion a job killer vs a warning and > is there anything I can tweak settings wise that may keep it at bay. > > I wouldn’t be surprised if this issue were exacerbated by the volume we do on > Kafka topics (~150k/sec on the persister that’s crashing). > > Thank you! > Justin > > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
