Hi Community, Seeing the below message in the logs and Spark application is getting terminated. There is an issue with our Kafka service and it auto restarts during which leader reelection happens.
*Exception:* assertion failed: Beginning offset 34242088 is after the ending offset 34242084 for topic <topic_name> partition 2. You either provided an invalid fromOffset, or the Kafka topic has been damaged. After analyzing it, I understand that the problem is that the Spark consumer has an earliest offset which it processed and Kafka returns an old offset (correct me if I am wrong). Is this behavior intended? Is there a way we can handle these situations without letting the app die? Any inputs are greatly appreciated. Thanks, Ravi -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
