JulianEberius commented on issue #17462: [SPARK-20050][DStream] Kafka 0.10 DirectStream doesn't commit last processed batch's offset when graceful shutdown URL: https://github.com/apache/spark/pull/17462#issuecomment-552943275 I have the same question. Why was this closed? It seems to make a lot of sense to me. I noticed the same issue in Kafka/Spark Streaming: offsets are only commited in the next batches compute() call. If there is no next batch (e.g. on shutdown), commits will be lost, and duplicates will arise on the next restart.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
