gaborgsomogyi commented on issue #24218: [SPARK-27281][DStreams] Change the way latest kafka offsets are retrieved to consumer#endOffsets URL: https://github.com/apache/spark/pull/24218#issuecomment-476675186 I think we've not yet reached the root cause of this issue. What do you want to solve exactly and why do you think this change would solve it? In general I've seen issues like this when `unclean.leader.election.enabled` turned on. Let's say there are 2 brokers. On broker1 position is 5 which is the leader and on broker2 position is 4 because its not in sync. When broker1 goes down suddenly broker2 will be the leader and when asked for position it will give back 4. In general it would be good to hit the root cause before fixing it.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
