fenil25 opened a new issue, #13995: URL: https://github.com/apache/iceberg/issues/13995
### Apache Iceberg version 1.9.2 (latest release) ### Query engine Kafka Connect ### Please describe the bug 🐞 At times, coordinator's consumer group's offsets resets to an older value after making any progress. This is seen at random times and the coordinator either recovers or the the older offset expires from the control topic and the consumer group resets its offsets. Because of this coordinator has to do lot of work again. Example - Say coordinator is at offset v1 and the latest value is v2. Then cg consumer group would consume all records from v1-v2 and then instead of committing the value to v2, it will switch back to v1. When new commit is started, if the latest value is v3, then coordinator will consume records from v1-v3 and then again reset to v1. The offset for coordinator's consumer group follows a pattern like this (see attached image) - <img width="798" height="235" alt="Image" src="https://github.com/user-attachments/assets/14493e17-c7e5-4826-9c2a-7cf25d661fb7"; /> Any idea what is causing this? Because of this, we have seen lot of issues like irregular iceberg commits and high number of kafka rebalances because coordinator's consumer group times out. ### Willingness to contribute - [ ] I can contribute a fix for this bug independently - [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
