twthorn opened a new issue, #17551: URL: https://github.com/apache/iceberg/issues/17551
### Apache Iceberg version 1.11.0 (latest release) ### Query engine None ### Please describe the bug 🐞 Data loss is observed with the following configs & timeline. ### Configs - iceberg.kafka.auto.offset.reset: latest (default) - control topic retention.ms / retention.bytes: some constant (not infinite retention) ### Timeline 1. Coordinator A elected, healthy, commits control-topic offset 100, assigned -coord group 0 partition in consumer group 1. Coordinator B elected, healthy, commits offset 105, assigned -coord group 0 partition in consumer group 1. Coordinator A fails to exit (eg due to any cause in https://github.com/apache/iceberg/issues/16016) now a zombie, ie assigned no partitions, last control-topic offset in memory is 100 1. Coordinator A regularly commits control-topic offset 100 on each commit, zero table commits 1. The record at coord offset 100 is removed from kafka due to retention (bytes or time), while coordinator A still exists (ie Coordinator A's pod & thread exist without crashing) 1. During this time sawtooth commit pattern oscillating between old and recent offset (see https://github.com/apache/iceberg/issues/13995) 1. Coordinator B dies 1. Coordinator A writes its old control-topic offset 100 once more, last writer wins 1. Coordinator A dies 1. Coordinator C elected, attempts to read from control-topic offset 100, no longer in retention 1. Coordinator C Auto offset reset to latest, skips all control topic records up to tip, data loss occurs for all records between B's last commit and C's election ### Solution We propose to make the coordinator read the currently committed coord group offset before committing, and never commit an offset lower than it. This provides cross-process monotonicity on the shared group -coord offset. The result will prevent data loss (note this is distinct from the in-process local monotonic map proposed in https://github.com/apache/iceberg/issues/17340 which cannot prevent this data loss scenario). The only potential functionality it will remove is an operator wanting to rewind a coordinator. But typically that is not done (only source offsets ie worker offsets are rewound). ### Willingness to contribute - [x] I can contribute a fix for this bug independently - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
