krishvishal opened a new issue, #4521: URL: https://github.com/apache/iggy/issues/4521
## Summary Iggy has no compaction. Tombstones (a key with a null value) are stored and served today. Once #4508 accepts `cleanup.policy=compact`, compacted topics work but grow without bound. Who hits it: every compacted topic. Connect reads its three internal topics in full at each start. A Kafka Streams restore replays the whole changelog. Schema Registry, CDC tables and KTables grow without bound. ## Open questions - Where the key lives: a field in the message frame, or a reserved header. The frame field is a wire change for every SDK and needs maintainer agreement - Cleaning on each replica, or a replicated trigger - Whether compaction waits for the lowest committed consumer offset, as retention does - What the cleaner does with records that have no key, after a topic changes to `compact` ## Scope - [ ] A key and a tombstone flag in the stored message - [ ] `cleanup_policy`, `delete_retention`, `min_compaction_lag` and `max_compaction_lag` as Iggy topic options. The option catalog takes new keys without a wire change - [ ] A cleaner that keeps the last record per key, run the same way on every replica - [ ] Offset gaps in writes, boot recovery and state transfer - [ ] Fetch and ListOffsets across gaps. Fetch must not answer `NOT_LEADER_OR_FOLLOWER` (6) for a range that compaction removed - [ ] Simulator coverage ## Acceptance criteria - [ ] A compacted topic keeps only the last record per key below the cleaned point, and a tombstone goes away after `delete.retention.ms` - [ ] A Connect restart and a Kafka Streams restore read the compacted log, and their results do not change -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
