krishvishal opened a new issue, #4525: URL: https://github.com/apache/iggy/issues/4525
Follow-up to #3547, which documented DeleteRecords as not advertised. ## Summary EARLIEST is always 0, Fetch sends `log_start_offset` -1, and DeleteRecords (21) is not served (`gateways/kafka/docs/SCOPE.md`, "DeleteRecords is not advertised"). Who hits it: - Kafka Streams purges its repartition topics every 30 seconds with DeleteRecords. The calls fail with only a debug line. The topics are created with `retention.ms=-1`, so they never shrink. - `kafka-delete-records` and `Admin.deleteRecords` fail. - Lag and data-loss detection reads EARLIEST: Spark `failOnDataLoss`, KEDA with `offsetResetPolicy=earliest`, and message counts in UIs. - If Fetch ever answers `OFFSET_OUT_OF_RANGE` at offset 0, Streams restores loop. Fetch avoids this today, because it reads from the oldest kept record. ## Scope - [ ] A log start offset in Iggy that retention and deletion move, and that clients can read - [ ] ListOffsets EARLIEST and Fetch `log_start_offset` report it - [ ] DeleteRecords as a replicated watermark that Fetch respects, at offset precision ## Acceptance criteria - [ ] After retention deletes a segment, EARLIEST and `log_start_offset` report the first kept offset - [ ] `kafka-delete-records` moves the start offset, and a Fetch below it answers `OFFSET_OUT_OF_RANGE` (1) - [ ] The repartition topics of a Kafka Streams app shrink ## References - Streams restore loop: [StoreChangelogReader.java:554-563](https://github.com/apache/kafka/blob/b15258c7398889248d7fbf2a2218cc01346ebf92/streams/src/main/java/org/apache/kafka/streams/processor/internals/StoreChangelogReader.java#L554-L563) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
