krishvishal opened a new issue, #4516: URL: https://github.com/apache/iggy/issues/4516
Follow-up to #3537. ## Summary ListOffsets answers `UNSUPPORTED_FOR_MESSAGE_FORMAT` (43) to every timestamp except EARLIEST (-2) and LATEST (-1) (`gateways/kafka/src/protocol/handlers/list_offsets.rs`). ListOffsets v7, which adds `MAX_TIMESTAMP`, is not served. Who hits it: - AKHQ finds the first offset of each partition with `offsetsForTimes(0)`. It gets null, uses the end offset instead, and shows every topic as empty. - Flink `OffsetsInitializer.timestamp()` and Spark `startingTimestamp` fail the job. - The Java consumer's `offsetsForTimes` returns null with no error. `auto.offset.reset=by_duration` never resolves. Kafka Streams with `by_duration` seeks to the end and skips data. - franz-go 1.22 rewinds by time after `OFFSET_OUT_OF_RANGE` by default, so its partition gets stuck. librdkafka's `offsets_for_times` fails the whole call, and `kcat -o s@<ts>` aborts. - `kafka-consumer-groups --reset-offsets --to-datetime` and `--by-duration` fail. `kafka-get-offsets --time <ts>` prints nothing and exits 0. Iggy already polls by timestamp, and the lookup uses the broker append time (`core/partitions/src/journal.rs`). ## Scope - [ ] Answer a timestamp with the first offset whose append time is at or after it. Answer -1 for a time past the last record - [ ] Document the difference from Kafka on CreateTime topics, where Kafka searches producer timestamps. If clocks agree, the difference is small - [ ] Serve ListOffsets v7 with `MAX_TIMESTAMP` from the last record - [ ] Never answer an EARLIEST or LATEST lookup with an error or an offset of -1. The async Java consumer throws a `NullPointerException` on such an entry ## Acceptance criteria - [ ] `offsetsForTimes` in the Java consumer returns the expected offsets - [ ] `kafka-consumer-groups --reset-offsets --to-datetime` moves a group to the expected offsets - [ ] AKHQ shows the records of a topic ## References - [OffsetFetcherUtils.java:219-234](https://github.com/apache/kafka/blob/b15258c7398889248d7fbf2a2218cc01346ebf92/clients/src/main/java/org/apache/kafka/clients/consumer/internals/OffsetFetcherUtils.java#L219-L234) - AKHQ [AbstractKafkaWrapper.java](https://github.com/tchiotludo/akhq/blob/4440816b892070fb7ba651a40d09a9e8bffac503/src/main/java/org/akhq/modules/AbstractKafkaWrapper.java), Flink [KafkaSourceEnumerator.java](https://github.com/apache/flink-connector-kafka/blob/e47cf425dcd6bd8b8d8d7d65b1f740b05f00ee8e/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/enumerator/KafkaSourceEnumerator.java), Spark [KafkaOffsetReaderAdmin.scala](https://github.com/apache/spark/blob/4117871cba1f5dfa0339eda14b6682d8bab5a521/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderAdmin.scala) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
