The GitHub Actions job "Flink Connector Kafka CI" on flink-connector-kafka.git/FLINK-33484 has succeeded. Run started by GitHub user dalelane (triggered by MartijnVisser).
Head commit for run: a94226763cf24fca816c095b503ec2679f417b7f / Dale Lane <[email protected]> [FLINK-33484][Connectors/Kafka] Correct offsets committed to Kafka Offsets that the Kafka source commits back to Kafka can lag the consumer's true position, because entries that the consumer reads but never delivers as records (e.g. transaction commit / abort markers) were not accounted for. The result is that the committed offset was stopping at the last delivered record, while the consumer had advanced past that. This wasn't impacting Flink processing guarantees, as restarts restore from checkpointed offsets, not offsets committed to Kafka. External tooling that uses the Kafka committed offset for monitoring would give the appearance of consumer lag. On low-traffic or bursty transactional topics, this would report lag that does not exist until new events are received. This is now changed to reconcile the offset to commit with the consumer's position: - KafkaPartitionSplitReader now tracks, per partition, the offset of the last record handed to the reader, and the consumer position observed at the end of each fetch() - At checkpoint commit, reconcileOffsetsToCommit advances the offset to commit to the consumer position when the reader is caught up to what it delivered, so the committed offset will reflect the non-record entries that the consumer read past. Reconciliation is skipped for partitions that aren't assigned and for bounded splits - Per-partition tracking state is cleaned up when partitions are unassigned - KafkaSourceReader records the committed-offset metric from the offsets that are committed (rather than those requested) so Kafka and Flink's committedOffsets metrics agree. Generated-by: Claude Code (Sonnet 5) Signed-off-by: Dale Lane <[email protected]> Report URL: https://github.com/apache/flink-connector-kafka/actions/runs/36442404713 With regards, GitHub Actions via GitBox
