The GitHub Actions job "Flink Connector Kafka CI" on flink-connector-kafka.git/FLINK-33484 has failed. Run started by GitHub user dalelane (triggered by dalelane).
Head commit for run: 9164260b9a74ac537ecab97ea5308a8aa395e628 / Dale Lane <[email protected]> [FLINK-33484][Connectors/Kafka] Correct offsets committed to Kafka Offsets that the Kafka source commits back to Kafka can lag the consumer's true position, because entries that the consumer reads but never delivers as records (e.g. transaction commit / abort markers, and records of aborted transactions) were not accounted for. The result is that the committed offset was stopping at the last delivered record, while the consumer had advanced past that. This wasn't impacting Flink processing guarantees, as restarts restore from checkpointed offsets, not offsets committed to Kafka. External tooling that uses the Kafka committed offset for monitoring would give the appearance of consumer lag. On low-traffic or bursty transactional topics, this would report lag that does not exist until new events are received. A partition holding only aborted transactions, read from the earliest offset, never had an offset committed at all. This is now changed to reconcile the offset to commit with the consumer's position: - KafkaPartitionSplitReader now tracks, per partition, the offset of the last record handed to the reader, and the consumer position observed at the end of each fetch() - At checkpoint commit, reconcileOffsetsToCommit advances the offset to commit to the consumer position when the reader is caught up to what it delivered, so the committed offset will reflect the non-record entries that the consumer read past. Reconciliation is skipped for partitions that aren't assigned and for running bounded splits - Splits that have not emitted a record still hold a starting offset placeholder such as EARLIEST_OFFSET, so KafkaSourceReader has no offset to commit for them. Their partitions are passed to the split reader separately, which commits the consumer position for them unless records have been fetched that the checkpoint may not include - The split reader remembers the offset it last committed for each assigned partition, and leaves a partition out of a commit rather than move its committed offset backwards. This can otherwise happen when a record is fetched between a checkpoint and its commit, so the offset can no longer be advanced. The offset is recorded when the commit is sent, so that a later commit cannot overtake one that is still in flight - A finished bounded split is committed at its stopping offset, rather than after its last emitted record, as it only finishes once the consumer position has reached the stopping offset. This only applies once the split holds a real offset: a split that is still at a starting offset placeholder may have been empty when it was assigned, with the consumer already past its stopping offset, so committing the stopping offset could move the offset backwards - Per-partition tracking state is cleaned up when partitions are unassigned - KafkaSourceReader records the committed-offset metric from the offsets that are committed (rather than those requested) so Kafka and Flink's committedOffsets metrics agree. Generated-by: Claude Code (Sonnet 5) Signed-off-by: Dale Lane <[email protected]> Report URL: https://github.com/apache/flink-connector-kafka/actions/runs/36937151384 With regards, GitHub Actions via GitBox
