The GitHub Actions job "Flink Connector Kafka CI" on 
flink-connector-kafka.git/FLINK-33484 has succeeded.
Run started by GitHub user dalelane (triggered by MartijnVisser).

Head commit for run:
a94226763cf24fca816c095b503ec2679f417b7f / Dale Lane <[email protected]>
[FLINK-33484][Connectors/Kafka] Correct offsets committed to Kafka

Offsets that the Kafka source commits back to Kafka can lag the
consumer's true position, because entries that the consumer reads
but never delivers as records (e.g. transaction commit / abort
markers) were not accounted for. The result is that the committed
offset was stopping at the last delivered record, while the
consumer had advanced past that.

This wasn't impacting Flink processing guarantees, as restarts
restore from checkpointed offsets, not offsets committed to Kafka.
External tooling that uses the Kafka committed offset for
monitoring would give the appearance of consumer lag. On
low-traffic or bursty transactional topics, this would report lag
that does not exist until new events are received.

This is now changed to reconcile the offset to commit with the
consumer's position:

- KafkaPartitionSplitReader now tracks, per partition, the offset
  of the last record handed to the reader, and the consumer
  position observed at the end of each fetch()
- At checkpoint commit, reconcileOffsetsToCommit advances the
  offset to commit to the consumer position when the reader is
  caught up to what it delivered, so the committed offset will
  reflect the non-record entries that the consumer read past.
  Reconciliation is skipped for partitions that aren't assigned and
  for bounded splits
- Per-partition tracking state is cleaned up when partitions are
  unassigned
- KafkaSourceReader records the committed-offset metric from the
  offsets that are committed (rather than those requested) so
  Kafka and Flink's committedOffsets metrics agree.

Generated-by: Claude Code (Sonnet 5)
Signed-off-by: Dale Lane <[email protected]>

Report URL: 
https://github.com/apache/flink-connector-kafka/actions/runs/36442404713

With regards,
GitHub Actions via GitBox

Reply via email to