kumarpritam863 commented on issue #16282:
URL: https://github.com/apache/iceberg/issues/16282#issuecomment-5082450519
Yeah I got your concern but I do not see how that can happen as:
1. No seek, no intra-session rewind. There is no .seek() anywhere on the
control-topic consumer (grep confirms: the only seek is
KafkaUtils.seekToLastCommittedOffsets on the source
consumer in CommitterImpl.java:120, unrelated to the floor).
consumeAvailable only ever calls consumer.poll(), whose offsets are
monotonically increasing per
partition within a session. So within one Channel, controlTopicOffsets[P]
is monotonically non-decreasing — the plain put is effectively a max because
records arrive in order.
2. The only way to resume at a lower control offset is a new Coordinator,
and that resets both structures together:
- startCoordinator → new Coordinator(...) → new CommitState(config)
(Coordinator.java:114) ⇒ fresh empty commitBuffer.
- new Coordinator → super(...) → new Channel ⇒ fresh empty
controlTopicOffsets (Channel.java:54, a final field never cleared during a
Channel's life).
So the buffer and the offsets map are never from different sessions. At
commit time the invariant
▎ controlTopicOffsets[P] ≥ max(control offset of every buffered envelope
for P) + 1
always holds, and Long::max with committedOffsets only raises the floor
further. The recorded floor is never below the snapshot's own contents. Your t3
state (buffer = 0–7,
controlTopicOffsets = {0:2}) has no code path that produces it.
Walking your timeline
- t1 {0:8}, buffer 0–7 — reachable, normal.
- t2 "rewinds to 0", buffer still 0–7 — impossible. A rewind means a new
Coordinator/Channel, which gives you controlTopicOffsets = {} and an empty
buffer, not {0:8}/buffer-0–7 with a
rewound consumer. Consumption and buffering can't desynchronize.
- Everything from t3 onward inherits the impossible t2 state.
You're right that the validator wouldn't catch it
Your read of offsetValidator (from #14510) is correct — at t5 both
expectedOffsets and lastCommittedOffsets would be 2, so it passes. It guards
against a concurrent writer during retry,
not a self-inflicted low floor. It's simply not the thing preventing this
— the monotonic-consumption + co-reset invariant above is.
Also to handle the split-brain scenario which can cause multiple
coordinators or even no coordinator in some cases, I am opening a PR where I
will mention this.
Also can you properly give the timeline where the backward shift is possible
which I may be missing. If that is a real issue I would be more than happy to
include that in the same PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
