vbhanuchander-lang commented on issue #17340:
URL: https://github.com/apache/iceberg/issues/17340#issuecomment-5287748553
I checked the mechanism against current main and the structural part of your
diagnosis holds:
`Channel.consumeAvailable` records the position with a plain `put`, so the
map is last-write-wins
rather than a high-water mark:
```java
// Channel.java:126
controlTopicOffsets.put(record.partition(), record.offset() + 1);
```
Nothing in that method compares against the value already stored, so any
path that re-reads a
partition from an earlier position — a seek back to committed group offsets
on restart, as you
describe — moves the recorded offset **backwards**. The value then flows into
`commitConsumerOffsets()` and into the `kafka.connect.offsets` stamped on
the snapshot, which is what
makes the regression durable rather than transient.
Your suggested change is consistent with how the value is used everywhere
else, since every consumer
of `controlTopicOffsets()` wants "furthest position consumed", never "most
recently seen":
```java
controlTopicOffsets.merge(record.partition(), record.offset() + 1,
Long::max);
```
Two things I could not confirm from the code alone, and which I think are
the open questions for a
reviewer rather than objections:
1. Whether making the watermark monotonic is sufficient on its own, or
whether the replay can also
deliver an envelope whose offset is genuinely above the last committed
watermark — in which case
the min-offset filter passes it legitimately and only content-level
deduplication would catch it.
Your step 5 suggests the former, but it depends on exactly where the seek
lands.
2. Whether the bounded "recently committed locations" set you float is
wanted at all. That is a
design call for the maintainers, and worth separating from the one-line
fix so the fix is not held
up by it.
I have not opened a PR — you said you can put one up and it is your finding,
so it should be yours.
Flagging the verification here in case it helps it move, since this has been
open three weeks with
no response and the production evidence you gathered (149 files, ~112k
duplicated rows) makes it
worth someone's attention.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
