JingsongLi commented on PR #10198:
URL: https://github.com/apache/paimon/pull/10198#issuecomment-5966220554
[P2] Select the first retained snapshot when qualifying watermarks repeat
(`CreateTagFromWatermarkProcedure.java:89`)
The expired-tag tie regression is resolved. There is another common tie case
in the new procedure's documented “first snapshot” contract: the retained
candidate can already be too late before the tag fallback runs.
At `f97d656738`, I committed five real stream snapshots with watermarks
`1000, 2000, 2000, 2000, 3000`, distinct rows `(1, 'row1')` through `(5,
'row5')`, and no pre-existing tags. The fixture uses the supported
`write-only=true` option to keep exactly five append snapshots. This call
creates a tag for snapshot **3**:
```sql
CALL paimon.sys.create_tag_from_watermark(
table => 'test.T', tag => 'boundary', watermark => 2000, time_retained =>
'1 d');
SELECT * FROM T VERSION AS OF 'boundary';
```
The persisted tag has snapshot ID 3 and the read returns rows 1, 2, and 3.
The first qualifying snapshot is **2**, whose dataset contains only rows 1 and
2. On a fresh table with the same snapshot chain, target `1999` correctly
chooses snapshot 2; target `2000` takes the binary-search equality shortcut and
chooses snapshot 3. Repeated watermarks occur during ordinary stream
checkpoints when the watermark stalls.
`SnapshotManager.laterOrEqualWatermark` stops on an equal midpoint instead
of continuing left. That helper behavior predates this PR, but this new Spark
API consumes its result and promises the first qualifying snapshot; the new tag
comparison cannot repair the result when there is no older qualifying tag.
Please make the retained search choose the earliest qualifying snapshot as
well, and add a retained repeated-watermark regression alongside the
expired-tag regression.
Validation: the two committed Spark 3 tests pass locally with normal Maven
checks, and both are also confirmed passing in the current Spark 4 CI log.
Seven additional actual Spark 3 SQL cases give six passes and this one failure.
An isolated control changing only the helper's equality branch to continue
searching left passes all seven cases, including expired-tag ties, the reverse
newer-tag tie, exact time-travel datasets, one-day tag metadata, and
missing-watermark cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
