PDGGK opened a new pull request, #10934:
URL: https://github.com/apache/paimon/pull/10934
### Purpose
Fixes #10886.
When `rollback_to` targets a tag whose snapshot has already been expired,
pypaimon writes the tag's snapshot file back into the snapshot directory but
leaves the `EARLIEST` hint on the old earliest id, which now points past the
restored snapshot. Once new commits reach that id, readers take it as the
earliest snapshot: time travel and incremental reads skip the restored snapshot
and the commits after it. Java readers are affected too, because
`HintFileUtils.findEarliest` trusts the hint when that snapshot file exists,
and for the same reason later snapshot expiration never sees the skipped
snapshots.
The Java `RollbackHelper.createSnapshotFileIfNeeded` commits the earliest
hint right after it writes the snapshot file, and the docstring of pypaimon's
`create_snapshot_file_if_needed` says it does the same, but the write was
missing. This PR adds it. `rollback_to(<snapshot id>)` for an expired snapshot
that a tag still holds goes through the same tag path, so it is fixed as well.
Reproduction from the issue on master (30d9127). pypaimon has no snapshot
expiration, so expiration was run with the Java API from paimon-bundle 2.1.0,
the same `ExpireSnapshots` call that `CALL sys.expire_snapshots(retain_max =>
1, retain_min => 1)` makes. pypaimon 2.1.0 gives the same output.
Before:
```
java expire_snapshots deleted 4 snapshot(s)
snapshot dir after rollback_to('v2'): ['EARLIEST', 'LATEST', 'snapshot-2']
snapshot dir after 3 more commits: ['EARLIEST', 'LATEST', 'snapshot-2',
'snapshot-3', 'snapshot-4', 'snapshot-5']
EARLIEST = 5 (expected 2)
incremental-between-timestamp s2,s5: [8] (expected [6, 7, 8])
scan.timestamp-millis s3: ValueError: No snapshot found with timestamp
earlier than or equal to 1793600819432ms. (expected [1, 2, 6])
```
After:
```
EARLIEST = 2 (expected 2)
incremental-between-timestamp s2,s5: [6, 7, 8] (expected [6, 7, 8])
scan.timestamp-millis s3: [1, 2, 6] (expected [1, 2, 6])
```
On the Java side, with paimon-bundle 2.1.0 on tables left by the same script:
- an `incremental-between-timestamp` read of the same window returned `[8]`
before the fix and `[6, 7, 8]` after it;
- running the expiration again (retain 1) deleted 0 snapshots before the
fix, leaving snapshot-2 to snapshot-4 behind `EARLIEST = 5`, and deleted 3
after it.
### Tests
- `SimpleTableTest#test_table_rollback_to_tag_after_snapshot_expired`:
writes 5 commits, tags snapshot 2, removes snapshots 1-4 the way expiration
leaves the snapshot directory (files deleted, `EARLIEST` = 5), rolls back to
the tag and writes 3 more commits. It then checks the two reads from the issue
and the hint. With the change to `rollback_helper.py` reverted, it fails with
`[8] != [6, 7, 8]`.
- Ran locally on Python 3.12: `simple_table_test.py`,
`snapshot_manager_test.py` and `filesystem_catalog_branch_test.py` (48 passed),
and the rollback tests in `cli_table_test.py`, `catalog_environment_test.py`
and the REST tests (`-k rollback`, 60 passed). flake8 with `dev/cfg.ini` is
clean. I left the full suite and the other Python versions to CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
