plusplusjiajia opened a new pull request, #8788:
URL: https://github.com/apache/paimon/pull/8788
### Purpose
Two defects make a commit's file removals invisible to Iceberg's incremental
changelog scans.
A DELETED manifest entry records the wrong snapshot. Entries rewritten as
DELETED keep the snapshot id of the snapshot that added the file, while the
spec requires the snapshot in which the file was deleted.
BaseIncrementalChangelogScan resolves each entry's snapshot id to a change
ordinal, so such a delete is dropped and the scan reports the rewritten files
as pure additions. EXISTING entries keep their original snapshot id; only
DELETED ones are restamped.
Manifest compaction then discards those entries. compactMetadataIfNeeded
identifies the current commit's entries by file sequence number, which for a
deletion still points at the older snapshot that added the file, so every
DELETED entry was dropped -- including the ones the same commit had just
recorded. Deletions are now also matched by snapshot id. Deletions carried over
from older snapshots are still dropped, as before.
### Tests
- testDeletedEntryRecordsTheDeletingSnapshot -- deletions carry the
current snapshot id.
- testCurrentDeletesSurviveManifestCompaction -- deletions survive a
commit that also compacts manifests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
