JingsongLi commented on PR #10127: URL: https://github.com/apache/paimon/pull/10127#issuecomment-5805885107
This closes a real data-integrity gap: a same-commit-dropped snapshot partition must not act as the surviving baseline for a delta follower. I traced the regular truncate path and rollback path into the pre-callback: regular OVERWRITE commits request conflict detection over changed partitions, and rollback supplies the latest snapshot’s base entries; the new fully-dropped check distinguishes whole-partition loss from a partial file delete. The overwrite scope is restored in `finally`, so standalone drop/rollback keeps the stricter check. Local verification on the PR patch: all 17 `ChainTablePartitionExpireTest` cases passed on JDK 8, including the new reject/allow regressions. Current PR checks are green (16 successful, 2 skipped). I could not run the Spark integration suite locally because this host blocks Spark’s local socket bind; CI covers that lane. One scale concern for production: `fullyDroppedPartitions` streams the entire `baseFiles` list for every deleted partition, adding O(deleted partitions × base files) work to a batch drop/rollback. Please group base file identities by partition once, or show a large-partition benchmark that makes this cost acceptable. The correctness fix has clear end-to-end value. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
