hbgstc123 opened a new pull request, #8631: URL: https://github.com/apache/paimon/pull/8631
### Purpose Follow up on [#7595](https://github.com/apache/paimon/pull/7595) per [this review comment](https://github.com/apache/paimon/pull/7595#issuecomment-4814778161): sort compact should not be modeled as an OVERWRITE commit with extra base-snapshot handling. Instead, it should produce a normal COMPACT commit so the existing commit protocol can perform compact validation and conflict detection. This PR fixes the concurrent-write data loss risk in sort compact by refactoring Flink/Spark sort compact for **append tables** to: 1. Write sorted output in write-only mode. 2. Rewrite the written append commit messages into CompactIncrement messages. 3. Commit them through the normal compact commit path. No new overwritePartition(baseSnapshotId) APIs or special overwrite conflict logic are introduced. This is a follow-up to [#8571](https://github.com/apache/paimon/pull/8571), scoped to append tables only per [review feedback](https://github.com/apache/paimon/pull/8571#issuecomment-4953835690). ### Tests Core - SortCompactCommitMessageRewriterTest - rewrite to compact messages - multi-bucket / cross-bucket rewrite - deletion-vector cleanup - hash-dynamic hash-index cleanup - snapshot sequence range preservation - inline compaction rejection - compact commit success detection, including expired snapshot gaps Flink - SortCompactCommitterTest - SortCompactActionForAppendTableITCase - latest snapshot kind is COMPACT - concurrent append between read and commit does not lose data - CommitterOperatorTest Spark - SortCompactSparkCommitTest - CompactProcedureTestBase updated to expect COMPACT -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
