hbgstc123 opened a new pull request, #8566: URL: https://github.com/apache/paimon/pull/8566
### Purpose Follow up on [#7595](https://github.com/apache/paimon/pull/7595) per [this review comment](https://github.com/apache/paimon/pull/7595#issuecomment-4814778161): sort compact should not be modeled as an OVERWRITE commit with extra base-snapshot handling. Instead, it should produce a normal COMPACT commit so the existing commit protocol can perform compact validation and conflict detection. This PR fixes the concurrent-write data loss risk in sort compact by refactoring Flink/Spark sort compact to: 1. Write sorted output in write-only mode. 2. Rewrite the written append commit messages into CompactIncrement messages. 3. Commit them through the normal compact commit path. No new overwritePartition(baseSnapshotId) APIs or special overwrite conflict logic are introduced. ### Tests Core - SortCompactCommitMessageRewriterTest - rewrite to compact messages - multi-bucket / cross-bucket rewrite - deletion-vector cleanup - hash-dynamic hash-index cleanup - snapshot sequence range preservation - inline compaction rejection - compact commit success detection, including expired snapshot gaps - ConflictDetectionTest - KeyValueTableReadSequenceTest - IncrementalDiffSplitReadTest - PrimaryKeyTableRawFileSplitReadProviderTest Flink - SortCompactCommitterTest - SortCompactSinkWriteTest - SortCompactActionForAppendTableITCase - latest snapshot kind is COMPACT - concurrent append between read and commit does not lose data - SortCompactActionForDynamicBucketITCase - CommitterOperatorTest - FlinkSourceBuilderTest Spark - SortCompactSparkCommitTest - CompactProcedureTestBase updated to expect COMPACT -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
