hbgstc123 opened a new pull request, #9790: URL: https://github.com/apache/paimon/pull/9790
### Purpose Stacked follow-up to [#8631](https://github.com/apache/paimon/pull/8631). Please review this PR **after** (or together with) the core rewriter. Until #8631 merges, GitHub's Files changed view includes the core commits. Engine-only diff: https://github.com/hbgstc123/paimon/compare/fix_sort_compact_append_only...fix_sort_compact_append_engines This PR wires Flink and Spark sort compact for **append tables** to the core rewriter: 1. Write sorted output in write-only mode. 2. Rewrite the written append commit messages into `CompactIncrement` messages. 3. Commit them through the normal compact commit path. Additional engine hardening: - Flink `SortCompactCommitter` merge/rewrite/abort and commit recovery. - Align Flink commit failure abort with Spark: abort newly written files when commit fails and the COMPACT snapshot is not yet visible. - Remove `SortCompactActionForDynamicBucketITCase` because sort compact is append-table only. No new `overwritePartition(baseSnapshotId)` APIs or special overwrite conflict logic are introduced. ### Tests Flink - `SortCompactCommitterTest` - merge partial committables before rewrite - delete-only compact when all input rows are filtered out - commit recovery does not delete planned input or create DV index files - commit failure aborts written messages only when COMPACT snapshot is not yet visible - `SortCompactActionForAppendTableITCase` - latest snapshot kind is COMPACT - concurrent append between read and commit does not lose data - `CommitterOperatorTest` - `CompactProcedureITCase` updated to expect COMPACT Spark - `SortCompactSparkCommitTest` - commit failure aborts written messages only when COMPACT snapshot is not yet visible - `CompactProcedureTestBase` updated to expect COMPACT -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
