JingsongLi opened a new pull request, #10684: URL: https://github.com/apache/paimon/pull/10684
With incremental clustering enabled, repeated Data Evolution column updates can accumulate even when their RowID groups do not need sorting. The same Flink/Spark maintenance job now plans ordinary compaction for complete groups that meet the file-count threshold and were not selected for clustering, then commits both kinds of work together. Ordinary compaction preserves RowIDs, trusted clustering levels and column sequences. Sorted rewrites still get new RowIDs. Central preparation moves or materializes DVs as appropriate, and conflict/index maintenance uses only ranges that actually lose their old RowIDs. This preserves safe concurrent partial updates to compacted groups while still rejecting stale clustering and conflicting DV changes. The task serializer advances to v5 while reading v4 tasks as L0. Validation: 12 common, 172 core, 20 Flink and 14 Spark cases passed locally with normal static checks enabled. Tests cover compact-only and mixed jobs, repeated updates, forward/reverse concurrency, optimistic retry, index coverage, DV handling, old task serialization and local/global sort ranges. Independent review completed. Final split from #10667 after #10683. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
