JingsongLi opened a new pull request, #10680: URL: https://github.com/apache/paimon/pull/10680
Data Evolution tables could not use incremental clustering. Reuse the existing clustering strategy and local/global sort paths to rewrite complete ordinary-column RowID groups, read their latest column values with deletion vectors applied, and assign new RowIDs on commit. Dedicated BLOB/VECTOR storage files remain unsupported; inline values keep their existing layouts. The DE rewriter delegates DV materialization to the existing rewrite flow. Snapshot-based conflict detection rejects overlapping concurrent updates and deletes, while independent appends survive. Global indexes in affected partitions are invalidated against the latest snapshot on each commit attempt; index RowID maintenance follows separately. Planning an empty table is also pinned to its initial empty snapshot: a concurrent first append is left for the next clustering invocation, avoiding a null snapshot dereference. A deterministic regression covers both ordinary and DE tables. Validation: 144 unique core cases, 15 Flink cases and 13 Spark cases passed locally with static checks enabled, including forward/reverse conflicts, index creation during optimistic retry, empty/deleted partitions, inline values and distributed sort file ranges. Independent review completed. Split from #10667 after #10679. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
