leaves12138 opened a new pull request, #8904: URL: https://github.com/apache/paimon/pull/8904
## What changed - Store live-file planning state as three primitive words per file and replace the monolithic comparison quicksort with fixed-size radix-sorted chunks plus merge. - Exclude strictly contiguous same-partition logical runs larger than one billion rows from row-id mappings and manifest rewriting. - Avoid a redundant reassignment when filtering leaves only one contiguous logical run in a partition. - Rebind an existing assignment plan to current manifests after a concurrent APPEND performs manifest merging, using projected `BinaryManifestEntry` scans. - Add boundary, retry, radix ordering, multi-chunk merge, failure cleanup, and second-run regression coverage. ## Why Large data-evolution tables could allocate a very large backing array and spend most of planning time in comparison sorting. In addition, an APPEND-side manifest merge changed manifest file names and invalidated an otherwise reusable plan. The planner now bounds temporary sorting arrays by chunk size, keeps primitive state, reduces mappings and rewrite work for large contiguous runs, and discovers the current manifests that contain already-planned ranges after an APPEND retry. ## Impact This changes only data-evolution row-id reassignment planning and retry handling. The large contiguous-run threshold defaults to 1,000,000,000 rows; setting `data-evolution.reassign.skip-contiguous-row-count` to `0` disables that filtering. COMPACT and OVERWRITE snapshots still abort plan reuse. ## Validation - `mvn -pl paimon-core -am -Dtest=DataEvolutionRowIdReassignerTest,LiveFileRowIdRangeCollectorTest -Dsurefire.failIfNoSpecifiedTests=false test` - 58 targeted tests passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
