leaves12138 opened a new pull request, #9241: URL: https://github.com/apache/paimon/pull/9241
Follow-up to #9135 and #9213. ## What changed - Replace the hand-written manifest field indexes and custom projected-row materialization in run merge with ProjectedManifestEntry.ENTRY_LAYOUT_PROJECTION. - Reuse the shared PartitionDictionary while preserving low-overhead single-threaded behavior for ordinary manifest merging and adding concurrent collection plus comparator-compatible ranks for run discovery. - Collect minor-compaction DELETE state per discovery task and combine it afterward instead of synchronizing every DELETE entry. - Keep Avro file-name views borrowed for the current entry and copy only keys that outlive the row into reusable owned buffers. - Add MAX_SEQUENCE_NUMBER to the shared entry layout because it is a row-id manifest sorting key. ## Why The run-merge implementation duplicated schema layout knowledge with numeric field constants and manual GenericRow projection. That made it easy for the optimized sort path to drift from the shared manifest representation and made schema changes require edits in multiple places. This follow-up routes the sort path through the same projected manifest abstractions used by block-aware ordinary merging. ## Validation - Formatting and repository checks passed with ~/bin/check. - Targeted manifest tests: 137 tests, 0 failures, 1 existing skip. - Added coverage for projected run-merge fields, concurrent partition collection, stable ranks, and comparator-equal but byte-distinct partitions. - dev2 large-table metadata benchmark without an Xmx limit: | Snapshot | Before | After | Result | | --- | ---: | ---: | --- | | 8795 | 5.26 s | 4.96 s | no regression | | 8816 | 19.03 s | 16.33 s | no regression | The output manifest counts and bytes remained unchanged in both benchmark cases. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
