leaves12138 opened a new pull request, #9135: URL: https://github.com/apache/paimon/pull/9135
## What changed - add a primitive Avro manifest reader which exposes only the fields required by row-id sorting - discover naturally sorted runs and merge them with a fixed selection tree - copy eligible encoded records and compressed Avro blocks directly into output manifests - retain the spillable external sorter as a bounded fallback for fragmented inputs - preserve exact DELETE identifier matching and manifest statistics ## Why Full row-id manifest compaction currently materializes every `ManifestEntry`, feeds all entries through the external sorter, and encodes every surviving entry again. Large data-evolution tables spend substantial CPU time and allocation volume in this path even when their manifests already consist of a small number of sorted runs. The new path keeps partition data interned, stores only primitive ordering fields, and avoids decoding or re-encoding blocks when their order and filtering eligibility are already known. ## Impact On the snapshot 7537 production metadata fixture with 26,100,963 input entries, using the same six-way manifest read parallelism and no `-Xmx` limit: | Metric | Existing external sort | New run merge | | --- | ---: | ---: | | Sort time | 181.01 s | 11.57 s | | Thread allocation | 147.53 GiB | 2.90 GiB | | Process peak RSS | 2.27 GiB | 1.20 GiB | The output count was 26,095,685 in both cases. The logical encoded-record SHA-256 was also identical: `afe6fcdaff7e8b2e7d047c199164c19457ced5b3cb2adaadd32c48cf11a1cdf5`. ## Validation - 11 targeted `ManifestFileMetaTest` cases covering row-id order, secondary keys, exact DELETE identifiers, raw identity fields, many partitions, fragmented fallback, block stats, configured partition fields, missing row-id stats, and minor compaction - 5 `ByteArrayKeyTest` cases - `mvn -pl paimon-format,paimon-core -am -DskipTests package` - packaged shaded-Avro ABI inspection - snapshot 7537 old/new output count and logical digest comparison on Linux -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
