JingsongLi opened a new pull request, #8890: URL: https://github.com/apache/paimon/pull/8890
## What changed - Rework manifest full and minor sort compaction to keep entries as projected binary rows through scan, external sort, delete matching, and manifest writing. - Add compact reusable binary identifiers and avoid repeatedly deserializing partitions and data file metadata. - Add a scoped Avro object-reuse reader for manifest scans. - Remove per-row temporary arrays and intermediate `Utf8` / `ByteBuffer` objects from Avro decoding. - Add end-to-end full and minor-delete allocation/throughput benchmarks, including configurable sort-buffer size for spill testing. ## Why Manifest sorting previously serialized entries into byte arrays, materialized large object sets, and repeatedly rebuilt manifest entry, partition, and Avro reader objects. These allocations increased GC pressure and made sort compaction slower as manifest counts grew. The new path retains the existing manifest format and compaction semantics while moving transient scan/sort state to binary views and compact identifiers. ## Benchmark Full sort, 24,000 entries in 16 manifests, one manifest reader, 64 MiB sort buffer: | | Before | After | |---|---:|---:| | Best / average time | 96.8 / 111.3 ms | 62.1 / 65.8 ms | | Allocated bytes | 271.8 MiB | 81.3 MiB | | Throughput | 247.9K entries/s | 386.8K entries/s | This reduces average time by 40.9%, allocated bytes by 70.1%, and increases throughput by 56.0%. The minor-delete benchmark allocates 110.2 MiB for 33,008 entries. A 4 MiB forced-spill run also passes. ## Validation - `paimon-format`: 463 tests passed, 6 skipped. - `BinaryManifestEntryTest`, `ManifestFileTest`, and `ManifestFileMetaTest`: 85 tests passed, 1 skipped. - Full and minor-delete benchmark runs passed with 64 MiB and 4 MiB sort buffers. - Spotless, Checkstyle, and Maven Enforcer checks passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
