JingsongLi opened a new pull request, #9160: URL: https://github.com/apache/paimon/pull/9160
## Purpose Speed up manifest reads by decoding Avro object-container blocks directly and applying partition, bucket, and projection pruning before materializing nested `DataFileMeta` rows. ## Changes - Add a public `AvroBlockReader` API that exposes independently owned decompressed `byte[]` blocks, the writer schema, and block record counts. - Add a schema-aware `ManifestAvroReader` with a fixed sequential top-level decoder and early partition/bucket filtering. - Build the nested `_FILE` reader from the writer Avro schema so manifests written with older, shorter `DataFileMeta` schemas remain readable. - Route normal manifest reads, cache loading, projected scans, and expire-file scans through the same reader path. - Return independently backed `BinaryManifestEntry` values from `ManifestFile.scan`, allowing callers to retain entries safely. - Generalize `ObjectsFile` around an iterator factory so `ManifestFile` no longer carries an unused generic format reader factory. - Add compatibility, lifecycle, projection, block API, and benchmark coverage. ## Benchmark With 30,000 manifest entries, 5 warmups, and 10 measured iterations on the development machine: - Full materialization: 1.02x throughput - Partition and bucket filtering: 2.09x throughput, 1.93x allocation reduction - Full projected scan: 1.11x throughput - Delete-entry projected scan: 1.16x throughput Full reads intentionally retain independently backed rows, trading a small allocation increase for safe retention by callers. ## Validation - `ManifestFileTest`: 32 passed - `ManifestFileMetaTest`: 52 passed, 1 skipped - `NoPartitionManifestFileMetaTest`: 3 passed - `BinaryManifestEntryTest`: 8 passed - `CachingCatalogTest#testManifestCache`: passed - `ManifestListTest`: 23 passed - `AvroFileFormatTest`: 7 passed - Manifest read benchmark completed successfully - Spotless, Checkstyle, Enforcer, and `mvn -pl paimon-core -am -DskipTests compile` passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
