weijietong opened a new pull request, #8745:
URL: https://github.com/apache/paimon/pull/8745
### Purpose
Optimize readPartitionEntries and readBucketEntries in
AbstractFileStoreScan to use per-entry converter during manifest reading, so
ManifestEntry objects (with heavy DataFileMeta including stats, key ranges,
embedded index, etc.) are immediately converted to lightweight
PartitionEntry/BucketEntry and never accumulated. This reduces peak memory per
manifest file from holding a full List<ManifestEntry> to only a
List<PartitionEntry> — each DataFileMeta (~20 fields, several KB) is discarded
after extracting the 3 needed values (rowCount, fileSize, creationTime), while
PartitionEntry only retains ~6 fields (~40 bytes). For tables with millions of
files the savings can be significant.
### Tests
Added FileStoreScanPartitionBucketEntryTest covering single/multiple
partition aggregation, multi-file merge within the same partition, partition
filter pushdown, append-only table variant, and equivalent bucket entry tests —
including the previously untested readBucketEntries() method which had zero
coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
