jianguotian opened a new pull request, #9784: URL: https://github.com/apache/paimon/pull/9784
## Purpose Bucket-key point lookups spanning many partitions can spend most of their time opening manifests whose bucket ranges cannot match. Manifest metadata currently lacks total-bucket bounds, and existing compacted manifests cannot be reorganized under an opt-in bucket-first layout. This supersedes #9783 with a clean, single-commit branch based on the latest `master`. ## Changes - persist nullable min/max bucket-count metadata in manifest lists - prune manifests conservatively with bucket and total-bucket ranges - push partition/bucket filters into projected delete-entry reads - add an opt-in bucket-first manifest sort order - add a one-shot forced manifest rewrite option for maintenance - expose the new ranges in the manifests system table - fall back to partition-only manifest comparison for a whole compaction pass when legacy bucket bounds are missing, preserving comparator transitivity - reject `manifest-sort.bucket-first` for data evolution tables instead of silently ignoring it - make manifest compact dry-run honor the bucket-first option Legacy manifests without the new nullable fields fall back to conservative reads. Both new options default to false. Bucket enumeration is bounded and also falls back conservatively. ## Performance evidence On a production-shaped 2.54 TB test branch using the same point-lookup SQL and result set, the combined layout and pruning path reduced opened manifests from 136 to 1 and planning from 22.6 s to 2.0 s. Resulted data files remained 54 and splits remained 25, isolating the change to planning rather than data scanning. ## Verification - focused `paimon-core` tests: 152 tests, 0 failures, 0 errors, 1 skipped - `paimon-core` plus required modules package: BUILD SUCCESS (9 modules) - Checkstyle, Spotless, and Maven Enforcer: BUILD SUCCESS - generated core option documentation updated -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
