jianguotian opened a new pull request, #9783: URL: https://github.com/apache/paimon/pull/9783
## Purpose Bucket-key point lookups spanning many partitions can spend most of their time opening manifests whose bucket ranges cannot match. Manifest metadata currently lacks total-bucket bounds, and existing compacted manifests cannot be reorganized under an opt-in bucket-first layout. ## Changes - persist nullable min/max bucket-count metadata in manifest lists - prune manifests conservatively with bucket and total-bucket ranges - push partition/bucket filters into projected delete-entry reads - add an opt-in bucket-first manifest sort order - add a one-shot forced manifest rewrite option for maintenance - expose the new ranges in the manifests system table Legacy manifests without the new nullable fields fall back to conservative reads. Both new options default to false. Bucket enumeration is bounded and also falls back conservatively. ## Performance evidence On a production-shaped 2.54 TB test branch using the same point-lookup SQL and result set, the combined layout and pruning path reduced opened manifests from 136 to 1 and planning from 22.6 s to 2.0 s. Resulted data files remained 54 and splits remained 25, isolating the change to planning rather than data scanning. ## Verification Focused paimon-core tests, formatting, and generated-option checks are running; this PR remains Draft until the final results are attached. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
