JingsongLi commented on PR #9784: URL: https://github.com/apache/paimon/pull/9784#issuecomment-5653368892
Please narrow this PR to manifest bucket pruning and complete that independently first. Manifest sorting, bucket-first layout, forced manifest rewrites, and other independent optimizations should be separate follow-up PRs. For the pruning metadata, I suggest a single nullable `totalBuckets` instead of `minTotalBuckets` / `maxTotalBuckets`: - Record `N` only when every ADD and DELETE entry in the manifest has the same positive total bucket count. - Record `null` for mixed or non-positive counts. Legacy metadata and copied manifests with unknown statistics must also remain unknown. - Keep the existing `minBucket` / `maxBucket`. When `N` is known and the predicate determines candidate buckets, reuse `BucketSelector`'s existing bucket-set cache for `N` and check whether any candidate falls within that range. - When the required metadata or predicate information is unavailable, or the manifest contains negative buckets, conservatively retain the manifest and continue the existing entry filtering. This removes total-bucket-count range enumeration and its combination limits. The tradeoff is deliberately narrower pruning coverage: manifests containing different bucket counts will fall back. Different partitions can retain different counts, so we must not substitute the current table's `bucket` option for this metadata. Let's implement and evaluate this simpler pruning change on its own first, without changing manifest sort behavior in the same PR. More general mixed-count pruning and layout changes can be considered separately with their own performance evidence. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
