taoran92 opened a new pull request, #10596:
URL: https://github.com/apache/paimon/pull/10596
### Purpose
Related to #10595 (PR1: Core physical bucket predicates and manifest
pruning).
Existing Core bucket filters already exclude unrelated data files, but
ordinary `Filter<Integer>` predicates do not use manifest bucket ranges to skip
metadata reads. This leaves unnecessary manifest or sidecar reads when
selecting multiple physical buckets or a bucket range.
Introduce an immutable, serializable `PhysicalBucketPredicate` through the
existing `ReadBuilder.withBucketFilter` API. Comparisons, membership, and
`intersect()` / `union()` / `negate()` composition preserve an exact set of
physical bucket IDs. The planner checks existing `minBucket` / `maxBucket`
statistics to skip disjoint manifests before opening their contents or
sidecars, then reuses existing filtering for retained manifests.
The change preserves ordinary filters and existing scan constraints, handles
physical bucket IDs independently of the current bucket count, and retains
manifests with missing or invalid range statistics. It requires no
storage-format changes. Spark SQL integration remains a follow-up under #10595.
### Tests
All 34 targeted Core tests passed, covering predicate boundaries and
serialization, filter compatibility, sidecars and caches, primary-key merge
reads, deleted/replaced files, and historical bucket counts:
```shell
mvn -pl paimon-core -DwildcardSuites=none \
-Dtest=PhysicalBucketPredicateTest,BucketFilterTest,PhysicalBucketFilterScanTest,BucketFilterScanTest
test
```
The supplementary Core probe documented in #10595 also verified identical
files and rows for ordinary and physical predicates across 16 scan cases.
Selecting two of four buckets reduced manifest opens from 4 to 2 without
sidecars, or sidecar opens from 4 to 2 with sidecars enabled. These are
input-stream open counts, not elapsed-time measurements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
