taoran92 opened a new issue, #10595:
URL: https://github.com/apache/paimon/issues/10595
## Search before asking
- [x] I searched in the issues and found nothing similar.
## Motivation
A customer's workload uses predicates such as `__paimon_bucket IN (0, 1)`
to query selected physical buckets through Spark without enumerating
business keys. This is useful for exploratory analysis, testing
transformations on a subset, and investigating particular buckets; it
does not require statistically representative sampling.
In the baseline below, Spark returns correct results but scans unrelated
buckets before applying the metadata filter. Translating this condition
into a Core bucket filter would avoid those data-file reads. PR1 adds
an optimization to the Core filtering path; Spark integration follows
separately.
### Spark baseline reproduction
The original local reproduction used Paimon 2.2-SNAPSHOT at commit
`937b5e1d51a5d346634df56794c562150aa1a2b5`, Spark 3.5.9,
Scala 2.12 and Java 11.
With a disposable Paimon catalog/database and the Paimon Spark extensions
enabled:
```sql
CREATE TABLE bucket_filter_repro (id INT, payload STRING)
USING paimon
TBLPROPERTIES (
'bucket' = '4',
'bucket-key' = 'id',
'file.format' = 'parquet'
);
INSERT INTO bucket_filter_repro
SELECT CAST(id AS INT), concat('v', CAST(id AS STRING))
FROM range(400);
SELECT id, payload, __paimon_bucket
FROM bucket_filter_repro;
SELECT id, payload, __paimon_bucket
FROM bucket_filter_repro WHERE __paimon_bucket = 1;
SELECT id, payload, __paimon_bucket
FROM bucket_filter_repro WHERE __paimon_bucket = 99;
SELECT id, payload, __paimon_bucket
FROM bucket_filter_repro WHERE id = 1;
```
The fixture produced four data files:
| Predicate | Planned buckets | Planned files | BatchScan output rows |
Final result rows |
| --- | --- | ---: | ---: | ---: |
| None | 0, 1, 2, 3 | 4 | 400 | 400 |
| `__paimon_bucket = 1` | 0, 1, 2, 3 | 4 | 400 | 95 |
| `__paimon_bucket = 99` | 0, 1, 2, 3 | 4 | 400 | 0 |
| `id = 1` | 2 | 1 | 100 | 1 |
Planned files were inspected through scan input splits. Scan output rows
were measured using BatchScan's `numOutputRows`. The `id = 1` query
demonstrates existing bucket-key pruning.
These are synthetic baseline results, not customer production measurements.
Exact file counts depend on the write configuration.
## Solution
### PR1 — Core physical bucket predicates and manifest pruning
Core already supports physical bucket filtering through
`ReadBuilder.withBucketFilter(Filter<Integer>)`. Ordinary filters exclude
data files using manifest entries or sidecar bucket indexes. Existing
`withBucket(int)` and supported bucket-key selectors can also prune
manifests using bucket-range statistics. PR1 extends the existing
[manifest bucket pruning](https://github.com/apache/paimon/pull/9791)
to structured physical bucket predicates.
PR1 introduces an immutable, serializable `PhysicalBucketPredicate` through
the same read API. It supports comparisons, membership, and set operations
through `intersect()`, `union()`, and `negate()`. These operations preserve
manifest-range pruning and serializability; the inherited `Filter.and()`
does not.
The planner compares the predicate with each manifest's `minBucket` and
`maxBucket`, already recorded in the manifest list. It can reject disjoint
manifests before opening their contents or sidecars. Retained manifests
continue through existing sidecar and entry-level filtering. A sidecar is
a separate companion index describing blocks within a manifest.
The implementation preserves existing filters and constraints, supports
historical physical bucket IDs independently of the current bucket count,
and retains manifests with missing or invalid range statistics. No
storage-format changes or mandatory manifest rewrites are required.
### Benefits
PR1 reduces metadata reads for multi-bucket, range and composed physical
predicates while selecting the same data files as an equivalent ordinary
bucket filter. Disjoint manifests can be skipped before opening their
contents or sidecars.
The benefit depends on how many manifest ranges are disjoint from the
requested buckets. Broadly overlapping or unavailable ranges limit it;
snapshot and manifest-list access is still required.
### Follow-up — Spark integration
PR2 will translate supported `__paimon_bucket` conditions into Core scan
filters and preserve Spark SQL semantics, including NULL handling.
PR1 alone does not change Spark SQL scan behavior; the Spark integration
must be completed to deliver the overall feature.
## Validation
A separate synthetic Core fixture contains four manifests, each with one
bucket, one data file and two rows. A supplementary local probe selected
buckets `IN (1, 3)` and measured these cold-cache input-stream open counts:
| Bucket filter | Sidecar enabled | Manifest opens | Sidecar opens |
| --- | --- | ---: | ---: |
| Ordinary bucket filter | No | 4 | 0 |
| Physical bucket predicate | No | 2 | 0 |
| Ordinary bucket filter | Yes | 2 | 4 |
| Physical bucket predicate | Yes | 2 | 2 |
All four cases plan two data files and return the same four rows; the
unfiltered scan plans four files. The same counts were verified for the
range `1 <= bucket <= 2`. Both selections were checked on append and
primary-key tables, covering 16 scan cases across filter forms and
sidecar settings.
These counters measure `FileIO.newInputStream` calls for manifests and
sidecars, not total metadata I/O, bytes read or elapsed-time speedup.
They validate Core pruning independently of Spark SQL pushdown.
The reviewed PR1 commit (`8aed1ea5d`) also passed 34 targeted Core tests,
covering predicate boundaries and serialization, filter compatibility,
sidecars and caches, primary-key merge reads, deleted/replaced files,
and historical bucket counts. The supplementary probe reused that test
fixture and was run separately from the committed test suite.
## Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
