taoran92 opened a new pull request, #10638:
URL: https://github.com/apache/paimon/pull/10638
### Purpose
Fixes #10595.
Push supported Spark `__paimon_bucket` equality, membership and range
predicates into the existing Core bucket-filter APIs. This avoids reading
unrelated data files when processing selected physical buckets, reducing
storage I/O and executor work while preserving Spark SQL results and residual
filters.
In the four-bucket example in the issue, `IN (0, 1)` reduces planned data
files from 4 to 2; an unmatched bucket plans zero files. Membership and range
predicates do not guarantee manifest-file pruning.
Change size (added + deleted lines, including license headers):
| Area | Files | Added | Deleted | Share of changed lines |
| --- | ---: | ---: | ---: | ---: |
| Production code | 9 | 153 | 21 | 16.3% |
| Tests | 8 | 873 | 0 | 81.9% |
| Documentation | 1 | 19 | 0 | 1.8% |
| Total | 18 | 1,045 | 21 | 100% |
The eight test files consist of one shared unit-test suite, one shared SQL
integration-test base, and six Spark-version entry classes (3.2–3.5, 4.0 and
4.1). Five entry classes are thin subclasses; the Spark 3.2 entry also checks
V1 NULL semantics. The integration cases are shared rather than duplicated
across versions.
### Tests
Passed locally with Java 11 and Scala 2.12, without `fast-build`:
- `PhysicalBucketFilterTest`: 5 tests passed.
- Spark 3.5.8 `PhysicalBucketPushDownTest`: 15 tests passed.
- Spark 3.2.4 `PhysicalBucketPushDownTest`: 15 tests passed.
- Checkstyle, Spotless, Enforcer and `git diff --check` passed.
Coverage checks both query results and planned buckets/data files, including
NULL and boolean composition, append/primary-key tables, existing filters,
historical bucket counts, dynamic buckets, fallback ownership, postpone
compacted reads, LIMIT/TopN/aggregation, grouped joins and scan copying. Other
Spark versions were not rerun locally in this verification.
```shell
mvn -pl paimon-spark/paimon-spark-3.5 -am -Pspark3 \
-DfailIfNoTests=false -Dtest=none \
-DwildcardSuites=org.apache.paimon.spark.PhysicalBucketFilterTest,org.apache.paimon.spark.sql.PhysicalBucketPushDownTest
test
mvn -pl paimon-spark/paimon-spark-3.2 -am -Pspark3 \
-DfailIfNoTests=false -Dtest=none \
-DwildcardSuites=org.apache.paimon.spark.sql.PhysicalBucketPushDownTest
test
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
