Zouxxyy opened a new pull request, #11158:
URL: https://github.com/apache/paimon/pull/11158
### Purpose
Implement the statistics overload of `SparkCatalystPartitionPredicate`,
which currently always returns `true`, so callers using this predicate through
Paimon's partition scan API can avoid reading provably irrelevant manifests.
- Reuse Paimon statistics predicates for compatible direct-field
comparisons, including non-singleton intervals.
- Evaluate CASTs and other eligible Catalyst expressions with the original
Spark semantics only when the referenced fields have statistics-proven finite
domains: one non-null value, NULL, or both. Enumerate the Cartesian product
rather than assuming cross-column correlation, with a bounded amount of work.
- Keep unknown, incomplete, inconsistent or unsupported statistics, and
evaluation failures. Do not cast string interval endpoints or invert a
may-match result for NOT.
- Capture query settings and share one lazily initialized, read-only SQLConf
per predicate. Keep mutable evaluators thread-local and restore the caller's
configuration, including after serialization and asynchronous evaluation.
Scope and limitations:
- This PR does not add a Catalyst pushdown rule to the SQL scan builder or
change native-versus-Catalyst routing. Integration tests explicitly pass the
predicate through Paimon's public scan API; ordinary SQL queries are not
claimed to automatically gain this optimization.
- CAST pruning is limited when a manifest spans multiple non-null values of
a referenced field. Merging manifests with the same referenced value can still
benefit, even if other partition columns vary. Direct-field predicates retain
native interval pruning.
- Conservative exception handling is local to this evaluator; independent
pushed predicates may still prune the same manifest.
- No new configuration, table-format change, or data-file rewrite is
introduced.
### Tests
Targeted suites: `SparkCatalystPartitionStatsTest`,
`SparkCatalystPartitionScanTest`, `SparkCatalystPartitionPredicateTest`, and
`SparkV2FilterConverterTest`.
- Apache Spark 3.5.9 / JDK 8 / Scala 2.12: **76 tests passed**.
- Apache Spark 4.1.4 / JDK 17 / Scala 2.13: **76 tests passed**.
- Each run includes **61,200 bounded exhaustive comparisons**, with 6,560
safely pruned cases and no false negatives within the tested space. These
compare statistics decisions against exact Spark evaluation, including NULL,
invalid/overflowing CASTs, comparisons, IN, NOT and compound expressions.
- Parquet/ORC integration tests compare identical partition predicates with
statistics enabled versus the former always-true statistics implementation,
checking both returned rows and scanned-manifest counts. Coverage includes
actual manifest merging (single-value and mixed-value domains), primary-key
merge reads, and overwrite ADD/DELETE entries.
- Additional coverage includes short statistics, unsupported types,
malformed UTF-8, shared SQLConf initialization, changed caller contexts, cold
serialization and concurrent evaluation.
Commands (run after bootstrapping codegen artifacts for each profile):
```sh
mvn -T 4 -pl paimon-spark/paimon-spark-3.5 -am -Pspark3 \
-DfailIfNoTests=false -Dtest=none \
-DwildcardSuites=org.apache.paimon.spark.predicate.SparkCatalystPartitionStatsTest,org.apache.paimon.spark.predicate.SparkCatalystPartitionScanTest,org.apache.paimon.spark.predicate.SparkCatalystPartitionPredicateTest,org.apache.paimon.spark.sql.SparkV2FilterConverterTest
test
mvn -T 4 -pl paimon-spark/paimon-spark-4.1 -am -Pspark4 \
-DfailIfNoTests=false -Dtest=none \
-DwildcardSuites=org.apache.paimon.spark.predicate.SparkCatalystPartitionStatsTest,org.apache.paimon.spark.predicate.SparkCatalystPartitionScanTest,org.apache.paimon.spark.predicate.SparkCatalystPartitionPredicateTest,org.apache.paimon.spark.sql.SparkV2FilterConverterTest
test
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]