Zouxxyy opened a new pull request, #11158:
URL: https://github.com/apache/paimon/pull/11158

   ### Purpose
   
   Implement the statistics overload of `SparkCatalystPartitionPredicate`, 
which currently always returns `true`, so callers using this predicate through 
Paimon's partition scan API can avoid reading provably irrelevant manifests.
   
   - Reuse Paimon statistics predicates for compatible direct-field 
comparisons, including non-singleton intervals.
   - Evaluate CASTs and other eligible Catalyst expressions with the original 
Spark semantics only when the referenced fields have statistics-proven finite 
domains: one non-null value, NULL, or both. Enumerate the Cartesian product 
rather than assuming cross-column correlation, with a bounded amount of work.
   - Keep unknown, incomplete, inconsistent or unsupported statistics, and 
evaluation failures. Do not cast string interval endpoints or invert a 
may-match result for NOT.
   - Capture query settings and share one lazily initialized, read-only SQLConf 
per predicate. Keep mutable evaluators thread-local and restore the caller's 
configuration, including after serialization and asynchronous evaluation.
   
   Scope and limitations:
   
   - This PR does not add a Catalyst pushdown rule to the SQL scan builder or 
change native-versus-Catalyst routing. Integration tests explicitly pass the 
predicate through Paimon's public scan API; ordinary SQL queries are not 
claimed to automatically gain this optimization.
   - CAST pruning is limited when a manifest spans multiple non-null values of 
a referenced field. Merging manifests with the same referenced value can still 
benefit, even if other partition columns vary. Direct-field predicates retain 
native interval pruning.
   - Conservative exception handling is local to this evaluator; independent 
pushed predicates may still prune the same manifest.
   - No new configuration, table-format change, or data-file rewrite is 
introduced.
   
   ### Tests
   
   Targeted suites: `SparkCatalystPartitionStatsTest`, 
`SparkCatalystPartitionScanTest`, `SparkCatalystPartitionPredicateTest`, and 
`SparkV2FilterConverterTest`.
   
   - Apache Spark 3.5.9 / JDK 8 / Scala 2.12: **76 tests passed**.
   - Apache Spark 4.1.4 / JDK 17 / Scala 2.13: **76 tests passed**.
   - Each run includes **61,200 bounded exhaustive comparisons**, with 6,560 
safely pruned cases and no false negatives within the tested space. These 
compare statistics decisions against exact Spark evaluation, including NULL, 
invalid/overflowing CASTs, comparisons, IN, NOT and compound expressions.
   - Parquet/ORC integration tests compare identical partition predicates with 
statistics enabled versus the former always-true statistics implementation, 
checking both returned rows and scanned-manifest counts. Coverage includes 
actual manifest merging (single-value and mixed-value domains), primary-key 
merge reads, and overwrite ADD/DELETE entries.
   - Additional coverage includes short statistics, unsupported types, 
malformed UTF-8, shared SQLConf initialization, changed caller contexts, cold 
serialization and concurrent evaluation.
   
   Commands (run after bootstrapping codegen artifacts for each profile):
   
   ```sh
   mvn -T 4 -pl paimon-spark/paimon-spark-3.5 -am -Pspark3 \
     -DfailIfNoTests=false -Dtest=none \
     
-DwildcardSuites=org.apache.paimon.spark.predicate.SparkCatalystPartitionStatsTest,org.apache.paimon.spark.predicate.SparkCatalystPartitionScanTest,org.apache.paimon.spark.predicate.SparkCatalystPartitionPredicateTest,org.apache.paimon.spark.sql.SparkV2FilterConverterTest
 test
   
   mvn -T 4 -pl paimon-spark/paimon-spark-4.1 -am -Pspark4 \
     -DfailIfNoTests=false -Dtest=none \
     
-DwildcardSuites=org.apache.paimon.spark.predicate.SparkCatalystPartitionStatsTest,org.apache.paimon.spark.predicate.SparkCatalystPartitionScanTest,org.apache.paimon.spark.predicate.SparkCatalystPartitionPredicateTest,org.apache.paimon.spark.sql.SparkV2FilterConverterTest
 test
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to