Szehon Ho created SPARK-60944:
---------------------------------
Summary: Skip physical evaluation of guaranteed DSv2 filters
Key: SPARK-60944
URL: https://issues.apache.org/jira/browse/SPARK-60944
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Szehon Ho
DSv2 sources can report inferred Catalyst filters for statistics estimation
after SPARK-58892, including a mode that keeps them on logical Filter nodes.
These predicates can remain in executable filters even when the source already
guarantees that every returned row satisfies them.
Add an internal table capability that proves whether a final Catalyst predicate
is SQL true for every row returned by a supplied built Scan.
DataSourceV2Strategy can omit proven conjuncts from physical filters and
Boolean projection aliases while leaving logical statistics unchanged. Keep
removal conservative when the source cannot prove the final expression true.
Tracking the originally inferred expressions is insufficient:
BooleanSimplification can factor them together with residual predicates;
MergeSubplans can put branch predicates into Boolean aliases and rebuild scans;
and projection, normalization, and schema pruning can change attribute
references. A guarantee from an old scan is not necessarily valid for its
replacement. The callback checks the final expression against the actual scan
instead of reconstructing its provenance.
The table capability also hosts fullyPushedFilterAttributes(), shared with
Catalyst runtime filtering. Its attribute declaration applies to predicates
actually sent to the runtime-filtering scan; static predicates require the
separate per-expression proof.
Original PR and discussion: https://github.com/apache/spark/pull/58145
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
