[
https://issues.apache.org/jira/browse/SPARK-60944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60944:
-----------------------------------
Labels: pull-request-available (was: )
> Skip physical evaluation of guaranteed DSv2 filters
> ---------------------------------------------------
>
> Key: SPARK-60944
> URL: https://issues.apache.org/jira/browse/SPARK-60944
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Szehon Ho
> Priority: Major
> Labels: pull-request-available
>
> DSv2 sources can report inferred Catalyst filters for statistics estimation
> after SPARK-58892, including a mode that keeps them on logical Filter nodes.
> These predicates can remain in executable filters even when the source
> already guarantees that every returned row satisfies them.
> Add an internal table capability that proves whether a final Catalyst
> predicate is SQL true for every row returned by a supplied built Scan.
> DataSourceV2Strategy can omit proven conjuncts from physical filters and
> Boolean projection aliases while leaving logical statistics unchanged. Keep
> removal conservative when the source cannot prove the final expression true.
> Tracking the originally inferred expressions is insufficient:
> BooleanSimplification can factor them together with residual predicates;
> MergeSubplans can put branch predicates into Boolean aliases and rebuild
> scans; and projection, normalization, and schema pruning can change attribute
> references. A guarantee from an old scan is not necessarily valid for its
> replacement. The callback checks the final expression against the actual scan
> instead of reconstructing its provenance.
> The table capability also hosts fullyPushedFilterAttributes(), shared with
> Catalyst runtime filtering. Its attribute declaration applies to predicates
> actually sent to the runtime-filtering scan; static predicates require the
> separate per-expression proof.
> Original PR and discussion: https://github.com/apache/spark/pull/58145
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
