cloud-fan commented on code in PR #57727:
URL: https://github.com/apache/spark/pull/57727#discussion_r3718486777
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala:
##########
@@ -213,6 +217,26 @@ object PushDownUtils extends Logging {
}
translatedFiltersPushed || partPredicatesPushed
+
+ case catalystScan: SupportsRuntimeCatalystFiltering if
runtimeFilters.nonEmpty =>
+ // A DPP filter degrades to TrueLiteral when its subquery is pruned
away; it carries no
+ // information for the source. The V2 path above drops these
implicitly because
+ // translateRuntimeFilterV2 returns None; here we push Catalyst
expressions directly,
+ // so filter them out explicitly.
+ // Screen with the same pushability guard as the V2 PartitionPredicate
path
+ // (deterministic, no subquery, no Python UDF). Keeps
non-deterministic filters
+ // from being the sole evaluator when fullyPushedFilterAttributes
drops FilterExec.
+ val catalystFilters = runtimeFilters
+ .flatMap(unwrapRuntimeFilterExpression)
+ .filterNot(_ == Literal.TrueLiteral)
+ .filter(isPushablePartitionFilter)
Review Comment:
This guard can now remove the predicate's only evaluator.
`DataSourceV2Strategy` puts a non-deterministic scalar-subquery filter in
`fullyPushedRuntimeFilters` based only on its references and removes it from
`postScanFilters`; this line then rejects it before `scan.filter`. For example,
`part = (SELECT max(val) FROM dim) OR rand() < 0.5` is evaluated nowhere when
`part` is fully pushed. Please apply the same pushability gate before removing
the post-scan filter, or keep that filter until delivery is guaranteed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
