dongjoon-hyun commented on code in PR #57235:
URL: https://github.com/apache/spark/pull/57235#discussion_r3580815086
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2Suite.scala:
##########
@@ -1351,6 +1354,20 @@ class DataSourceV2Suite extends SharedSparkSession with
AdaptiveSparkPlanHelper
"pushedFilters should contain the pushed filter on column i")
}
+ test("catalyst filter pushdown skips non-deterministic filters") {
+ val df =
spark.read.format(classOf[CatalystFilterDataSourceV2].getName).load()
+ val q = df.filter($"i" > 3 && rand() > 0.5)
+
+ val scanRelation = getScanRelation(q)
+ assert(scanRelation.pushedFilters.nonEmpty,
+ "pushedFilters should contain the deterministic pushed filter")
+ assert(scanRelation.pushedFilters.forall(_.deterministic),
+ "pushedFilters should not contain non-deterministic filters")
+ val referencedCols =
scanRelation.pushedFilters.flatMap(_.references.map(_.name)).toSet
+ assert(referencedCols.contains("i"),
+ "pushedFilters should contain the pushed filter on column i")
+ }
Review Comment:
Could you verify the post-scan Filters, too, @szehon-ho ?
```suggestion
// The non-deterministic filter must not be dropped: it should remain as
a
// post-scan Filter evaluated by Spark.
val postScanCondition = q.queryExecution.optimizedPlan.collectFirst {
case f: LogicalFilter => f.condition
}
assert(postScanCondition.exists(_.exists(!_.deterministic)),
"non-deterministic filter should remain as a post-scan filter")
}
```
Import statement is also needed to be adjust for the above.
```
-import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Project}
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Filter =>
LogicalFilter, Project}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
