Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/19947#discussion_r156250376
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -695,7 +695,9 @@ object CombineUnions extends Rule[LogicalPlan] {
*/
object CombineFilters extends Rule[LogicalPlan] with PredicateHelper {
def apply(plan: LogicalPlan): LogicalPlan = plan transform {
- case Filter(fc, nf @ Filter(nc, grandChild)) =>
+ // The query execution/optimization does not guarantee the expressions
are evaluated in order.
+ // We only can combine them if and only if both are deterministic.
+ case Filter(fc, nf @ Filter(nc, grandChild)) if fc.deterministic &&
nc.deterministic =>
--- End diff --
if `nc` is deterministic and `fc` is non-deterministic, I think we can
still combine `nc` with part of `fc` (partitioned by deterministic)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
