[spark] branch master updated: [SPARK-41017][SQL][FOLLOWUP] Respect the original Filter operator order

[wenchen]

This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new ac84e1462c6 [SPARK-41017][SQL][FOLLOWUP] Respect the original Filter 
operator order
ac84e1462c6 is described below

commit ac84e1462c6cc74b86d727a625e5242c107d3844
Author: Wenchen Fan <wenc...@databricks.com>
AuthorDate: Thu Nov 17 23:49:34 2022 +0800

    [SPARK-41017][SQL][FOLLOWUP] Respect the original Filter operator order
    
    ### What changes were proposed in this pull request?
    
    This is a followup of https://github.com/apache/spark/pull/38511 to fix a 
mistake: we should respect the original `Filter` operator order when 
re-constructing the query plan.
    
    ### Why are the changes needed?
    
    bug fix
    
    ### Does this PR introduce _any_ user-facing change?
    
    No
    
    ### How was this patch tested?
    
    N/A
    
    Closes #38684 from cloud-fan/column-pruning.
    
    Lead-authored-by: Wenchen Fan <wenc...@databricks.com>
    Co-authored-by: Wenchen Fan <cloud0...@gmail.com>
    Signed-off-by: Wenchen Fan <wenc...@databricks.com>
---
 .../spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala  | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala
index 24ffe4b887d..87b11da5d5c 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala
@@ -350,7 +350,7 @@ object V2ScanRelationPushDown extends Rule[LogicalPlan] 
with PredicateHelper {
       val normalizedProjects = DataSourceStrategy
         .normalizeExprs(project, sHolder.output)
         .asInstanceOf[Seq[NamedExpression]]
-      val allFilters = filtersStayUp ++ filtersPushDown.reduceOption(And)
+      val allFilters = filtersPushDown.reduceOption(And).toSeq ++ filtersStayUp
       val normalizedFilters = DataSourceStrategy.normalizeExprs(allFilters, 
sHolder.output)
       val (scan, output) = PushDownUtils.pruneColumns(
         sHolder.builder, sHolder.relation, normalizedProjects, 
normalizedFilters)
@@ -371,7 +371,8 @@ object V2ScanRelationPushDown extends Rule[LogicalPlan] 
with PredicateHelper {
       }
 
       val finalFilters = normalizedFilters.map(projectionFunc)
-      val withFilter = 
finalFilters.foldRight[LogicalPlan](scanRelation)((cond, plan) => {
+      // bottom-most filters are put in the left of the list.
+      val withFilter = finalFilters.foldLeft[LogicalPlan](scanRelation)((plan, 
cond) => {
         Filter(cond, plan)
       })
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org