[GitHub] [arrow-datafusion] Ted-Jiang commented on a diff in pull request #2729: Filter push down need consider alias columns

Tue, 14 Jun 2022 20:20:52 -0700 


Ted-Jiang commented on code in PR #2729:
URL: https://github.com/apache/arrow-datafusion/pull/2729#discussion_r897490349


##########
datafusion/optimizer/src/filter_push_down.rs:
##########
@@ -1831,4 +1870,111 @@ mod tests {
 
         Ok(())
     }
+

Review Comment:
   @alamb  you are right! without the code change the ut still passed
   
   > I am still a little confused about where the issue is -- is the filter 
push down logic adding an extra alias? Or is it not adding an alias it should?
   
   I think it is `not` adding an alias it should,  but the ut show it doesn't 😂
   
   But I got a log from send logical plan to ballista-scheduler
   ```
   After apply projection_push_down rule:
   
   Optimized logical plan:
   Limit: 50000
     Projection: #LINEORDER.LO_SHIPMODE, #ASS
       Projection: #LO_SHIPMODE AS LINEORDER.LO_SHIPMODE, 
#KYLINAPPROXCOUNTDISTINCT(_KY_COUNT_DISTINCT_LINEORDER_LO_SHIPPRIORITY_LINEORDER_LO_SUPPKEY_,UInt8(10))
 AS ASS
         Aggregate: groupBy=[[#LO_SHIPMODE]], 
aggr=[[KYLINAPPROXCOUNTDISTINCT(#_KY_COUNT_DISTINCT_LINEORDER_LO_SHIPPRIORITY_LINEORDER_LO_SUPPKEY_,
 UInt8(10))]]
           Projection: #LO_SHIPMODE, 
#_KY_COUNT_DISTINCT_LINEORDER_LO_SHIPPRIORITY_LINEORDER_LO_SUPPKEY_
     hi      Filter: #LO_SHIPMODE IN ([Utf8("SHIP"), Utf8("Rail"), 
Utf8("2321"), Utf8("MAIL")])
               Projection: #LO_SHIPMODE, 
#_KY_COUNT_DISTINCT_LINEORDER_LO_SHIPPRIORITY_LINEORDER_LO_SUPPKEY_
    alias         Projection: #ssb@[email protected] AS 
LO_SHIPMODE, #ssb@[email protected] AS 
_KY_COUNT_DISTINCT_LINEORDER_LO_SHIPPRIORITY_LINEORDER_LO_SUPPKEY_
                   TableScan: ssb@kylin_udaf_cube_update@17179869183 
projection=Some([12, 36])
   
   After apply filter_push_down rule:
   
   Optimized logical plan:
   Limit: 50000
     Projection: #LINEORDER.LO_SHIPMODE, #ASS
       Projection: #LO_SHIPMODE AS LINEORDER.LO_SHIPMODE, 
#KYLINAPPROXCOUNTDISTINCT(_KY_COUNT_DISTINCT_LINEORDER_LO_SHIPPRIORITY_LINEORDER_LO_SUPPKEY_,UInt8(10))
 AS ASS
         Aggregate: groupBy=[[#LO_SHIPMODE]], 
aggr=[[KYLINAPPROXCOUNTDISTINCT(#_KY_COUNT_DISTINCT_LINEORDER_LO_SHIPPRIORITY_LINEORDER_LO_SUPPKEY_,
 UInt8(10))]]
           Projection: #LO_SHIPMODE, 
#_KY_COUNT_DISTINCT_LINEORDER_LO_SHIPPRIORITY_LINEORDER_LO_SUPPKEY_
             Projection: #LO_SHIPMODE, 
#_KY_COUNT_DISTINCT_LINEORDER_LO_SHIPPRIORITY_LINEORDER_LO_SUPPKEY_
     alias     Projection: #ssb@[email protected] AS 
LO_SHIPMODE, #ssb@[email protected] AS 
_KY_COUNT_DISTINCT_LINEORDER_LO_SHIPPRIORITY_LINEORDER_LO_SUPPKEY_
     hi          Filter: #LO_SHIPMODE IN ([Utf8("SHIP"), Utf8("Rail"), 
Utf8("2321"), Utf8("MAIL")])
                   TableScan: ssb@kylin_udaf_cube_update@17179869183 
projection=Some([12, 36]), partial_filters=[#LO_SHIPMODE IN ([Utf8("SHIP"), 
Utf8("Rail"), Utf8("2321"), Utf8("MAIL")])]
   
   ```
   It show not adding an alias it should 🤔, in our internal version datafusion 
may lost some commit, but add this pr ut still passed😭
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to