andygrove opened a new issue, #908: URL: https://github.com/apache/datafusion-comet/issues/908
### What is the problem the feature request solves? DataFusion has an optimization where projections can be pushed down into hash joins. This implemented in the projection pushdown optimizer rule and was added in https://github.com/apache/datafusion/pull/9236. We do not currently apply this rule, so each HashJoin will output all columns and then we have a projection to discard the unwanted columns. Here is an example plan from TPC-DS q3 showing two projections after the outermost join. Pushing the projection into the join may show some small performance improvement. ``` AggregateExec: mode=Partial, gby=[col_0@0 as col_0, col_3@3 as col_1, col_2@2 as col_2], aggr=[sum] ProjectionExec: expr=[col_0@0 as col_0, col_2@2 as col_1, col_1@4 as col_2, col_2@5 as col_3] ProjectionExec: expr=[col_0@3 as col_0, col_1@4 as col_1, col_2@5 as col_2, col_0@0 as col_0, col_1@1 as col_1, col_2@2 as col_2] HashJoinExec: mode=Partitioned, join_type=Inner, on=[(col_0@0, col_1@1)] CopyExec [UnpackOrDeepCopy] ScanExec: source=[BroadcastExchange (unknown)], schema=[col_0: Int32, col_1: Int32, col_2: Utf8] CopyExec [UnpackOrClone] ProjectionExec: expr=[col_1@1 as col_0, col_1@3 as col_1, col_2@4 as col_2] HashJoinExec: mode=Partitioned, join_type=Inner, on=[(col_0@0, col_0@0)] CopyExec [UnpackOrDeepCopy] ScanExec: source=[BroadcastExchange (unknown)], schema=[col_0: Int32, col_1: Int32] CopyExec [UnpackOrClone] FilterExec: col_0@0 IS NOT NULL AND col_1@1 IS NOT NULL ScanExec: source=[CometScan parquet (unknown)], schema=[col_0: Int32, col_1: Int32, col_2: Decimal128(7, 2)] ``` ### Describe the potential solution _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
