Luka Zdravic created SPARK-61201:
------------------------------------
Summary: Add whole-stage codegen to the sort-merge ASOF join
Key: SPARK-61201
URL: https://issues.apache.org/jira/browse/SPARK-61201
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.4.0
Reporter: Luka Zdravic
SortMergeAsOfJoinExec does not support whole-stage codegen. Every row goes
through the interpreted SortMergeAsOfJoinScanner, and the join ends the codegen
stage of the operators above it.
Add whole-stage codegen for backward and forward ASOF joins. Generate the
cursor scan, the buffered group scan, the ON key compare and the join
conditions, and pass the output columns straight to the parent. Keep the
spillable right-side buffer as a Scala object, as SortMergeJoinExec does.
Nearest joins keep the interpreted scan. An internal config,
spark.sql.codegen.join.sortMergeAsOfJoin.enabled (default true), turns it off.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
