Eugene Koifman created SPARK-37193:
--------------------------------------
Summary: DynamicJoinSelection.shouldDemoteBroadcastHashJoin should
not apply to outer joins
Key: SPARK-37193
URL: https://issues.apache.org/jira/browse/SPARK-37193
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.1.1
Reporter: Eugene Koifman
{{DynamicJoinSelection.shouldDemoteBroadcastHashJoin}} will prevent AQE from
converting Sort merge join into a broadcast join because SMJ is faster when the
side that would be broadcast has a lot of empty partitions.
This makes sense for inner joins which can short circuit if one side is empty.
For (left,right) outer join, the streaming side still has to be processed so
demoting broadcast join doesn't have the same advantage.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
