Leanken.Lin created SPARK-32573:
-----------------------------------
Summary: Eliminate Anti Join when BuildSide is Empty
Key: SPARK-32573
URL: https://issues.apache.org/jira/browse/SPARK-32573
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.0.0
Reporter: Leanken.Lin
In [SPARK-32290|https://issues.apache.org/jira/browse/SPARK-32290], we
introduced several new types of HashedRelation
* EmptyHashedRelation
* EmptyHashedRelationWithAllNullKeys
They were all limited to used only in NAAJ scenario. But as for a improvement,
EmptyHashedRelation could also be used in Normal AntiJoin for fast stop, and as
for in AQE, we can even eliminate anti join when we knew that buildSide is
empty.
This Patch including two changes.
In Non-AQE, using EmptyHashedRelation to do fast stop for common anti join as
well
In AQE, eliminate anti join if buildSide is a EmptyHashedRelation of
ShuffleWriteRecord is 0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
