Leanken.Lin created SPARK-32494:
-----------------------------------
Summary: Null Aware Anti Join Optimize Support Multi-Column
Key: SPARK-32494
URL: https://issues.apache.org/jira/browse/SPARK-32494
Project: Spark
Issue Type: New Feature
Components: SQL
Affects Versions: 3.0.0
Reporter: Leanken.Lin
In Issue [SPARK-32290|https://issues.apache.org/jira/browse/SPARK-32290], we
managed to optimize BroadcastNestedLoopJoin into BroadcastHashJoin within the
Single-Column NAAJ scenario, by using hash lookup instead of loop join.
It's simple to just fulfill a "NOT IN" logical when it's a single key, but
multi-column not in is much more complicated with all these null aware compare.
Hence, proposed with a New type HashedRelation, NullAwareHashedRelation.
For NullAwareHashedRelation
# it will not skip anyNullColumn key like LongHashedRelation and
UnsafeHashedRelation
# while building NullAwareHashedRelation, will put extra keys into the
relation, just to make null aware columns comparison in hash lookup style.
the duplication would be 2^numKeys - 1, for example, if we are to support NAAJ
with 3 column join key, the buildSide would be expanded into (2^3 - 1) times,
7X.
For example, if there is a UnsafeRow key (1,2,3)
In NullAware Mode, it should be expanded into 7 keys with extra C(3,1), C(3,2)
combinations, within the combinations, we duplicated these record with null
padding as following.
Original record
(1,2,3)
Extra record to be appended into HashedRelation
(null, 2, 3) (1, null, 3) (1, 2, null)
(null, null, 3) (null, 2, null) (1, null, null))
with the expanded data we can extract a common pattern for both single and
multi column. allNull refer to a unsafeRow which has all null columns.
* buildSide is empty input => return all rows
* allNullColumnKey Exists In buildSide input => reject all rows
* if streamedSideRow.allNull is true => drop the row
* if streamedSideRow.allNull is false & findMatch in NullAwareHashedRelation
=> drop the row
* if streamedSideRow.allNull is false & notFindMatch in
NullAwareHashedRelation => return the row
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
