leanken commented on pull request #29614:
URL: https://github.com/apache/spark/pull/29614#issuecomment-685032785
After some code investigation and manual test, found that Optimizer will put
Repartition after LeftAnti & LeftSemi Join with Some rules. So we can see that
in UT, LeftAnti & LeftSemi could still be converted from Join to EmptyRelation.
FYI, @cloud-fan
## NotInSubQuery NAAJ streamedSide repartition
```
=== Applying Rule
org.apache.spark.sql.catalyst.optimizer.RewritePredicateSubquery ===
Repartition 77, true
+- Project [a#218]
! +- Filter NOT a#218 IN (list#230 [])
! : +- Project [c#220]
! : +- LocalRelation [c#220, d#221]
! +- LocalRelation [a#218, b#219]
Repartition 77, true
+- Project [a#218]
+- Join LeftAnti, ((a#218 = c#220) OR isnull((a#218 = c#220)))
:- LocalRelation [a#218, b#219]
+- Project [c#220]
+- LocalRelation [c#220, d#221]
```
## hand-rewritten NAAJ streamedSide repartition
```
=== Applying Rule
org.apache.spark.sql.catalyst.optimizer.PushDownLeftSemiAntiJoin ===
Project [a#218]
!+- Join LeftAnti, ((a#218 = c#220) OR isnull((a#218 = c#220)))
! :- Repartition 77, true
! : +- Project [a#218, b#219]
! : +- LocalRelation [a#218, b#219]
! +- Project [c#220, d#221]
! +- LocalRelation [c#220, d#221]
Project [a#218]
+- Repartition 77, true
+- Project [a#218, b#219]
+- Join LeftAnti, ((a#218 = c#220) OR isnull((a#218 = c#220)))
:- LocalRelation [a#218, b#219]
+- Project [c#220, d#221]
+- LocalRelation [c#220, d#221]
```
## LEFT SEMI streamedSide repartition
```
=== Applying Rule
org.apache.spark.sql.catalyst.optimizer.PushDownLeftSemiAntiJoin ===
+- Project [a#218]
! +- Join LeftSemi, (a#218 = c#220)
! :- Repartition 77, true
! : +- Project [a#218, b#219]
! : +- LocalRelation [a#218, b#219]
! +- Project [c#220, d#221]
! +- Project [c#220, d#221]
! +- Filter (cast(d#221 as decimal(4,1)) > cast(cast(100 as
decimal(3,0)) as decimal(4,1)))
! +- LocalRelation [c#220, d#221]
+- Project [a#218]
+- Repartition 77, true
+- Project [a#218, b#219]
+- Join LeftSemi, (a#218 = c#220)
:- LocalRelation [a#218, b#219]
+- Project [c#220, d#221]
+- Project [c#220, d#221]
+- Filter (cast(d#221 as decimal(4,1)) > cast(cast(100 as
decimal(3,0)) as decimal(4,1)))
+- LocalRelation [c#220, d#221]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
