EnricoMi commented on PR #38676:
URL: https://github.com/apache/spark/pull/38676#issuecomment-1317220559
I did not manage to test this in `LeftSemiAntiJoinPushDownSuite`, which
would be preferrably.
My approach is
```
test("Aggregate: LeftAnti join no pushdown on ambiguity") {
val relation = testRelation
.groupBy($"b")($"b", sum($"c").as("sum"))
val relationPlusOne = relation.select(($"b" + 1).as("b"))
val originalQuery = relationPlusOne
.join(relation, joinType = LeftAnti, usingColumns = Seq("b"))
val optimized = Optimize.execute(originalQuery.analyze)
comparePlans(optimized, originalQuery.analyze)
}
```
This creates plan
```
Project [b#7]
+- Join LeftAnti, (b#7 = b#12)
:- Project [(b#1 + 1) AS b#7]
: +- Aggregate [b#1], [b#1, sum(c#2) AS sum#6L]
: +- LocalRelation <empty>, [a#0, b#1, c#2]
+- Aggregate [b#12], [b#12, sum(c#13) AS sum#6L]
+- LocalRelation <empty>, [a#11, b#12, c#13]
```
while this would be required to expose the bug (both `Aggregate` plans have
to have identical references):
```
Project [b#7]
+- Join LeftAnti, (b#7 = b#1)
:- Project [(b#1 + 1) AS b#7]
: +- Aggregate [b#1], [b#1, sum(c#2) AS sum#6L]
: +- LocalRelation <empty>, [a#0, b#1, c#2]
+- Aggregate [b#1], [b#1, sum(c#2) AS sum#6L]
+- LocalRelation <empty>, [a#0, b#1, c#2]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
