EnricoMi opened a new pull request, #38676: URL: https://github.com/apache/spark/pull/38676
### What changes were proposed in this pull request? Rule `PushDownLeftSemiAntiJoin` should not push an anti-join below an `Aggregate` when the translated (cf. `aliasMap`) join conditions become ambiguous w.r.t. to both join sides. ### Why are the changes needed? The following example fails with `distinct()`, and succeeds without `distinct()`, but both queries are identical. ```scala val ids = Seq(1, 2, 3).toDF("id").distinct() val result = ids.withColumn("id", $"id" + 1).join(ids, "id", "left_anti").collect() assert(result.length == 1) ``` With `distinct()`, rule `PushDownLeftSemiAntiJoin` creates a join condition `(id#750 + 1) = id#750`, which can never be true. This effectively removes the anti-join. ### Does this PR introduce _any_ user-facing change? It fixes correctness. ### How was this patch tested? Unit test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org