[GitHub] [spark] EnricoMi commented on pull request #38676: [SPARK-41162][SQL] Do not push down anti-join predicates that become ambiguous

EnricoMi commented on PR #38676: URL: https://github.com/apache/spark/pull/38676#issuecomment-1371192971 Closed in favour of #39131. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] EnricoMi commented on pull request #38676: [SPARK-41162][SQL] Do not push down anti-join predicates that become ambiguous

EnricoMi commented on PR #38676: URL: https://github.com/apache/spark/pull/38676#issuecomment-1358011350 @shardulm94 you are right, `canPushThroughCondition` already guards `Project` and `Union` against this situation, so that should be the natural way to fix this for `Aggregate` as well.

[GitHub] [spark] EnricoMi commented on pull request #38676: [SPARK-41162][SQL] Do not push down anti-join predicates that become ambiguous

EnricoMi commented on PR #38676: URL: https://github.com/apache/spark/pull/38676#issuecomment-1346167823 @wangyum @cloud-fan do you consider this issue a correctness bug? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] EnricoMi commented on pull request #38676: [SPARK-41162][SQL] Do not push down anti-join predicates that become ambiguous

EnricoMi commented on PR #38676: URL: https://github.com/apache/spark/pull/38676#issuecomment-1335354876 @wangyum @cloud-fan what do you think about my approach? Do you have a suggestion for a better strategy? -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] EnricoMi commented on pull request #38676: [SPARK-41162][SQL] Do not push down anti-join predicates that become ambiguous

EnricoMi commented on PR #38676: URL: https://github.com/apache/spark/pull/38676#issuecomment-1323236232 @wangyum @cloud-fan I am not sure if this is the right approach to fix `DeduplicateRelations`. Please advise. Problem is that `DeduplicateRelations` is only considering duplicates

[GitHub] [spark] EnricoMi commented on pull request #38676: [SPARK-41162][SQL] Do not push down anti-join predicates that become ambiguous

EnricoMi commented on PR #38676: URL: https://github.com/apache/spark/pull/38676#issuecomment-1320300231 Problem is that `DeduplicateRelations` is only considering duplicates between left `output` and right `output`, and not duplicates between left `references` and right `output`. I have

[GitHub] [spark] EnricoMi commented on pull request #38676: [SPARK-41162][SQL] Do not push down anti-join predicates that become ambiguous

EnricoMi commented on PR #38676: URL: https://github.com/apache/spark/pull/38676#issuecomment-1319839395 > Could we fix the `DeduplicateRelations`? Interesting, that sounds like a better solution. I'll look into it. -- This is an automated message from the Apache Git Service. To

[GitHub] [spark] EnricoMi commented on pull request #38676: [SPARK-41162][SQL] Do not push down anti-join predicates that become ambiguous

EnricoMi commented on PR #38676: URL: https://github.com/apache/spark/pull/38676#issuecomment-1318255174 @wangyum @cloud-fan appreciate your suggestion on how to test this bug in `LeftSemiAntiJoinPushDownSuite` (see https://github.com/apache/spark/pull/38676#issuecomment-1317220559). --