Github user maryannxue commented on the issue:
https://github.com/apache/spark/pull/21083
Thank you for you reply, @cloud-fan! I was not clear when you had become
aware of the effort on SPARK-21479 so it might be a misunderstanding on my side
and I apologize. Anyway, if you had had a closer look at the PR, you would have
probably got the idea that it's basically the same approach as what you have
here, only that you have covered more join types.
Here's another note. There's two types of constraint-to-filter inference
for joins going on here:
1. Infer from the Join node constraints, which is covered by the
`PushPredicateThroughJoin` rule;
2. Infer from the sibling child node combined with the join condition,
which is what you've added here.
That said, the InnerLike joins should already be covered by 1 and might not
be worth being considered again in this optimization rule. Not sure about
LeftSemi joins, so it would be nice if we could have a test case that proves
this optimization does something that has not yet been covered by
`PushPredicateThroughJoin` rule.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
