Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18692
> After adding the inferred join conditions, it might lead to the child
node's partitioning NOT satisfying the JOIN node's requirements which otherwise
could have.
Isn't it an existing problem? the current constraint propagation framework
infers as many predicates as possible, so we may already hit this problem. I
think we should revisit the constraint propagation framework to think about how
to avoid adding more shuffles, instead of stopping improving this framework to
infer more predicates.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
