Github user nsyca commented on the issue:
https://github.com/apache/spark/pull/17138
You are right. The plans generated at n-join level comes from the join of
the plans in (n-1)-join level as well as (n-2)-join level and so on. So it
should be able to generate {A,B} join {C,D} plan.
The fallback of this algorithm is the assumption that a lowest cost plan
can only be generated among the lowest cost plans from its previous levels. An
example is an {A,B} plan that could preserve the order of A that makes the join
between {A,B} join {C,D} much less expensive may be discarded by this
algorithm. In addition, because the decision to choose a sorted-merge join,
which preserves the order of its input, versus other join methods is done in
the `QueryPlanner` phase we do not have this information to make a good
decision in the Optimizer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
