Github user wzhfy commented on the issue:
https://github.com/apache/spark/pull/16228
@Tagar We can always find extreme cases to which these formula can't apply.
In my opinion, it's better to over-estimate than under-estimate, which can lead
to OOM problems, e.g. broadcast a very large result.
If A is a big table and B is a small one, every A.k has a match in B (a
common case for PK and FK), then
> cardinality(A) + cardinality(B) - inner_join_cardinality(table_A,
table_B))
becomes card(B), which is dramatically smaller than the real outer join
card. Even more, it can be negative if all A.k and B.k has the same value, the
inner join part becomes a cartesian product.
This formula,
> cardinality = MAX(card(A) + card(B), innerCard(AB))
although over estimates sometimes, it's still obviously better than the
original one in spark: card(A) * card(B).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
