Github user ioana-delaney commented on the issue:
https://github.com/apache/spark/pull/15363
The following updates were made:
1. Incorporate table and column statistics into the star join detection
algorithm. Fact table is chosen based on table cardinality, and dimensions are
chosen based on the RI constraints. To infer column uniqueness, the algorithm
uses table and column statistics. It compares the number of distinct values
with the total number of rows in the table. If their relative difference is
within certain limits, the column is assumed to be unique. The updated design
document is uploaded to
https://issues.apache.org/jira/secure/attachment/12843316/StarJoinReordering1214.doc.
2. Move star join test cases under Hive test suite, which currently
supports statistics.
3. Rerun TPCDS 1TB with the new table and column statistics. The results
are shown in the design doc.
@wzhfy and @davies Would you please review the changes?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
