GitHub user davies opened a pull request:
https://github.com/apache/spark/pull/11210
[SPARK-13329] [SQL] considering output for statistics of logical plan
The current implementation of statistics of UnaryNode does not considering
output (for example, Project may product much less columns than it's child), we
should considering it to have a better guess.
We usually only join with few columns from a parquet table, the size of
projected plan could be much smaller than the original parquet files. Having a
better guess of size help we choose between broadcast join or sort merge join.
After this PR, I saw a few queries choose broadcast join other than sort
merge join without turning spark.sql.autoBroadcastJoinThreshold for every
query, ended up with about 6-8X improvements on end-to-end time.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/davies/spark statics
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11210.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11210
----
commit 4f96863457ec220bea70113bb3477729be411f07
Author: Davies Liu <[email protected]>
Date: 2016-02-16T01:27:25Z
considering output for statistics
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
