[
https://issues.apache.org/jira/browse/SPARK-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-14588:
---------------------------------
Labels: bulk-closed (was: )
> Consider getting column stats from files (wherever feasible) to get better
> stats for joins
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-14588
> URL: https://issues.apache.org/jira/browse/SPARK-14588
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Reporter: Rajesh Balamohan
> Priority: Major
> Labels: bulk-closed
>
> Broadcast join is determined by "spark.sql.autoBroadcastJoinThreshold". Stats
> for this is determined from the files and by determining the projected
> columns (internally it assumes 20 bytes for string columns). However,
> estimated stats could be invalid if the dataset contains greater than 20
> bytes for string columns . In such instances, broadcast join would not be
> invoked.
> File formats like ORC would be able to provide the raw data size for the
> projected columns. It might be good to consider those (whenever available) to
> determine the accurate stats for broadcast threshold.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
