[
https://issues.apache.org/jira/browse/SPARK-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289899#comment-15289899
]
Apache Spark commented on SPARK-15392:
--------------------------------------
User 'davies' has created a pull request for this issue:
https://github.com/apache/spark/pull/13179
> The default value of size estimation is not good
> ------------------------------------------------
>
> Key: SPARK-15392
> URL: https://issues.apache.org/jira/browse/SPARK-15392
> Project: Spark
> Issue Type: Bug
> Affects Versions: 2.0.0
> Reporter: Davies Liu
> Assignee: Davies Liu
>
> We use autoBroadcastJoinThreshold + 1L as the default value of size
> estimation, that is not good in 2.0, because we will calculate the size based
> on size of schema, then the estimation could be less than
> autoBroadcastJoinThreshold if you have an SELECT on top of an DataFrame
> created from RDD.
> We should use an even bigger default value, for example, MaxLong.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
