[
https://issues.apache.org/jira/browse/SPARK-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14987589#comment-14987589
]
Yin Huai commented on SPARK-6377:
---------------------------------
This is for setting the number of map output partitions. Right now, it is still
using the static conf (spark.sql.shuffle.partitions). What we have done is to
set the number of post-shuffle partitions. I have updated the title.
> Set the number of map output partitions for Exchange operator automatically
> based on the size of input tables and the reduce-side operation.
> --------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-6377
> URL: https://issues.apache.org/jira/browse/SPARK-6377
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Reporter: Yin Huai
>
> It will be helpful to automatically set the number of shuffle partitions
> based on the size of input tables and the operation at the reduce side for an
> Exchange operator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
