[
https://issues.apache.org/jira/browse/SPARK-2503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reynold Xin updated SPARK-2503:
-------------------------------
Labels: stability (was: )
> Lower shuffle output buffer (spark.shuffle.file.buffer.kb)
> ----------------------------------------------------------
>
> Key: SPARK-2503
> URL: https://issues.apache.org/jira/browse/SPARK-2503
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle, Spark Core
> Reporter: Reynold Xin
> Assignee: Reynold Xin
> Labels: stability
>
> Each shuffle output stream uses a 100KB buffer. This is simply too large. For
> 10k reducers, 32 cores, that adds up to 32GB in buffer alone.
> The change is simple, but we should make sure there isn't much performance
> regression when lowering the buffer size.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
