Reynold Xin created SPARK-2503:
----------------------------------
Summary: Lower shuffle output buffer (spark.shuffle.file.buffer.kb)
Key: SPARK-2503
URL: https://issues.apache.org/jira/browse/SPARK-2503
Project: Spark
Issue Type: Improvement
Components: Shuffle, Spark Core
Reporter: Reynold Xin
Assignee: Reynold Xin
Each shuffle output stream uses a 100KB buffer. This is simply too large. For
10k reducers, 32 cores, that adds up to 32GB in buffer alone.
The change is simple, but we should make sure there isn't much performance
regression when lowering the buffer size.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
