Hong Shen created SPARK-6104:
--------------------------------
Summary: spark sql shuffle OOM
Key: SPARK-6104
URL: https://issues.apache.org/jira/browse/SPARK-6104
Project: Spark
Issue Type: Improvement
Reporter: Hong Shen
Currently, spark shuffle can use ExternalAppendOnlyMap to combine data that
have fetched, but it depend on ShuffleDependency's aggregator, while spark
sql's shuffle have not define aggregator, so it will easily cause OOM in large
scale of data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
