jin xing created SPARK-20801:
--------------------------------
Summary: Store accurate size of blocks in MapStatus when it's
above threshold.
Key: SPARK-20801
URL: https://issues.apache.org/jira/browse/SPARK-20801
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 2.1.1
Reporter: jin xing
Currently, when number of reduces is above 2000, HighlyCompressedMapStatus is
used to store size of blocks. in HighlyCompressedMapStatus, only average size
is stored for non empty blocks. Which is not good for memory control when we
shuffle blocks. It makes sense to store the accurate size of block when it's
above threshold.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
