Andrew Or created SPARK-3493:
--------------------------------
Summary: Unrolling behavior is too aggressive in dropping blocks
Key: SPARK-3493
URL: https://issues.apache.org/jira/browse/SPARK-3493
Project: Spark
Issue Type: Improvement
Components: Block Manager
Affects Versions: 1.1.0
Reporter: Andrew Or
Assignee: Andrew Or
Priority: Critical
When unrolling a new block, we (almost) always call `ensureFreeSpace` with a
fixed fraction (spark.storage.unrollFraction) of the block manager's total
memory minus a small delta. This means we're essentially reserving this chunk
of memory in advance because we allocate all of this space even if the new
block is very small.
We need to call `ensureFreeSpace` incrementally and more cautiously to avoid
unnecessarily dropping blocks where possible.
Thanks to [~tdas] for originally reporting this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
