[
https://issues.apache.org/jira/browse/SPARK-10342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14940315#comment-14940315
]
FangzhouXing commented on SPARK-10342:
--------------------------------------
In my understanding, inactive spark programs will receive memory warning when
memory runs low. And then a handler implemented by programmer will be called to
reduce their program memory usage, just like what happened in iOS app.
Is this correct?
Also, what's an example use-case for this?
> Cooperative memory management
> -----------------------------
>
> Key: SPARK-10342
> URL: https://issues.apache.org/jira/browse/SPARK-10342
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, SQL
> Affects Versions: 1.5.0
> Reporter: Davies Liu
> Priority: Critical
>
> We have memory starving problems for a long time, it become worser in 1.5
> since we use larger page.
> In order to increase the memory usage (reduce unnecessary spilling) also
> reduce the risk of OOM, we should manage the memory in a cooperative way, it
> means all the memory consume should be also responsive to release memory
> (spilling) upon others' requests.
> The requests of memory could be different, hard requirement (will crash if
> not allocated) or soft requirement (worse performance if not allocated). Also
> the costs of spilling are also different. We could introduce some kind of
> priority to make them work together better.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
