[
https://issues.apache.org/jira/browse/SPARK-10342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14942475#comment-14942475
]
FangzhouXing commented on SPARK-10342:
--------------------------------------
We are interested in working on this issue. Do you have any suggestions on
where to start?
Sorry for all the questions, this is my first time contributing to Spark.
> Cooperative memory management
> -----------------------------
>
> Key: SPARK-10342
> URL: https://issues.apache.org/jira/browse/SPARK-10342
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, SQL
> Affects Versions: 1.5.0
> Reporter: Davies Liu
> Priority: Critical
>
> We have memory starving problems for a long time, it become worser in 1.5
> since we use larger page.
> In order to increase the memory usage (reduce unnecessary spilling) also
> reduce the risk of OOM, we should manage the memory in a cooperative way, it
> means all the memory consume should be also responsive to release memory
> (spilling) upon others' requests.
> The requests of memory could be different, hard requirement (will crash if
> not allocated) or soft requirement (worse performance if not allocated). Also
> the costs of spilling are also different. We could introduce some kind of
> priority to make them work together better.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
