[
https://issues.apache.org/jira/browse/SPARK-1882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14205068#comment-14205068
]
Andrew Ash commented on SPARK-1882:
-----------------------------------
Another major issue is to dynamically scale hardware usage as jobs come online,
you need to scale both memory and cores up and down.
Scaling cores is pretty easy via starting/stopping JVM threads (as Spark does
already) but scaling memory in a JVM is more difficult. Scaling up is somewhat
feasible (set Xms to initial size and Xmx to something huge, then JVM scaling
up happens automatically and could be moderated by the block manager
self-regulating), but scaling memory down is hard.
I started a thread on jdk9-dev here and think this will be more possible with
JVMs in the future but it's not quite there yet:
http://mail.openjdk.java.net/pipermail/jdk9-dev/2014-August/001237.html
Until scaling a JVM heap size down is possible this may just be the best that's
possible -- scaling cores at will and memory up but not down.
> Support dynamic memory sharing in Mesos
> ---------------------------------------
>
> Key: SPARK-1882
> URL: https://issues.apache.org/jira/browse/SPARK-1882
> Project: Spark
> Issue Type: Improvement
> Components: Mesos
> Affects Versions: 1.0.0
> Reporter: Andrew Ash
>
> Fine grained mode Mesos currently supports sharing CPUs very well, but
> requires that memory be pre-partitioned according to the executor memory
> parameter. Mesos supports dynamic memory allocation in addition to dynamic
> CPU allocation, so we should utilize this feature in Spark.
> See below where when the Mesos backend accepts a resource offer it only
> checks that there's enough memory to cover sc.executorMemory, and doesn't
> ever take a fraction of the memory available. The memory offer is accepted
> all or nothing from a pre-defined parameter.
> Coarse mode:
> https://github.com/apache/spark/blob/3ce526b168050c572a1feee8e0121e1426f7d9ee/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala#L208
> Fine mode:
> https://github.com/apache/spark/blob/a5150d199ca97ab2992bc2bb221a3ebf3d3450ba/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala#L114
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
