[
https://issues.apache.org/jira/browse/MESOS-1688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14111125#comment-14111125
]
Vinod Kone commented on MESOS-1688:
-----------------------------------
We can change the allocator to send offers with only cpus or only memory but we
need to make sure frameworks DO NOT launch executors with only cpus or only
memory. This is because it doesn't make sense to launch an executor with only
cpus/memory (esp. when there are no tasks running on it).
This is something we've been talking for a while and we could add a check in
the master to reject such task/executor launches. This likely needs a heads up
to the dev list in case there are any frameworks out there that launch such
executors.
> No offers if no memory is allocatable
> -------------------------------------
>
> Key: MESOS-1688
> URL: https://issues.apache.org/jira/browse/MESOS-1688
> Project: Mesos
> Issue Type: Bug
> Components: master
> Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1
> Reporter: Martin Weindel
> Priority: Critical
>
> The [Spark
> scheduler|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala]
> allocates memory only for the executor and cpu only for its tasks.
> So it can happen that all memory is nearly completely allocated by Spark
> executors, but all cpu resources are idle.
> In this case Mesos does not offer resources anymore, as less than MIN_MEM
> (=32MB) memory is allocatable.
> This effectively causes a dead lock in the Spark job, as it is not offered
> cpu resources needed for launching new tasks.
> see {{HierarchicalAllocatorProcess::allocatable(const Resources&)}} called in
> {{HierarchicalAllocatorProcess::allocate(const hashset<SlaveID>&)}}
> {code}
> template <class RoleSorter, class FrameworkSorter>
> bool
> HierarchicalAllocatorProcess<RoleSorter, FrameworkSorter>::allocatable(
> const Resources& resources)
> {
> ...
> Option<double> cpus = resources.cpus();
> Option<Bytes> mem = resources.mem();
> if (cpus.isSome() && mem.isSome()) {
> return cpus.get() >= MIN_CPUS && mem.get() > MIN_MEM;
> }
> return false;
> }
> {code}
> A possible solution may to completely drop the condition on allocatable
> memory.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
