[
https://issues.apache.org/jira/browse/YARN-624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13736894#comment-13736894
]
Robert Joseph Evans commented on YARN-624:
------------------------------------------
>From my perspective it does not really solve the problem for me. It comes
>close but is not perfect. I am interested in gang scheduling to support
>[storm on yarn|https://github.com/yahoo/storm-yarn/]
The biggest issue I have with this design is knowing the size before the
application is launched. The ultimate goal with storm is to have a system
where multiple separate, but related, storm topologies are processing data
using the same application. We would configure the queues so that if storm
sees a spike in demand it can steal containers from batch processing to grow a
topology and when the spike goes away it would release those containers back.
If the number of containers changes dynamically, by both submitting new
topologies and growing/shrinking existing ones it is impossible to tell YARN
what I need at the beginning.
Gang scheduling is interesting for me because there is a specific number of
containers that each topology is configured to need when that topology is
launched. Without all of those containers there is no reason to launch a
single part of the topology. I can see this happening with a modification to
your approach where the all or nothing happens when the AM submits a request,
and not when the AM is submitted.
I also have a hard time seeing how this would work well with other advanced
features like preemption. For preemption to work well with gang scheduling it
needs to take into account that if it shoots a container in a gang of
containers it is likely going to get back a lot more resources then just one
container. If it is aware of this then it can still shoot the container, but
avoid shooting other containers needlessly because it knows what it is going to
get back.
> Support gang scheduling in the AM RM protocol
> ---------------------------------------------
>
> Key: YARN-624
> URL: https://issues.apache.org/jira/browse/YARN-624
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, scheduler
> Affects Versions: 2.0.4-alpha
> Reporter: Sandy Ryza
> Assignee: Sandy Ryza
>
> Per discussion on YARN-392 and elsewhere, gang scheduling, in which a
> scheduler runs a set of tasks when they can all be run at the same time,
> would be a useful feature for YARN schedulers to support.
> Currently, AMs can approximate this by holding on to containers until they
> get all the ones they need. However, this lends itself to deadlocks when
> different AMs are waiting on the same containers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
