[
https://issues.apache.org/jira/browse/YARN-373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13569659#comment-13569659
]
Hitesh Shah commented on YARN-373:
----------------------------------
@Alejandro, had a few questions regarding the jira description:
- is this only for re-sizing an existing allocated container in terms of the
cpu/memory resources allocated to it? i.e either increase or decrease the size
of the container ( reduce memory as you mentioned for shuffle or increase it
for allocating a reduce task on it for example ) ?
- the container today exits as soon as the process completes as part of the
container. Is the proposal also to change the MR tasks today to convert and
launch a shuffle handler within the same process space after the map task
completes? Or change the current notion of what constitutes completion of a
container?
- would the new 'task' running on the container need to run with the same
existing set of localized resources or would there be new localization needed?
> Allow an AM to reuse the resources allocated to container for a new container
> -----------------------------------------------------------------------------
>
> Key: YARN-373
> URL: https://issues.apache.org/jira/browse/YARN-373
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
> Affects Versions: 2.0.3-alpha
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> When a container completes, instead the corresponding resources being freed
> up, it should be possible for the AM to reuse the assigned resources for a
> new container.
> As part of the reallocation, the AM would notify the RM about partial
> resources being freed up and the RM would make the necessary corrections in
> the corresponding node.
> With this functionality, an AM can ensure it gets a container in the same
> node where previous containers run.
> This will allow getting rid of the ShuffleHandler as a service in the NMs and
> run it as regular container task of the corresponding AM. In this case, the
> reallocation would reduce the CPU/MEM obtained for the original container to
> the what is needed for serving the shuffle. Note that in this example the MR
> AM would only do this reallocation for one of the many tasks that may have
> run in a particular node (as a single shuffle task could serve all the map
> outputs from all map tasks run in that node).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
