[
https://issues.apache.org/jira/browse/YARN-382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhijie Shen updated YARN-382:
-----------------------------
Attachment: YARN-382_1.patch
YARN-382_demo.patch
I've investigated the method of embedding the allocated container into a
ContainerLaunchContext. Please have a look at the attached YARN-382_demo.patch.
IMHO, the modification is not necessary, because it doesn't not solve the
problem essentially. On the other side, the modification is huge and
wide-reaching, such that it raises the risk of causing additional bugs.
AFAIK, the root problem described in YARN-370 is that the CLC for AM container
is created during the application submission stage, before container
allocation, while the CLC for the consequent containers is created after
container allocation (the resource of CLC is set at the creating stage). When
YarnScheduler allocates a container, the requested resource (for AM container,
the number is extracted from the CLC created in advance) will be normalized.
Therefore, the CLC for AM container doesn't know the updated resource, while
the CLC for consequent containers always sees the updated resource. Therefore,
the essential solution to this problem is to update the resource of the CLC for
AM container after container allocation, which is same as what Arun has
proposed in YARN-370. Also, please refer to YARN-382_1.patch here.
Additionally, even if we are to embed the container in CLC directly, to fix the
problem, we still need to update the encapsulated container of the CLC for AM
container after container allocation.
> SchedulerUtils improve way normalizeRequest sets the resource capabilities
> --------------------------------------------------------------------------
>
> Key: YARN-382
> URL: https://issues.apache.org/jira/browse/YARN-382
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: scheduler
> Affects Versions: 2.0.3-alpha
> Reporter: Thomas Graves
> Assignee: Zhijie Shen
> Attachments: YARN-382_1.patch, YARN-382_demo.patch
>
>
> In YARN-370, we changed it from setting the capability to directly setting
> memory and cores:
> - ask.setCapability(normalized);
> + ask.getCapability().setMemory(normalized.getMemory());
> + ask.getCapability().setVirtualCores(normalized.getVirtualCores());
> We did this because it is directly setting the values in the original
> resource object passed in when the AM gets allocated and without it the AM
> doesn't get the resource normalized correctly in the submission context. See
> YARN-370 for more details.
> I think we should find a better way of doing this long term, one so we don't
> have to keep adding things there when new resources are added, two because
> its a bit confusing as to what its doing and prone to someone accidentally
> breaking it in the future again. Something closer to what Arun suggested in
> YARN-370 would be better but we need to make sure all the places work and get
> some more testing on it before putting it in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
