[
https://issues.apache.org/jira/browse/YARN-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15321672#comment-15321672
]
Inigo Goiri commented on YARN-5215:
-----------------------------------
Yes, I realized that the original title didn't mention external load. Fixed
now, sorry about that; I think it's more clear. Feel free to tweak the
description more.
As you mention, we could achieve this by tweaking the "guaranteed" size.
However, I think that having the explicit concept regarding external
utilization makes it simpler and it's compatible with the overcommit approach
(both can be enabled/disabled independently). In addition, the concept of node
utilization is not planned to be used in YARN-1011 for now.
I'm going to post during the next hour a patch with:
* Unit tests
* Conf switches
* Boundary checks
Then, I agree that we need to report this properly to the user. I was thinking
on exposing the {{getExternalUtilization()}} or the updated
{{getUnallocated()}} through the Web UI, etc. If we decide this feature should
go ahead, I would add here or in a new JIRA.
To summarize the issues to discuss/finalize are:
* Decide if this should be a separate feature or within overcommit
* Add unit tests
* Add conf switches
* Add boundary checks
* Interface to expose this information
Regarding YARN-5202 vs YARN-1011, it looks to me like there's a lot of overlap
between them. I think it'd be better to port most of YARN-5202 into YARN-1011.
We probably should move this discussion into one of them.
> Scheduling containers based on external load in the servers
> -----------------------------------------------------------
>
> Key: YARN-5215
> URL: https://issues.apache.org/jira/browse/YARN-5215
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Inigo Goiri
> Attachments: YARN-5215.000.patch
>
>
> Currently YARN runs containers in the servers assuming that they own all the
> resources. The proposal is to use the utilization information in the node and
> the containers to estimate how much is consumed by external processes and
> schedule based on this estimation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
