[jira] [Commented] (YARN-5215) Scheduling containers based on load in the servers
[
https://issues.apache.org/jira/browse/YARN-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321355#comment-15321355
]
Inigo Goiri commented on YARN-5215:
---
This could be a subtask in YARN-1011. However, I thought it was isolated enough
to make it independent. I think YARN-5202 also goes into the same overcommit
direction. To clarify the differences to YARN-1011: here we propose to estimate
the utilization from external processes (e.g., HDFS DataNode or Impala daemons)
and schedule based on that, so I think is orthogonal to the overcommit work.
Happy to move it to a subtask if you guys think it makes more sense there.
Regarding the questions from [~curino]:
# This is just at scheduling time and with the proposed approach we will
allocate the same or less as the current schedulers; so it's conservative in
terms of scheduling and the only issue is it wouldn't use as many resources as
it could. The estimation is: {{externalUtilization = nodeUtilization -
containersUtilization}} and given that the container and node utilization are
captured at different intervals, we could have containersUtilization >
nodeUtilization; I think adding a check for negative values should be enough.
In any case, I don't see issues with enforcing the resources as the only thing
we do is estimating the external utilization.
# This patch only prevents scheduling, further discussion in #4.
# As I mentioned in the first paragraph, I think this is orthogonal to
overcommit, we can run this without overcommiting resources and just prevent
impact on the external load. If overcommitting is enabled, we can still play
this trick.
# For the functionality described in this patch, this is it; we can open other
tasks to do preemption at (1) scheduler level and (2) NM level. We could add
the first one to this patch if needed.
# We have variations of this implemented and running in our cluster for the
last year. In our scenario, we have other latency sensitive load running in
those machines, and we want to guarantee they get as many resources as they
need. Regarding unit testing, I can try to play with the MiniYarnCluster to
fake external load; it shouldn't be too bad to extend
{{TestMiniYarnClusterNodeUtilization}}.
(I tried to use bq to reply but it got messy, I hope this is comprehensive.)
Just to highlight my point on the major discussion, I think this can be a
subtask of YARN-1011 but it's orthogonal to overcommit.
> Scheduling containers based on load in the servers
> --
>
> Key: YARN-5215
> URL: https://issues.apache.org/jira/browse/YARN-5215
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Inigo Goiri
> Attachments: YARN-5215.000.patch
>
>
> Currently YARN runs containers in the servers assuming that they own all the
> resources. The proposal is to use the utilization information in the node and
> the containers to estimate how much is actually available in the NMs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5215) Scheduling containers based on load in the servers
[ https://issues.apache.org/jira/browse/YARN-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321318#comment-15321318 ] Jason Lowe commented on YARN-5215: -- Besides YARN-1011 this is also very similar to the dynamic overcommit prototype we're running, see YARN-5202. It simply has the RM dynamically size the nodes based on the node's reported utilization. Nodes will protect themselves from too much overcommit by preempting containers when utilization gets too high. > Scheduling containers based on load in the servers > -- > > Key: YARN-5215 > URL: https://issues.apache.org/jira/browse/YARN-5215 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Inigo Goiri > Attachments: YARN-5215.000.patch > > > Currently YARN runs containers in the servers assuming that they own all the > resources. The proposal is to use the utilization information in the node and > the containers to estimate how much is actually available in the NMs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5215) Scheduling containers based on load in the servers
[
https://issues.apache.org/jira/browse/YARN-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321233#comment-15321233
]
Hadoop QA commented on YARN-5215:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 26s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 40s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 14s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 38s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 39s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 2
new + 5 unchanged - 0 fixed = 7 total (was 5) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 27s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 34m 41s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 63m 18s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| | hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart |
| | hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore |
| | hadoop.yarn.server.resourcemanager.TestClientRMTokens |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12808984/YARN-5215.000.patch |
| JIRA Issue | YARN-5215 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ceaf562e1a6e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 5a43583 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Commented] (YARN-5215) Scheduling containers based on load in the servers
[ https://issues.apache.org/jira/browse/YARN-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321105#comment-15321105 ] Naganarasimha G R commented on YARN-5215: - almost same set of queries from my end too... i think YARN-1011 is more comprehensive, thoughts? > Scheduling containers based on load in the servers > -- > > Key: YARN-5215 > URL: https://issues.apache.org/jira/browse/YARN-5215 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Inigo Goiri > Attachments: YARN-5215.000.patch > > > Currently YARN runs containers in the servers assuming that they own all the > resources. The proposal is to use the utilization information in the node and > the containers to estimate how much is actually available in the NMs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5215) Scheduling containers based on load in the servers
[ https://issues.apache.org/jira/browse/YARN-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321091#comment-15321091 ] Carlo Curino commented on YARN-5215: This sounds a generally good idea (patch obviously need work). It would be able for example to take into account of HDFS resource consumption, or more generally if other services are run on the same box and have non-constant resource utilization we would not need to pessimistically bound the resources given to YARN. Questions: # How do we ensure that there are no weird feedback loops, e.g., a task is schedule and consume lots of resources, and as a consequence the scheduler lower the load on the node, and this task graphs even more resources? For CPU/Mem we might rely on enforcement, but what about adding non-enforced resources? # Would you also trigger preemption based on this? Or only avoid scheduling more load if the node is busy? # What is the interplay between this and the work on Overcommit? # Patch looks very simple/small for this feature, is that all there is needed here? More dependencies? # How do we test this till we are convinced works? (are you using it anywhere?) [~kasha], [~kkaranasos], [~asuresh] can you guys comment on this? > Scheduling containers based on load in the servers > -- > > Key: YARN-5215 > URL: https://issues.apache.org/jira/browse/YARN-5215 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Inigo Goiri > Attachments: YARN-5215.000.patch > > > Currently YARN runs containers in the servers assuming that they own all the > resources. The proposal is to use the utilization information in the node and > the containers to estimate how much is actually available in the NMs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5215) Scheduling containers based on load in the servers
[ https://issues.apache.org/jira/browse/YARN-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321085#comment-15321085 ] Rohith Sharma K S commented on YARN-5215: - Is this more or less similar to YARN-1011? > Scheduling containers based on load in the servers > -- > > Key: YARN-5215 > URL: https://issues.apache.org/jira/browse/YARN-5215 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Inigo Goiri > Attachments: YARN-5215.000.patch > > > Currently YARN runs containers in the servers assuming that they own all the > resources. The proposal is to use the utilization information in the node and > the containers to estimate how much is actually available in the NMs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
