[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14565934#comment-14565934 ] Hudson commented on YARN-3547: -- FAILURE: Integrated in Hadoop-Yarn-trunk #943 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/943/]) YARN-3547. FairScheduler: Apps that have no resource demand should not participate scheduling. (Xianyin Xin via kasha) (kasha: rev 3ae2a625018bc8cf431aa19da5bf8fe4ef8c1ad4) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java * hadoop-yarn-project/CHANGES.txt FairScheduler: Apps that have no resource demand should not participate scheduling -- Key: YARN-3547 URL: https://issues.apache.org/jira/browse/YARN-3547 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Reporter: Xianyin Xin Assignee: Xianyin Xin Fix For: 2.8.0 Attachments: YARN-3547.001.patch, YARN-3547.002.patch, YARN-3547.003.patch, YARN-3547.004.patch, YARN-3547.005.patch, YARN-3547.006.patch At present, all of the 'running' apps participate the scheduling process, however, most of them may have no resource demand on a production cluster, as the app's status is running other than waiting for resource at the most of the app's lifetime. It's not a wise way we sort all the 'running' apps and try to fulfill them, especially on a large-scale cluster which has heavy scheduling load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14565959#comment-14565959 ] Hudson commented on YARN-3547: -- SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #213 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/213/]) YARN-3547. FairScheduler: Apps that have no resource demand should not participate scheduling. (Xianyin Xin via kasha) (kasha: rev 3ae2a625018bc8cf431aa19da5bf8fe4ef8c1ad4) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java FairScheduler: Apps that have no resource demand should not participate scheduling -- Key: YARN-3547 URL: https://issues.apache.org/jira/browse/YARN-3547 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Reporter: Xianyin Xin Assignee: Xianyin Xin Fix For: 2.8.0 Attachments: YARN-3547.001.patch, YARN-3547.002.patch, YARN-3547.003.patch, YARN-3547.004.patch, YARN-3547.005.patch, YARN-3547.006.patch At present, all of the 'running' apps participate the scheduling process, however, most of them may have no resource demand on a production cluster, as the app's status is running other than waiting for resource at the most of the app's lifetime. It's not a wise way we sort all the 'running' apps and try to fulfill them, especially on a large-scale cluster which has heavy scheduling load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14566007#comment-14566007 ] Hudson commented on YARN-3547: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #2141 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2141/]) YARN-3547. FairScheduler: Apps that have no resource demand should not participate scheduling. (Xianyin Xin via kasha) (kasha: rev 3ae2a625018bc8cf431aa19da5bf8fe4ef8c1ad4) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java FairScheduler: Apps that have no resource demand should not participate scheduling -- Key: YARN-3547 URL: https://issues.apache.org/jira/browse/YARN-3547 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Reporter: Xianyin Xin Assignee: Xianyin Xin Fix For: 2.8.0 Attachments: YARN-3547.001.patch, YARN-3547.002.patch, YARN-3547.003.patch, YARN-3547.004.patch, YARN-3547.005.patch, YARN-3547.006.patch At present, all of the 'running' apps participate the scheduling process, however, most of them may have no resource demand on a production cluster, as the app's status is running other than waiting for resource at the most of the app's lifetime. It's not a wise way we sort all the 'running' apps and try to fulfill them, especially on a large-scale cluster which has heavy scheduling load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14566016#comment-14566016 ] Hudson commented on YARN-3547: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #202 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/202/]) YARN-3547. FairScheduler: Apps that have no resource demand should not participate scheduling. (Xianyin Xin via kasha) (kasha: rev 3ae2a625018bc8cf431aa19da5bf8fe4ef8c1ad4) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java FairScheduler: Apps that have no resource demand should not participate scheduling -- Key: YARN-3547 URL: https://issues.apache.org/jira/browse/YARN-3547 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Reporter: Xianyin Xin Assignee: Xianyin Xin Fix For: 2.8.0 Attachments: YARN-3547.001.patch, YARN-3547.002.patch, YARN-3547.003.patch, YARN-3547.004.patch, YARN-3547.005.patch, YARN-3547.006.patch At present, all of the 'running' apps participate the scheduling process, however, most of them may have no resource demand on a production cluster, as the app's status is running other than waiting for resource at the most of the app's lifetime. It's not a wise way we sort all the 'running' apps and try to fulfill them, especially on a large-scale cluster which has heavy scheduling load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14566043#comment-14566043 ] Hudson commented on YARN-3547: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #211 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/211/]) YARN-3547. FairScheduler: Apps that have no resource demand should not participate scheduling. (Xianyin Xin via kasha) (kasha: rev 3ae2a625018bc8cf431aa19da5bf8fe4ef8c1ad4) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java * hadoop-yarn-project/CHANGES.txt FairScheduler: Apps that have no resource demand should not participate scheduling -- Key: YARN-3547 URL: https://issues.apache.org/jira/browse/YARN-3547 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Reporter: Xianyin Xin Assignee: Xianyin Xin Fix For: 2.8.0 Attachments: YARN-3547.001.patch, YARN-3547.002.patch, YARN-3547.003.patch, YARN-3547.004.patch, YARN-3547.005.patch, YARN-3547.006.patch At present, all of the 'running' apps participate the scheduling process, however, most of them may have no resource demand on a production cluster, as the app's status is running other than waiting for resource at the most of the app's lifetime. It's not a wise way we sort all the 'running' apps and try to fulfill them, especially on a large-scale cluster which has heavy scheduling load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14566054#comment-14566054 ] Hudson commented on YARN-3547: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2159 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2159/]) YARN-3547. FairScheduler: Apps that have no resource demand should not participate scheduling. (Xianyin Xin via kasha) (kasha: rev 3ae2a625018bc8cf431aa19da5bf8fe4ef8c1ad4) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java FairScheduler: Apps that have no resource demand should not participate scheduling -- Key: YARN-3547 URL: https://issues.apache.org/jira/browse/YARN-3547 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Reporter: Xianyin Xin Assignee: Xianyin Xin Fix For: 2.8.0 Attachments: YARN-3547.001.patch, YARN-3547.002.patch, YARN-3547.003.patch, YARN-3547.004.patch, YARN-3547.005.patch, YARN-3547.006.patch At present, all of the 'running' apps participate the scheduling process, however, most of them may have no resource demand on a production cluster, as the app's status is running other than waiting for resource at the most of the app's lifetime. It's not a wise way we sort all the 'running' apps and try to fulfill them, especially on a large-scale cluster which has heavy scheduling load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14565028#comment-14565028 ] Karthik Kambatla commented on YARN-3547: +1, pending Jenkins. At commit time, I plan to add the following comment: Note that we might consider a non-runnable app for scheduling if it was just removed from runnableApps. This should be okay in practice - (1) because that is rare, and (2) holding the lock to disallow that doesn't alter the reality where conditions for an app to be made non-runnable have already been met. FairScheduler: Apps that have no resource demand should not participate scheduling -- Key: YARN-3547 URL: https://issues.apache.org/jira/browse/YARN-3547 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Reporter: Xianyin Xin Assignee: Xianyin Xin Attachments: YARN-3547.001.patch, YARN-3547.002.patch, YARN-3547.003.patch, YARN-3547.004.patch, YARN-3547.005.patch, YARN-3547.006.patch At present, all of the 'running' apps participate the scheduling process, however, most of them may have no resource demand on a production cluster, as the app's status is running other than waiting for resource at the most of the app's lifetime. It's not a wise way we sort all the 'running' apps and try to fulfill them, especially on a large-scale cluster which has heavy scheduling load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14565733#comment-14565733 ] Xianyin Xin commented on YARN-3547: --- Thanks [~kasha]! Thanks [~kasha], [~leftnoteasy] and [~vinodkv] for your valuable comments! FairScheduler: Apps that have no resource demand should not participate scheduling -- Key: YARN-3547 URL: https://issues.apache.org/jira/browse/YARN-3547 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Reporter: Xianyin Xin Assignee: Xianyin Xin Fix For: 2.8.0 Attachments: YARN-3547.001.patch, YARN-3547.002.patch, YARN-3547.003.patch, YARN-3547.004.patch, YARN-3547.005.patch, YARN-3547.006.patch At present, all of the 'running' apps participate the scheduling process, however, most of them may have no resource demand on a production cluster, as the app's status is running other than waiting for resource at the most of the app's lifetime. It's not a wise way we sort all the 'running' apps and try to fulfill them, especially on a large-scale cluster which has heavy scheduling load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14565584#comment-14565584 ] Hudson commented on YARN-3547: -- FAILURE: Integrated in Hadoop-trunk-Commit #7928 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7928/]) YARN-3547. FairScheduler: Apps that have no resource demand should not participate scheduling. (Xianyin Xin via kasha) (kasha: rev 3ae2a625018bc8cf431aa19da5bf8fe4ef8c1ad4) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java * hadoop-yarn-project/CHANGES.txt FairScheduler: Apps that have no resource demand should not participate scheduling -- Key: YARN-3547 URL: https://issues.apache.org/jira/browse/YARN-3547 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Reporter: Xianyin Xin Assignee: Xianyin Xin Fix For: 2.8.0 Attachments: YARN-3547.001.patch, YARN-3547.002.patch, YARN-3547.003.patch, YARN-3547.004.patch, YARN-3547.005.patch, YARN-3547.006.patch At present, all of the 'running' apps participate the scheduling process, however, most of them may have no resource demand on a production cluster, as the app's status is running other than waiting for resource at the most of the app's lifetime. It's not a wise way we sort all the 'running' apps and try to fulfill them, especially on a large-scale cluster which has heavy scheduling load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[
https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14565452#comment-14565452
]
Hadoop QA commented on YARN-3547:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 8s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 41s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 44s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 46s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 26s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 50m 20s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 88m 39s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12736049/YARN-3547.006.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 7817674 |
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8130/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8130/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8130/console |
This message was automatically generated.
FairScheduler: Apps that have no resource demand should not participate
scheduling
--
Key: YARN-3547
URL: https://issues.apache.org/jira/browse/YARN-3547
Project: Hadoop YARN
Issue Type: Improvement
Components: fairscheduler
Reporter: Xianyin Xin
Assignee: Xianyin Xin
Attachments: YARN-3547.001.patch, YARN-3547.002.patch,
YARN-3547.003.patch, YARN-3547.004.patch, YARN-3547.005.patch,
YARN-3547.006.patch
At present, all of the 'running' apps participate the scheduling process,
however, most of them may have no resource demand on a production cluster, as
the app's status is running other than waiting for resource at the most of
the app's lifetime. It's not a wise way we sort all the 'running' apps and
try to fulfill them, especially on a large-scale cluster which has heavy
scheduling load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14564039#comment-14564039 ] Xianyin Xin commented on YARN-3547: --- Thanks [~kasha], [~leftnoteasy]. [~kasha], do you have any other concerns on YARN-3547.005.patch? FairScheduler: Apps that have no resource demand should not participate scheduling -- Key: YARN-3547 URL: https://issues.apache.org/jira/browse/YARN-3547 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Reporter: Xianyin Xin Assignee: Xianyin Xin Attachments: YARN-3547.001.patch, YARN-3547.002.patch, YARN-3547.003.patch, YARN-3547.004.patch, YARN-3547.005.patch At present, all of the 'running' apps participate the scheduling process, however, most of them may have no resource demand on a production cluster, as the app's status is running other than waiting for resource at the most of the app's lifetime. It's not a wise way we sort all the 'running' apps and try to fulfill them, especially on a large-scale cluster which has heavy scheduling load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[
https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14564128#comment-14564128
]
Karthik Kambatla commented on YARN-3547:
Per recent discussion, let us go with the approach in YARN-3547.004.patch?
Review comments for 004 patch:
# What is the readLock protecting? If it is only runnableApps, we should
release the lock as soon as we are done iterating through runnableApps.
# Use {{(!pending.equals(Resources.none())}} instead of an elaborate check?
# We can avoid importing {{Set}} by just using {{TreeSet}}. Since it is all
local, that should be fine.
FairScheduler: Apps that have no resource demand should not participate
scheduling
--
Key: YARN-3547
URL: https://issues.apache.org/jira/browse/YARN-3547
Project: Hadoop YARN
Issue Type: Improvement
Components: fairscheduler
Reporter: Xianyin Xin
Assignee: Xianyin Xin
Attachments: YARN-3547.001.patch, YARN-3547.002.patch,
YARN-3547.003.patch, YARN-3547.004.patch, YARN-3547.005.patch
At present, all of the 'running' apps participate the scheduling process,
however, most of them may have no resource demand on a production cluster, as
the app's status is running other than waiting for resource at the most of
the app's lifetime. It's not a wise way we sort all the 'running' apps and
try to fulfill them, especially on a large-scale cluster which has heavy
scheduling load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14563934#comment-14563934 ] Wangda Tan commented on YARN-3547: -- I'm fine with the approach suggested by [~kasha]. [~xinxianyin], you can figure out with Karthik about how to continue with the patch. FairScheduler: Apps that have no resource demand should not participate scheduling -- Key: YARN-3547 URL: https://issues.apache.org/jira/browse/YARN-3547 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Reporter: Xianyin Xin Assignee: Xianyin Xin Attachments: YARN-3547.001.patch, YARN-3547.002.patch, YARN-3547.003.patch, YARN-3547.004.patch, YARN-3547.005.patch At present, all of the 'running' apps participate the scheduling process, however, most of them may have no resource demand on a production cluster, as the app's status is running other than waiting for resource at the most of the app's lifetime. It's not a wise way we sort all the 'running' apps and try to fulfill them, especially on a large-scale cluster which has heavy scheduling load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[
https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14564211#comment-14564211
]
Xianyin Xin commented on YARN-3547:
---
Sorry [~kasha], i correct the comment in
https://issues.apache.org/jira/browse/YARN-3547?focusedCommentId=14564204page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14564204.
I firstly use Resources.greaterThan(Resources.none()), which is not suitable
for zero memory container request. But {{(!pending.equals(Resources.none())}}
can.
FairScheduler: Apps that have no resource demand should not participate
scheduling
--
Key: YARN-3547
URL: https://issues.apache.org/jira/browse/YARN-3547
Project: Hadoop YARN
Issue Type: Improvement
Components: fairscheduler
Reporter: Xianyin Xin
Assignee: Xianyin Xin
Attachments: YARN-3547.001.patch, YARN-3547.002.patch,
YARN-3547.003.patch, YARN-3547.004.patch, YARN-3547.005.patch,
YARN-3547.006.patch
At present, all of the 'running' apps participate the scheduling process,
however, most of them may have no resource demand on a production cluster, as
the app's status is running other than waiting for resource at the most of
the app's lifetime. It's not a wise way we sort all the 'running' apps and
try to fulfill them, especially on a large-scale cluster which has heavy
scheduling load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[
https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14564204#comment-14564204
]
Xianyin Xin commented on YARN-3547:
---
Sorry [~kasha], i misunderstood the discussion. Anyway, thanks for your
reviewing.
Only comment on point 2:
In fact i firstly use {{(!pending.equals(Resources.none())}}, however, a test
case in {{TestFairScheduler}} is tesing zero memory container request 0MB,
1core. To avoid this, i changed the check. But it seems the test has been
removed recently.
Now all of the 3 points are addressed.
FairScheduler: Apps that have no resource demand should not participate
scheduling
--
Key: YARN-3547
URL: https://issues.apache.org/jira/browse/YARN-3547
Project: Hadoop YARN
Issue Type: Improvement
Components: fairscheduler
Reporter: Xianyin Xin
Assignee: Xianyin Xin
Attachments: YARN-3547.001.patch, YARN-3547.002.patch,
YARN-3547.003.patch, YARN-3547.004.patch, YARN-3547.005.patch
At present, all of the 'running' apps participate the scheduling process,
however, most of them may have no resource demand on a production cluster, as
the app's status is running other than waiting for resource at the most of
the app's lifetime. It's not a wise way we sort all the 'running' apps and
try to fulfill them, especially on a large-scale cluster which has heavy
scheduling load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[
https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14561894#comment-14561894
]
Wangda Tan commented on YARN-3547:
--
[~kasha], do you have any concern if using
{{SchedulerApplicationAttempt.getAppAttemptResourceUsage().getPending()}}.
FairScheduler: Apps that have no resource demand should not participate
scheduling
--
Key: YARN-3547
URL: https://issues.apache.org/jira/browse/YARN-3547
Project: Hadoop YARN
Issue Type: Improvement
Components: fairscheduler
Reporter: Xianyin Xin
Assignee: Xianyin Xin
Attachments: YARN-3547.001.patch, YARN-3547.002.patch,
YARN-3547.003.patch, YARN-3547.004.patch, YARN-3547.005.patch
At present, all of the 'running' apps participate the scheduling process,
however, most of them may have no resource demand on a production cluster, as
the app's status is running other than waiting for resource at the most of
the app's lifetime. It's not a wise way we sort all the 'running' apps and
try to fulfill them, especially on a large-scale cluster which has heavy
scheduling load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14561924#comment-14561924 ] Karthik Kambatla commented on YARN-3547: Sorry for the delay in response here. I am okay with using SchedulerApplicationAttempt.getAppAttemptResourceUsage().getPending(). I don't anticipate any issues due to potential mismatch between this value and getDemand - getResourceUsage. We can always fix them if we run into them. FairScheduler: Apps that have no resource demand should not participate scheduling -- Key: YARN-3547 URL: https://issues.apache.org/jira/browse/YARN-3547 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Reporter: Xianyin Xin Assignee: Xianyin Xin Attachments: YARN-3547.001.patch, YARN-3547.002.patch, YARN-3547.003.patch, YARN-3547.004.patch, YARN-3547.005.patch At present, all of the 'running' apps participate the scheduling process, however, most of them may have no resource demand on a production cluster, as the app's status is running other than waiting for resource at the most of the app's lifetime. It's not a wise way we sort all the 'running' apps and try to fulfill them, especially on a large-scale cluster which has heavy scheduling load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14560255#comment-14560255 ] Xianyin Xin commented on YARN-3547: --- Hi [~kasha], [~leftnoteasy], can we reach a consensus as the patch is just a simple fix? FairScheduler: Apps that have no resource demand should not participate scheduling -- Key: YARN-3547 URL: https://issues.apache.org/jira/browse/YARN-3547 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Reporter: Xianyin Xin Assignee: Xianyin Xin Attachments: YARN-3547.001.patch, YARN-3547.002.patch, YARN-3547.003.patch, YARN-3547.004.patch, YARN-3547.005.patch At present, all of the 'running' apps participate the scheduling process, however, most of them may have no resource demand on a production cluster, as the app's status is running other than waiting for resource at the most of the app's lifetime. It's not a wise way we sort all the 'running' apps and try to fulfill them, especially on a large-scale cluster which has heavy scheduling load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[
https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547461#comment-14547461
]
Xianyin Xin commented on YARN-3547:
---
Agree, [~leftnoteasy]. Now we have YARN-3547.004.patch using
{{SchedulerApplicationAttempt.getAppAttemptResourceUsage().getPending()}} and
YARN-3547.005.patch using {{getDemand() - getResourceUsage()}}.
FairScheduler: Apps that have no resource demand should not participate
scheduling
--
Key: YARN-3547
URL: https://issues.apache.org/jira/browse/YARN-3547
Project: Hadoop YARN
Issue Type: Improvement
Components: fairscheduler
Reporter: Xianyin Xin
Assignee: Xianyin Xin
Attachments: YARN-3547.001.patch, YARN-3547.002.patch,
YARN-3547.003.patch, YARN-3547.004.patch, YARN-3547.005.patch
At present, all of the 'running' apps participate the scheduling process,
however, most of them may have no resource demand on a production cluster, as
the app's status is running other than waiting for resource at the most of
the app's lifetime. It's not a wise way we sort all the 'running' apps and
try to fulfill them, especially on a large-scale cluster which has heavy
scheduling load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14544995#comment-14544995 ] Xianyin Xin commented on YARN-3547: --- [~kasha], can you please have a look? FairScheduler: Apps that have no resource demand should not participate scheduling -- Key: YARN-3547 URL: https://issues.apache.org/jira/browse/YARN-3547 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Reporter: Xianyin Xin Assignee: Xianyin Xin Attachments: YARN-3547.001.patch, YARN-3547.002.patch, YARN-3547.003.patch, YARN-3547.004.patch, YARN-3547.005.patch At present, all of the 'running' apps participate the scheduling process, however, most of them may have no resource demand on a production cluster, as the app's status is running other than waiting for resource at the most of the app's lifetime. It's not a wise way we sort all the 'running' apps and try to fulfill them, especially on a large-scale cluster which has heavy scheduling load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14545821#comment-14545821 ] Wangda Tan commented on YARN-3547: -- SchedulerApplicationAttempt.getAppAttemptResourceUsage().getPending() is always up-to-dated, FS can use it to do pending resource updating. FairScheduler: Apps that have no resource demand should not participate scheduling -- Key: YARN-3547 URL: https://issues.apache.org/jira/browse/YARN-3547 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Reporter: Xianyin Xin Assignee: Xianyin Xin Attachments: YARN-3547.001.patch, YARN-3547.002.patch, YARN-3547.003.patch, YARN-3547.004.patch, YARN-3547.005.patch At present, all of the 'running' apps participate the scheduling process, however, most of them may have no resource demand on a production cluster, as the app's status is running other than waiting for resource at the most of the app's lifetime. It's not a wise way we sort all the 'running' apps and try to fulfill them, especially on a large-scale cluster which has heavy scheduling load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[
https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14539692#comment-14539692
]
Xianyin Xin commented on YARN-3547:
---
Sorry for late response, [~kasha]. {{getDemand() - getResourceUsage()}} is
another choice, but i think
{{SchedulerApplicationAttempt.getAppAttemptResourceUsage().getPending()}} would
be better, because the pending information is immediately updated as the
request is update/fulfilled, while the former was derived from request in
another update thread. The request is changed when there's resource request
update. If AM submits resource requests
{code}
location=* containers=1
location=rack containers=1
location=node1 containers=1
location=node2 containers=1
{code}
and then it is fulfilled on node1, the request becomes
{code}
location=* containers=0
location=node2 containers=1
{code}
Later the AM doesn't submit a new request (in which the #container set to 0),
this app will also participate scheduling, even though it's demand has been
fulfilled.
Besides, when i try use {{getDemand() - getResourceUsage()}}, i got some test
failures. Now i was checking the cause.
FairScheduler: Apps that have no resource demand should not participate
scheduling
--
Key: YARN-3547
URL: https://issues.apache.org/jira/browse/YARN-3547
Project: Hadoop YARN
Issue Type: Improvement
Components: fairscheduler
Reporter: Xianyin Xin
Assignee: Xianyin Xin
Attachments: YARN-3547.001.patch, YARN-3547.002.patch,
YARN-3547.003.patch, YARN-3547.004.patch
At present, all of the 'running' apps participate the scheduling process,
however, most of them may have no resource demand on a production cluster, as
the app's status is running other than waiting for resource at the most of
the app's lifetime. It's not a wise way we sort all the 'running' apps and
try to fulfill them, especially on a large-scale cluster which has heavy
scheduling load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14532131#comment-14532131 ] Xianyin Xin commented on YARN-3547: --- Thanks for your feedback [~vinodkv]. We use SchedulerApplicationAttempt.getAppAttemptResourceUsage().getPending(), not hasPendingResourceRequest(). In fact i don't quite understand what you mean. attemptResourceUsage records the resource information of an attempt, no matter it is scheduled by CS or Fair. Would you please explain your concern? FairScheduler: Apps that have no resource demand should not participate scheduling -- Key: YARN-3547 URL: https://issues.apache.org/jira/browse/YARN-3547 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Reporter: Xianyin Xin Assignee: Xianyin Xin Labels: BB2015-05-TBR Attachments: YARN-3547.001.patch, YARN-3547.002.patch, YARN-3547.003.patch, YARN-3547.004.patch At present, all of the 'running' apps participate the scheduling process, however, most of them may have no resource demand on a production cluster, as the app's status is running other than waiting for resource at the most of the app's lifetime. It's not a wise way we sort all the 'running' apps and try to fulfill them, especially on a large-scale cluster which has heavy scheduling load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14531225#comment-14531225 ] Vinod Kumar Vavilapalli commented on YARN-3547: --- [~leftnoteasy] / [~xinxianyin], can we not use SchedulerApplicationAttempt.hasPendingResourceRequest() the way CS's LeafQueue uses it? FairScheduler: Apps that have no resource demand should not participate scheduling -- Key: YARN-3547 URL: https://issues.apache.org/jira/browse/YARN-3547 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Reporter: Xianyin Xin Assignee: Xianyin Xin Labels: BB2015-05-TBR Attachments: YARN-3547.001.patch, YARN-3547.002.patch, YARN-3547.003.patch, YARN-3547.004.patch At present, all of the 'running' apps participate the scheduling process, however, most of them may have no resource demand on a production cluster, as the app's status is running other than waiting for resource at the most of the app's lifetime. It's not a wise way we sort all the 'running' apps and try to fulfill them, especially on a large-scale cluster which has heavy scheduling load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[
https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14529974#comment-14529974
]
Hadoop QA commented on YARN-3547:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 41s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 35s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 33s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 45s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 17s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | yarn tests | 53m 39s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 90m 2s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12730699/YARN-3547.004.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 90b3845 |
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7724/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/7724/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7724/console |
This message was automatically generated.
FairScheduler: Apps that have no resource demand should not participate
scheduling
--
Key: YARN-3547
URL: https://issues.apache.org/jira/browse/YARN-3547
Project: Hadoop YARN
Issue Type: Improvement
Components: fairscheduler
Reporter: Xianyin Xin
Assignee: Xianyin Xin
Labels: BB2015-05-TBR
Attachments: YARN-3547.001.patch, YARN-3547.002.patch,
YARN-3547.003.patch, YARN-3547.004.patch
At present, all of the 'running' apps participate the scheduling process,
however, most of them may have no resource demand on a production cluster, as
the app's status is running other than waiting for resource at the most of
the app's lifetime. It's not a wise way we sort all the 'running' apps and
try to fulfill them, especially on a large-scale cluster which has heavy
scheduling load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[
https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527827#comment-14527827
]
Hadoop QA commented on YARN-3547:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 39s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 35s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 36s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 49s | The applied patch generated 1
new checkstyle issues (total was 9, now 10). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 15s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:red}-1{color} | yarn tests | 52m 58s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 89m 29s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12730098/YARN-3547.003.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 551615f |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/7696/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7696/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/7696/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7696/console |
This message was automatically generated.
FairScheduler: Apps that have no resource demand should not participate
scheduling
--
Key: YARN-3547
URL: https://issues.apache.org/jira/browse/YARN-3547
Project: Hadoop YARN
Issue Type: Improvement
Components: fairscheduler
Reporter: Xianyin Xin
Assignee: Xianyin Xin
Attachments: YARN-3547.001.patch, YARN-3547.002.patch,
YARN-3547.003.patch
At present, all of the 'running' apps participate the scheduling process,
however, most of them may have no resource demand on a production cluster, as
the app's status is running other than waiting for resource at the most of
the app's lifetime. It's not a wise way we sort all the 'running' apps and
try to fulfill them, especially on a large-scale cluster which has heavy
scheduling load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527541#comment-14527541 ] Wangda Tan commented on YARN-3547: -- [~xinxianyin], Thanks for updating, I think using either TreeSet or List (needs resort) should be fine. +[~kasha]. FairScheduler: Apps that have no resource demand should not participate scheduling -- Key: YARN-3547 URL: https://issues.apache.org/jira/browse/YARN-3547 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Reporter: Xianyin Xin Assignee: Xianyin Xin Attachments: YARN-3547.001.patch, YARN-3547.002.patch, YARN-3547.003.patch At present, all of the 'running' apps participate the scheduling process, however, most of them may have no resource demand on a production cluster, as the app's status is running other than waiting for resource at the most of the app's lifetime. It's not a wise way we sort all the 'running' apps and try to fulfill them, especially on a large-scale cluster which has heavy scheduling load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526210#comment-14526210 ] Xianyin Xin commented on YARN-3547: --- Thanks [~leftnoteasy]! Attached a new patch using Set. When i review the patch, I found patch-001 is ill-considered, because unlike CS, Fair uses list to organize the apps at the very beginning, thus the newly added sorted set may become un-sorted because of the update resource usage or update fair share operations. We have two choices, the first is we continue to use the solution in patch-001, but fix it with considerations that we reorder the set when we update resource usage or fair shares just like in CS; the second is like patch-003, before we do assign, we scan the runningApps, and pick out the apps with real request to construct a set. Patch-003 is more efficient than patch-002, and the change of code is less than patch-001 (if fixed). What do you think, [~leftnoteasy]? Change the name pendingApps to pendingForResourceApps because pending is ambiguous. Maybe we need a better name. FairScheduler: Apps that have no resource demand should not participate scheduling -- Key: YARN-3547 URL: https://issues.apache.org/jira/browse/YARN-3547 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Reporter: Xianyin Xin Assignee: Xianyin Xin Attachments: YARN-3547.001.patch, YARN-3547.002.patch At present, all of the 'running' apps participate the scheduling process, however, most of them may have no resource demand on a production cluster, as the app's status is running other than waiting for resource at the most of the app's lifetime. It's not a wise way we sort all the 'running' apps and try to fulfill them, especially on a large-scale cluster which has heavy scheduling load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522518#comment-14522518 ] Wangda Tan commented on YARN-3547: -- [~xinxianyin], just added you to contributor list so that you can assign JIRAs to yourself. Looked at your patch, now it make the pendingApps to be apps participate scheduling. It's an ArrayList so every add/remove takes O(n). I think it's better to make it to be a Set to do faster add/remove. FairScheduler: Apps that have no resource demand should not participate scheduling -- Key: YARN-3547 URL: https://issues.apache.org/jira/browse/YARN-3547 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Reporter: Xianyin Xin Assignee: Xianyin Xin Attachments: YARN-3547.001.patch, YARN-3547.002.patch At present, all of the 'running' apps participate the scheduling process, however, most of them may have no resource demand on a production cluster, as the app's status is running other than waiting for resource at the most of the app's lifetime. It's not a wise way we sort all the 'running' apps and try to fulfill them, especially on a large-scale cluster which has heavy scheduling load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3547) FairScheduler: Apps that have no resource demand should not participate scheduling
[ https://issues.apache.org/jira/browse/YARN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14516714#comment-14516714 ] Xianyin Xin commented on YARN-3547: --- Thanks for pointing out the duplication, [~jlowe], and thanks for comment, [~jlowe], [~leftnoteasy]. [~leftnoteasy], are we going to implement non-exclusive node labels support in FairScheduler or is there any plan? If so, this problem can be solved incidentally just like YARN-2176. FairScheduler: Apps that have no resource demand should not participate scheduling -- Key: YARN-3547 URL: https://issues.apache.org/jira/browse/YARN-3547 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Reporter: Xianyin Xin Attachments: YARN-3547.001.patch At present, all of the 'running' apps participate the scheduling process, however, most of them may have no resource demand on a production cluster, as the app's status is running other than waiting for resource at the most of the app's lifetime. It's not a wise way we sort all the 'running' apps and try to fulfill them, especially on a large-scale cluster which has heavy scheduling load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
