[jira] [Commented] (YARN-10535) Make queue placement in CapacityScheduler compliant with auto-queue-placement
[
https://issues.apache.org/jira/browse/YARN-10535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266473#comment-17266473
]
Hadoop QA commented on YARN-10535:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
13s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
4s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 24s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
0s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config;
considering switching to SpotBugs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
58s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/492/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html{color}
| {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
in trunk has 1 extant findbugs warnings. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/492/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 5 new + 1 unchanged - 1 fixed = 6 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} |
{color:
[jira] [Commented] (YARN-10512) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS
[
https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266444#comment-17266444
]
Hadoop QA commented on YARN-10512:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 26m
30s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 5
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
15s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 34s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
46s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 36s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/490/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 37 new + 149 unchanged - 0 fixed = 186 total (was 149)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/490/artifact/out/whitespace-eol.txt{color}
| {color:red} The patch has 7 line(s) that end in whitespace. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:
[jira] [Commented] (YARN-10512) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS
[
https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266418#comment-17266418
]
Hadoop QA commented on YARN-10512:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
19s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 5
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
11s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 27s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
50s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/489/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 37 new + 149 unchanged - 0 fixed = 186 total (was 149)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/489/artifact/out/whitespace-eol.txt{color}
| {color:red} The patch has 7 line(s) that end in whitespace. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:
[jira] [Commented] (YARN-10535) Make queue placement in CapacityScheduler compliant with auto-queue-placement
[ https://issues.apache.org/jira/browse/YARN-10535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266413#comment-17266413 ] Gergely Pollak commented on YARN-10535: --- Dependency merged patchset 3 is a reupload to retrigger the build. > Make queue placement in CapacityScheduler compliant with auto-queue-placement > - > > Key: YARN-10535 > URL: https://issues.apache.org/jira/browse/YARN-10535 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Wangda Tan >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10535.001.patch, YARN-10535.002.patch, > YARN-10535.003.patch > > > Once YARN-10506 is done, we need to call the API from the queue placement > policy to create queues. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10535) Make queue placement in CapacityScheduler compliant with auto-queue-placement
[ https://issues.apache.org/jira/browse/YARN-10535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Pollak updated YARN-10535: -- Attachment: YARN-10535.003.patch > Make queue placement in CapacityScheduler compliant with auto-queue-placement > - > > Key: YARN-10535 > URL: https://issues.apache.org/jira/browse/YARN-10535 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Wangda Tan >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10535.001.patch, YARN-10535.002.patch, > YARN-10535.003.patch > > > Once YARN-10506 is done, we need to call the API from the queue placement > policy to create queues. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[
https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266391#comment-17266391
]
Wangda Tan commented on YARN-10532:
---
[~zhuqi], thanks for the patch, I took a brief look (nothing in detail yet). I
think here're my overall thoughts:
1) Instead of changing GuaranteedOrZeroCapacityOverTimePolicy, I suggest
creating a new Policy (maybe we can make it runnable by default so we don't
have to create another config). The policy just simply monitor queue last used
time and delete queues when needed.
2) The latest patch only remove queue when reinitialize queue is called, it is
not a frequent used method, we have cluster which doesn't have queue
reinitialize called for a long period of time. Can we just call it from the
policy (then we may need to introduce a method in CapacityScheduler to delete
queue).
3) When we delete a queue, we need to check inside Scheduler to make sure 1)
there's nothing running in the queue; 2) last usage timestamp is expired. We
need to do this to avoid racing condition that Policy think a queue is
deletable but Scheduler doesn't.
4) Policy modifies "expiredQueue" field, I suggest to make Policy just read the
state, and let Scheduler delete it.
5) An additional requirement we should keep in mind:
Scenario A:
{code:java}
- At time T0, policy signals scheduler to delete queue A (an auto created
queue).
- Before the signal arrives to scheduler, an app submitted to scheduler (T1).
T1 > T0
- When at T2 (T2 > T1), the signal arrived at scheduler, scheduler should avoid
removing the queue A because now it is used.{code}
Scenario B:
{code:java}
- At time T0, policy signals scheduler to delete queue A (an auto created
queue).
- At T1 (T1 > T0), scheduler got the signal and deleted the queue.
- At T2 (T2 > T1), an app submitted to scheduler.
Scheduler should immediately recreate the queue, in another word, deleting an
dynamic queue should NEVER fail a submitted application.{code}
> Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is
> not being used
>
>
> Key: YARN-10532
> URL: https://issues.apache.org/jira/browse/YARN-10532
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10532.001.patch
>
>
> It's better if we can delete auto-created queues when they are not in use for
> a period of time (like 5 mins). It will be helpful when we have a large
> number of auto-created queues (e.g. from 500 users), but only a small subset
> of queues are actively used.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266385#comment-17266385 ] Wangda Tan commented on YARN-10506: --- Verified the test failure is an intermittent one which is not caused by the patch, but I still added a minor change to call mockRM.shutdown to cleanup mockRM before starting a new one (see ver.17). Committed the ver.17 patch to trunk. Thanks for the contribution from [~zhuqi] and [~gandras]! And really appreciate reviews from [~shuzirra], [~pbacsko], and [~bteke]! > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch, > YARN-10506.014.patch, YARN-10506.015.patch, YARN-10506.016.patch, > YARN-10506.017.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266384#comment-17266384
]
Hadoop QA commented on YARN-10506:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 11s{color}
| {color:red}{color} | {color:red} YARN-10506 does not apply to trunk. Rebase
required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-10506 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13018848/YARN-10506.017.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/491/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch,
> YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch,
> YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch,
> YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch,
> YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch,
> YARN-10506.014.patch, YARN-10506.015.patch, YARN-10506.016.patch,
> YARN-10506.017.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-10506: -- Attachment: YARN-10506.017.patch > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch, > YARN-10506.014.patch, YARN-10506.015.patch, YARN-10506.016.patch, > YARN-10506.017.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10512) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS
[
https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266376#comment-17266376
]
Hadoop QA commented on YARN-10512:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
23s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 5
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
59s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 11s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
48s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/488/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 37 new + 149 unchanged - 0 fixed = 186 total (was 149)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/488/artifact/out/whitespace-eol.txt{color}
| {color:red} The patch has 7 line(s) that end in whitespace. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:
[jira] [Updated] (YARN-7200) SLS generates a realtimetrack.json file but that file is missing the closing ']'
[
https://issues.apache.org/jira/browse/YARN-7200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth updated YARN-7200:
-
Fix Version/s: 3.4.0
> SLS generates a realtimetrack.json file but that file is missing the closing
> ']'
>
>
> Key: YARN-7200
> URL: https://issues.apache.org/jira/browse/YARN-7200
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler-load-simulator
>Reporter: Grant Sohn
>Assignee: Agshin Kazimli
>Priority: Minor
> Labels: newbie, newbie++
> Fix For: 3.4.0
>
> Attachments: YARN-7200-branch-trunk.patch, YARN-7200.002.patch,
> YARN-7200.003.patch, YARN-7200.004.patch, YARN-7200.005.patch,
> snemeth-testing-20201113.zip
>
>
> File
> hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SchedulerMetrics.java
> shows:
> {noformat}
> void tearDown() throws Exception {
> if (metricsLogBW != null) {
> metricsLogBW.write("]");
> metricsLogBW.close();
> }
>
> {noformat}
> So the exit logic is flawed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7200) SLS generates a realtimetrack.json file but that file is missing the closing ']'
[
https://issues.apache.org/jira/browse/YARN-7200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266361#comment-17266361
]
Szilard Nemeth commented on YARN-7200:
--
Thanks [~akshink] for the new patch and the explanation.
Makes sense for me.
Latest patch LGTM, committed to trunk.
> SLS generates a realtimetrack.json file but that file is missing the closing
> ']'
>
>
> Key: YARN-7200
> URL: https://issues.apache.org/jira/browse/YARN-7200
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler-load-simulator
>Reporter: Grant Sohn
>Assignee: Agshin Kazimli
>Priority: Minor
> Labels: newbie, newbie++
> Attachments: YARN-7200-branch-trunk.patch, YARN-7200.002.patch,
> YARN-7200.003.patch, YARN-7200.004.patch, YARN-7200.005.patch,
> snemeth-testing-20201113.zip
>
>
> File
> hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SchedulerMetrics.java
> shows:
> {noformat}
> void tearDown() throws Exception {
> if (metricsLogBW != null) {
> metricsLogBW.write("]");
> metricsLogBW.close();
> }
>
> {noformat}
> So the exit logic is flawed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10490) "yarn top" command not quitting completely with ctrl+c
[ https://issues.apache.org/jira/browse/YARN-10490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266356#comment-17266356 ] Szilard Nemeth commented on YARN-10490: --- Hi [~akshink], Thanks for working on this. I'm a bit confused. Looking at the source code of TopCLI, there's a method named "clearScreen": https://github.com/apache/hadoop/blob/630f8ddd2c72c27b02032ed3cf9dde15745ed605/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/TopCLI.java#L473 It's implementation prints the CLEAR string from this class to the System.out stream. You introduced a string called CLEAR_SCREEN. So the namings are quite confusing. Did you test to call the clearScreen method from the shutdown hook? If not, please test it and you don't need the new constant. Thanks. > "yarn top" command not quitting completely with ctrl+c > -- > > Key: YARN-10490 > URL: https://issues.apache.org/jira/browse/YARN-10490 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Reporter: Agshin Kazimli >Assignee: Agshin Kazimli >Priority: Minor > Attachments: YARN-10490-001.patch, YARN-10490-001.patch, > YARN-10490-002.patch > > > When we quit "yarn top" command using ctrl+c, even though the command prompt > returns, the old statistics info is still visible below the prompt. > And we need to manually type clear command or multiple return keys to get > around it. It is reported as a inconvenience for the customer from a > usability perspective. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10512) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS
[ https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-10512: -- Attachment: YARN-10512.003.patch > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include > mode of operation for CS > -- > > Key: YARN-10512 > URL: https://issues.apache.org/jira/browse/YARN-10512 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-10512.001.patch, YARN-10512.002.patch, > YARN-10512.003.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > We would like to expose the mode of operation with the RM's /scheduler REST > endpoint. > The field name will be 'mode'. > All queue representations in the response will be uniformly hold any of the > mode values of: "percentage", "absolute", "weight". -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10512) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS
[ https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266342#comment-17266342 ] Szilard Nemeth commented on YARN-10512: --- Just realized I left some unnecessary setters in class: CapacitySchedulerInfo Removed those with the latest patch (003). > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include > mode of operation for CS > -- > > Key: YARN-10512 > URL: https://issues.apache.org/jira/browse/YARN-10512 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-10512.001.patch, YARN-10512.002.patch, > YARN-10512.003.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > We would like to expose the mode of operation with the RM's /scheduler REST > endpoint. > The field name will be 'mode'. > All queue representations in the response will be uniformly hold any of the > mode values of: "percentage", "absolute", "weight". -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10562) Follow up changes for YARN-9833
[
https://issues.apache.org/jira/browse/YARN-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger updated YARN-10562:
---
Fix Version/s: 2.10.2
I committed this to branch-2.10 after backporting YARN-9833. this JIRA has now
been committed to trunk (3.4), branch-3.3, branch-3.2, branch-3.1, and
branch-2.10
> Follow up changes for YARN-9833
> ---
>
> Key: YARN-10562
> URL: https://issues.apache.org/jira/browse/YARN-10562
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Affects Versions: 3.4.0
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Major
> Labels: resourcemanager
> Fix For: 3.4.0, 3.3.1, 3.1.5, 2.10.2, 3.2.3
>
> Attachments: YARN-10562.001.patch, YARN-10562.002.patch,
> YARN-10562.003.patch, YARN-10562.004.patch
>
>
> In YARN-9833, a race condition in DirectoryCollection. {{getGoodDirs()}} and
> related methods were returning an unmodifiable view of the lists. These
> accesses were protected by read/write locks, but because the lists are
> CopyOnWriteArrayLists, subsequent changes to the list, even when done under
> the writelock, were exposed when a caller started iterating the list view.
> CopyOnWriteArrayLists cache the current underlying list in the iterator, so
> it is safe to iterate them even while they are being changed - at least the
> view will be consistent.
> The problem was that checkDirs() was clearing the lists and rebuilding them
> from scratch every time, so if a caller called getGoodDirs() just before
> checkDirs cleared it, and then started iterating right after the clear, they
> could get an empty list.
> The fix in YARN-9833 was to change {{getGoodDirs()}} and related methods to
> return a copy of the list, which definitely fixes the race condition. The
> disadvantage is that now we create a new copy of these lists every time we
> launch a container. The advantage using CopyOnWriteArrayList was that the
> lists should rarely ever change, and we can avoid all the copying.
> Unfortunately, the way checkDirs() was written, it guaranteed that it would
> modify those lists multiple times every time.
> So this Jira proposes an alternate solution for YARN-9833, which mainly just
> rewrites checkDirs() to minimize the changes to the underlying lists. There
> are still some small windows where a disk will have been added to one list,
> but not yet removed from another if you hit it just right, but I think these
> should be pretty rare and relatively harmless, and in the vast majority of
> cases I suspect only one disk will be moving from one list to another at any
> time. The question is whether this type of inconsistency (which was always
> there before -YARN-9833- is worth reducing all the copying.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9833) Race condition when DirectoryCollection.checkDirs() runs during container launch
[
https://issues.apache.org/jira/browse/YARN-9833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger updated YARN-9833:
--
Fix Version/s: 2.10.2
I backported this to branch-2.10
> Race condition when DirectoryCollection.checkDirs() runs during container
> launch
>
>
> Key: YARN-9833
> URL: https://issues.apache.org/jira/browse/YARN-9833
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.3.0, 3.2.2, 3.1.4, 2.10.2
>
> Attachments: YARN-9833-001.patch
>
>
> During endurance testing, we found a race condition that cause an empty
> {{localDirs}} being passed to container-executor.
> The problem is that {{DirectoryCollection.checkDirs()}} clears three
> collections:
> {code:java}
> this.writeLock.lock();
> try {
> localDirs.clear();
> errorDirs.clear();
> fullDirs.clear();
> ...
> {code}
> This happens in critical section guarded by a write lock. When we start a
> container, we retrieve the local dirs by calling
> {{dirsHandler.getLocalDirs();}} which in turn invokes
> {{DirectoryCollection.getGoodDirs()}}. The implementation of this method is:
> {code:java}
> List getGoodDirs() {
> this.readLock.lock();
> try {
> return Collections.unmodifiableList(localDirs);
> } finally {
> this.readLock.unlock();
> }
> }
> {code}
> So we're also in a critical section guarded by the lock. But
> {{Collections.unmodifiableList()}} only returns a _view_ of the collection,
> not a copy. After we get the view, {{MonitoringTimerTask.run()}} might be
> scheduled to run and immediately clears {{localDirs}}.
> This caused a weird behaviour in container-executor, which exited with error
> code 35 (COULD_NOT_CREATE_WORK_DIRECTORIES).
> Therefore we can't just return a view, we must return a copy with
> {{ImmutableList.copyOf()}}.
> Credits to [~snemeth] for analyzing and determining the root cause.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266315#comment-17266315
]
Hadoop QA commented on YARN-10506:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
12s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
49s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 11s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
48s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 53s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/487/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 2 new + 40
unchanged - 2 fixed = 42 total (was 42) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green}{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 738 unchanged - 1 fixed = 738 total (was 739)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:gre
[jira] [Commented] (YARN-10512) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS
[ https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266299#comment-17266299 ] Szilard Nemeth commented on YARN-10512: --- Thanks for the comments. [~gandras], [~bteke] Please take a final look as I had to move the tests and rewrite the assertion logic for the queues. > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include > mode of operation for CS > -- > > Key: YARN-10512 > URL: https://issues.apache.org/jira/browse/YARN-10512 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-10512.001.patch, YARN-10512.002.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > We would like to expose the mode of operation with the RM's /scheduler REST > endpoint. > The field name will be 'mode'. > All queue representations in the response will be uniformly hold any of the > mode values of: "percentage", "absolute", "weight". -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10512) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS
[ https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-10512: -- Attachment: YARN-10512.002.patch > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include > mode of operation for CS > -- > > Key: YARN-10512 > URL: https://issues.apache.org/jira/browse/YARN-10512 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-10512.001.patch, YARN-10512.002.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > We would like to expose the mode of operation with the RM's /scheduler REST > endpoint. > The field name will be 'mode'. > All queue representations in the response will be uniformly hold any of the > mode values of: "percentage", "absolute", "weight". -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[
https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266294#comment-17266294
]
Wangda Tan commented on YARN-10531:
---
[~zhuqi], I haven't reviewed details of the patch yet, I'm not sure if we need
rebase after YARN-10506. Please share your thoughts so we can move it forward.
> Be able to disable user limit factor for CapacityScheduler Leaf Queue
> -
>
> Key: YARN-10531
> URL: https://issues.apache.org/jira/browse/YARN-10531
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10531.001.patch, YARN-10531.002.patch
>
>
> User limit factor is used to define max cap of how much resource can be
> consumed by single user.
> Under Auto Queue Creation context, it doesn't make much sense to set user
> limit factor, because initially every queue will set weight to 1.0, we want
> user can consume more resource if possible. It is hard to pre-determine how
> to set up user limit factor. So it makes more sense to add a new value (like
> -1) to indicate we will disable user limit factor
> Logic need to be changed is below:
> (Inside LeafQueue.java)
> {code}
> Resource maxUserLimit = Resources.none();
> if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) {
> maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity,
> getUserLimitFactor());
> } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY)
> {
> maxUserLimit = partitionResource;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266292#comment-17266292 ] Wangda Tan commented on YARN-10506: --- Thanks for the additional comments, I +1 to the latest patch, if no objections I will get it in by this afternoon. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch, > YARN-10506.014.patch, YARN-10506.015.patch, YARN-10506.016.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266265#comment-17266265
]
Hadoop QA commented on YARN-10506:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
25s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 32m
49s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 53s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
50s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 52s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/485/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 1 new + 41
unchanged - 1 fixed = 42 total (was 42) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 737 unchanged - 1 fixed = 737 total (was 738)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:gre
[jira] [Commented] (YARN-10535) Make queue placement in CapacityScheduler compliant with auto-queue-placement
[
https://issues.apache.org/jira/browse/YARN-10535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266186#comment-17266186
]
Hadoop QA commented on YARN-10535:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
13s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
13s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 26s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
51s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
26s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/486/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
29s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/486/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 29s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/486/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
27s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/486/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 26s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/486/artifact/out
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266181#comment-17266181 ] Andras Gyori commented on YARN-10506: - Thanks [~zhuqi], you are right, but I realized the whole logic was flawed because of the parent == null check. I have removed this check and added your suggestion, also, fixed the corresponding test. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch, > YARN-10506.014.patch, YARN-10506.015.patch, YARN-10506.016.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Gyori updated YARN-10506: Attachment: YARN-10506.016.patch > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch, > YARN-10506.014.patch, YARN-10506.015.patch, YARN-10506.016.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266170#comment-17266170
]
zhuqi commented on YARN-10506:
--
[~gandras]
Here should be changed to :
{code:java}
if (parentQueue != null &&
conf.isAutoCreateChildQueueEnabled(parentQueue.getQueuePath())) {
// Case 1: Handle ManagedParentQueue
AutoCreatedLeafQueue autoCreatedLeafQueue = null;
ManagedParentQueue autoCreateEnabledParentQueue =
(ManagedParentQueue) parentQueue;
autoCreatedLeafQueue = new AutoCreatedLeafQueue(this, leafQueueName,
autoCreateEnabledParentQueue);
addQueue(autoCreatedLeafQueue);
return autoCreatedLeafQueue;
}
{code}
Because parentQueue be null, it should be handled in new auto created, we can
created parent then it may passed. if throw exception, the case will not be
handled.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch,
> YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch,
> YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch,
> YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch,
> YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch,
> YARN-10506.014.patch, YARN-10506.015.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10535) Make queue placement in CapacityScheduler compliant with auto-queue-placement
[
https://issues.apache.org/jira/browse/YARN-10535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266139#comment-17266139
]
Gergely Pollak commented on YARN-10535:
---
[~pbacsko] [~gandras] Thank you for the review, I fixed the most issues
(checkstyle might find some more). Also when jenkins can run the full test
suite we might get some test failures which needs to be fixed.
*7. {{MappingRuleValidationContextImpl}}: {{staticPartParent}} is initially
{{null}}. Any change of it remaining {{null}} after the {{while}} loop? If so,
a null check is warranted.*
There is an explicit check for null state of staticPartParent, but null is
totally legit, if there is no parent of the static part. But that case is
handled.
*I can't really comment on the validation logic because there a lot of string
operations and hopefully the unit tests will do their job.*
Tests only cover the current cases, they won't provide enough coverage, for
validation I've already added a few test, but I still need a few for the
CSMappingPlacementRule class.
*Talking about unit tests: {{TestMappingRuleValidationContextImpl}} was
extended with some new cases, does that provide enough coverage? Have you run
it through a coverage tool?*
Tests are still being written
*EDIT: ok, the patch does not compile yet, so probably the answer is "no".*
It won't compile until YARN-10506 is merged.
> Make queue placement in CapacityScheduler compliant with auto-queue-placement
> -
>
> Key: YARN-10535
> URL: https://issues.apache.org/jira/browse/YARN-10535
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Wangda Tan
>Assignee: Gergely Pollak
>Priority: Major
> Attachments: YARN-10535.001.patch, YARN-10535.002.patch
>
>
> Once YARN-10506 is done, we need to call the API from the queue placement
> policy to create queues.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10535) Make queue placement in CapacityScheduler compliant with auto-queue-placement
[ https://issues.apache.org/jira/browse/YARN-10535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Pollak updated YARN-10535: -- Attachment: YARN-10535.002.patch > Make queue placement in CapacityScheduler compliant with auto-queue-placement > - > > Key: YARN-10535 > URL: https://issues.apache.org/jira/browse/YARN-10535 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Wangda Tan >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10535.001.patch, YARN-10535.002.patch > > > Once YARN-10506 is done, we need to call the API from the queue placement > policy to create queues. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10562) Follow up changes for YARN-9833
[
https://issues.apache.org/jira/browse/YARN-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266098#comment-17266098
]
Jim Brennan commented on YARN-10562:
Given that the original bug exists in branch-2 as well, I think back-porting to
branch-2 is a good idea in this case.
> Follow up changes for YARN-9833
> ---
>
> Key: YARN-10562
> URL: https://issues.apache.org/jira/browse/YARN-10562
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Affects Versions: 3.4.0
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Major
> Labels: resourcemanager
> Fix For: 3.4.0, 3.3.1, 3.1.5, 3.2.3
>
> Attachments: YARN-10562.001.patch, YARN-10562.002.patch,
> YARN-10562.003.patch, YARN-10562.004.patch
>
>
> In YARN-9833, a race condition in DirectoryCollection. {{getGoodDirs()}} and
> related methods were returning an unmodifiable view of the lists. These
> accesses were protected by read/write locks, but because the lists are
> CopyOnWriteArrayLists, subsequent changes to the list, even when done under
> the writelock, were exposed when a caller started iterating the list view.
> CopyOnWriteArrayLists cache the current underlying list in the iterator, so
> it is safe to iterate them even while they are being changed - at least the
> view will be consistent.
> The problem was that checkDirs() was clearing the lists and rebuilding them
> from scratch every time, so if a caller called getGoodDirs() just before
> checkDirs cleared it, and then started iterating right after the clear, they
> could get an empty list.
> The fix in YARN-9833 was to change {{getGoodDirs()}} and related methods to
> return a copy of the list, which definitely fixes the race condition. The
> disadvantage is that now we create a new copy of these lists every time we
> launch a container. The advantage using CopyOnWriteArrayList was that the
> lists should rarely ever change, and we can avoid all the copying.
> Unfortunately, the way checkDirs() was written, it guaranteed that it would
> modify those lists multiple times every time.
> So this Jira proposes an alternate solution for YARN-9833, which mainly just
> rewrites checkDirs() to minimize the changes to the underlying lists. There
> are still some small windows where a disk will have been added to one list,
> but not yet removed from another if you hit it just right, but I think these
> should be pretty rare and relatively harmless, and in the vast majority of
> cases I suspect only one disk will be moving from one list to another at any
> time. The question is whether this type of inconsistency (which was always
> there before -YARN-9833- is worth reducing all the copying.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266096#comment-17266096 ] Andras Gyori commented on YARN-10506: - Thank you [~shuzirra] for the feedback, I too find out troubling and fixed this concern, and also found an additional line where we used short name instead of full path. As for the rest, I will handle them in the followup refactoring jira, but in general I agree with each of them. Also thank you [~bteke] for the comments, I have removed all the unused variables. [~zhuqi] will reintroduce the fields that are needed in an other jira. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch, > YARN-10506.014.patch, YARN-10506.015.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Gyori updated YARN-10506: Attachment: YARN-10506.015.patch > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch, > YARN-10506.014.patch, YARN-10506.015.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266081#comment-17266081
]
Gergely Pollak edited comment on YARN-10506 at 1/15/21, 2:46 PM:
-
Thank you [~gandras] for the fix, here are a few comments, most of them can
wait for a followup JIRA:
*CapacitySchedulerAutoQueueHandler class*:
Why is it a separate class, this is the functionality of the actual queue, I
think it would have been better to extend the parentQueue and add this
functionality there, this is a class which automagicly changes the ParentQueue
class and enhances it with new functionality, but I think it would be better to
solve this via inheritance. Also that would make differentiating between
ParentQueue and the DynamicParentQueue much easier. Probably we should fix it
in a followup JIRA.
*CapacitySchedulerAutoQueueHandler#getQueue*
This method is unnecessary, since CSQueueStore.get method handles null paths,
and returns null. See CapacitySchedulerQueueManager#getQueue and
CSQueueStore#get
*CapacitySchedulerAutoQueueHandler#autoCreateQueue*
I might be wrong, but this is concerning:
{code:java}
CSQueue existingQueueCandidate =
getQueue(queueCandidateContext.getQueue());{code}
The getQueue will always return the leaf name of the queue, which might be
ambiguous so this might return null even if the queue exists on the path we are
checking, and might identify an already existing queue invalid.
As a rule of thumb, wherever possible use the full path of the queues please!
If we have the following structure:
{code:java}
root
+--alice
| +--test
| +--dev
+--bob
| +--test
| +--dev
{code}
Where both _dev_ and _test_ queues are dynamic parents, then when I try to
submit something to root.bob.dev.something, this part of the code
{code:java}
ApplicationPlacementContext queueCandidateContext = parentContext; //
(Parent context is root.bob.dev)
CSQueue existingQueueCandidate =
getQueue(queueCandidateContext.getQueue()); //queueCandidateContext.getQueue()
= 'dev'
getQueue(queueCandidateContext.getQueue()) == null //because dev is
ambiguous
{code}
Will identify root.bob.dev as a not existing queue.
So please use _ApplciationPlacementContext#getFullQueuePath_
*CapacitySchedulerAutoQueueHandler#isDynamicQueue*
Is not fool/future proof, while I'm aware currently only the AbstractCSQueue
class implements the interface, but still it would be better to check before
casting it to AbstractCSQueue. Or move the isDynamic down to the interface,
then you don't have to cast
was (Author: shuzirra):
Thank you [~gandras] for the fix, here are a few comments, most of them can
wait for a followup JIRA:
*CapacitySchedulerAutoQueueHandler class*:
Why is it a separate class, this is the functionality of the actual queue, I
think it would have been better to extend the parentQueue and add this
functionality there, this is a class which automagicly changes the ParentQueue
class and enhances it with new functionality, but I think it would be better to
solve this via inheritance. Also that would make differentiating between
ParentQueue and the DynamicParentQueue much easier. Probably we should fix it
in a followup JIRA.
*CapacitySchedulerAutoQueueHandler#getQueue*
This method is unnecessary, since CSQueueStore.get method handles null paths,
and returns null. See CapacitySchedulerQueueManager#getQueue and
CSQueueStore#get
*CapacitySchedulerAutoQueueHandler#autoCreateQueue*
I might be wrong, but this is concerning:
{code:java}
CSQueue existingQueueCandidate =
getQueue(queueCandidateContext.getQueue());{code}
The getQueue will always return the leaf name of the queue, which might be
ambiguous so this might return null even if the queue exists on the path we are
checking, and might identify an already existing queue invalid.
As a rule of thumb, wherever possible use the full path of the queues please!
If we have the following structure:
{code:java}
root
+--alice
| +--test
| +--dev
+--bob
| +--test
| +--dev
{code}
Where both _dev_ and _test_ queues are dynamic parents, then when I try to
submit something to root.bob.dev.something, this part of the code
{code:java}
ApplicationPlacementContext queueCandidateContext = parentContext; //
(Parent context is root.bob.dev)
CSQueue existingQueueCandidate =
getQueue(queueCandidateContext.getQueue()); //queueCandidateContext.getQueue()
= 'dev'
getQueue(queueCandidateContext.getQueue()) == null //because dev is
ambiguous
{code}
Will identify root.bob.dev as a not existing queue.
So please use _ApplciationPlacementContext#getFullQueuePath_
*CapacitySchedulerAutoQueueHandler#isDynamicQueue*
Is not fool/future proof, while I'm aware currently only the AbstractCSQueue
class implements the interface, but still it would be bett
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266081#comment-17266081
]
Gergely Pollak commented on YARN-10506:
---
Thank you [~gandras] for the fix, here are a few comments, most of them can
wait for a followup JIRA:
*CapacitySchedulerAutoQueueHandler class*:
Why is it a separate class, this is the functionality of the actual queue, I
think it would have been better to extend the parentQueue and add this
functionality there, this is a class which automagicly changes the ParentQueue
class and enhances it with new functionality, but I think it would be better to
solve this via inheritance. Also that would make differentiating between
ParentQueue and the DynamicParentQueue much easier. Probably we should fix it
in a followup JIRA.
*CapacitySchedulerAutoQueueHandler#getQueue*
This method is unnecessary, since CSQueueStore.get method handles null paths,
and returns null. See CapacitySchedulerQueueManager#getQueue and
CSQueueStore#get
*CapacitySchedulerAutoQueueHandler#autoCreateQueue*
I might be wrong, but this is concerning:
{code:java}
CSQueue existingQueueCandidate =
getQueue(queueCandidateContext.getQueue());{code}
The getQueue will always return the leaf name of the queue, which might be
ambiguous so this might return null even if the queue exists on the path we are
checking, and might identify an already existing queue invalid.
As a rule of thumb, wherever possible use the full path of the queues please!
If we have the following structure:
{code:java}
root
+--alice
| +--test
| +--dev
+--bob
| +--test
| +--dev
{code}
Where both _dev_ and _test_ queues are dynamic parents, then when I try to
submit something to root.bob.dev.something, this part of the code
{code:java}
ApplicationPlacementContext queueCandidateContext = parentContext; //
(Parent context is root.bob.dev)
CSQueue existingQueueCandidate =
getQueue(queueCandidateContext.getQueue()); //queueCandidateContext.getQueue()
= 'dev'
getQueue(queueCandidateContext.getQueue()) == null //because dev is
ambiguous
{code}
Will identify root.bob.dev as a not existing queue.
So please use _ApplciationPlacementContext#getFullQueuePath_
*CapacitySchedulerAutoQueueHandler#isDynamicQueue*
Is not fool/future proof, while I'm aware currently only the AbstractCSQueue
class implements the interface, but still it would be better to check before
casting it to AbstractCSQueue. Or move the isDynamic down to the interface,
then you don't have to cast
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch,
> YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch,
> YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch,
> YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch,
> YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch,
> YARN-10506.014.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10573) Enhance placement rule conversion in fs2cs in weight mode
[
https://issues.apache.org/jira/browse/YARN-10573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10573:
Description:
If we're using weight mode, we have much more freedom when it comes to
placement rules.
In YARN-10525, weight conversion became the default in {{fs2cs}}. This also
means that we can support nested rules properly and also queues can be created
under {{root}}.
Therefore, a lot of warnings and validations inside {{QueuePlacementConverter}}
are not necessary and only relevant if the user chose percentage-based
conversion in the command line.
Also remove unnecessary stuff in the RuleHandler which are already supported:
* SPECIFIED_NOT_FIRST
* USER_MAX_APPS_DEFAULT
* USER_MAX_RUNNING_APPS
was:
If we're using weight mode, we have much more freedom when it comes to
placement rules.
In YARN-10525, weight conversion became the default in {{fs2cs}}. This also
means that we can support nested rules properly and also queues can be created
under {{root}}.
Therefore, a lot of warnings and validations inside {{QueuePlacementConverter}}
are not necessary and only relevant if the user chose percentage-based
conversion in the command line.
Also remove unnecessary stuff which are already supported:
* SPECIFIED_NOT_FIRST
* USER_MAX_APPS_DEFAULT
* USER_MAX_RUNNING_APPS
> Enhance placement rule conversion in fs2cs in weight mode
> -
>
> Key: YARN-10573
> URL: https://issues.apache.org/jira/browse/YARN-10573
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>
> If we're using weight mode, we have much more freedom when it comes to
> placement rules.
> In YARN-10525, weight conversion became the default in {{fs2cs}}. This also
> means that we can support nested rules properly and also queues can be
> created under {{root}}.
> Therefore, a lot of warnings and validations inside
> {{QueuePlacementConverter}} are not necessary and only relevant if the user
> chose percentage-based conversion in the command line.
> Also remove unnecessary stuff in the RuleHandler which are already supported:
> * SPECIFIED_NOT_FIRST
> * USER_MAX_APPS_DEFAULT
> * USER_MAX_RUNNING_APPS
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10513) CS Flexible Auto Queue Creation RM UIv2 modifications
[ https://issues.apache.org/jira/browse/YARN-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266069#comment-17266069 ] Andras Gyori commented on YARN-10513: - I will analyse the impact of AQC and weight mode on UI2 after the major parts got into trunk. > CS Flexible Auto Queue Creation RM UIv2 modifications > - > > Key: YARN-10513 > URL: https://issues.apache.org/jira/browse/YARN-10513 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10513) CS Flexible Auto Queue Creation RM UIv2 modifications
[ https://issues.apache.org/jira/browse/YARN-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Gyori reassigned YARN-10513: --- Assignee: Andras Gyori > CS Flexible Auto Queue Creation RM UIv2 modifications > - > > Key: YARN-10513 > URL: https://issues.apache.org/jira/browse/YARN-10513 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10535) Make queue placement in CapacityScheduler compliant with auto-queue-placement
[ https://issues.apache.org/jira/browse/YARN-10535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266068#comment-17266068 ] Andras Gyori commented on YARN-10535: - Thank you [~shuzirra] for the patch. I do not have any major comment at this point and I understand that you will add more test cases in the following patches. One minor feedback: * MappingRuleValidationContextImpl#isParentQueue: I think this method is not about being a ParentQueue, but being a dynamic parent candidate, and I found it a bit ambiguous. Also, the javadoc of the method talks about child queues, but the ManagedParentQueues and AQC-enabled ParentQueues are different in that they can have dynamic child queues. This is only a nit, and only address it if you have enough time at hand. > Make queue placement in CapacityScheduler compliant with auto-queue-placement > - > > Key: YARN-10535 > URL: https://issues.apache.org/jira/browse/YARN-10535 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Wangda Tan >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10535.001.patch > > > Once YARN-10506 is done, we need to call the API from the queue placement > policy to create queues. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266061#comment-17266061 ] zhuqi commented on YARN-10506: -- Thanks for [~gandras] fix. Thanks for [~bteke] review. The lastSubmittedTimestamp, i will use it for YARN-10532. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch, > YARN-10506.014.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266036#comment-17266036
]
Benjamin Teke commented on YARN-10506:
--
Thanks for working on this! Checked the latest patch, generally seems good to
me. I have two minor nits about it:
* In AbstractCSQueue the field {{lastSubmittedTimestamp}} is unused. It has a
"setter" and a getter, but neither of them are called. Shouldn't this be
removed to avoid confusion?
* The same goes for CapacitySchedulerAutoQueueHandler's conf field.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch,
> YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch,
> YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch,
> YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch,
> YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch,
> YARN-10506.014.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10512) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS
[ https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266033#comment-17266033 ] Andras Gyori commented on YARN-10512: - I agree with this approach, +1 LGTM. > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include > mode of operation for CS > -- > > Key: YARN-10512 > URL: https://issues.apache.org/jira/browse/YARN-10512 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-10512.001.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > We would like to expose the mode of operation with the RM's /scheduler REST > endpoint. > The field name will be 'mode'. > All queue representations in the response will be uniformly hold any of the > mode values of: "percentage", "absolute", "weight". -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10565) Refactor CS queue initialization to simplify weight mode calculation
[ https://issues.apache.org/jira/browse/YARN-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10565: - Attachment: YARN-10565.002.patch > Refactor CS queue initialization to simplify weight mode calculation > > > Key: YARN-10565 > URL: https://issues.apache.org/jira/browse/YARN-10565 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-10565.001.patch, YARN-10565.002.patch > > > In YARN-10504 weight mode support was introduced to CS. This jira is a > followup to simplify and restructure the initialization, so that the weight > calculation/absolute/percentage mode is easier to understand and modify. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10573) Enhance placement rule conversion in fs2cs in weight mode
[
https://issues.apache.org/jira/browse/YARN-10573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10573:
Description:
If we're using weight mode, we have much more freedom when it comes to
placement rules.
In YARN-10525, weight conversion became the default in {{fs2cs}}. This also
means that we can support nested rules properly and also queues can be created
under {{root}}.
Therefore, a lot of warnings and validations inside {{QueuePlacementConverter}}
are not necessary and only relevant if the user chose percentage-based
conversion in the command line.
Also remove unnecessary stuff which are already supported:
* SPECIFIED_NOT_FIRST
* USER_MAX_APPS_DEFAULT
* USER_MAX_RUNNING_APPS
was:
If we're using weight mode, we have much more freedom when it comes to
placement rules.
In YARN-10525, weight conversion became the default in {{fs2cs}}. This also
means that we can support nested rules properly and also queues can be created
under {{root}}.
Therefore, a lot of warnings and validations inside {{QueuePlacementConverter}}
are not necessary and only relevant if the user chose percentage-based
conversion in the command line.
> Enhance placement rule conversion in fs2cs in weight mode
> -
>
> Key: YARN-10573
> URL: https://issues.apache.org/jira/browse/YARN-10573
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>
> If we're using weight mode, we have much more freedom when it comes to
> placement rules.
> In YARN-10525, weight conversion became the default in {{fs2cs}}. This also
> means that we can support nested rules properly and also queues can be
> created under {{root}}.
> Therefore, a lot of warnings and validations inside
> {{QueuePlacementConverter}} are not necessary and only relevant if the user
> chose percentage-based conversion in the command line.
> Also remove unnecessary stuff which are already supported:
> * SPECIFIED_NOT_FIRST
> * USER_MAX_APPS_DEFAULT
> * USER_MAX_RUNNING_APPS
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266018#comment-17266018
]
Hadoop QA commented on YARN-10506:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
23s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
49s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 28s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
50s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 53s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/484/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 2 new + 40
unchanged - 2 fixed = 42 total (was 42) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green}{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 738 unchanged - 1 fixed = 738 total (was 739)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:gre
[jira] [Assigned] (YARN-10565) Refactor CS queue initialization to simplify weight mode calculation
[ https://issues.apache.org/jira/browse/YARN-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke reassigned YARN-10565: Assignee: Benjamin Teke > Refactor CS queue initialization to simplify weight mode calculation > > > Key: YARN-10565 > URL: https://issues.apache.org/jira/browse/YARN-10565 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-10565.001.patch > > > In YARN-10504 weight mode support was introduced to CS. This jira is a > followup to simplify and restructure the initialization, so that the weight > calculation/absolute/percentage mode is easier to understand and modify. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10512) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS
[
https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266010#comment-17266010
]
Benjamin Teke commented on YARN-10512:
--
Thank you [~snemeth] for working on this.
Regarding the 2nd point in [~gandras]'s comment: as I see it
{{getCapacityConfigurationTypeForQueues}} basically uses the same approach
(check whether the QueueCapacities' weight or capacity is set), except it
doesn't consider the queue's {{getCapacityConfigType()}} method. That method
should be sufficient for differentiating ABSOLUTE and PERCENTAGE+WEIGHT mode,
as {{AbstractCSQueue.updateEffectiveResources}} uses it as well. And the
dynamic queues will have their weight set in their respective QueueCapacities
object, as it holds the calculated values, not the ones read from the
configuration. All in all I suggest to keep this patch as simple as possible
and consider refactoring this after the YARN-10504 refactor is finished
(YARN-10565).
> CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include
> mode of operation for CS
> --
>
> Key: YARN-10512
> URL: https://issues.apache.org/jira/browse/YARN-10512
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-10512.001.patch
>
>
> Under this umbrella (YARN-10496), weight-mode has been implemented for CS
> with YARN-10504.
> We would like to expose the mode of operation with the RM's /scheduler REST
> endpoint.
> The field name will be 'mode'.
> All queue representations in the response will be uniformly hold any of the
> mode values of: "percentage", "absolute", "weight".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10573) Enhance placement rule conversion in fs2cs in weight mode
[
https://issues.apache.org/jira/browse/YARN-10573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10573:
Description:
If we're using weight mode, we have much more freedom when it comes to
placement rules.
In YARN-10525, weight conversion became the default in {{fs2cs}}. This also
means that we can support nested rules properly and also queues can be created
under {{root}}.
Therefore, a lot of warnings and validations inside {{QueuePlacementConverter}}
are not necessary and only relevant if the user chose percentage-based
conversion in the command line.
was:
If we're using weight mode, we have much more freedom when it comes to
placement rules.
In YARN-10525, weight conversion is the default in {{fs2cs}}. This also means
that we can support nested rules properly and also queues can be created under
{{root}}.
Therefore, a lot of warnings and validations inside {{QueuePlacementConverter}}
are not necessary and only relevant if the user chose percentage-based
conversion in the command line.
> Enhance placement rule conversion in fs2cs in weight mode
> -
>
> Key: YARN-10573
> URL: https://issues.apache.org/jira/browse/YARN-10573
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>
> If we're using weight mode, we have much more freedom when it comes to
> placement rules.
> In YARN-10525, weight conversion became the default in {{fs2cs}}. This also
> means that we can support nested rules properly and also queues can be
> created under {{root}}.
> Therefore, a lot of warnings and validations inside
> {{QueuePlacementConverter}} are not necessary and only relevant if the user
> chose percentage-based conversion in the command line.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10573) Enhance placement rule conversion in fs2cs in weight mode
Peter Bacsko created YARN-10573:
---
Summary: Enhance placement rule conversion in fs2cs in weight mode
Key: YARN-10573
URL: https://issues.apache.org/jira/browse/YARN-10573
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Peter Bacsko
Assignee: Peter Bacsko
If we're using weight mode, we have much more freedom when it comes to
placement rules.
In YARN-10525, weight conversion is the default in {{fs2cs}}. This also means
that we can support nested rules properly and also queues can be created under
{{root}}.
Therefore, a lot of warnings and validations inside {{QueuePlacementConverter}}
are not necessary and only relevant if the user chose percentage-based
conversion in the command line.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10512) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS
[
https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17265986#comment-17265986
]
Andras Gyori commented on YARN-10512:
-
While ParentQueue#getCapacityConfigurationTypeForQueues has a
Collection signature, it is used for single queues as well, by calling
it:
{code:java}
getCapacityConfigurationTypeForQueues(ImmutableList.of(queue))
{code}
> CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include
> mode of operation for CS
> --
>
> Key: YARN-10512
> URL: https://issues.apache.org/jira/browse/YARN-10512
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-10512.001.patch
>
>
> Under this umbrella (YARN-10496), weight-mode has been implemented for CS
> with YARN-10504.
> We would like to expose the mode of operation with the RM's /scheduler REST
> endpoint.
> The field name will be 'mode'.
> All queue representations in the response will be uniformly hold any of the
> mode values of: "percentage", "absolute", "weight".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10512) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS
[
https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17265984#comment-17265984
]
Andras Gyori commented on YARN-10512:
-
Thank you [~snemeth] for the patch! I have some comment, but I am not entirely
sure if this could be done in this patch.
* First is simple. I would advise to use ParentQueue's QueueCapacityType(it
has the weight mode) in CapacitySchedulerInfoHelper#getMode. That said,
QueueCapacityType and AbstractCSQueue.CapacityConfigType is almost a same, we
might need to merge them together. I would advise to store its string
representation in a similar fashion:
{code:java}
private enum QueueCapacityType {
WEIGHT("weight"),
ABSOLUTE_RESOURCE("absolute"),
PERCENT("percentage");
private String internal;
QueueCapacityType(String internal) {
this.internal = internal;
}
@Override
public String toString() {
return internal;
}
}
{code}
* I think this is a more serious issue, and we should think about it. The way
getMode discovers the mode might be problematic. Dynamic queues do not have a
weight mode config, therefore they will be considered as they were in weight
mode. There is a method ParentQueue#getCapacityConfigurationTypeForQueues that
is used internally, and handles all corner cases as well. My suggestion would
be to put it in AbstractCSQueue and use it instead.
> CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include
> mode of operation for CS
> --
>
> Key: YARN-10512
> URL: https://issues.apache.org/jira/browse/YARN-10512
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-10512.001.patch
>
>
> Under this umbrella (YARN-10496), weight-mode has been implemented for CS
> with YARN-10504.
> We would like to expose the mode of operation with the RM's /scheduler REST
> endpoint.
> The field name will be 'mode'.
> All queue representations in the response will be uniformly hold any of the
> mode values of: "percentage", "absolute", "weight".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10535) Make queue placement in CapacityScheduler compliant with auto-queue-placement
[
https://issues.apache.org/jira/browse/YARN-10535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17265947#comment-17265947
]
Peter Bacsko edited comment on YARN-10535 at 1/15/21, 11:41 AM:
Initial review:
1. Remove star imports: {{import java.util.*;}}
2. Remove star imports: {{import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.*;}}
3. Is this class used? {{import org.apache.hadoop.yarn.webapp.hamlet2.Hamlet;}}
4. Nit: some exceptions ends with ".", some with "!", some with no punctuation
at all. Let's have some consistency in this area.
5. Nit: I'd replace "Unknown dynamic path lookup error" to sth like "Unknown
queue path validation result"
6. {{MappingRuleValidationContextImpl}}: I'd definitely call {{hasNext()}}
before calling {{String staticPart = pointer.next();}} just to be 100% safe
7. {{MappingRuleValidationContextImpl}}: {{staticPartParent}} is initially
{{null}}. Any change of it remaining {{null}} after the {{while}} loop? If so,
a null check is warranted.
8. Nit: I'd rename "QueueCreationValidation" to
"QueueCreationValidationResult" or "AutoQueueCreateValidationResult" to reflect
that it is a result of something. Or simply "ValidationResult", because it's a
nested enum.
I can't really comment on the validation logic because there a lot of string
operations and hopefully the unit tests will do their job.
Talking about unit tests: {{TestMappingRuleValidationContextImpl}} was extended
with some new cases, does that provide enough coverage? Have you run it through
a coverage tool? EDIT: ok, the patch does not compile yet, so probably the
answer is "no".
Also, many checkstyle issue, pls fix them if time permits (mostly indentation).
was (Author: pbacsko):
Initial review:
1. Remove star imports: {{import java.util.*;}}
2. Remove star imports: {{import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.*;}}
3. Is this class used? {{import org.apache.hadoop.yarn.webapp.hamlet2.Hamlet;}}
4. Nit: some exceptions ends with ".", some with "!", some with no punctuation
at all. Let's have some consistency in this area.
5. Nit: I'd replace "Unknown dynamic path lookup error" to sth like "Unknown
queue path validation result"
6. {{MappingRuleValidationContextImpl}}: I'd definitely call {{hasNext()}}
before calling {{String staticPart = pointer.next();}} just to be 100% safe
7. {{MappingRuleValidationContextImpl}}: {{staticPartParent}} is initially
{{null}}. Any change of it remaining {{null}} after the {{while}} loop? If so,
a null check is warranted.
8. Nit: I'd rename "QueueCreationValidation" to
"QueueCreationValidationResult" or "AutoQueueCreateValidationResult" to reflect
that it is a result of something. Or simply "ValidationResult", because it's a
nested enum.
I can't really comment on the validation logic because there a lot of string
operations and hopefully the unit tests do their job.
Talking about unit tests: {{TestMappingRuleValidationContextImpl}} was extended
with some new cases, does that provide enough coverage? Have you run it through
a coverage tool?
Also, many checkstyle issue, pls fix them if time permits (mostly indentation).
> Make queue placement in CapacityScheduler compliant with auto-queue-placement
> -
>
> Key: YARN-10535
> URL: https://issues.apache.org/jira/browse/YARN-10535
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Wangda Tan
>Assignee: Gergely Pollak
>Priority: Major
> Attachments: YARN-10535.001.patch
>
>
> Once YARN-10506 is done, we need to call the API from the queue placement
> policy to create queues.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10535) Make queue placement in CapacityScheduler compliant with auto-queue-placement
[
https://issues.apache.org/jira/browse/YARN-10535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17265947#comment-17265947
]
Peter Bacsko commented on YARN-10535:
-
Initial review:
1. Remove star imports: {{import java.util.*;}}
2. Remove star imports: {{import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.*;}}
3. Is this class used? {{import org.apache.hadoop.yarn.webapp.hamlet2.Hamlet;}}
4. Nit: some exceptions ends with ".", some with "!", some with no punctuation
at all. Let's have some consistency in this area.
5. Nit: I'd replace "Unknown dynamic path lookup error" to sth like "Unknown
queue path validation result"
6. {{MappingRuleValidationContextImpl}}: I'd definitely call {{hasNext()}}
before calling {{String staticPart = pointer.next();}} just to be 100% safe
7. {{MappingRuleValidationContextImpl}}: {{staticPartParent}} is initially
{{null}}. Any change of it remaining {{null}} after the {{while}} loop? If so,
a null check is warranted.
8. Nit: I'd rename "QueueCreationValidation" to
"QueueCreationValidationResult" or "AutoQueueCreateValidationResult" to reflect
that it is a result of something. Or simply "ValidationResult", because it's a
nested enum.
I can't really comment on the validation logic because there a lot of string
operations and hopefully the unit tests do their job.
Talking about unit tests: {{TestMappingRuleValidationContextImpl}} was extended
with some new cases, does that provide enough coverage? Have you run it through
a coverage tool?
Also, many checkstyle issue, pls fix them if time permits (mostly indentation).
> Make queue placement in CapacityScheduler compliant with auto-queue-placement
> -
>
> Key: YARN-10535
> URL: https://issues.apache.org/jira/browse/YARN-10535
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Wangda Tan
>Assignee: Gergely Pollak
>Priority: Major
> Attachments: YARN-10535.001.patch
>
>
> Once YARN-10506 is done, we need to call the API from the queue placement
> policy to create queues.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10490) "yarn top" command not quitting completely with ctrl+c
[ https://issues.apache.org/jira/browse/YARN-10490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17265890#comment-17265890 ] Andras Gyori commented on YARN-10490: - Thank you [~akshink], the patch looks good to me +1. > "yarn top" command not quitting completely with ctrl+c > -- > > Key: YARN-10490 > URL: https://issues.apache.org/jira/browse/YARN-10490 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Reporter: Agshin Kazimli >Assignee: Agshin Kazimli >Priority: Minor > Attachments: YARN-10490-001.patch, YARN-10490-001.patch, > YARN-10490-002.patch > > > When we quit "yarn top" command using ctrl+c, even though the command prompt > returns, the old statistics info is still visible below the prompt. > And we need to manually type clear command or multiple return keys to get > around it. It is reported as a inconvenience for the customer from a > usability perspective. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17265868#comment-17265868 ] Andras Gyori commented on YARN-10506: - Thank you [~shuzirra] [~pbacsko] [~zhuqi] [~wangda] for the patches, reviews and discussion details. I have fixed some minor things (comments and findbug fixes), also reintroduced the locking on in autoCreateLeafQueue at LoC 3403, I think it is needed here, but might be subject to change. Apart from this, I have no additional comments +1. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch, > YARN-10506.014.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Gyori updated YARN-10506: Attachment: YARN-10506.014.patch > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch, > YARN-10506.014.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
