[jira] [Commented] (YARN-10645) Fix queue state related update for auto created queue.
[ https://issues.apache.org/jira/browse/YARN-10645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289716#comment-17289716 ] Qi Zhu commented on YARN-10645: --- [~snemeth] [~bteke] [~pbacsko] I tested in my local cluster, i confirmed that: If we set parent queue of auto created queue to stopped , then open it, but the auto created queue will never be running. If we set parent queue of auto created queue to running , then set it to stopped both the auto created queue set stopped, but the auto created queue will never be stopped state, it will always be running. > Fix queue state related update for auto created queue. > -- > > Key: YARN-10645 > URL: https://issues.apache.org/jira/browse/YARN-10645 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Critical > Attachments: YARN-10645.001.patch > > > Now the queue state in auto created queue can't be updated after refactor in > YARN-10504. > We should support fix the queue state related logic. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10645) Fix queue state related update for auto created queue.
[ https://issues.apache.org/jira/browse/YARN-10645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289716#comment-17289716 ] Qi Zhu edited comment on YARN-10645 at 2/24/21, 7:01 AM: - [~snemeth] [~bteke] [~pbacsko] I tested in my local cluster, i confirmed that: If we set parent queue of auto created queue to stopped , then open it, but the auto created queue will never be running. If we set parent queue of auto created queue to running , then set it to stopped both the auto created queue set stopped, but the auto created queue will never be stopped state, it will always be running state. was (Author: zhuqi): [~snemeth] [~bteke] [~pbacsko] I tested in my local cluster, i confirmed that: If we set parent queue of auto created queue to stopped , then open it, but the auto created queue will never be running. If we set parent queue of auto created queue to running , then set it to stopped both the auto created queue set stopped, but the auto created queue will never be stopped state, it will always be running. > Fix queue state related update for auto created queue. > -- > > Key: YARN-10645 > URL: https://issues.apache.org/jira/browse/YARN-10645 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Critical > Attachments: YARN-10645.001.patch > > > Now the queue state in auto created queue can't be updated after refactor in > YARN-10504. > We should support fix the queue state related logic. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10258) Add metrics for 'ApplicationsRunning' in NodeManager
[
https://issues.apache.org/jira/browse/YARN-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289672#comment-17289672
]
Hadoop QA commented on YARN-10258:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
16s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
58s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 21s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
19s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
20s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
17s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 1s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green}{color} | {color:green} the patch
[jira] [Commented] (YARN-10125) In Federation, kill application from client does not kill Unmanaged AM's and containers launched by Unmanaged AM
[ https://issues.apache.org/jira/browse/YARN-10125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289661#comment-17289661 ] Brahma Reddy Battula commented on YARN-10125: - [~subru] thanks for prompt reply.. [~dmmkr] could address the check-style issues..? > In Federation, kill application from client does not kill Unmanaged AM's and > containers launched by Unmanaged AM > > > Key: YARN-10125 > URL: https://issues.apache.org/jira/browse/YARN-10125 > Project: Hadoop YARN > Issue Type: Bug > Components: client, federation, router >Reporter: D M Murali Krishna Reddy >Assignee: D M Murali Krishna Reddy >Priority: Major > Attachments: YARN-10125.001.patch > > > In Federation, killing an application from client using "bin/yarn application > -kill ", kills the containers only of the home subcluster, > the Unmanaged AM and the containers launched in other subcluster are not > being killed causing blocking of resources. > The containers get killed after the task gets completed and The unmanaged AM > gets killed after 10 minutes of killing the application, killing any > remaining running containers in that subcluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10258) Add metrics for 'ApplicationsRunning' in NodeManager
[ https://issues.apache.org/jira/browse/YARN-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-10258: -- Attachment: YARN-10258-006.patch > Add metrics for 'ApplicationsRunning' in NodeManager > > > Key: YARN-10258 > URL: https://issues.apache.org/jira/browse/YARN-10258 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.1.3 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Minor > Attachments: YARN-10258-001.patch, YARN-10258-002.patch, > YARN-10258-003.patch, YARN-10258-005.patch, YARN-10258-006.patch, > YARN-10258_004.patch > > > Add metrics for 'ApplicationsRunning' in NodeManagers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10629) Add a timeout when the AM container is not allocated resources within the timeout period then kill the application.
[ https://issues.apache.org/jira/browse/YARN-10629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-10629: -- Attachment: YARN-10629-001.patch > Add a timeout when the AM container is not allocated resources within the > timeout period then kill the application. > --- > > Key: YARN-10629 > URL: https://issues.apache.org/jira/browse/YARN-10629 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.1.1 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Minor > Labels: resourcemanager > Fix For: 3.1.1 > > Attachments: YARN-10629-001.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10649) ContainerIdPBImpl & ContainerStatusPBImpl objects can be leaked in RMNodeImpl.updatedExistContainers
[ https://issues.apache.org/jira/browse/YARN-10649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-10649: -- Labels: pull-request-available (was: ) > ContainerIdPBImpl & ContainerStatusPBImpl objects can be leaked in > RMNodeImpl.updatedExistContainers > > > Key: YARN-10649 > URL: https://issues.apache.org/jira/browse/YARN-10649 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.3.0 >Reporter: Max Xie >Assignee: Max Xie >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > YARN-5168 the patch added RMNodeImpl.updatedExistContainers, but it didn't > remove completed containers. > These objects stay in RMNodeImpl.completedContainers forever. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10649) ContainerIdPBImpl & ContainerStatusPBImpl objects can be leaked in RMNodeImpl.updatedExistContainers
Max Xie created YARN-10649: --- Summary: ContainerIdPBImpl & ContainerStatusPBImpl objects can be leaked in RMNodeImpl.updatedExistContainers Key: YARN-10649 URL: https://issues.apache.org/jira/browse/YARN-10649 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 3.3.0 Reporter: Max Xie Assignee: Max Xie https://issues.apache.org/jira/browse/YARN-5168 the patch added RMNodeImpl.updatedExistContainers, but it didn't remove completed containers. These objects stay in RMNodeImpl.completedContainers forever. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10649) ContainerIdPBImpl & ContainerStatusPBImpl objects can be leaked in RMNodeImpl.updatedExistContainers
[ https://issues.apache.org/jira/browse/YARN-10649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Xie updated YARN-10649: Description: YARN-5168 the patch added RMNodeImpl.updatedExistContainers, but it didn't remove completed containers. These objects stay in RMNodeImpl.completedContainers forever. was: https://issues.apache.org/jira/browse/YARN-5168 the patch added RMNodeImpl.updatedExistContainers, but it didn't remove completed containers. These objects stay in RMNodeImpl.completedContainers forever. > ContainerIdPBImpl & ContainerStatusPBImpl objects can be leaked in > RMNodeImpl.updatedExistContainers > > > Key: YARN-10649 > URL: https://issues.apache.org/jira/browse/YARN-10649 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.3.0 >Reporter: Max Xie >Assignee: Max Xie >Priority: Major > > YARN-5168 the patch added RMNodeImpl.updatedExistContainers, but it didn't > remove completed containers. > These objects stay in RMNodeImpl.completedContainers forever. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10613) Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF
[
https://issues.apache.org/jira/browse/YARN-10613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289617#comment-17289617
]
Hadoop QA commented on YARN-10613:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 22m
13s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
7s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 44s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
51s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 39s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/666/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 103 unchanged - 1 fixed = 104 total (was 104)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/666/artifact/out/whitespace-eol.txt{color}
| {color:red} The patch has 5 line(s) that end in whitespace. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient
[jira] [Updated] (YARN-10613) Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF
[
https://issues.apache.org/jira/browse/YARN-10613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated YARN-10613:
--
Attachment: YARN-10613.trunk.001.patch
> Config to allow Intra- and Inter-queue preemption to enable/disable
> conservativeDRF
>
>
> Key: YARN-10613
> URL: https://issues.apache.org/jira/browse/YARN-10613
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 3.3.0, 3.2.2, 3.1.4, 2.10.1
>Reporter: Eric Payne
>Assignee: Eric Payne
>Priority: Minor
> Attachments: YARN-10613.trunk.001.patch
>
>
> YARN-8292 added code that prevents CS intra-queue preemption from preempting
> containers from an app unless all of the major resources used by the app are
> greater than the user limit for that user.
> Ex:
> | Used | User Limit |
> | <58GB, 58> | <30GB, 300> |
> In this example, only used memory is above the user limit, not used vcores.
> So, intra-queue preemption will not occur.
> YARN-8292 added the {{conservativeDRF}} flag to
> {{CapacitySchedulerPreemptionUtils#tryPreemptContainerAndDeductResToObtain}}.
> If {{conservativeDRF}} is false, containers will be preempted from apps in
> the example state. If true, containers will not be preempted.
> This flag is hard-coded to false for Inter-queue (cross-queue) preemption and
> true for intra-queue (in-queue) preemption.
> I propose that in some cases, we want intra-queue preemption to be more
> aggressive and preempt in the example case. To accommodate that, I propose
> the addition of a config property.
> Also, we may want inter-queue (cross-queue) preemption to be more
> conservative, so I propose also making that a configuration property:
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10613) Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF
[
https://issues.apache.org/jira/browse/YARN-10613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289407#comment-17289407
]
Eric Payne commented on YARN-10613:
---
I was wrong in my original assessment. To be consistent with the existing
property names, one would have the {{intra-queue-preemption}} prefix and the
other would not.
So, it would look like this:
{code}
yarn.resourcemanager.monitor.capacity.preemption.conservative-drf
yarn.resourcemanager.monitor.capacity.preemption.inter-queue-preemption.conservative-drf
{code}
We should probably be consistent, even though it's ugly.
> Config to allow Intra- and Inter-queue preemption to enable/disable
> conservativeDRF
>
>
> Key: YARN-10613
> URL: https://issues.apache.org/jira/browse/YARN-10613
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 3.3.0, 3.2.2, 3.1.4, 2.10.1
>Reporter: Eric Payne
>Assignee: Eric Payne
>Priority: Minor
>
> YARN-8292 added code that prevents CS intra-queue preemption from preempting
> containers from an app unless all of the major resources used by the app are
> greater than the user limit for that user.
> Ex:
> | Used | User Limit |
> | <58GB, 58> | <30GB, 300> |
> In this example, only used memory is above the user limit, not used vcores.
> So, intra-queue preemption will not occur.
> YARN-8292 added the {{conservativeDRF}} flag to
> {{CapacitySchedulerPreemptionUtils#tryPreemptContainerAndDeductResToObtain}}.
> If {{conservativeDRF}} is false, containers will be preempted from apps in
> the example state. If true, containers will not be preempted.
> This flag is hard-coded to false for Inter-queue (cross-queue) preemption and
> true for intra-queue (in-queue) preemption.
> I propose that in some cases, we want intra-queue preemption to be more
> aggressive and preempt in the example case. To accommodate that, I propose
> the addition of a config property.
> Also, we may want inter-queue (cross-queue) preemption to be more
> conservative, so I propose also making that a configuration property:
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10613) Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF
[
https://issues.apache.org/jira/browse/YARN-10613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289400#comment-17289400
]
Jim Brennan commented on YARN-10613:
I think you should stick with INTRA_QUEUE_PREEMPTION_CONFIG_PREFIX instead of
adding the new one.
We already have that redundancy for other configs.
> Config to allow Intra- and Inter-queue preemption to enable/disable
> conservativeDRF
>
>
> Key: YARN-10613
> URL: https://issues.apache.org/jira/browse/YARN-10613
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 3.3.0, 3.2.2, 3.1.4, 2.10.1
>Reporter: Eric Payne
>Assignee: Eric Payne
>Priority: Minor
>
> YARN-8292 added code that prevents CS intra-queue preemption from preempting
> containers from an app unless all of the major resources used by the app are
> greater than the user limit for that user.
> Ex:
> | Used | User Limit |
> | <58GB, 58> | <30GB, 300> |
> In this example, only used memory is above the user limit, not used vcores.
> So, intra-queue preemption will not occur.
> YARN-8292 added the {{conservativeDRF}} flag to
> {{CapacitySchedulerPreemptionUtils#tryPreemptContainerAndDeductResToObtain}}.
> If {{conservativeDRF}} is false, containers will be preempted from apps in
> the example state. If true, containers will not be preempted.
> This flag is hard-coded to false for Inter-queue (cross-queue) preemption and
> true for intra-queue (in-queue) preemption.
> I propose that in some cases, we want intra-queue preemption to be more
> aggressive and preempt in the example case. To accommodate that, I propose
> the addition of a config property.
> Also, we may want inter-queue (cross-queue) preemption to be more
> conservative, so I propose also making that a configuration property:
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10613) Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF
[
https://issues.apache.org/jira/browse/YARN-10613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289393#comment-17289393
]
Eric Payne commented on YARN-10613:
---
Actually, I want to change my suggestions to the following, since thee is no
need to have the word "preemption" twice:
{code}
yarn.resourcemanager.monitor.capacity.preemption.in-queue.conservative-drf
yarn.resourcemanager.monitor.capacity.preemption.cross-queue.conservative-drf
{code}
bq. For the sake of readability, I suggest the following instead:
However, adding these property names has its own set of messiness.
With my changes, we will have both of the following:
{code}
private static final String INTRA_QUEUE_PREEMPTION_CONFIG_PREFIX =
"intra-queue-preemption.";
private static final String IN_QUEUE_PREEMPTION_CONFIG_PREFIX =
"in-queue.";
{code}
> Config to allow Intra- and Inter-queue preemption to enable/disable
> conservativeDRF
>
>
> Key: YARN-10613
> URL: https://issues.apache.org/jira/browse/YARN-10613
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 3.3.0, 3.2.2, 3.1.4, 2.10.1
>Reporter: Eric Payne
>Assignee: Eric Payne
>Priority: Minor
>
> YARN-8292 added code that prevents CS intra-queue preemption from preempting
> containers from an app unless all of the major resources used by the app are
> greater than the user limit for that user.
> Ex:
> | Used | User Limit |
> | <58GB, 58> | <30GB, 300> |
> In this example, only used memory is above the user limit, not used vcores.
> So, intra-queue preemption will not occur.
> YARN-8292 added the {{conservativeDRF}} flag to
> {{CapacitySchedulerPreemptionUtils#tryPreemptContainerAndDeductResToObtain}}.
> If {{conservativeDRF}} is false, containers will be preempted from apps in
> the example state. If true, containers will not be preempted.
> This flag is hard-coded to false for Inter-queue (cross-queue) preemption and
> true for intra-queue (in-queue) preemption.
> I propose that in some cases, we want intra-queue preemption to be more
> aggressive and preempt in the example case. To accommodate that, I propose
> the addition of a config property.
> Also, we may want inter-queue (cross-queue) preemption to be more
> conservative, so I propose also making that a configuration property:
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10613) Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF
[
https://issues.apache.org/jira/browse/YARN-10613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289325#comment-17289325
]
Jim Brennan commented on YARN-10613:
I like your proposal for the more easily distinguished property names.
> Config to allow Intra- and Inter-queue preemption to enable/disable
> conservativeDRF
>
>
> Key: YARN-10613
> URL: https://issues.apache.org/jira/browse/YARN-10613
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 3.3.0, 3.2.2, 3.1.4, 2.10.1
>Reporter: Eric Payne
>Assignee: Eric Payne
>Priority: Minor
>
> YARN-8292 added code that prevents CS intra-queue preemption from preempting
> containers from an app unless all of the major resources used by the app are
> greater than the user limit for that user.
> Ex:
> | Used | User Limit |
> | <58GB, 58> | <30GB, 300> |
> In this example, only used memory is above the user limit, not used vcores.
> So, intra-queue preemption will not occur.
> YARN-8292 added the {{conservativeDRF}} flag to
> {{CapacitySchedulerPreemptionUtils#tryPreemptContainerAndDeductResToObtain}}.
> If {{conservativeDRF}} is false, containers will be preempted from apps in
> the example state. If true, containers will not be preempted.
> This flag is hard-coded to false for Inter-queue (cross-queue) preemption and
> true for intra-queue (in-queue) preemption.
> I propose that in some cases, we want intra-queue preemption to be more
> aggressive and preempt in the example case. To accommodate that, I propose
> the addition of a config property.
> Also, we may want inter-queue (cross-queue) preemption to be more
> conservative, so I propose also making that a configuration property:
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10613) Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF
[
https://issues.apache.org/jira/browse/YARN-10613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289323#comment-17289323
]
Eric Payne commented on YARN-10613:
---
As far as property names go, the logical thing to do would be to add the
following two properties (one that affects inter-queue and one that affectsr
intra-queue preemption):
{code}
yarn.resourcemanager.monitor.capacity.preemption.intra-queue-preemption.conservative-drf
yarn.resourcemanager.monitor.capacity.preemption.inter-queue-preemption.conservative-drf
{code}
However, those very long names are exactly the same except for 1 character. For
the sake of readability, I suggest the following instead:
{code}
yarn.resourcemanager.monitor.capacity.preemption.in-queue-preemption.conservative-drf
yarn.resourcemanager.monitor.capacity.preemption.cross-queue-preemption.conservative-drf
{code}
> Config to allow Intra- and Inter-queue preemption to enable/disable
> conservativeDRF
>
>
> Key: YARN-10613
> URL: https://issues.apache.org/jira/browse/YARN-10613
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 3.3.0, 3.2.2, 3.1.4, 2.10.1
>Reporter: Eric Payne
>Assignee: Eric Payne
>Priority: Minor
>
> YARN-8292 added code that prevents CS intra-queue preemption from preempting
> containers from an app unless all of the major resources used by the app are
> greater than the user limit for that user.
> Ex:
> | Used | User Limit |
> | <58GB, 58> | <30GB, 300> |
> In this example, only used memory is above the user limit, not used vcores.
> So, intra-queue preemption will not occur.
> YARN-8292 added the {{conservativeDRF}} flag to
> {{CapacitySchedulerPreemptionUtils#tryPreemptContainerAndDeductResToObtain}}.
> If {{conservativeDRF}} is false, containers will be preempted from apps in
> the example state. If true, containers will not be preempted.
> This flag is hard-coded to false for Inter-queue (cross-queue) preemption and
> true for intra-queue (in-queue) preemption.
> I propose that in some cases, we want intra-queue preemption to be more
> aggressive and preempt in the example case. To accommodate that, I propose
> the addition of a config property.
> Also, we may want inter-queue (cross-queue) preemption to be more
> conservative, so I propose also making that a configuration property:
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10613) Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF
[
https://issues.apache.org/jira/browse/YARN-10613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289319#comment-17289319
]
Eric Payne commented on YARN-10613:
---
Thanks for the suggestion, [~Jim_Brennan]! I have updated the summary and
description to reflect that conservative DRF should be configurable for both
inter and intra queue preemption.
> Config to allow Intra- and Inter-queue preemption to enable/disable
> conservativeDRF
>
>
> Key: YARN-10613
> URL: https://issues.apache.org/jira/browse/YARN-10613
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 3.3.0, 3.2.2, 3.1.4, 2.10.1
>Reporter: Eric Payne
>Assignee: Eric Payne
>Priority: Minor
>
> YARN-8292 added code that prevents CS intra-queue preemption from preempting
> containers from an app unless all of the major resources used by the app are
> greater than the user limit for that user.
> Ex:
> | Used | User Limit |
> | <58GB, 58> | <30GB, 300> |
> In this example, only used memory is above the user limit, not used vcores.
> So, intra-queue preemption will not occur.
> YARN-8292 added the {{conservativeDRF}} flag to
> {{CapacitySchedulerPreemptionUtils#tryPreemptContainerAndDeductResToObtain}}.
> If {{conservativeDRF}} is false, containers will be preempted from apps in
> the example state. If true, containers will not be preempted.
> This flag is hard-coded to false for Inter-queue (cross-queue) preemption and
> true for intra-queue (in-queue) preemption.
> I propose that in some cases, we want intra-queue preemption to be more
> aggressive and preempt in the example case. To accommodate that, I propose
> the addition of a config property.
> Also, we may want inter-queue (cross-queue) preemption to be more
> conservative, so I propose also making that a configuration property:
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10613) Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF
[
https://issues.apache.org/jira/browse/YARN-10613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated YARN-10613:
--
Description:
YARN-8292 added code that prevents CS intra-queue preemption from preempting
containers from an app unless all of the major resources used by the app are
greater than the user limit for that user.
Ex:
| Used | User Limit |
| <58GB, 58> | <30GB, 300> |
In this example, only used memory is above the user limit, not used vcores. So,
intra-queue preemption will not occur.
YARN-8292 added the {{conservativeDRF}} flag to
{{CapacitySchedulerPreemptionUtils#tryPreemptContainerAndDeductResToObtain}}.
If {{conservativeDRF}} is false, containers will be preempted from apps in the
example state. If true, containers will not be preempted.
This flag is hard-coded to false for Inter-queue (cross-queue) preemption and
true for intra-queue (in-queue) preemption.
I propose that in some cases, we want intra-queue preemption to be more
aggressive and preempt in the example case. To accommodate that, I propose the
addition of a config property.
Also, we may want inter-queue (cross-queue) preemption to be more conservative,
so I propose also making that a configuration property:
was:
YARN-8292 added code that prevents CS intra-queue preemption from preempting
containers from an app unless all of the major resources used by the app are
greater than the user limit for that user.
Ex:
| Used | User Limit |
| <58GB, 58> | <30GB, 300> |
In this example, only used memory is above the user limit, not used vcores. So,
intra-queue preemption will not occur.
YARN-8292 added the {{conservativeDRF}} flag to
{{CapacitySchedulerPreemptionUtils#tryPreemptContainerAndDeductResToObtain}}.
If {{conservativeDRF}} is false, containers will be preempted from apps in the
example state. If true, containers will not be preempted.
This flag is hard-coded to false for Inter-queue (cross-queue) preemption and
true for intra-queue (in-queue) preemption.
I propose that in some cases, we want intra-queue preemption to be more
aggressive and preempt in the example case. To accommodate that, I propose the
addition of the following config property:
{code:xml}
yarn.resourcemanager.monitor.capacity.preemption.intra-queue-preemption.conservative-drf
true
{code}
> Config to allow Intra- and Inter-queue preemption to enable/disable
> conservativeDRF
>
>
> Key: YARN-10613
> URL: https://issues.apache.org/jira/browse/YARN-10613
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 3.3.0, 3.2.2, 3.1.4, 2.10.1
>Reporter: Eric Payne
>Assignee: Eric Payne
>Priority: Minor
>
> YARN-8292 added code that prevents CS intra-queue preemption from preempting
> containers from an app unless all of the major resources used by the app are
> greater than the user limit for that user.
> Ex:
> | Used | User Limit |
> | <58GB, 58> | <30GB, 300> |
> In this example, only used memory is above the user limit, not used vcores.
> So, intra-queue preemption will not occur.
> YARN-8292 added the {{conservativeDRF}} flag to
> {{CapacitySchedulerPreemptionUtils#tryPreemptContainerAndDeductResToObtain}}.
> If {{conservativeDRF}} is false, containers will be preempted from apps in
> the example state. If true, containers will not be preempted.
> This flag is hard-coded to false for Inter-queue (cross-queue) preemption and
> true for intra-queue (in-queue) preemption.
> I propose that in some cases, we want intra-queue preemption to be more
> aggressive and preempt in the example case. To accommodate that, I propose
> the addition of a config property.
> Also, we may want inter-queue (cross-queue) preemption to be more
> conservative, so I propose also making that a configuration property:
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10613) Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF
[
https://issues.apache.org/jira/browse/YARN-10613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated YARN-10613:
--
Summary: Config to allow Intra- and Inter-queue preemption to
enable/disable conservativeDRF (was: Config to allow Intra-queue preemption to
enable/disable conservativeDRF)
> Config to allow Intra- and Inter-queue preemption to enable/disable
> conservativeDRF
>
>
> Key: YARN-10613
> URL: https://issues.apache.org/jira/browse/YARN-10613
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 3.3.0, 3.2.2, 3.1.4, 2.10.1
>Reporter: Eric Payne
>Assignee: Eric Payne
>Priority: Minor
>
> YARN-8292 added code that prevents CS intra-queue preemption from preempting
> containers from an app unless all of the major resources used by the app are
> greater than the user limit for that user.
> Ex:
> | Used | User Limit |
> | <58GB, 58> | <30GB, 300> |
> In this example, only used memory is above the user limit, not used vcores.
> So, intra-queue preemption will not occur.
> YARN-8292 added the {{conservativeDRF}} flag to
> {{CapacitySchedulerPreemptionUtils#tryPreemptContainerAndDeductResToObtain}}.
> If {{conservativeDRF}} is false, containers will be preempted from apps in
> the example state. If true, containers will not be preempted.
> This flag is hard-coded to false for Inter-queue (cross-queue) preemption and
> true for intra-queue (in-queue) preemption.
> I propose that in some cases, we want intra-queue preemption to be more
> aggressive and preempt in the example case. To accommodate that, I propose
> the addition of the following config property:
> {code:xml}
>
>
> yarn.resourcemanager.monitor.capacity.preemption.intra-queue-preemption.conservative-drf
> true
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10633) setup yarn federation failed
[ https://issues.apache.org/jira/browse/YARN-10633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289299#comment-17289299 ] Subramaniam Krishnan commented on YARN-10633: - [~hanfrank], there are multiple other configs require to enable federation as well. Can you follow the detailed steps in the Configuration section under: https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/Federation.html > setup yarn federation failed > > > Key: YARN-10633 > URL: https://issues.apache.org/jira/browse/YARN-10633 > Project: Hadoop YARN > Issue Type: Bug > Components: federation >Affects Versions: 3.2.2 >Reporter: yuguang >Priority: Major > > Hi > I am trying to setup yarn federation mode. But after I add below > configuration in etc/hadoop/yarn-site.xml > > yarn.federation.enabled > true > > then when I run yarn node -list . Get below error . Also the historyserver > service can not be started either . > I am using hadoop-3.2.2 version . > [root@yarna hadoop-3.2.2]# yarn node -list > 2021-02-18 05:51:39,178 INFO service.AbstractService: Service > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl failed in state > STARTEDjava.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for > length 0 at > org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.init(ConfiguredRMFailoverProxyProvider.java:62) > at > org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:175) > at org.apache.hadoop.yarn.client.RMProxy.newProxyInstance(RMProxy.java:130) > at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:103) at > org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStart(YarnClientImpl.java:233) > at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) > at > org.apache.hadoop.yarn.client.cli.YarnCLI.createAndStartYarnClient(YarnCLI.java:55) > at org.apache.hadoop.yarn.client.cli.NodeCLI.run(NodeCLI.java:110) at > org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at > org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at > org.apache.hadoop.yarn.client.cli.NodeCLI.main(NodeCLI.java:62)Exception in > thread "main" java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds > for length 0 at > org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.init(ConfiguredRMFailoverProxyProvider.java:62) > at > org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:175) > at org.apache.hadoop.yarn.client.RMProxy.newProxyInstance(RMProxy.java:130) > at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:103) at > org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStart(YarnClientImpl.java:233) > at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) > at > org.apache.hadoop.yarn.client.cli.YarnCLI.createAndStartYarnClient(YarnCLI.java:55) > at org.apache.hadoop.yarn.client.cli.NodeCLI.run(NodeCLI.java:110) at > org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at > org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at > org.apache.hadoop.yarn.client.cli.NodeCLI.main(NodeCLI.java:62) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10125) In Federation, kill application from client does not kill Unmanaged AM's and containers launched by Unmanaged AM
[ https://issues.apache.org/jira/browse/YARN-10125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289294#comment-17289294 ] Subramaniam Krishnan commented on YARN-10125: - Thanks [~brahmareddy] for looping me in, agree it should be handled. I recall testing this as well in the initial version years back as well, not sure how it got dropped in the interim. Thanks [~dmmkr] for working on this! > In Federation, kill application from client does not kill Unmanaged AM's and > containers launched by Unmanaged AM > > > Key: YARN-10125 > URL: https://issues.apache.org/jira/browse/YARN-10125 > Project: Hadoop YARN > Issue Type: Bug > Components: client, federation, router >Reporter: D M Murali Krishna Reddy >Assignee: D M Murali Krishna Reddy >Priority: Major > Attachments: YARN-10125.001.patch > > > In Federation, killing an application from client using "bin/yarn application > -kill ", kills the containers only of the home subcluster, > the Unmanaged AM and the containers launched in other subcluster are not > being killed causing blocking of resources. > The containers get killed after the task gets completed and The unmanaged AM > gets killed after 10 minutes of killing the application, killing any > remaining running containers in that subcluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode
[
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289236#comment-17289236
]
Hadoop QA commented on YARN-10627:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
51s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
48s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 53s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
49s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/665/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 4 new + 147 unchanged - 15 fixed = 151 total (was 162)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 49s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
|
[jira] [Commented] (YARN-10544) AMParams.java having un-necessary access identifier static final
[
https://issues.apache.org/jira/browse/YARN-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289095#comment-17289095
]
Hadoop QA commented on YARN-10544:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
52s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to
include any new or modified tests. Please justify why no new tests are needed
for this patch. Also please list what manual steps were performed to verify
this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
54s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 33s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
9s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config;
considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
6s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green}{color} | {color:green}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:
The patch generated 0 new + 1 unchanged - 9 fixed = 1 total (was 10) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 8s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
[jira] [Commented] (YARN-10544) AMParams.java having un-necessary access identifier static final
[ https://issues.apache.org/jira/browse/YARN-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289057#comment-17289057 ] Brahma Reddy Battula commented on YARN-10544: - [~gb.ana...@gmail.com] thanks report and working on this. Patch LGTM. > AMParams.java having un-necessary access identifier static final > > > Key: YARN-10544 > URL: https://issues.apache.org/jira/browse/YARN-10544 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Trivial > Attachments: YARN-10544_001.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10258) Add metrics for 'ApplicationsRunning' in NodeManager
[ https://issues.apache.org/jira/browse/YARN-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289054#comment-17289054 ] Brahma Reddy Battula commented on YARN-10258: - [~gb.ana...@gmail.com] thanks for report and working on this. Could please address the jenkins report. > Add metrics for 'ApplicationsRunning' in NodeManager > > > Key: YARN-10258 > URL: https://issues.apache.org/jira/browse/YARN-10258 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.1.3 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Minor > Attachments: YARN-10258-001.patch, YARN-10258-002.patch, > YARN-10258-003.patch, YARN-10258-005.patch, YARN-10258_004.patch > > > Add metrics for 'ApplicationsRunning' in NodeManagers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10258) Add metrics for 'ApplicationsRunning' in NodeManager
[
https://issues.apache.org/jira/browse/YARN-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289002#comment-17289002
]
Hadoop QA commented on YARN-10258:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
10s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
37s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 42s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
21s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
17s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 25s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/663/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 2 new + 71 unchanged - 0 fixed = 73 total (was 71) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/663/artifact/out/whitespace-tabs.txt{color}
| {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 23s{color} | {color:green}{color} | {color:green} patch has no errors when
[jira] [Commented] (YARN-10648) NM local logs are not cleared after uploading to hdfs
[
https://issues.apache.org/jira/browse/YARN-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288999#comment-17288999
]
Brahma Reddy Battula commented on YARN-10648:
-
[~dmmkr] thanks for reporting this. Looks some how it's missed in YARN-8273.
Patch LGTM.
[~grepas]/[~rkanter]/[~snemeth] can you also look into this as you worked on
YARN-8273.
> NM local logs are not cleared after uploading to hdfs
> -
>
> Key: YARN-10648
> URL: https://issues.apache.org/jira/browse/YARN-10648
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 3.2.0
>Reporter: D M Murali Krishna Reddy
>Assignee: D M Murali Krishna Reddy
>Priority: Major
> Attachments: YARN-10648.001.patch
>
>
> YARN-8273 has induced the following issues.
> # The {color:#00}delService.delete(deletionTask){color} has been removed
> from the for loop, and added at the end in finally block. Inside the for loop
> we are creating FileDeletionTask for each container, but not storing it, due
> to this, only the last container log files will be present in the
> deletionTask and only those files will be removed. Ideally all the container
> log files which are uploaded must be deleted.
> # The LogAggregationDFSException is caught in the closeswriter, but when we
> configure LogAggregationTFileController as logAggregationFileController,
> this.logAggregationFileController.closeWriter() itself calls closeWriter,
> which throws LogAggregationDFSException if any, and the exception is not
> saved. Again when we try to do closeWriter we dont get any exception and, we
> are not throwing the LogAggregationDFSException in this scenario.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10648) NM local logs are not cleared after uploading to hdfs
[
https://issues.apache.org/jira/browse/YARN-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288991#comment-17288991
]
D M Murali Krishna Reddy edited comment on YARN-10648 at 2/23/21, 10:20 AM:
Hi [~snemeth] [~brahmareddy] Can you have a look over the issue and review the
patch.
Thanks
was (Author: dmmkr):
Hi [~snemeth] [~brahma] Can you have a look over the issue and review the
patch.
Thanks
> NM local logs are not cleared after uploading to hdfs
> -
>
> Key: YARN-10648
> URL: https://issues.apache.org/jira/browse/YARN-10648
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 3.2.0
>Reporter: D M Murali Krishna Reddy
>Assignee: D M Murali Krishna Reddy
>Priority: Major
> Attachments: YARN-10648.001.patch
>
>
> YARN-8273 has induced the following issues.
> # The {color:#00}delService.delete(deletionTask){color} has been removed
> from the for loop, and added at the end in finally block. Inside the for loop
> we are creating FileDeletionTask for each container, but not storing it, due
> to this, only the last container log files will be present in the
> deletionTask and only those files will be removed. Ideally all the container
> log files which are uploaded must be deleted.
> # The LogAggregationDFSException is caught in the closeswriter, but when we
> configure LogAggregationTFileController as logAggregationFileController,
> this.logAggregationFileController.closeWriter() itself calls closeWriter,
> which throws LogAggregationDFSException if any, and the exception is not
> saved. Again when we try to do closeWriter we dont get any exception and, we
> are not throwing the LogAggregationDFSException in this scenario.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10648) NM local logs are not cleared after uploading to hdfs
[
https://issues.apache.org/jira/browse/YARN-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288996#comment-17288996
]
Hadoop QA commented on YARN-10648:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
16s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to
include any new or modified tests. Please justify why no new tests are needed
for this patch. Also please list what manual steps were performed to verify
this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
59s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
56s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
27s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
22s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
36s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
38s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 43s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
28s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
58s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/662/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html{color}
| {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has
1 extant findbugs warnings. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
29s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
29s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
33s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
33s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
41s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
42s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green}
[jira] [Commented] (YARN-10648) NM local logs are not cleared after uploading to hdfs
[
https://issues.apache.org/jira/browse/YARN-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288991#comment-17288991
]
D M Murali Krishna Reddy commented on YARN-10648:
-
Hi [~snemeth] [~brahma] Can you have a look over the issue and review the
patch.
Thanks
> NM local logs are not cleared after uploading to hdfs
> -
>
> Key: YARN-10648
> URL: https://issues.apache.org/jira/browse/YARN-10648
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 3.2.0
>Reporter: D M Murali Krishna Reddy
>Assignee: D M Murali Krishna Reddy
>Priority: Major
> Attachments: YARN-10648.001.patch
>
>
> YARN-8273 has induced the following issues.
> # The {color:#00}delService.delete(deletionTask){color} has been removed
> from the for loop, and added at the end in finally block. Inside the for loop
> we are creating FileDeletionTask for each container, but not storing it, due
> to this, only the last container log files will be present in the
> deletionTask and only those files will be removed. Ideally all the container
> log files which are uploaded must be deleted.
> # The LogAggregationDFSException is caught in the closeswriter, but when we
> configure LogAggregationTFileController as logAggregationFileController,
> this.logAggregationFileController.closeWriter() itself calls closeWriter,
> which throws LogAggregationDFSException if any, and the exception is not
> saved. Again when we try to do closeWriter we dont get any exception and, we
> are not throwing the LogAggregationDFSException in this scenario.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10641) Refactor the max app related update, and fix maxApllications update error when add new queues.
[ https://issues.apache.org/jira/browse/YARN-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288972#comment-17288972 ] Qi Zhu commented on YARN-10641: --- The test not related, passed in my local test. > Refactor the max app related update, and fix maxApllications update error > when add new queues. > -- > > Key: YARN-10641 > URL: https://issues.apache.org/jira/browse/YARN-10641 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Critical > Attachments: YARN-10641.001.patch, YARN-10641.002.patch, > YARN-10641.003.patch, YARN-10641.004.patch, > image-2021-02-20-15-49-58-677.png, image-2021-02-20-15-53-51-099.png, > image-2021-02-20-15-55-44-780.png, image-2021-02-20-16-29-18-519.png, > image-2021-02-20-16-31-13-714.png > > > When refactor the update logic in YARN-10504 . > The update max applications based abs/cap is wrong, this should be fixed, > because the max applications is key part to limit applications in CS. > For example: > When adding a dynamic queue, the other children's max app of parent queue are > not updated correctly: > !image-2021-02-20-15-53-51-099.png|width=639,height=509! > The new added queue's max app will updated correctly: > !image-2021-02-20-15-55-44-780.png|width=542,height=426! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10258) Add metrics for 'ApplicationsRunning' in NodeManager
[ https://issues.apache.org/jira/browse/YARN-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ANANDA G B updated YARN-10258: -- Attachment: YARN-10258-005.patch > Add metrics for 'ApplicationsRunning' in NodeManager > > > Key: YARN-10258 > URL: https://issues.apache.org/jira/browse/YARN-10258 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.1.3 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Minor > Attachments: YARN-10258-001.patch, YARN-10258-002.patch, > YARN-10258-003.patch, YARN-10258-005.patch, YARN-10258_004.patch > > > Add metrics for 'ApplicationsRunning' in NodeManagers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10258) Add metrics for 'ApplicationsRunning' in NodeManager
[
https://issues.apache.org/jira/browse/YARN-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288934#comment-17288934
]
Hadoop QA commented on YARN-10258:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
49s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
44s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 3s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
36s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
34s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 27s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/661/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 2 new + 72 unchanged - 0 fixed = 74 total (was 72) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/661/artifact/out/whitespace-eol.txt{color}
| {color:red} The patch has 1 line(s) that end in whitespace. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} |
[jira] [Commented] (YARN-10466) Fix NullPointerException in yarn-services Component.java
[
https://issues.apache.org/jira/browse/YARN-10466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288923#comment-17288923
]
Hadoop QA commented on YARN-10466:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
29s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to
include any new or modified tests. Please justify why no new tests are needed
for this patch. Also please list what manual steps were performed to verify
this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
22s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 8s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
57s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 57s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
|
