[jira] [Updated] (YARN-7563) Invalid event: FINISH_APPLICATION at NEW may make some application level resource be not cleaned
[
https://issues.apache.org/jira/browse/YARN-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lujie updated YARN-7563:
Attachment: YARN-7563.txt
> Invalid event: FINISH_APPLICATION at NEW may make some application level
> resource be not cleaned
> -
>
> Key: YARN-7563
> URL: https://issues.apache.org/jira/browse/YARN-7563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 2.6.0, 3.0.0-beta1
>Reporter: lujie
> Attachments: YARN-7563.png, YARN-7563.txt
>
>
> I send kill command to application, nodemanager log shows:
> {code:java}
> 2017-11-25 19:18:48,126 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> couldn't find container container_1511608703018_0001_01_01 while
> processing FINISH_CONTAINERS event
> 2017-11-25 19:18:48,146 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> FINISH_APPLICATION at NEW
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-25 19:18:48,151 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1511608703018_0001 transitioned from NEW to INITING
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7563) Invalid event: FINISH_APPLICATION at NEW may make some application level resource be not cleaned
[
https://issues.apache.org/jira/browse/YARN-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lujie updated YARN-7563:
Attachment: (was: YARN-7563.txt)
> Invalid event: FINISH_APPLICATION at NEW may make some application level
> resource be not cleaned
> -
>
> Key: YARN-7563
> URL: https://issues.apache.org/jira/browse/YARN-7563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 2.6.0, 3.0.0-beta1
>Reporter: lujie
> Attachments: YARN-7563.png
>
>
> I send kill command to application, nodemanager log shows:
> {code:java}
> 2017-11-25 19:18:48,126 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> couldn't find container container_1511608703018_0001_01_01 while
> processing FINISH_CONTAINERS event
> 2017-11-25 19:18:48,146 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> FINISH_APPLICATION at NEW
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-25 19:18:48,151 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1511608703018_0001 transitioned from NEW to INITING
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7563) Invalid event: FINISH_APPLICATION at NEW may make some application level resource be not cleaned
[
https://issues.apache.org/jira/browse/YARN-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lujie updated YARN-7563:
Attachment: YARN-7563.txt
> Invalid event: FINISH_APPLICATION at NEW may make some application level
> resource be not cleaned
> -
>
> Key: YARN-7563
> URL: https://issues.apache.org/jira/browse/YARN-7563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 2.6.0, 3.0.0-beta1
>Reporter: lujie
> Attachments: YARN-7563.png, YARN-7563.txt
>
>
> I send kill command to application, nodemanager log shows:
> {code:java}
> 2017-11-25 19:18:48,126 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> couldn't find container container_1511608703018_0001_01_01 while
> processing FINISH_CONTAINERS event
> 2017-11-25 19:18:48,146 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> FINISH_APPLICATION at NEW
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-25 19:18:48,151 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1511608703018_0001 transitioned from NEW to INITING
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7563) Invalid event: FINISH_APPLICATION at NEW may make some application level resource be not cleaned
[
https://issues.apache.org/jira/browse/YARN-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lujie updated YARN-7563:
Summary: Invalid event: FINISH_APPLICATION at NEW may make some
application level resource be not cleaned (was: Invalid event:
FINISH_APPLICATION at NEW may)
> Invalid event: FINISH_APPLICATION at NEW may make some application level
> resource be not cleaned
> -
>
> Key: YARN-7563
> URL: https://issues.apache.org/jira/browse/YARN-7563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 2.6.0, 3.0.0-beta1
>Reporter: lujie
> Attachments: YARN-7563.png
>
>
> I send kill command to application, nodemanager log shows:
> {code:java}
> 2017-11-25 19:18:48,126 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> couldn't find container container_1511608703018_0001_01_01 while
> processing FINISH_CONTAINERS event
> 2017-11-25 19:18:48,146 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> FINISH_APPLICATION at NEW
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-25 19:18:48,151 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1511608703018_0001 transitioned from NEW to INITING
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7563) Invalid event: FINISH_APPLICATION at NEW may
[
https://issues.apache.org/jira/browse/YARN-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lujie updated YARN-7563:
Summary: Invalid event: FINISH_APPLICATION at NEW may (was: Invalid event:
FINISH_APPLICATION at NEW)
> Invalid event: FINISH_APPLICATION at NEW may
>
>
> Key: YARN-7563
> URL: https://issues.apache.org/jira/browse/YARN-7563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 2.6.0, 3.0.0-beta1
>Reporter: lujie
> Attachments: YARN-7563.png
>
>
> I send kill command to application, nodemanager log shows:
> {code:java}
> 2017-11-25 19:18:48,126 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> couldn't find container container_1511608703018_0001_01_01 while
> processing FINISH_CONTAINERS event
> 2017-11-25 19:18:48,146 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> FINISH_APPLICATION at NEW
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-25 19:18:48,151 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1511608703018_0001 transitioned from NEW to INITING
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7563) Invalid event: FINISH_APPLICATION at NEW
[
https://issues.apache.org/jira/browse/YARN-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lujie updated YARN-7563:
Affects Version/s: 2.6.0
> Invalid event: FINISH_APPLICATION at NEW
>
>
> Key: YARN-7563
> URL: https://issues.apache.org/jira/browse/YARN-7563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 2.6.0, 3.0.0-beta1
>Reporter: lujie
> Attachments: YARN-7563.png
>
>
> I send kill command to application, nodemanager log shows:
> {code:java}
> 2017-11-25 19:18:48,126 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> couldn't find container container_1511608703018_0001_01_01 while
> processing FINISH_CONTAINERS event
> 2017-11-25 19:18:48,146 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> FINISH_APPLICATION at NEW
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-25 19:18:48,151 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1511608703018_0001 transitioned from NEW to INITING
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7562) queuePlacementPolicy should not match parent queue
[
https://issues.apache.org/jira/browse/YARN-7562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268199#comment-16268199
]
Wilfred Spiegelenburg commented on YARN-7562:
-
The behaviour you are seeing is the correct behaviour. You have defined the
_algo_ queue implicitly as a parent queue.
Your configuration is incomplete but even that already shows what you did wrong:
{code}
...
{code}
Since you have not closed the definition of the queue named _algo_ with a
{{}} tag the queue test is now seen as a parent queue.
Your first rule in the placement policy tries to find a queue directly below
the root queue with the user name. The user name _algo_ matches the existing
queue _algo_ and thus returns as expected. However you can not submit to a
parent queue and thus you get the exception. It is also impossible to have a
parent and leaf queue with the same name at one level. Not returning a *parent*
queue in the rules would break the whole concept of hierarchical queues and the
nested placement rules.
Your patch also breaks all existing queue setups with more than one level of
queues.
> queuePlacementPolicy should not match parent queue
> --
>
> Key: YARN-7562
> URL: https://issues.apache.org/jira/browse/YARN-7562
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, resourcemanager
>Affects Versions: 2.7.1
>Reporter: chuanjie.duan
> Attachments: YARN-7562.patch
>
>
> User algo submit a mapreduce job, console log said "root.algo is not a leaf
> queue exception".
> root.algo is a parent queue, it's meanless for me. Not sure why parent queue
> added before
>
>
> 300
>
>
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7572) Make the service status output more readable
Jian He created YARN-7572: - Summary: Make the service status output more readable Key: YARN-7572 URL: https://issues.apache.org/jira/browse/YARN-7572 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Currently the service status output is just a JSON spec, we can make it more human readable -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7564) Fix checkstyle issues of YARN-5881 branch
[ https://issues.apache.org/jira/browse/YARN-7564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268150#comment-16268150 ] Wangda Tan commented on YARN-7564: -- +1 to latest patch, thanks [~sunilg] > Fix checkstyle issues of YARN-5881 branch > - > > Key: YARN-7564 > URL: https://issues.apache.org/jira/browse/YARN-7564 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil G >Assignee: Sunil G >Priority: Minor > Attachments: YARN-7564-YARN-5881.001.patch > > > fix jenkins issues. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7562) queuePlacementPolicy should not match parent queue
[
https://issues.apache.org/jira/browse/YARN-7562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268118#comment-16268118
]
genericqa commented on YARN-7562:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 0s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 17s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}104m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestQueuePlacementPolicy |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7562 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12899372/YARN-7562.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 005cb3da2960 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d8923cd |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/18682/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
|
[jira] [Commented] (YARN-7491) Make sure AM is not scheduled on an opportunistic container
[
https://issues.apache.org/jira/browse/YARN-7491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268065#comment-16268065
]
Haibo Chen commented on YARN-7491:
--
That's because of this
{code}
@@ -491,7 +491,7 @@ private RMAppImpl createAndPopulateNewRMApp(
for (ResourceRequest amReq : amReqs) {
amReq.setCapability(anyReq.getCapability());
amReq.setExecutionTypeRequest(
- ExecutionTypeRequest.newInstance(ExecutionType.GUARANTEED));
+ ExecutionTypeRequest.newInstance(ExecutionType.GUARANTEED,
true));
amReq.setNumContainers(1);
amReq.setPriority(RMAppAttemptImpl.AM_CONTAINER_PRIORITY);
}
{code}
which is the core of this patch.
All AM resource requests, whether specified in submission context or not, will
be normalized as above.
If you revert that core change, TestAppManager will fail.
Though not absolutely necessary,
{code}
if (amReqs == null || amReqs.isEmpty()) {
if (submissionContext.getResource() != null) {
amReqs = Collections.singletonList(BuilderUtils
- .newResourceRequest(RMAppAttemptImpl.AM_CONTAINER_PRIORITY,
- ResourceRequest.ANY, submissionContext.getResource(), 1));
+ .newAMResourceRequest(RMAppAttemptImpl.AM_CONTAINER_PRIORITY,
+ ResourceRequest.ANY, submissionContext.getResource()));
{code}
makes it very clear that we always set the enforceExecutionType to true for AM
resource requests.
> Make sure AM is not scheduled on an opportunistic container
> ---
>
> Key: YARN-7491
> URL: https://issues.apache.org/jira/browse/YARN-7491
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: YARN-7491-YARN-1011.00.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6124) Make SchedulingEditPolicy can be enabled / disabled / updated with RMAdmin -refreshQueues
[
https://issues.apache.org/jira/browse/YARN-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268040#comment-16268040
]
genericqa commented on YARN-6124:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 5s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 34s{color}
| {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
generated 1 new + 13 unchanged - 0 fixed = 14 total (was 13) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 33s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 23 new + 715 unchanged - 1 fixed = 738 total (was 716)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 57s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 38s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}106m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerLazyPreemption
|
| |
hadoop.yarn.server.resourcemanager.monitor.capacity.TestProportionalCapacityPreemptionPolicy
|
| | hadoop.yarn.server.resourcemanager.monitor.TestSchedulingMonitor |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerSurgicalPreemption
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-6124 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12899521/YARN-6124.4.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 1f5379afaddd 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (YARN-6669) Support security for YARN service framework
[
https://issues.apache.org/jira/browse/YARN-6669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267999#comment-16267999
]
genericqa commented on YARN-6669:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 3s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
0s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
14s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 1s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 29 new + 297 unchanged - 47 fixed = 326 total (was 344) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 14s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
1s{color} | {color:green} hadoop-yarn-registry in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
50s{color} | {color:green} hadoop-yarn-services in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
51s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} hadoop-yarn-services-api in the patch passed.
{color} |
| {color:green}+1{color} |
[jira] [Comment Edited] (YARN-7229) Add a metric for the size of event queue in AsyncDispatcher
[ https://issues.apache.org/jira/browse/YARN-7229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267978#comment-16267978 ] sandflee edited comment on YARN-7229 at 11/28/17 2:30 AM: -- yes, planed to add this to our cluster, assign to myself was (Author: sandflee): yes, planed to add this to our cluster, assign this to myself > Add a metric for the size of event queue in AsyncDispatcher > --- > > Key: YARN-7229 > URL: https://issues.apache.org/jira/browse/YARN-7229 > Project: Hadoop YARN > Issue Type: Improvement > Components: metrics, nodemanager, resourcemanager >Affects Versions: 3.1.0 >Reporter: Yufei Gu >Assignee: sandflee > > The size of event queue in AsyncDispatcher is a good point to monitor daemon > performance. Let's make it a RM metric. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7229) Add a metric for the size of event queue in AsyncDispatcher
[ https://issues.apache.org/jira/browse/YARN-7229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267978#comment-16267978 ] sandflee commented on YARN-7229: yes, planed to add this to our cluster, assign this to myself > Add a metric for the size of event queue in AsyncDispatcher > --- > > Key: YARN-7229 > URL: https://issues.apache.org/jira/browse/YARN-7229 > Project: Hadoop YARN > Issue Type: Improvement > Components: metrics, nodemanager, resourcemanager >Affects Versions: 3.1.0 >Reporter: Yufei Gu >Assignee: sandflee > > The size of event queue in AsyncDispatcher is a good point to monitor daemon > performance. Let's make it a RM metric. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-7535) We should display origin value of demand in fair scheduler page
[
https://issues.apache.org/jira/browse/YARN-7535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xuekang updated YARN-7535:
--
Comment: was deleted
(was: I understand that a queue cannot use more resource than its max
resources, but demand is not use.
The users should know how many resources their apps are requesting, and the
YARN admin should know how many extra resources should be added to the queue,
if the demand is always larger than the max resources. Now, when the demand is
larger than the max value, the demand value in the RM web UI is just a limit
instead of a real demand, which is unreasonable.
)
> We should display origin value of demand in fair scheduler page
> ---
>
> Key: YARN-7535
> URL: https://issues.apache.org/jira/browse/YARN-7535
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Reporter: YunFan Zhou
>Assignee: YunFan Zhou
>
> The value of *demand* of leaf queue that we now view on the fair scheduler
> page shows only the value of *maxResources* when the demand value is greater
> than *maxResources*. It doesn't reflect the real situation. Most of the time,
> when we expand the queue, we often rely on seeing the current demand real
> value.
> {code:java}
> private void updateDemandForApp(FSAppAttempt sched, Resource maxRes) {
> sched.updateDemand();
> Resource toAdd = sched.getDemand();
> if (LOG.isDebugEnabled()) {
> LOG.debug("Counting resource from " + sched.getName() + " " + toAdd
> + "; Total resource consumption for " + getName() + " now "
> + demand);
> }
> demand = Resources.add(demand, toAdd);
> demand = Resources.componentwiseMin(demand, maxRes);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6669) Support security for YARN service framework
[
https://issues.apache.org/jira/browse/YARN-6669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267972#comment-16267972
]
genericqa commented on YARN-6669:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 33s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
7s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 3s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 28 new + 309 unchanged - 47 fixed = 337 total (was 356) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 7s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
58s{color} | {color:green} hadoop-yarn-registry in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
42s{color} | {color:green} hadoop-yarn-services in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
48s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
26s{color} | {color:green} hadoop-yarn-services-api in the patch passed.
{color} |
| {color:green}+1{color} |
[jira] [Commented] (YARN-6124) Make SchedulingEditPolicy can be enabled / disabled / updated with RMAdmin -refreshQueues
[ https://issues.apache.org/jira/browse/YARN-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267950#comment-16267950 ] Zian Chen commented on YARN-6124: - Hi [~leftnoteasy] , thanks for the comments, I fixed the failed test cases and updated the patch. Let's see if we get any other failed cases. > Make SchedulingEditPolicy can be enabled / disabled / updated with RMAdmin > -refreshQueues > - > > Key: YARN-6124 > URL: https://issues.apache.org/jira/browse/YARN-6124 > Project: Hadoop YARN > Issue Type: Task >Reporter: Wangda Tan >Assignee: Zian Chen > Attachments: YARN-6124.4.patch, YARN-6124.wip.1.patch, > YARN-6124.wip.2.patch, YARN-6124.wip.3.patch > > > Now enabled / disable / update SchedulingEditPolicy config requires restart > RM. This is inconvenient when admin wants to make changes to > SchedulingEditPolicies. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7558) "yarn logs" command fails to get logs for running containers if UI authentication is enabled.
[
https://issues.apache.org/jira/browse/YARN-7558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267945#comment-16267945
]
genericqa commented on YARN-7558:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 34s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 51s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 21m 27s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.TestApplicationClientProtocolOnHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7558 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12899512/YARN-7558.2.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 7a9c5562c40b 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d8923cd |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/18677/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/18677/testReport/ |
| Max. process+thread count | 627 (vs. ulimit of 5000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
[jira] [Updated] (YARN-6124) Make SchedulingEditPolicy can be enabled / disabled / updated with RMAdmin -refreshQueues
[ https://issues.apache.org/jira/browse/YARN-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zian Chen updated YARN-6124: Attachment: YARN-6124.4.patch > Make SchedulingEditPolicy can be enabled / disabled / updated with RMAdmin > -refreshQueues > - > > Key: YARN-6124 > URL: https://issues.apache.org/jira/browse/YARN-6124 > Project: Hadoop YARN > Issue Type: Task >Reporter: Wangda Tan >Assignee: Zian Chen > Attachments: YARN-6124.4.patch, YARN-6124.wip.1.patch, > YARN-6124.wip.2.patch, YARN-6124.wip.3.patch > > > Now enabled / disable / update SchedulingEditPolicy config requires restart > RM. This is inconvenient when admin wants to make changes to > SchedulingEditPolicies. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6669) Support security for YARN service framework
[ https://issues.apache.org/jira/browse/YARN-6669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-6669: -- Attachment: YARN-6669.07.patch fixed some check style issues, and made client to throw explicit exception if it cannot talks to a AM that doesn't do kerberos login. > Support security for YARN service framework > --- > > Key: YARN-6669 > URL: https://issues.apache.org/jira/browse/YARN-6669 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-6669.01.patch, YARN-6669.02.patch, > YARN-6669.03.patch, YARN-6669.04.patch, YARN-6669.05.patch, > YARN-6669.06.patch, YARN-6669.07.patch, > YARN-6669.yarn-native-services.01.patch, > YARN-6669.yarn-native-services.03.patch, > YARN-6669.yarn-native-services.04.patch, > YARN-6669.yarn-native-services.05.patch > > > Changes include: > - Make registry client to programmatically generate the jaas conf for secure > access ZK quorum > - Create a KerberosPrincipal resource object in REST API for user to supply > keberos keytab and principal > - User has two ways to configure: > -- If keytab starts with "hdfs://", the keytab will be localized by YARN > -- If keytab starts with "file://", it is assumed that the keytab are > available on the localhost. > - AM will use the keytab to log in > - ServiceClient is changed to ask hdfs delegation token when submitting the > service > - AM code will use the tokens when launching containers > - Support kerberized communication between client and AM -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7541) Node updates don't update the maximum cluster capability for resources other than CPU and memory
[
https://issues.apache.org/jira/browse/YARN-7541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267925#comment-16267925
]
Yufei Gu commented on YARN-7541:
+1 for patch v5. Please fix the style issue before commit.
> Node updates don't update the maximum cluster capability for resources other
> than CPU and memory
>
>
> Key: YARN-7541
> URL: https://issues.apache.org/jira/browse/YARN-7541
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 3.0.0-beta1, 3.1.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: YARN-7541.001.patch, YARN-7541.002.patch,
> YARN-7541.003.patch, YARN-7541.004.patch, YARN-7541.005.patch
>
>
> When I submit an MR job that asks for too much memory or CPU for the map or
> reduce, the AM will fail because it recognizes that the request is too large.
> With any other resources, however, the resource requests will instead be
> made and remain pending forever. Looks like we forgot to update the code
> that tracks the maximum container allocation in {{ClusterNodeTracker}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7491) Make sure AM is not scheduled on an opportunistic container
[
https://issues.apache.org/jira/browse/YARN-7491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267920#comment-16267920
]
Miklos Szegedi commented on YARN-7491:
--
I reverted the AM request line in the patch to the following:
{code}
amReqs = Collections.singletonList(BuilderUtils
.newResourceRequest(RMAppAttemptImpl.AM_CONTAINER_PRIORITY,
ResourceRequest.ANY, submissionContext.getResource(), 1));
{code}
TestAppManager still succeeded. That means that the change is not covered by
the unit tests to me.
> Make sure AM is not scheduled on an opportunistic container
> ---
>
> Key: YARN-7491
> URL: https://issues.apache.org/jira/browse/YARN-7491
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: YARN-7491-YARN-1011.00.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7571) YARN has multiple Queue Configuration Providers
Xuan Gong created YARN-7571: --- Summary: YARN has multiple Queue Configuration Providers Key: YARN-7571 URL: https://issues.apache.org/jira/browse/YARN-7571 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong After YARN-5734, we have multiple Queue Configuration Providers: * queue configuration XML file from local conf directory * OrgQueue: queue configuration from zk/leveldb backing store. * Remote Configuration Provider(YARN-1611) We should be careful if all of them are enabled by accident -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6669) Support security for YARN service framework
[ https://issues.apache.org/jira/browse/YARN-6669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-6669: -- Attachment: (was: YARN-6669.07.patch) > Support security for YARN service framework > --- > > Key: YARN-6669 > URL: https://issues.apache.org/jira/browse/YARN-6669 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-6669.01.patch, YARN-6669.02.patch, > YARN-6669.03.patch, YARN-6669.04.patch, YARN-6669.05.patch, > YARN-6669.06.patch, YARN-6669.yarn-native-services.01.patch, > YARN-6669.yarn-native-services.03.patch, > YARN-6669.yarn-native-services.04.patch, > YARN-6669.yarn-native-services.05.patch > > > Changes include: > - Make registry client to programmatically generate the jaas conf for secure > access ZK quorum > - Create a KerberosPrincipal resource object in REST API for user to supply > keberos keytab and principal > - User has two ways to configure: > -- If keytab starts with "hdfs://", the keytab will be localized by YARN > -- If keytab starts with "file://", it is assumed that the keytab are > available on the localhost. > - AM will use the keytab to log in > - ServiceClient is changed to ask hdfs delegation token when submitting the > service > - AM code will use the tokens when launching containers > - Support kerberized communication between client and AM -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6669) Support security for YARN service framework
[ https://issues.apache.org/jira/browse/YARN-6669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-6669: -- Attachment: YARN-6669.07.patch > Support security for YARN service framework > --- > > Key: YARN-6669 > URL: https://issues.apache.org/jira/browse/YARN-6669 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-6669.01.patch, YARN-6669.02.patch, > YARN-6669.03.patch, YARN-6669.04.patch, YARN-6669.05.patch, > YARN-6669.06.patch, YARN-6669.07.patch, > YARN-6669.yarn-native-services.01.patch, > YARN-6669.yarn-native-services.03.patch, > YARN-6669.yarn-native-services.04.patch, > YARN-6669.yarn-native-services.05.patch > > > Changes include: > - Make registry client to programmatically generate the jaas conf for secure > access ZK quorum > - Create a KerberosPrincipal resource object in REST API for user to supply > keberos keytab and principal > - User has two ways to configure: > -- If keytab starts with "hdfs://", the keytab will be localized by YARN > -- If keytab starts with "file://", it is assumed that the keytab are > available on the localhost. > - AM will use the keytab to log in > - ServiceClient is changed to ask hdfs delegation token when submitting the > service > - AM code will use the tokens when launching containers > - Support kerberized communication between client and AM -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6731) Add ability to export scheduler configuration XML
[
https://issues.apache.org/jira/browse/YARN-6731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267876#comment-16267876
]
genericqa commented on YARN-6731:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} YARN-6731 does not apply to YARN-5734. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-6731 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12887813/YARN-6731-YARN-5734.004.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/18678/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add ability to export scheduler configuration XML
> -
>
> Key: YARN-6731
> URL: https://issues.apache.org/jira/browse/YARN-6731
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Attachments: YARN-6731-YARN-5734.001.patch,
> YARN-6731-YARN-5734.002.patch, YARN-6731-YARN-5734.003.patch,
> YARN-6731-YARN-5734.004.patch
>
>
> This is useful for debugging/cluster migration/peace of mind.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7570) Merge Scheduler configuration Store and ZK RM State Store
Xuan Gong created YARN-7570: --- Summary: Merge Scheduler configuration Store and ZK RM State Store Key: YARN-7570 URL: https://issues.apache.org/jira/browse/YARN-7570 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong In RMHA, we are using zookeeper based store for scheduler configuration updates which is separately from the ZK RM State Store. Given they have many similarities, it is better to merge the implementations together. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7569) Add ability to dump and upload scheduler configuration XML
Xuan Gong created YARN-7569: --- Summary: Add ability to dump and upload scheduler configuration XML Key: YARN-7569 URL: https://issues.apache.org/jira/browse/YARN-7569 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7568) Add audit log for add/remove queues
Xuan Gong created YARN-7568: --- Summary: Add audit log for add/remove queues Key: YARN-7568 URL: https://issues.apache.org/jira/browse/YARN-7568 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7567) Make sure that OrgQueue works in secure ZK environment
[ https://issues.apache.org/jira/browse/YARN-7567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-7567: Description: In YARN-5734, we introduced OrgQueue for easy CapacityScheduler queue management. And we are using zookeeper based store for scheduler configuration updates. We need to make sure that it can work in a secure zk environment (was: In YARN-5734, we introduced ) > Make sure that OrgQueue works in secure ZK environment > -- > > Key: YARN-7567 > URL: https://issues.apache.org/jira/browse/YARN-7567 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong > > In YARN-5734, we introduced OrgQueue for easy CapacityScheduler queue > management. And we are using zookeeper based store for scheduler > configuration updates. We need to make sure that it can work in a secure zk > environment -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7567) Make sure that OrgQueue works in secure ZK environment
[ https://issues.apache.org/jira/browse/YARN-7567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-7567: Description: In YARN-5734, we introduced > Make sure that OrgQueue works in secure ZK environment > -- > > Key: YARN-7567 > URL: https://issues.apache.org/jira/browse/YARN-7567 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong > > In YARN-5734, we introduced -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7567) Make sure that OrgQueue works in secure ZK environment
Xuan Gong created YARN-7567: --- Summary: Make sure that OrgQueue works in secure ZK environment Key: YARN-7567 URL: https://issues.apache.org/jira/browse/YARN-7567 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7566) [Umbrella] More improvements on YARN Queue Management
Xuan Gong created YARN-7566: --- Summary: [Umbrella] More improvements on YARN Queue Management Key: YARN-7566 URL: https://issues.apache.org/jira/browse/YARN-7566 Project: Hadoop YARN Issue Type: Task Reporter: Xuan Gong Assignee: Xuan Gong This serves as an umbrella ticket for tasks related to adding more improvements on YARN Queue Management. * YARN-5734: OrgQueue for easy CapacityScheduler queue configuration management * YARN-5724: Configuration-based Queue Management -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7558) "yarn logs" command fails to get logs for running containers if UI authentication is enabled.
[ https://issues.apache.org/jira/browse/YARN-7558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267837#comment-16267837 ] Xuan Gong commented on YARN-7558: - Fix the checkstyle issue. The testcase failure is not related > "yarn logs" command fails to get logs for running containers if UI > authentication is enabled. > - > > Key: YARN-7558 > URL: https://issues.apache.org/jira/browse/YARN-7558 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Namit Maheshwari >Assignee: Xuan Gong >Priority: Critical > Attachments: YARN-7558.1.patch, YARN-7558.2.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7558) "yarn logs" command fails to get logs for running containers if UI authentication is enabled.
[ https://issues.apache.org/jira/browse/YARN-7558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-7558: Attachment: YARN-7558.2.patch > "yarn logs" command fails to get logs for running containers if UI > authentication is enabled. > - > > Key: YARN-7558 > URL: https://issues.apache.org/jira/browse/YARN-7558 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Namit Maheshwari >Assignee: Xuan Gong >Priority: Critical > Attachments: YARN-7558.1.patch, YARN-7558.2.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7491) Make sure AM is not scheduled on an opportunistic container
[ https://issues.apache.org/jira/browse/YARN-7491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267819#comment-16267819 ] Haibo Chen commented on YARN-7491: -- Thanks for the review [~miklos.szeg...@cloudera.com]! I think it is already covered in the unit tests that are changed in the patch. Specifically, we check in the existing unit tests that the ResourceRequests generated for AM containers are always guaranteed with enforceExecutionType set to true. This is also why YARN-7531 is a dependency. > Make sure AM is not scheduled on an opportunistic container > --- > > Key: YARN-7491 > URL: https://issues.apache.org/jira/browse/YARN-7491 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-7491-YARN-1011.00.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7363) ContainerLocalizer doesn't have a valid log4j config when using LinuxContainerExecutor
[ https://issues.apache.org/jira/browse/YARN-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267740#comment-16267740 ] Yufei Gu edited comment on YARN-7363 at 11/27/17 10:56 PM: --- Uploaded the branch-2 version. There is a minor conflict. Committed to branch-2. was (Author: yufeigu): Uploaded the branch-2 version. There is a minor conflict. > ContainerLocalizer doesn't have a valid log4j config when using > LinuxContainerExecutor > -- > > Key: YARN-7363 > URL: https://issues.apache.org/jira/browse/YARN-7363 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.1.0 >Reporter: Yufei Gu >Assignee: Yufei Gu > Fix For: 3.0.0, 3.1.0, 2.10.0 > > Attachments: YARN-7363.001.patch, YARN-7363.002.patch, > YARN-7363.003.patch, YARN-7363.004.patch, YARN-7363.005.patch, > YARN-7363.branch-2.001.patch > > > In case of Linux container executor, ContainerLocalizer run as a separated > process. It doesn't access a valid log4j.properties when the application user > is not in the "hadoop" group. The log4j.properties of node manager is in its > classpath, but it isn't readable by users not in hadoop group due to the > security concern. In that case, ContainerLocalizer doesn't have a valid log4j > configuration, and normally no log output. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7363) ContainerLocalizer doesn't have a valid log4j config when using LinuxContainerExecutor
[ https://issues.apache.org/jira/browse/YARN-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated YARN-7363: --- Fix Version/s: 2.10.0 > ContainerLocalizer doesn't have a valid log4j config when using > LinuxContainerExecutor > -- > > Key: YARN-7363 > URL: https://issues.apache.org/jira/browse/YARN-7363 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.1.0 >Reporter: Yufei Gu >Assignee: Yufei Gu > Fix For: 3.0.0, 3.1.0, 2.10.0 > > Attachments: YARN-7363.001.patch, YARN-7363.002.patch, > YARN-7363.003.patch, YARN-7363.004.patch, YARN-7363.005.patch, > YARN-7363.branch-2.001.patch > > > In case of Linux container executor, ContainerLocalizer run as a separated > process. It doesn't access a valid log4j.properties when the application user > is not in the "hadoop" group. The log4j.properties of node manager is in its > classpath, but it isn't readable by users not in hadoop group due to the > security concern. In that case, ContainerLocalizer doesn't have a valid log4j > configuration, and normally no log output. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7363) ContainerLocalizer doesn't have a valid log4j config when using LinuxContainerExecutor
[ https://issues.apache.org/jira/browse/YARN-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated YARN-7363: --- Attachment: YARN-7363.branch-2.001.patch Uploaded the branch-2 version. There is a minor conflict. > ContainerLocalizer doesn't have a valid log4j config when using > LinuxContainerExecutor > -- > > Key: YARN-7363 > URL: https://issues.apache.org/jira/browse/YARN-7363 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.1.0 >Reporter: Yufei Gu >Assignee: Yufei Gu > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-7363.001.patch, YARN-7363.002.patch, > YARN-7363.003.patch, YARN-7363.004.patch, YARN-7363.005.patch, > YARN-7363.branch-2.001.patch > > > In case of Linux container executor, ContainerLocalizer run as a separated > process. It doesn't access a valid log4j.properties when the application user > is not in the "hadoop" group. The log4j.properties of node manager is in its > classpath, but it isn't readable by users not in hadoop group due to the > security concern. In that case, ContainerLocalizer doesn't have a valid log4j > configuration, and normally no log output. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7363) ContainerLocalizer doesn't have a valid log4j config when using LinuxContainerExecutor
[ https://issues.apache.org/jira/browse/YARN-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267737#comment-16267737 ] Hudson commented on YARN-7363: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13280 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13280/]) YARN-7363. ContainerLocalizer don't have a valid log4j config in case of (yufei: rev d8923cdbf1567aee10a54f144fef734d1465ebed) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/WindowsSecureContainerExecutor.java > ContainerLocalizer doesn't have a valid log4j config when using > LinuxContainerExecutor > -- > > Key: YARN-7363 > URL: https://issues.apache.org/jira/browse/YARN-7363 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.1.0 >Reporter: Yufei Gu >Assignee: Yufei Gu > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-7363.001.patch, YARN-7363.002.patch, > YARN-7363.003.patch, YARN-7363.004.patch, YARN-7363.005.patch > > > In case of Linux container executor, ContainerLocalizer run as a separated > process. It doesn't access a valid log4j.properties when the application user > is not in the "hadoop" group. The log4j.properties of node manager is in its > classpath, but it isn't readable by users not in hadoop group due to the > security concern. In that case, ContainerLocalizer doesn't have a valid log4j > configuration, and normally no log output. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7363) ContainerLocalizer doesn't have a valid log4j config when using LinuxContainerExecutor
[ https://issues.apache.org/jira/browse/YARN-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267717#comment-16267717 ] Yufei Gu commented on YARN-7363: Committed to trunk and branch-3.0. Thanks for the review, [~miklos.szeg...@cloudera.com] and [~rchiang]! > ContainerLocalizer doesn't have a valid log4j config when using > LinuxContainerExecutor > -- > > Key: YARN-7363 > URL: https://issues.apache.org/jira/browse/YARN-7363 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.1.0 >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-7363.001.patch, YARN-7363.002.patch, > YARN-7363.003.patch, YARN-7363.004.patch, YARN-7363.005.patch > > > In case of Linux container executor, ContainerLocalizer run as a separated > process. It doesn't access a valid log4j.properties when the application user > is not in the "hadoop" group. The log4j.properties of node manager is in its > classpath, but it isn't readable by users not in hadoop group due to the > security concern. In that case, ContainerLocalizer doesn't have a valid log4j > configuration, and normally no log output. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7363) ContainerLocalizer doesn't have a valid log4j config when using LinuxContainerExecutor
[ https://issues.apache.org/jira/browse/YARN-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated YARN-7363: - Summary: ContainerLocalizer doesn't have a valid log4j config when using LinuxContainerExecutor (was: ContainerLocalizer don't have a valid log4j config in case of Linux container executor) > ContainerLocalizer doesn't have a valid log4j config when using > LinuxContainerExecutor > -- > > Key: YARN-7363 > URL: https://issues.apache.org/jira/browse/YARN-7363 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.1.0 >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-7363.001.patch, YARN-7363.002.patch, > YARN-7363.003.patch, YARN-7363.004.patch, YARN-7363.005.patch > > > In case of Linux container executor, ContainerLocalizer run as a separated > process. It doesn't access a valid log4j.properties when the application user > is not in the "hadoop" group. The log4j.properties of node manager is in its > classpath, but it isn't readable by users not in hadoop group due to the > security concern. In that case, ContainerLocalizer doesn't have a valid log4j > configuration, and normally no log output. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7561) Why hasContainerForNode() return false directly when there is no request of ANY locality without considering NODE_LOCAL and RACK_LOCAL?
[
https://issues.apache.org/jira/browse/YARN-7561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266564#comment-16266564
]
Yufei Gu edited comment on YARN-7561 at 11/27/17 10:30 PM:
---
You might need to look at YARN-392. The logic doesn't change since then. Also
[~rkanter] has more context since he did YARN-6050.
was (Author: yufeigu):
You might need to look at YARN-392. The logic doesn't change since then.
> Why hasContainerForNode() return false directly when there is no request of
> ANY locality without considering NODE_LOCAL and RACK_LOCAL?
> ---
>
> Key: YARN-7561
> URL: https://issues.apache.org/jira/browse/YARN-7561
> Project: Hadoop YARN
> Issue Type: Task
> Components: fairscheduler
>Affects Versions: 2.7.3
>Reporter: wuchang
>
> I am studying the FairScheduler source cod of yarn 2.7.3.
> By the code of class FSAppAttempt:
> {code}
> public boolean hasContainerForNode(Priority prio, FSSchedulerNode node) {
> ResourceRequest anyRequest = getResourceRequest(prio,
> ResourceRequest.ANY);
> ResourceRequest rackRequest = getResourceRequest(prio,
> node.getRackName());
> ResourceRequest nodeRequest = getResourceRequest(prio,
> node.getNodeName());
>
> return
> // There must be outstanding requests at the given priority:
> anyRequest != null && anyRequest.getNumContainers() > 0 &&
> // If locality relaxation is turned off at *-level, there must be
> a
> // non-zero request for the node's rack:
> (anyRequest.getRelaxLocality() ||
> (rackRequest != null && rackRequest.getNumContainers() > 0))
> &&
> // If locality relaxation is turned off at rack-level, there must
> be a
> // non-zero request at the node:
> (rackRequest == null || rackRequest.getRelaxLocality() ||
> (nodeRequest != null && nodeRequest.getNumContainers() > 0))
> &&
> // The requested container must be able to fit on the node:
> Resources.lessThanOrEqual(RESOURCE_CALCULATOR, null,
> anyRequest.getCapability(),
> node.getRMNode().getTotalCapability());
> }
> {code}
> I really cannot understand why when there is no anyRequest ,
> *hasContainerForNode()* return false directly without considering whether
> there is NODE_LOCAL or RACK_LOCAL requests.
> And , *AppSchedulingInfo.allocateNodeLocal()* and
> *AppSchedulingInfo.allocateRackLocal()* will also decrease the number of
> containers for *ResourceRequest.ANY*, this is another place where I feel
> confused.
> Really thanks for some prompt.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7491) Make sure AM is not scheduled on an opportunistic container
[ https://issues.apache.org/jira/browse/YARN-7491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267702#comment-16267702 ] Miklos Szegedi commented on YARN-7491: -- Thank you for the patch [~haibochen]. Even though there are changes in unit tests, so Jenkins does not complain, I think we should have a unit test for the change. > Make sure AM is not scheduled on an opportunistic container > --- > > Key: YARN-7491 > URL: https://issues.apache.org/jira/browse/YARN-7491 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-7491-YARN-1011.00.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7363) ContainerLocalizer don't have a valid log4j config in case of Linux container executor
[ https://issues.apache.org/jira/browse/YARN-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267618#comment-16267618 ] Ray Chiang commented on YARN-7363: -- Looks good to me [~yufeigu]. +1 (binding) pending Jenkins. I'll commit this soon. > ContainerLocalizer don't have a valid log4j config in case of Linux container > executor > -- > > Key: YARN-7363 > URL: https://issues.apache.org/jira/browse/YARN-7363 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.1.0 >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-7363.001.patch, YARN-7363.002.patch, > YARN-7363.003.patch, YARN-7363.004.patch, YARN-7363.005.patch > > > In case of Linux container executor, ContainerLocalizer run as a separated > process. It doesn't access a valid log4j.properties when the application user > is not in the "hadoop" group. The log4j.properties of node manager is in its > classpath, but it isn't readable by users not in hadoop group due to the > security concern. In that case, ContainerLocalizer doesn't have a valid log4j > configuration, and normally no log output. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7363) ContainerLocalizer don't have a valid log4j config in case of Linux container executor
[
https://issues.apache.org/jira/browse/YARN-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267597#comment-16267597
]
genericqa commented on YARN-7363:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
16s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
52s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 0s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 14 new + 281 unchanged - 10 fixed = 295 total (was 291) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 43s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
39s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
2s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 17m
19s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 99m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7363 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12899479/YARN-7363.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux ef510d998436 3.13.0-135-generic
[jira] [Commented] (YARN-7558) "yarn logs" command fails to get logs for running containers if UI authentication is enabled.
[
https://issues.apache.org/jira/browse/YARN-7558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267518#comment-16267518
]
genericqa commented on YARN-7558:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 32s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 13s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: The patch generated 11 new
+ 5 unchanged - 0 fixed = 16 total (was 5) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 5s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 22m 3s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.api.impl.TestAMRMClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7558 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12899480/YARN-7558.1.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux ffc4aa9c29b4 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / fedabca |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/18675/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| unit |
[jira] [Commented] (YARN-7564) Fix checkstyle issues of YARN-5881 branch
[
https://issues.apache.org/jira/browse/YARN-7564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267440#comment-16267440
]
genericqa commented on YARN-7564:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} YARN-5881 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 5m
55s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
33s{color} | {color:green} YARN-5881 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
11s{color} | {color:green} YARN-5881 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
12s{color} | {color:green} YARN-5881 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
43s{color} | {color:green} YARN-5881 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 17s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
4s{color} | {color:green} YARN-5881 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
13s{color} | {color:green} YARN-5881 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
12s{color} | {color:green} root: The patch generated 0 new + 353 unchanged - 34
fixed = 353 total (was 387) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 1s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
7s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
4s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 55s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}169m 46s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7564 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12899457/YARN-7564-YARN-5881.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux b88e9b8dbdd4 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
|
[jira] [Commented] (YARN-7509) AsyncScheduleThread and ResourceCommitterService are still running after RM is transitioned to standby
[ https://issues.apache.org/jira/browse/YARN-7509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267425#comment-16267425 ] Subru Krishnan commented on YARN-7509: -- [~leftnoteasy], the fix version says 2.9.1 but it has not been cherry-picked to branch-2.9. Can you go ahead and do that? Thanks. > AsyncScheduleThread and ResourceCommitterService are still running after RM > is transitioned to standby > -- > > Key: YARN-7509 > URL: https://issues.apache.org/jira/browse/YARN-7509 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0-alpha4, 2.9.1 >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Critical > Fix For: 3.1.0, 2.9.1, 3.0.1 > > Attachments: YARN-7509.001.patch > > > After RM is transitioned to standby, AsyncScheduleThread and > ResourceCommitterService will receive interrupt signal. When thread is > sleeping, it will ignore the interrupt signal since InterruptedException is > catched inside and the interrupt signal is cleared. > For AsyncScheduleThread, InterruptedException was catched and ignored in > CapacityScheduler#schedule. > For ResourceCommitterService, InterruptedException was catched inside and > ignored in ResourceCommitterService#run. > We should let the interrupt signal out and make these threads exit. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7541) Node updates don't update the maximum cluster capability for resources other than CPU and memory
[
https://issues.apache.org/jira/browse/YARN-7541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267397#comment-16267397
]
genericqa commented on YARN-7541:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 22s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
10s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
17s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 54s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 1 new + 23 unchanged - 3 fixed = 24 total (was 26) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 8s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
38s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 29s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}128m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7541 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12899193/YARN-7541.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 063d545013af 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31
19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool |
[jira] [Updated] (YARN-7558) "yarn logs" command fails to get logs for running containers if UI authentication is enabled.
[ https://issues.apache.org/jira/browse/YARN-7558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-7558: Reporter: Namit Maheshwari (was: Xuan Gong) > "yarn logs" command fails to get logs for running containers if UI > authentication is enabled. > - > > Key: YARN-7558 > URL: https://issues.apache.org/jira/browse/YARN-7558 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Namit Maheshwari >Assignee: Xuan Gong >Priority: Critical > Attachments: YARN-7558.1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7558) "yarn logs" command fails to get logs for running containers if UI authentication is enabled.
[ https://issues.apache.org/jira/browse/YARN-7558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267364#comment-16267364 ] Xuan Gong commented on YARN-7558: - Test the patch in secure/un-secure environment, and it fixes the issue > "yarn logs" command fails to get logs for running containers if UI > authentication is enabled. > - > > Key: YARN-7558 > URL: https://issues.apache.org/jira/browse/YARN-7558 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Critical > Attachments: YARN-7558.1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7558) "yarn logs" command fails to get logs for running containers if UI authentication is enabled.
[ https://issues.apache.org/jira/browse/YARN-7558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-7558: Attachment: YARN-7558.1.patch > "yarn logs" command fails to get logs for running containers if UI > authentication is enabled. > - > > Key: YARN-7558 > URL: https://issues.apache.org/jira/browse/YARN-7558 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Critical > Attachments: YARN-7558.1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7363) ContainerLocalizer don't have a valid log4j config in case of Linux container executor
[ https://issues.apache.org/jira/browse/YARN-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated YARN-7363: --- Attachment: YARN-7363.005.patch Thanks for the review, [~rchiang]. Uploaded patch v5 for your comments. > ContainerLocalizer don't have a valid log4j config in case of Linux container > executor > -- > > Key: YARN-7363 > URL: https://issues.apache.org/jira/browse/YARN-7363 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.1.0 >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-7363.001.patch, YARN-7363.002.patch, > YARN-7363.003.patch, YARN-7363.004.patch, YARN-7363.005.patch > > > In case of Linux container executor, ContainerLocalizer run as a separated > process. It doesn't access a valid log4j.properties when the application user > is not in the "hadoop" group. The log4j.properties of node manager is in its > classpath, but it isn't readable by users not in hadoop group due to the > security concern. In that case, ContainerLocalizer doesn't have a valid log4j > configuration, and normally no log output. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7473) Implement Framework and policy for capacity management of auto created queues
[
https://issues.apache.org/jira/browse/YARN-7473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267317#comment-16267317
]
Wangda Tan commented on YARN-7473:
--
Thanks [~suma.shivaprasad], majorly reviewed configs and GuaranteedOrZeroPolicy
implementation.
1) observeObly can be removed from
QueueManagementDynamicEditPolicy/CapacitySchedulerConfiguration.
2) GuaranteedOrZeroCapacityOverTimePolicy
- Javadocs of the class is not accurate, please add more details to the
Javadocs.
- In Init, It's better not to cache {{parentQueueCapacities}}, instead should
always fetch from ParentQueue.getQueueCapacities. Because we don't have
guarantee that queue.queueCapacities ref won't be updated.
- ZERO_CAPACITY_ENTITLEMENT, should set capacity = 0, and maximum-capcaity
equals to leaf queue template's maximum-capacity (for all labels).
- deactivateLeafQueuesIfInActive: it's better to add a LOG.debug when
deactivate queues.
- It's better to calculate deactivatedCapacity from actual deactivated queue's
capacity for better maintenanbility. I understand this logic is correct now,
but we could have: a. queue with different capacity. b. some other actions
added before the deactive actions.
- {{isDeactivatedQueue}} is too simple to be a method.
- {{//acquires read lock on leaf queue}}, is not accurate, it just acquires
readlock of the policy itself.
- There're many variables/methods named {{*entitlement*}} inside the policy,
some of them should be renamed.
- Does the {{reinitialize}} will be used after YARN-6142? If yes, could you add
a TODO comment? And the signature should be same as init:
(CapacitySchedulerContext, ParentQueue)
- Add a TODO comment saying node label will be supported in a separate patch.
3) AutoCreatedLeafQueueConfig doesn't include maximum-am-percent, should we
include it?
> Implement Framework and policy for capacity management of auto created queues
> --
>
> Key: YARN-7473
> URL: https://issues.apache.org/jira/browse/YARN-7473
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Attachments: YARN-7473.1.patch, YARN-7473.2.patch, YARN-7473.3.patch,
> YARN-7473.4.patch, YARN-7473.5.patch, YARN-7473.6.patch, YARN-7473.7.patch,
> YARN-7473.8.patch, YARN-7473.9.patch
>
>
> This jira mainly addresses the following
>
> 1.Support adding pluggable policies on parent queue for dynamically managing
> capacity/state for leaf queues.
> 2. Implement a default policy that manages capacity based on pending
> applications and either grants guaranteed or zero capacity to queues based on
> parent's available guaranteed capacity.
> 3. Integrate with SchedulingEditPolicy framework to trigger this periodically
> and signal scheduler to take necessary actions for capacity/queue management.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6483) Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes returned to the AM
[ https://issues.apache.org/jira/browse/YARN-6483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267314#comment-16267314 ] ASF GitHub Bot commented on YARN-6483: -- Github user juanrh commented on the issue: https://github.com/apache/hadoop/pull/289 Pushed in https://github.com/apache/hadoop/commit/b46ca7e73b8bac3fdbff0b13afe009308078acf2 > Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes > returned to the AM > > > Key: YARN-6483 > URL: https://issues.apache.org/jira/browse/YARN-6483 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Juan RodrÃguez Hortalá >Assignee: Juan RodrÃguez Hortalá > Fix For: 3.1.0 > > Attachments: YARN-6483-v1.patch, YARN-6483.002.patch, > YARN-6483.003.patch > > > The DECOMMISSIONING node state is currently used as part of the graceful > decommissioning mechanism to give time for tasks to complete in a node that > is scheduled for decommission, and for reducer tasks to read the shuffle > blocks in that node. Also, YARN effectively blacklists nodes in > DECOMMISSIONING state by assigning them a capacity of 0, to prevent > additional containers to be launched in those nodes, so no more shuffle > blocks are written to the node. This blacklisting is not effective for > applications like Spark, because a Spark executor running in a YARN container > will keep receiving more tasks after the corresponding node has been > blacklisted at the YARN level. We would like to propose a modification of the > YARN heartbeat mechanism so nodes transitioning to DECOMMISSIONING are added > to the list of updated nodes returned by the Resource Manager as a response > to the Application Master heartbeat. This way a Spark application master > would be able to blacklist a DECOMMISSIONING at the Spark level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6483) Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes returned to the AM
[ https://issues.apache.org/jira/browse/YARN-6483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267316#comment-16267316 ] ASF GitHub Bot commented on YARN-6483: -- Github user juanrh closed the pull request at: https://github.com/apache/hadoop/pull/289 > Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes > returned to the AM > > > Key: YARN-6483 > URL: https://issues.apache.org/jira/browse/YARN-6483 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Juan RodrÃguez Hortalá >Assignee: Juan RodrÃguez Hortalá > Fix For: 3.1.0 > > Attachments: YARN-6483-v1.patch, YARN-6483.002.patch, > YARN-6483.003.patch > > > The DECOMMISSIONING node state is currently used as part of the graceful > decommissioning mechanism to give time for tasks to complete in a node that > is scheduled for decommission, and for reducer tasks to read the shuffle > blocks in that node. Also, YARN effectively blacklists nodes in > DECOMMISSIONING state by assigning them a capacity of 0, to prevent > additional containers to be launched in those nodes, so no more shuffle > blocks are written to the node. This blacklisting is not effective for > applications like Spark, because a Spark executor running in a YARN container > will keep receiving more tasks after the corresponding node has been > blacklisted at the YARN level. We would like to propose a modification of the > YARN heartbeat mechanism so nodes transitioning to DECOMMISSIONING are added > to the list of updated nodes returned by the Resource Manager as a response > to the Application Master heartbeat. This way a Spark application master > would be able to blacklist a DECOMMISSIONING at the Spark level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7363) ContainerLocalizer don't have a valid log4j config in case of Linux container executor
[ https://issues.apache.org/jira/browse/YARN-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267274#comment-16267274 ] Ray Chiang edited comment on YARN-7363 at 11/27/17 7:08 PM: Minor nit: * The yarn.nodemanager.container-localizer.log.level property is missing a description. * The method getContaierLogDir misspells the word "Container" was (Author: rchiang): Minor nit: The yarn.nodemanager.container-localizer.log.level property is missing a description. > ContainerLocalizer don't have a valid log4j config in case of Linux container > executor > -- > > Key: YARN-7363 > URL: https://issues.apache.org/jira/browse/YARN-7363 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.1.0 >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-7363.001.patch, YARN-7363.002.patch, > YARN-7363.003.patch, YARN-7363.004.patch > > > In case of Linux container executor, ContainerLocalizer run as a separated > process. It doesn't access a valid log4j.properties when the application user > is not in the "hadoop" group. The log4j.properties of node manager is in its > classpath, but it isn't readable by users not in hadoop group due to the > security concern. In that case, ContainerLocalizer doesn't have a valid log4j > configuration, and normally no log output. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7473) Implement Framework and policy for capacity management of auto created queues
[
https://issues.apache.org/jira/browse/YARN-7473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267277#comment-16267277
]
genericqa commented on YARN-7473:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 2s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 94 new + 208 unchanged - 8 fixed = 302 total (was 216)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 9s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
26s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 7s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}107m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| |
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.queuemanagement.GuaranteedOrZeroCapacityOverTimePolicy$PendingApplicationComparator
is serializable but also an inner class of a non-serializable class At
GuaranteedOrZeroCapacityOverTimePolicy.java:an inner class of a
non-serializable class At GuaranteedOrZeroCapacityOverTimePolicy.java:[lines
223-239] |
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMAdminService |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7473 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12899454/YARN-7473.9.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle
[jira] [Commented] (YARN-7363) ContainerLocalizer don't have a valid log4j config in case of Linux container executor
[ https://issues.apache.org/jira/browse/YARN-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267274#comment-16267274 ] Ray Chiang commented on YARN-7363: -- Minor nit: The yarn.nodemanager.container-localizer.log.level property is missing a description. > ContainerLocalizer don't have a valid log4j config in case of Linux container > executor > -- > > Key: YARN-7363 > URL: https://issues.apache.org/jira/browse/YARN-7363 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.1.0 >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-7363.001.patch, YARN-7363.002.patch, > YARN-7363.003.patch, YARN-7363.004.patch > > > In case of Linux container executor, ContainerLocalizer run as a separated > process. It doesn't access a valid log4j.properties when the application user > is not in the "hadoop" group. The log4j.properties of node manager is in its > classpath, but it isn't readable by users not in hadoop group due to the > security concern. In that case, ContainerLocalizer doesn't have a valid log4j > configuration, and normally no log output. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7495) Improve robustness of the AggregatedLogDeletionService
[ https://issues.apache.org/jira/browse/YARN-7495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267235#comment-16267235 ] Jason Lowe commented on YARN-7495: -- Thanks for updating the patch! It would be good to address the checkstyle issues wrt. line length. The RETENTION_SECS one might be fixed by marking the variable final. > Improve robustness of the AggregatedLogDeletionService > -- > > Key: YARN-7495 > URL: https://issues.apache.org/jira/browse/YARN-7495 > Project: Hadoop YARN > Issue Type: Improvement > Components: log-aggregation >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: YARN-7495.001.patch, YARN-7495.002.patch > > > The deletion tasks are scheduled with a TimerTask whose scheduler is a Timer > scheduleAtFixedRate. If an exception occurs in the log deletion task, the > Timer scheduler interprets this as a task cancelation and stops scheduling > future deletion tasks. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6168) Restarted RM may not inform AM about all existing containers
[ https://issues.apache.org/jira/browse/YARN-6168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267224#comment-16267224 ] Hudson commented on YARN-6168: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13279 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13279/]) YARN-6168. Restarted RM may not inform AM about all existing containers. (jianhe: rev fedabcad42067ac7dd24de40fab6be2d3485a540) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/Allocation.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateResponsePBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateResponse.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/DefaultAMSProcessor.java > Restarted RM may not inform AM about all existing containers > > > Key: YARN-6168 > URL: https://issues.apache.org/jira/browse/YARN-6168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Chandni Singh > Fix For: 3.1.0 > > Attachments: YARN-6168.001.patch, YARN-6168.002.patch, > YARN-6168.003.patch, YARN-6168.004.patch > > > There appears to be a race condition when an RM is restarted. I had a > situation where the RMs and AM were down, but NMs and app containers were > still running. When I restarted the RM, the AM restarted, registered with the > RM, and received its list of existing containers before the NMs had reported > all of their containers to the RM. The AM was only told about some of the > app's existing containers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6168) Restarted RM may not inform AM about all existing containers
[ https://issues.apache.org/jira/browse/YARN-6168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267191#comment-16267191 ] Chandni Singh commented on YARN-6168: - Thanks [~jianhe] > Restarted RM may not inform AM about all existing containers > > > Key: YARN-6168 > URL: https://issues.apache.org/jira/browse/YARN-6168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Chandni Singh > Fix For: 3.1.0 > > Attachments: YARN-6168.001.patch, YARN-6168.002.patch, > YARN-6168.003.patch, YARN-6168.004.patch > > > There appears to be a race condition when an RM is restarted. I had a > situation where the RMs and AM were down, but NMs and app containers were > still running. When I restarted the RM, the AM restarted, registered with the > RM, and received its list of existing containers before the NMs had reported > all of their containers to the RM. The AM was only told about some of the > app's existing containers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7565) Yarn service pre-maturely releases the container after AM restart
Chandni Singh created YARN-7565: --- Summary: Yarn service pre-maturely releases the container after AM restart Key: YARN-7565 URL: https://issues.apache.org/jira/browse/YARN-7565 Project: Hadoop YARN Issue Type: Sub-task Reporter: Chandni Singh Assignee: Chandni Singh With YARN-6168, recovered containers can be reported to AM in response to the AM heartbeat. Currently, the Service Master will release the containers, that are not reported in the AM registration response, immediately. Instead, the master can wait for a configured amount of time for the containers to be recovered by RM. These containers are sent to AM in the heartbeat response. Once a container is not reported in the configured interval, it can be released by the master. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6168) Restarted RM may not inform AM about all existing containers
[ https://issues.apache.org/jira/browse/YARN-6168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267181#comment-16267181 ] Jian He commented on YARN-6168: --- I committed this into trunk. Thanks [~csingh] ! > Restarted RM may not inform AM about all existing containers > > > Key: YARN-6168 > URL: https://issues.apache.org/jira/browse/YARN-6168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Chandni Singh > Fix For: 3.1.0 > > Attachments: YARN-6168.001.patch, YARN-6168.002.patch, > YARN-6168.003.patch, YARN-6168.004.patch > > > There appears to be a race condition when an RM is restarted. I had a > situation where the RMs and AM were down, but NMs and app containers were > still running. When I restarted the RM, the AM restarted, registered with the > RM, and received its list of existing containers before the NMs had reported > all of their containers to the RM. The AM was only told about some of the > app's existing containers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6168) Restarted RM may not inform AM about all existing containers
[ https://issues.apache.org/jira/browse/YARN-6168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267173#comment-16267173 ] Chandni Singh commented on YARN-6168: - Test and findbug failure are not related to the change > Restarted RM may not inform AM about all existing containers > > > Key: YARN-6168 > URL: https://issues.apache.org/jira/browse/YARN-6168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Chandni Singh > Attachments: YARN-6168.001.patch, YARN-6168.002.patch, > YARN-6168.003.patch, YARN-6168.004.patch > > > There appears to be a race condition when an RM is restarted. I had a > situation where the RMs and AM were down, but NMs and app containers were > still running. When I restarted the RM, the AM restarted, registered with the > RM, and received its list of existing containers before the NMs had reported > all of their containers to the RM. The AM was only told about some of the > app's existing containers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7558) "yarn logs" command fails to get logs for running containers if UI authentication is enabled.
[ https://issues.apache.org/jira/browse/YARN-7558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-7558: -- Priority: Critical (was: Major) Summary: "yarn logs" command fails to get logs for running containers if UI authentication is enabled. (was: YARN log command fails to get logs for running containers if the url authentication is enabled.) Fixing title. Debugged this with [~xgong] and [~djp]. "yarn logs" never worked with running containers once it started depending on NM REST APIs. The underlying issue is that the logs CLI code failed to do SPNEGO auth to pass along the kerberos credentials to the REST API. > "yarn logs" command fails to get logs for running containers if UI > authentication is enabled. > - > > Key: YARN-7558 > URL: https://issues.apache.org/jira/browse/YARN-7558 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Critical > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7564) Fix checkstyle issues of YARN-5881 branch
[ https://issues.apache.org/jira/browse/YARN-7564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7564: -- Attachment: YARN-7564-YARN-5881.001.patch Attaching patch addressing this issue. cc/[~leftnoteasy] > Fix checkstyle issues of YARN-5881 branch > - > > Key: YARN-7564 > URL: https://issues.apache.org/jira/browse/YARN-7564 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil G >Assignee: Sunil G >Priority: Minor > Attachments: YARN-7564-YARN-5881.001.patch > > > fix jenkins issues. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7564) Fix checkstyle issues of YARN-5881 branch
Sunil G created YARN-7564: - Summary: Fix checkstyle issues of YARN-5881 branch Key: YARN-7564 URL: https://issues.apache.org/jira/browse/YARN-7564 Project: Hadoop YARN Issue Type: Sub-task Reporter: Sunil G Assignee: Sunil G Priority: Minor fix jenkins issues. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7473) Implement Framework and policy for capacity management of auto created queues
[ https://issues.apache.org/jira/browse/YARN-7473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated YARN-7473: --- Attachment: YARN-7473.9.patch > Implement Framework and policy for capacity management of auto created queues > -- > > Key: YARN-7473 > URL: https://issues.apache.org/jira/browse/YARN-7473 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Attachments: YARN-7473.1.patch, YARN-7473.2.patch, YARN-7473.3.patch, > YARN-7473.4.patch, YARN-7473.5.patch, YARN-7473.6.patch, YARN-7473.7.patch, > YARN-7473.8.patch, YARN-7473.9.patch > > > This jira mainly addresses the following > > 1.Support adding pluggable policies on parent queue for dynamically managing > capacity/state for leaf queues. > 2. Implement a default policy that manages capacity based on pending > applications and either grants guaranteed or zero capacity to queues based on > parent's available guaranteed capacity. > 3. Integrate with SchedulingEditPolicy framework to trigger this periodically > and signal scheduler to take necessary actions for capacity/queue management. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7473) Implement Framework and policy for capacity management of auto created queues
[ https://issues.apache.org/jira/browse/YARN-7473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated YARN-7473: --- Attachment: (was: YARN-7394.9.patch) > Implement Framework and policy for capacity management of auto created queues > -- > > Key: YARN-7473 > URL: https://issues.apache.org/jira/browse/YARN-7473 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Attachments: YARN-7473.1.patch, YARN-7473.2.patch, YARN-7473.3.patch, > YARN-7473.4.patch, YARN-7473.5.patch, YARN-7473.6.patch, YARN-7473.7.patch, > YARN-7473.8.patch > > > This jira mainly addresses the following > > 1.Support adding pluggable policies on parent queue for dynamically managing > capacity/state for leaf queues. > 2. Implement a default policy that manages capacity based on pending > applications and either grants guaranteed or zero capacity to queues based on > parent's available guaranteed capacity. > 3. Integrate with SchedulingEditPolicy framework to trigger this periodically > and signal scheduler to take necessary actions for capacity/queue management. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7473) Implement Framework and policy for capacity management of auto created queues
[ https://issues.apache.org/jira/browse/YARN-7473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated YARN-7473: --- Attachment: YARN-7394.9.patch Fixed signature for AutoCreatedQueueManagementPolicy.init/reinitialize. > Implement Framework and policy for capacity management of auto created queues > -- > > Key: YARN-7473 > URL: https://issues.apache.org/jira/browse/YARN-7473 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Attachments: YARN-7394.9.patch, YARN-7473.1.patch, YARN-7473.2.patch, > YARN-7473.3.patch, YARN-7473.4.patch, YARN-7473.5.patch, YARN-7473.6.patch, > YARN-7473.7.patch, YARN-7473.8.patch > > > This jira mainly addresses the following > > 1.Support adding pluggable policies on parent queue for dynamically managing > capacity/state for leaf queues. > 2. Implement a default policy that manages capacity based on pending > applications and either grants guaranteed or zero capacity to queues based on > parent's available guaranteed capacity. > 3. Integrate with SchedulingEditPolicy framework to trigger this periodically > and signal scheduler to take necessary actions for capacity/queue management. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7213) [Umbrella] Test and validate HBase-2.0.x with Atsv2
[ https://issues.apache.org/jira/browse/YARN-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267009#comment-16267009 ] Rohith Sharma K S commented on YARN-7213: - thanks [~haibo.chen] and other HBase folks for detailed investigations! I would suggest following items # Lets create official HBase JIRA saying filters output mismatch from HBaes-1.* to HBase-2.* with your test case demostrated. Since they have already investigated it, for them it is easier to comment on the JIRA and they can close it officially. If JIRA is already exist, that is well and good. # Lets fork this discussion in separate new sub task with your detailed analysis as input. Though modifications are much lesser, it would required detailed review as it going to affect ALL filters. # Use YARN-7346 for fixing compilation issue which is current patch. > [Umbrella] Test and validate HBase-2.0.x with Atsv2 > --- > > Key: YARN-7213 > URL: https://issues.apache.org/jira/browse/YARN-7213 > Project: Hadoop YARN > Issue Type: Task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-7213.prelim.patch, YARN-7213.prelim.patch, > YARN-7213.wip.patch > > > Hbase-2.0.x officially support hadoop-alpha compilations. And also they are > getting ready for Hadoop-beta release so that HBase can release their > versions compatible with Hadoop-beta. So, this JIRA is to keep track of > HBase-2.0 integration issues. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7563) Invalid event: FINISH_APPLICATION at NEW
[
https://issues.apache.org/jira/browse/YARN-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266987#comment-16266987
]
lujie edited comment on YARN-7563 at 11/27/17 4:10 PM:
---
I have find the reason by analysis code and logs
[^YARN-7563.png]
above figure has shown the reason:client submit a application and then send
kill command. NM will start Container by
ContainerManagerImpl.startContainerInternal, this method will (1)put appID in
context and then (4)send INIT_APPLICATION. Meanwhile NodeManager apperceives
the app that need to be cleaned by ResourceTrackerService.nodeHeartbeat, and
send FINISH_APPS event to ContainerManagerImpl. ContainerManagerImpl will
first (2)check the appID if exists in context, if it does, then (3) send
FINISH_APPLICATION.
This bug manifests needing two condition: (1) happens before(2) and (3)
happens before(4). one of them is violated, this bug will be hidden.
I need to future check the ApplicationImpl code, make sure whether
AppFinishTriggeredTransition needed to fix this bug.
was (Author: xiaoheipangzi):
I have find the reason by analysis code and logs
[^YARN-7563.png]
above figure has shown the reason:client submit a application and then send
kill command. NM will start Container by
ContainerManagerImpl.startContainerInternal, this method will (1)put appID in
context and then (4)send INIT_APPLICATION. Meanwhile NodeManager apperceives
the app that need to be cleaned by ResourceTrackerService.nodeHeartbeat, and
send FINISH_APPS event to ContainerManagerImpl. ContainerManagerImpl will
first (2)check the appID if exists in context, if it dose, (3) send
FINISH_APPLICATION.
This bug manifests needing two condition: (1) happens before(2) and (3)
happens before(4). one of them is violated, this bug will be hidden.
I need to future check the ApplicationImpl code, make sure whether
AppFinishTriggeredTransition needed to fix this bug.
> Invalid event: FINISH_APPLICATION at NEW
>
>
> Key: YARN-7563
> URL: https://issues.apache.org/jira/browse/YARN-7563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0-beta1
>Reporter: lujie
> Attachments: YARN-7563.png
>
>
> I send kill command to application, nodemanager log shows:
> {code:java}
> 2017-11-25 19:18:48,126 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> couldn't find container container_1511608703018_0001_01_01 while
> processing FINISH_CONTAINERS event
> 2017-11-25 19:18:48,146 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> FINISH_APPLICATION at NEW
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-25 19:18:48,151 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1511608703018_0001 transitioned from NEW to INITING
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-7563) Invalid event: FINISH_APPLICATION at NEW
[
https://issues.apache.org/jira/browse/YARN-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lujie updated YARN-7563:
Comment: was deleted
(was: !YARN-7563.png!)
> Invalid event: FINISH_APPLICATION at NEW
>
>
> Key: YARN-7563
> URL: https://issues.apache.org/jira/browse/YARN-7563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0-beta1
>Reporter: lujie
> Attachments: YARN-7563.png
>
>
> I send kill command to application, nodemanager log shows:
> {code:java}
> 2017-11-25 19:18:48,126 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> couldn't find container container_1511608703018_0001_01_01 while
> processing FINISH_CONTAINERS event
> 2017-11-25 19:18:48,146 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> FINISH_APPLICATION at NEW
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-25 19:18:48,151 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1511608703018_0001 transitioned from NEW to INITING
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7563) Invalid event: FINISH_APPLICATION at NEW
[
https://issues.apache.org/jira/browse/YARN-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266987#comment-16266987
]
lujie edited comment on YARN-7563 at 11/27/17 4:09 PM:
---
I have find the reason by analysis code and logs
[^YARN-7563.png]
above figure has shown the reason:client submit a application and then send
kill command. NM will start Container by
ContainerManagerImpl.startContainerInternal, this method will (1)put appID in
context and then (4)send INIT_APPLICATION. Meanwhile NodeManager apperceives
the app that need to be cleaned by ResourceTrackerService.nodeHeartbeat, and
send FINISH_APPS event to ContainerManagerImpl. ContainerManagerImpl will
first (2)check the appID if exists in context, if it dose, (3) send
FINISH_APPLICATION.
This bug manifests needing two condition: (1) happens before(2) and (3)
happens before(4). one of them is violated, this bug will be hidden.
I need to future check the ApplicationImpl code, make sure whether
AppFinishTriggeredTransition needed to fix this bug.
was (Author: xiaoheipangzi):
I have find the reason by analysis code and logs
[^YARN-7563.png]
above figure has shown the reason:client submit a application and then send
kill command. NM will start Container by ContainerManagerImpl
.startContainerInternal, this method will (1)put appID in context and then
(4)send INIT_APPLICATION. Meanwhile NodeManager apperceives the app that need
to be cleaned by ResourceTrackerService.nodeHeartbeat, and send FINISH_APPS
event to ContainerManagerImpl. ContainerManagerImpl will first (2)check the
appID if exists in context, if it dose, (3) send FINISH_APPLICATION.
This bug manifests needing two condition: (1) happens before(2) and (3)
happens before(4). one of them is violated, this bug will be hidden.
I need to future check the ApplicationImpl code, make sure whether
AppFinishTriggeredTransition needed to fix this bug.
> Invalid event: FINISH_APPLICATION at NEW
>
>
> Key: YARN-7563
> URL: https://issues.apache.org/jira/browse/YARN-7563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0-beta1
>Reporter: lujie
> Attachments: YARN-7563.png
>
>
> I send kill command to application, nodemanager log shows:
> {code:java}
> 2017-11-25 19:18:48,126 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> couldn't find container container_1511608703018_0001_01_01 while
> processing FINISH_CONTAINERS event
> 2017-11-25 19:18:48,146 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> FINISH_APPLICATION at NEW
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-25 19:18:48,151 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1511608703018_0001 transitioned from NEW to INITING
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7563) Invalid event: FINISH_APPLICATION at NEW
[
https://issues.apache.org/jira/browse/YARN-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266994#comment-16266994
]
lujie commented on YARN-7563:
-
!YARN-7563.png!
> Invalid event: FINISH_APPLICATION at NEW
>
>
> Key: YARN-7563
> URL: https://issues.apache.org/jira/browse/YARN-7563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0-beta1
>Reporter: lujie
> Attachments: YARN-7563.png
>
>
> I send kill command to application, nodemanager log shows:
> {code:java}
> 2017-11-25 19:18:48,126 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> couldn't find container container_1511608703018_0001_01_01 while
> processing FINISH_CONTAINERS event
> 2017-11-25 19:18:48,146 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> FINISH_APPLICATION at NEW
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-25 19:18:48,151 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1511608703018_0001 transitioned from NEW to INITING
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7563) Invalid event: FINISH_APPLICATION at NEW
[
https://issues.apache.org/jira/browse/YARN-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266987#comment-16266987
]
lujie edited comment on YARN-7563 at 11/27/17 4:08 PM:
---
I have find the reason by analysis code and logs
[^YARN-7563.png]
above figure has shown the reason:client submit a application and then send
kill command. NM will start Container by ContainerManagerImpl
.startContainerInternal, this method will (1)put appID in context and then
(4)send INIT_APPLICATION. Meanwhile NodeManager apperceives the app that need
to be cleaned by ResourceTrackerService.nodeHeartbeat, and send FINISH_APPS
event to ContainerManagerImpl. ContainerManagerImpl will first (2)check the
appID if exists in context, if it dose, (3) send FINISH_APPLICATION.
This bug manifests needing two condition: (1) happens before(2) and (3)
happens before(4). one of them is violated, this bug will be hidden.
I need to future check the ApplicationImpl code, make sure whether
AppFinishTriggeredTransition needed to fix this bug.
was (Author: xiaoheipangzi):
I have find the reason by analysis code and logs
!YARN-7563.png!
above figure has shown the reason:client submit a application and then send
kill command. NM will start Container by ContainerManagerImpl
.startContainerInternal, this method will (1)put appID in context and then
(4)send INIT_APPLICATION. Meanwhile NodeManager apperceives the app that need
to be cleaned by ResourceTrackerService.nodeHeartbeat, and send FINISH_APPS
event to ContainerManagerImpl. ContainerManagerImpl will first (2)check the
appID if exists in context, if it dose, (3) send FINISH_APPLICATION.
This bug manifests needing two condition: (1) happens before(2) and (3)
happens before(4). one of them is violated, this bug will be hidden.
I need to future check the ApplicationImpl code, make sure whether
AppFinishTriggeredTransition needed to fix this bug.
> Invalid event: FINISH_APPLICATION at NEW
>
>
> Key: YARN-7563
> URL: https://issues.apache.org/jira/browse/YARN-7563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0-beta1
>Reporter: lujie
> Attachments: YARN-7563.png
>
>
> I send kill command to application, nodemanager log shows:
> {code:java}
> 2017-11-25 19:18:48,126 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> couldn't find container container_1511608703018_0001_01_01 while
> processing FINISH_CONTAINERS event
> 2017-11-25 19:18:48,146 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> FINISH_APPLICATION at NEW
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-25 19:18:48,151 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1511608703018_0001 transitioned from NEW to INITING
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7563) Invalid event: FINISH_APPLICATION at NEW
[
https://issues.apache.org/jira/browse/YARN-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266987#comment-16266987
]
lujie edited comment on YARN-7563 at 11/27/17 4:07 PM:
---
I have find the reason by analysis code and logs
!YARN-7563.png!
above figure has shown the reason:client submit a application and then send
kill command. NM will start Container by ContainerManagerImpl
.startContainerInternal, this method will (1)put appID in context and then
(4)send INIT_APPLICATION. Meanwhile NodeManager apperceives the app that need
to be cleaned by ResourceTrackerService.nodeHeartbeat, and send FINISH_APPS
event to ContainerManagerImpl. ContainerManagerImpl will first (2)check the
appID if exists in context, if it dose, (3) send FINISH_APPLICATION.
This bug manifests needing two condition: (1) happens before(2) and (3)
happens before(4). one of them is violated, this bug will be hidden.
I need to future check the ApplicationImpl code, make sure whether
AppFinishTriggeredTransition needed to fix this bug.
was (Author: xiaoheipangzi):
I have find the reason by analysis code and logs
!YARN-7536.png!
above figure has shown the reason:client submit a application and then send
kill command. NM will start Container by ContainerManagerImpl
.startContainerInternal, this method will (1)put appID in context and then
(4)send INIT_APPLICATION. Meanwhile NodeManager apperceives the app that need
to be cleaned by ResourceTrackerService.nodeHeartbeat, and send FINISH_APPS
event to ContainerManagerImpl. ContainerManagerImpl will first (2)check the
appID if exists in context, if it dose, (3) send FINISH_APPLICATION.
This bug manifests needing two condition: (1) happens before(2) and (3)
happens before(4). one of them is violated, this bug will be hidden.
I need to future check the ApplicationImpl code, make sure whether
AppFinishTriggeredTransition needed to fix this bug.
> Invalid event: FINISH_APPLICATION at NEW
>
>
> Key: YARN-7563
> URL: https://issues.apache.org/jira/browse/YARN-7563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0-beta1
>Reporter: lujie
> Attachments: YARN-7563.png
>
>
> I send kill command to application, nodemanager log shows:
> {code:java}
> 2017-11-25 19:18:48,126 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> couldn't find container container_1511608703018_0001_01_01 while
> processing FINISH_CONTAINERS event
> 2017-11-25 19:18:48,146 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> FINISH_APPLICATION at NEW
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-25 19:18:48,151 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1511608703018_0001 transitioned from NEW to INITING
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7563) Invalid event: FINISH_APPLICATION at NEW
[
https://issues.apache.org/jira/browse/YARN-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lujie updated YARN-7563:
Attachment: YARN-7563.png
> Invalid event: FINISH_APPLICATION at NEW
>
>
> Key: YARN-7563
> URL: https://issues.apache.org/jira/browse/YARN-7563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0-beta1
>Reporter: lujie
> Attachments: YARN-7563.png
>
>
> I send kill command to application, nodemanager log shows:
> {code:java}
> 2017-11-25 19:18:48,126 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> couldn't find container container_1511608703018_0001_01_01 while
> processing FINISH_CONTAINERS event
> 2017-11-25 19:18:48,146 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> FINISH_APPLICATION at NEW
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-25 19:18:48,151 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1511608703018_0001 transitioned from NEW to INITING
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7563) Invalid event: FINISH_APPLICATION at NEW
[
https://issues.apache.org/jira/browse/YARN-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lujie updated YARN-7563:
Attachment: (was: YARN-7536.png)
> Invalid event: FINISH_APPLICATION at NEW
>
>
> Key: YARN-7563
> URL: https://issues.apache.org/jira/browse/YARN-7563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0-beta1
>Reporter: lujie
>
> I send kill command to application, nodemanager log shows:
> {code:java}
> 2017-11-25 19:18:48,126 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> couldn't find container container_1511608703018_0001_01_01 while
> processing FINISH_CONTAINERS event
> 2017-11-25 19:18:48,146 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> FINISH_APPLICATION at NEW
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-25 19:18:48,151 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1511608703018_0001 transitioned from NEW to INITING
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7563) Invalid event: FINISH_APPLICATION at NEW
[
https://issues.apache.org/jira/browse/YARN-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266987#comment-16266987
]
lujie edited comment on YARN-7563 at 11/27/17 4:06 PM:
---
I have find the reason by analysis code and logs
!YARN-7536.png!
above figure has shown the reason:client submit a application and then send
kill command. NM will start Container by ContainerManagerImpl
.startContainerInternal, this method will (1)put appID in context and then
(4)send INIT_APPLICATION. Meanwhile NodeManager apperceives the app that need
to be cleaned by ResourceTrackerService.nodeHeartbeat, and send FINISH_APPS
event to ContainerManagerImpl. ContainerManagerImpl will first (2)check the
appID if exists in context, if it dose, (3) send FINISH_APPLICATION.
This bug manifests needing two condition: (1) happens before(2) and (3)
happens before(4). one of them is violated, this bug will be hidden.
I need to future check the ApplicationImpl code, make sure whether
AppFinishTriggeredTransition needed to fix this bug.
was (Author: xiaoheipangzi):
I have find the reason by analysis code and logs
[^YARN-7536.png]
above figure has shown the reason:client submit a application and then send
kill command. NM will start Container by ContainerManagerImpl
.startContainerInternal, this method will (1)put appID in context and then
(4)send INIT_APPLICATION. Meanwhile NodeManager apperceives the app that need
to be cleaned by ResourceTrackerService.nodeHeartbeat, and send FINISH_APPS
event to ContainerManagerImpl. ContainerManagerImpl will first (2)check the
appID if exists in context, if it dose, (3) send FINISH_APPLICATION.
This bug manifests needing two condition: (1) happens before(2) and (3)
happens before(4). one of them is violated, this bug will be hidden.
I need to future check the ApplicationImpl code, make sure whether
AppFinishTriggeredTransition needed to fix this bug.
> Invalid event: FINISH_APPLICATION at NEW
>
>
> Key: YARN-7563
> URL: https://issues.apache.org/jira/browse/YARN-7563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0-beta1
>Reporter: lujie
>
> I send kill command to application, nodemanager log shows:
> {code:java}
> 2017-11-25 19:18:48,126 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> couldn't find container container_1511608703018_0001_01_01 while
> processing FINISH_CONTAINERS event
> 2017-11-25 19:18:48,146 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> FINISH_APPLICATION at NEW
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-25 19:18:48,151 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1511608703018_0001 transitioned from NEW to INITING
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7563) Invalid event: FINISH_APPLICATION at NEW
[
https://issues.apache.org/jira/browse/YARN-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lujie updated YARN-7563:
Attachment: YARN-7536.png
> Invalid event: FINISH_APPLICATION at NEW
>
>
> Key: YARN-7563
> URL: https://issues.apache.org/jira/browse/YARN-7563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0-beta1
>Reporter: lujie
> Attachments: YARN-7536.png
>
>
> I send kill command to application, nodemanager log shows:
> {code:java}
> 2017-11-25 19:18:48,126 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> couldn't find container container_1511608703018_0001_01_01 while
> processing FINISH_CONTAINERS event
> 2017-11-25 19:18:48,146 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> FINISH_APPLICATION at NEW
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-25 19:18:48,151 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1511608703018_0001 transitioned from NEW to INITING
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7563) Invalid event: FINISH_APPLICATION at NEW
[
https://issues.apache.org/jira/browse/YARN-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266987#comment-16266987
]
lujie edited comment on YARN-7563 at 11/27/17 4:05 PM:
---
I have find the reason by analysis code and logs
[^YARN-7536.png]
above figure has shown the reason:client submit a application and then send
kill command. NM will start Container by ContainerManagerImpl
.startContainerInternal, this method will (1)put appID in context and then
(4)send INIT_APPLICATION. Meanwhile NodeManager apperceives the app that need
to be cleaned by ResourceTrackerService.nodeHeartbeat, and send FINISH_APPS
event to ContainerManagerImpl. ContainerManagerImpl will first (2)check the
appID if exists in context, if it dose, (3) send FINISH_APPLICATION.
This bug manifests needing two condition: (1) happens before(2) and (3)
happens before(4). one of them is violated, this bug will be hidden.
I need to future check the ApplicationImpl code, make sure whether
AppFinishTriggeredTransition needed to fix this bug.
was (Author: xiaoheipangzi):
I have find the reason by analysis code and logs
above figure has shown the reason:client submit a application and then send
kill command. NM will start Container by ContainerManagerImpl
.startContainerInternal, this method will (1)put appID in context and then
(4)send INIT_APPLICATION. Meanwhile NodeManager apperceives the app that need
to be cleaned by ResourceTrackerService.nodeHeartbeat, and send FINISH_APPS
event to ContainerManagerImpl. ContainerManagerImpl will first (2)check the
appID if exists in context, if it dose, (3) send FINISH_APPLICATION.
This bug manifests needing two condition: (1) happens before(2) and (3)
happens before(4). one of them is violated, this bug will be hidden.
I need to future check the ApplicationImpl code, make sure whether
AppFinishTriggeredTransition needed to fix this bug.
> Invalid event: FINISH_APPLICATION at NEW
>
>
> Key: YARN-7563
> URL: https://issues.apache.org/jira/browse/YARN-7563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0-beta1
>Reporter: lujie
> Attachments: YARN-7536.png
>
>
> I send kill command to application, nodemanager log shows:
> {code:java}
> 2017-11-25 19:18:48,126 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> couldn't find container container_1511608703018_0001_01_01 while
> processing FINISH_CONTAINERS event
> 2017-11-25 19:18:48,146 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> FINISH_APPLICATION at NEW
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-25 19:18:48,151 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1511608703018_0001 transitioned from NEW to INITING
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7563) Invalid event: FINISH_APPLICATION at NEW
[
https://issues.apache.org/jira/browse/YARN-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266987#comment-16266987
]
lujie edited comment on YARN-7563 at 11/27/17 4:04 PM:
---
I have find the reason by analysis code and logs
above figure has shown the reason:client submit a application and then send
kill command. NM will start Container by ContainerManagerImpl
.startContainerInternal, this method will (1)put appID in context and then
(4)send INIT_APPLICATION. Meanwhile NodeManager apperceives the app that need
to be cleaned by ResourceTrackerService.nodeHeartbeat, and send FINISH_APPS
event to ContainerManagerImpl. ContainerManagerImpl will first (2)check the
appID if exists in context, if it dose, (3) send FINISH_APPLICATION.
This bug manifests needing two condition: (1) happens before(2) and (3)
happens before(4). one of them is violated, this bug will be hidden.
I need to future check the ApplicationImpl code, make sure whether
AppFinishTriggeredTransition needed to fix this bug.
was (Author: xiaoheipangzi):
I have find the reason by analysis code and logs
above figure has shown the reason:client submit a application and then send
kill command. NM will start Container by ContainerManagerImpl
.startContainerInternal, this method will (1)put appID in context and then
(4)send INIT_APPLICATION. Meanwhile NodeManager apperceives the app that need
to be cleaned by ResourceTrackerService.nodeHeartbeat, and send FINISH_APPS
event to ContainerManagerImpl. ContainerManagerImpl will first (2)check the
appID if exists in context, if it dose, (3) send FINISH_APPLICATION.
This bug manifests needing two condition: (1) happens before(2) and (3)
happens before(4). one of them is violated, this bug will be hidden.
I need to future check the ApplicationImpl code, make sure whether
AppFinishTriggeredTransition needed to fix this bug.
> Invalid event: FINISH_APPLICATION at NEW
>
>
> Key: YARN-7563
> URL: https://issues.apache.org/jira/browse/YARN-7563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0-beta1
>Reporter: lujie
>
> I send kill command to application, nodemanager log shows:
> {code:java}
> 2017-11-25 19:18:48,126 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> couldn't find container container_1511608703018_0001_01_01 while
> processing FINISH_CONTAINERS event
> 2017-11-25 19:18:48,146 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> FINISH_APPLICATION at NEW
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-25 19:18:48,151 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1511608703018_0001 transitioned from NEW to INITING
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7563) Invalid event: FINISH_APPLICATION at NEW
[
https://issues.apache.org/jira/browse/YARN-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266987#comment-16266987
]
lujie edited comment on YARN-7563 at 11/27/17 4:03 PM:
---
I have find the reason by analysis code and logs
above figure has shown the reason:client submit a application and then send
kill command. NM will start Container by ContainerManagerImpl
.startContainerInternal, this method will (1)put appID in context and then
(4)send INIT_APPLICATION. Meanwhile NodeManager apperceives the app that need
to be cleaned by ResourceTrackerService.nodeHeartbeat, and send FINISH_APPS
event to ContainerManagerImpl. ContainerManagerImpl will first (2)check the
appID if exists in context, if it dose, (3) send FINISH_APPLICATION.
This bug manifests needing two condition: (1) happens before(2) and (3)
happens before(4). one of them is violated, this bug will be hidden.
I need to future check the ApplicationImpl code, make sure whether
AppFinishTriggeredTransition needed to fix this bug.
was (Author: xiaoheipangzi):
I have find the reason by analysis code and logs
!YARN-7536.png!
above figure has shown the reason:client submit a application and then send
kill command. NM will start Container by ContainerManagerImpl
.startContainerInternal, this method will (1)put appID in context and then
(4)send INIT_APPLICATION. Meanwhile NodeManager apperceives the app that need
to be cleaned by ResourceTrackerService.nodeHeartbeat, and send FINISH_APPS
event to ContainerManagerImpl. ContainerManagerImpl will first (2)check the
appID if exists in context, if it dose, (3) send FINISH_APPLICATION.
This bug manifests needing two condition: (1) happens before(2) and (3)
happens before(4). one of them is violated, this bug will be hidden.
I need to future check the ApplicationImpl code, make sure whether
AppFinishTriggeredTransition needed to fix this bug.
> Invalid event: FINISH_APPLICATION at NEW
>
>
> Key: YARN-7563
> URL: https://issues.apache.org/jira/browse/YARN-7563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0-beta1
>Reporter: lujie
>
> I send kill command to application, nodemanager log shows:
> {code:java}
> 2017-11-25 19:18:48,126 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> couldn't find container container_1511608703018_0001_01_01 while
> processing FINISH_CONTAINERS event
> 2017-11-25 19:18:48,146 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> FINISH_APPLICATION at NEW
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-25 19:18:48,151 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1511608703018_0001 transitioned from NEW to INITING
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7563) Invalid event: FINISH_APPLICATION at NEW
[
https://issues.apache.org/jira/browse/YARN-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266987#comment-16266987
]
lujie commented on YARN-7563:
-
I have find the reason by analysis code and logs
!YARN-7536.png!
above figure has shown the reason:client submit a application and then send
kill command. NM will start Container by ContainerManagerImpl
.startContainerInternal, this method will (1)put appID in context and then
(4)send INIT_APPLICATION. Meanwhile NodeManager apperceives the app that need
to be cleaned by ResourceTrackerService.nodeHeartbeat, and send FINISH_APPS
event to ContainerManagerImpl. ContainerManagerImpl will first (2)check the
appID if exists in context, if it dose, (3) send FINISH_APPLICATION.
This bug manifests needing two condition: (1) happens before(2) and (3)
happens before(4). one of them is violated, this bug will be hidden.
I need to future check the ApplicationImpl code, make sure whether
AppFinishTriggeredTransition needed to fix this bug.
> Invalid event: FINISH_APPLICATION at NEW
>
>
> Key: YARN-7563
> URL: https://issues.apache.org/jira/browse/YARN-7563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0-beta1
>Reporter: lujie
> Attachments: YARN-7536.png
>
>
> I send kill command to application, nodemanager log shows:
> {code:java}
> 2017-11-25 19:18:48,126 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> couldn't find container container_1511608703018_0001_01_01 while
> processing FINISH_CONTAINERS event
> 2017-11-25 19:18:48,146 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> FINISH_APPLICATION at NEW
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-25 19:18:48,151 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1511608703018_0001 transitioned from NEW to INITING
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7563) Invalid event: FINISH_APPLICATION at NEW
[
https://issues.apache.org/jira/browse/YARN-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lujie updated YARN-7563:
Attachment: (was: YARN-7536.png)
> Invalid event: FINISH_APPLICATION at NEW
>
>
> Key: YARN-7563
> URL: https://issues.apache.org/jira/browse/YARN-7563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0-beta1
>Reporter: lujie
>
> I send kill command to application, nodemanager log shows:
> {code:java}
> 2017-11-25 19:18:48,126 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> couldn't find container container_1511608703018_0001_01_01 while
> processing FINISH_CONTAINERS event
> 2017-11-25 19:18:48,146 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> FINISH_APPLICATION at NEW
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-25 19:18:48,151 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1511608703018_0001 transitioned from NEW to INITING
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7563) Invalid event: FINISH_APPLICATION at NEW
[
https://issues.apache.org/jira/browse/YARN-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lujie updated YARN-7563:
Attachment: (was: YARN-7563.png)
> Invalid event: FINISH_APPLICATION at NEW
>
>
> Key: YARN-7563
> URL: https://issues.apache.org/jira/browse/YARN-7563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0-beta1
>Reporter: lujie
> Attachments: YARN-7536.png
>
>
> I send kill command to application, nodemanager log shows:
> {code:java}
> 2017-11-25 19:18:48,126 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> couldn't find container container_1511608703018_0001_01_01 while
> processing FINISH_CONTAINERS event
> 2017-11-25 19:18:48,146 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> FINISH_APPLICATION at NEW
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-25 19:18:48,151 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1511608703018_0001 transitioned from NEW to INITING
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7563) Invalid event: FINISH_APPLICATION at NEW
[
https://issues.apache.org/jira/browse/YARN-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lujie updated YARN-7563:
Attachment: YARN-7536.png
> Invalid event: FINISH_APPLICATION at NEW
>
>
> Key: YARN-7563
> URL: https://issues.apache.org/jira/browse/YARN-7563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0-beta1
>Reporter: lujie
> Attachments: YARN-7536.png
>
>
> I send kill command to application, nodemanager log shows:
> {code:java}
> 2017-11-25 19:18:48,126 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> couldn't find container container_1511608703018_0001_01_01 while
> processing FINISH_CONTAINERS event
> 2017-11-25 19:18:48,146 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> FINISH_APPLICATION at NEW
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-25 19:18:48,151 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1511608703018_0001 transitioned from NEW to INITING
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7473) Implement Framework and policy for capacity management of auto created queues
[
https://issues.apache.org/jira/browse/YARN-7473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266974#comment-16266974
]
genericqa commented on YARN-7473:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 56s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
22s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
23s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 23s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 27s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 94 new + 208 unchanged - 8 fixed = 302 total (was 216)
{color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
24s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 3m
45s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
19s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 24s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7473 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12899415/YARN-7473.8.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 7a393f6ce6a9 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2bde3ae |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| mvninstall |
https://builds.apache.org/job/PreCommit-YARN-Build/18671/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| compile |
[jira] [Updated] (YARN-7563) Invalid event: FINISH_APPLICATION at NEW
[
https://issues.apache.org/jira/browse/YARN-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lujie updated YARN-7563:
Attachment: screenshot-1.png
> Invalid event: FINISH_APPLICATION at NEW
>
>
> Key: YARN-7563
> URL: https://issues.apache.org/jira/browse/YARN-7563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0-beta1
>Reporter: lujie
> Attachments: YARN-7563.png
>
>
> I send kill command to application, nodemanager log shows:
> {code:java}
> 2017-11-25 19:18:48,126 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> couldn't find container container_1511608703018_0001_01_01 while
> processing FINISH_CONTAINERS event
> 2017-11-25 19:18:48,146 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> FINISH_APPLICATION at NEW
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-25 19:18:48,151 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1511608703018_0001 transitioned from NEW to INITING
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7563) Invalid event: FINISH_APPLICATION at NEW
[
https://issues.apache.org/jira/browse/YARN-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lujie updated YARN-7563:
Attachment: (was: screenshot-1.png)
> Invalid event: FINISH_APPLICATION at NEW
>
>
> Key: YARN-7563
> URL: https://issues.apache.org/jira/browse/YARN-7563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0-beta1
>Reporter: lujie
> Attachments: YARN-7563.png
>
>
> I send kill command to application, nodemanager log shows:
> {code:java}
> 2017-11-25 19:18:48,126 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> couldn't find container container_1511608703018_0001_01_01 while
> processing FINISH_CONTAINERS event
> 2017-11-25 19:18:48,146 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> FINISH_APPLICATION at NEW
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-25 19:18:48,151 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1511608703018_0001 transitioned from NEW to INITING
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7563) Invalid event: FINISH_APPLICATION at NEW
[
https://issues.apache.org/jira/browse/YARN-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lujie updated YARN-7563:
Attachment: YARN-7563.png
> Invalid event: FINISH_APPLICATION at NEW
>
>
> Key: YARN-7563
> URL: https://issues.apache.org/jira/browse/YARN-7563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0-beta1
>Reporter: lujie
> Attachments: YARN-7563.png
>
>
> I send kill command to application, nodemanager log shows:
> {code:java}
> 2017-11-25 19:18:48,126 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> couldn't find container container_1511608703018_0001_01_01 while
> processing FINISH_CONTAINERS event
> 2017-11-25 19:18:48,146 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> FINISH_APPLICATION at NEW
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:745)
> 2017-11-25 19:18:48,151 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Application application_1511608703018_0001 transitioned from NEW to INITING
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7497) Add HDFSSchedulerConfigurationStore for RM HA
[ https://issues.apache.org/jira/browse/YARN-7497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266947#comment-16266947 ] Greg Phillips commented on YARN-7497: - [~yangjiandan] - Would it make sense to move the two configuration statics (SCHEDULER_CONFIGURATION_HFDS_PATH & SCHEDULER_CONFIGURATION_MAX_VERSION) into YarnConfiguration with entries in yarn-defaults? I would like to reuse these for a fair scheduler implementation of an HDFS ConfigStore. > Add HDFSSchedulerConfigurationStore for RM HA > - > > Key: YARN-7497 > URL: https://issues.apache.org/jira/browse/YARN-7497 > Project: Hadoop YARN > Issue Type: New Feature > Components: yarn >Reporter: Jiandan Yang > Attachments: YARN-7497.001.patch, YARN-7497.002.patch, > YARN-7497.003.patch, YARN-7497.004.patch > > > YARN-5947 add LeveldbConfigurationStore using Leveldb as backing store, but > it does not support Yarn RM HA. > YARN-6840 supports RM HA, but too many scheduler configurations may exceed > znode limit, for example 10 thousand queues. > HDFSSchedulerConfigurationStore store conf file in HDFS, when RM failover, > new active RM can load scheduler configuration from HDFS. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7563) Invalid event: FINISH_APPLICATION at NEW
[
https://issues.apache.org/jira/browse/YARN-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lujie updated YARN-7563:
Description:
I send kill command to application, nodemanager log shows:
{code:java}
2017-11-25 19:18:48,126 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
couldn't find container container_1511608703018_0001_01_01 while
processing FINISH_CONTAINERS event
2017-11-25 19:18:48,146 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
FINISH_APPLICATION at NEW
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
at java.lang.Thread.run(Thread.java:745)
2017-11-25 19:18:48,151 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Application application_1511608703018_0001 transitioned from NEW to INITING
{code}
was:
I send kill command to application, nodemanager log shows:
{code:java}
2017-11-25 19:18:48,126 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
couldn't find container container_1511608703018_0001_01_01 while
processing FINISH_CONTAINERS event
2017-11-25 19:18:48,146 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
FINISH_APPLICATION at NEW
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
at java.lang.Thread.run(Thread.java:745)
2017-11-25 19:18:48,151 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Application application_1511608703018_0001 transitioned from NEW to INITING
{code}
> Invalid event: FINISH_APPLICATION at NEW
>
>
> Key: YARN-7563
> URL: https://issues.apache.org/jira/browse/YARN-7563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0-beta1
>Reporter: lujie
>
> I send kill command to application, nodemanager log shows:
> {code:java}
> 2017-11-25 19:18:48,126 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> couldn't find container container_1511608703018_0001_01_01 while
> processing FINISH_CONTAINERS event
> 2017-11-25 19:18:48,146 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> FINISH_APPLICATION at NEW
> at
>
[jira] [Updated] (YARN-7473) Implement Framework and policy for capacity management of auto created queues
[ https://issues.apache.org/jira/browse/YARN-7473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated YARN-7473: --- Attachment: YARN-7473.8.patch Fixed license, findbugs and javac issues > Implement Framework and policy for capacity management of auto created queues > -- > > Key: YARN-7473 > URL: https://issues.apache.org/jira/browse/YARN-7473 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Attachments: YARN-7473.1.patch, YARN-7473.2.patch, YARN-7473.3.patch, > YARN-7473.4.patch, YARN-7473.5.patch, YARN-7473.6.patch, YARN-7473.7.patch, > YARN-7473.8.patch > > > This jira mainly addresses the following > > 1.Support adding pluggable policies on parent queue for dynamically managing > capacity/state for leaf queues. > 2. Implement a default policy that manages capacity based on pending > applications and either grants guaranteed or zero capacity to queues based on > parent's available guaranteed capacity. > 3. Integrate with SchedulingEditPolicy framework to trigger this periodically > and signal scheduler to take necessary actions for capacity/queue management. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7563) Invalid event: FINISH_APPLICATION at NEW
lujie created YARN-7563:
---
Summary: Invalid event: FINISH_APPLICATION at NEW
Key: YARN-7563
URL: https://issues.apache.org/jira/browse/YARN-7563
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 3.0.0-beta1
Reporter: lujie
I send kill command to application, nodemanager log shows:
{code:java}
2017-11-25 19:18:48,126 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
couldn't find container container_1511608703018_0001_01_01 while
processing FINISH_CONTAINERS event
2017-11-25 19:18:48,146 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
FINISH_APPLICATION at NEW
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:627)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:75)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1508)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:1501)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
at java.lang.Thread.run(Thread.java:745)
2017-11-25 19:18:48,151 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Application application_1511608703018_0001 transitioned from NEW to INITING
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7535) We should display origin value of demand in fair scheduler page
[
https://issues.apache.org/jira/browse/YARN-7535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266791#comment-16266791
]
Wilfred Spiegelenburg commented on YARN-7535:
-
The demand for an application is not limited and is we do not stop adding up
demand at any limit for an application.
This is purely about the maximum demand a queue is assigned for the purpose of
scheduling. Showing a demand in a queue that is higher than what could ever be
allocated in the queue also does not make sense from a scheduling perspective.
I agree that it is one of the possible factors you look at from a tuning
perspective.
Demand should not be the one and only number to base a queue size on. An
applications demand is just the requested resources for the Application Master
until the application runs. When the AM is not started there is no request for
any other resources beside that one AM container. There is thus no demand
beside that one container. This means that the demand in a queue is of limited
value for tuning the size of a queue. Non runnable applications always skew the
demand number. If you run into the maximum size of a queue demand will be
skewed unless all applications in all queues are running. Most likely it will
give the wrong info and your tuning would be off. If you are not running into
the maximum queue size and still have unsatisfied demand something else is
going on. Again demand will not be the value you are tuning on in that case.
> We should display origin value of demand in fair scheduler page
> ---
>
> Key: YARN-7535
> URL: https://issues.apache.org/jira/browse/YARN-7535
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Reporter: YunFan Zhou
>Assignee: YunFan Zhou
>
> The value of *demand* of leaf queue that we now view on the fair scheduler
> page shows only the value of *maxResources* when the demand value is greater
> than *maxResources*. It doesn't reflect the real situation. Most of the time,
> when we expand the queue, we often rely on seeing the current demand real
> value.
> {code:java}
> private void updateDemandForApp(FSAppAttempt sched, Resource maxRes) {
> sched.updateDemand();
> Resource toAdd = sched.getDemand();
> if (LOG.isDebugEnabled()) {
> LOG.debug("Counting resource from " + sched.getName() + " " + toAdd
> + "; Total resource consumption for " + getName() + " now "
> + demand);
> }
> demand = Resources.add(demand, toAdd);
> demand = Resources.componentwiseMin(demand, maxRes);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
