[jira] [Commented] (YARN-9694) UI always show default-rack for all the nodes while running SLS.
[
https://issues.apache.org/jira/browse/YARN-9694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894259#comment-16894259
]
Hadoop QA commented on YARN-9694:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 54s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 24s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 26s{color}
| {color:red} hadoop-sls in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 9s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.sls.TestSLSRunner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9694 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12976019/YARN-9694.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c2662973b50e 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2fe450c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/24430/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24430/testReport/ |
| Max. process+thread count | 449 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-sls U: hadoop-tools/hadoop-sls |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24430/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> UI always show
[jira] [Comment Edited] (YARN-9690) Invalid AMRM token when distributed scheduling is enabled.
[
https://issues.apache.org/jira/browse/YARN-9690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894019#comment-16894019
]
Babble Shack edited comment on YARN-9690 at 7/26/19 6:08 PM:
-
I also tried creating a new configuration directory and disabling
_yarn.resourcemanager.scheduler_ and using it default value at the resource
manager.
{code:java}
{code}
The new configuration:
{code:java}
yarn.resourcemanager.scheduler.address
localhost:8049
{code}
And launching the job with
{code:java}
yarn --config /debug/hadoop/etc/hadoop jar $MAPRED_EXAMPLES pi
-Dmapreduce.job.num-opportunistic-maps-percent="40" 10 10
{code}
The nodemanager then tries to connect with scheduler on 0.0.0.0:8030, however
the amrmproxy has already started, and so the amrmproxy should respond by
redirecting the request to the Local RM
{code:java}
//Nodemanager logs
2019-07-26 17:32:25,480 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
container_1564162333195_0001_01_01's ip = 10.244.29.84, and hostname =
yarn-worker-2.yarn-service.yarn.svc.cluster.local
2019-07-26 17:32:26,962 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth
successful for appattempt_1564162333195_0001_01 (auth:SIMPLE)
2019-07-26 17:32:26,965 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyService:
Registering application master.
Host:yarn-worker-2.yarn-service.yarn.svc.cluster.local Port:42172 Tracking
Url:http://yarn-worker-2.yarn-service.yarn.svc.cluster.local:37110
2019-07-26 17:32:26,965 INFO
org.apache.hadoop.yarn.server.nodemanager.scheduler.DistributedScheduler:
Forwarding registration request to theDistributed Scheduler Service on YARN RM
2019-07-26 17:32:26,966 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.DefaultRequestInterceptor:
Forwarding registerApplicationMasterForDistributedSchedulingrequest to the real
YARN RM
2019-07-26 17:32:27,971 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: 0.0.0.0/0.0.0.0:8030. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-07-26 17:32:28,485 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Skipping monitoring container container_1564162333195_0001_01_01 since CPU
usage is not yet available.
2019-07-26 17:32:28,971 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: 0.0.0.0/0.0.0.0:8030. Already tried 1 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
{code}
However there is no service listening on 0.0.0.0:8030
The application log, shows AM is connecting to AMRMproxy
{code:java}
// AM Logs
2019-07-26 17:50:17,238 INFO [main] org.apache.hadoop.yarn.client.RMProxy:
Connecting to ResourceManager at localhost/127.0.0.1:8049
End of LogType:syslog.This log file belongs to a running container
(container_1564163395606_0001_01_01) and so may not be complete.
***
{code}
was (Author: babbleshack):
I also tried disabling _yarn.resourcemanager.scheduler_ and using it default
value at the resource manager.
{code:java}
{code}
I then create a new config directory with the following
{code:java}
yarn.resourcemanager.scheduler.address
localhost:8049
{code}
And launching the job with
{code:java}
yarn --config /debug/hadoop/etc/hadoop jar $MAPRED_EXAMPLES pi
-Dmapreduce.job.num-opportunistic-maps-percent="40" 10 10
{code}
The nodemanager then tries to connect with scheduler on 0.0.0.0:8030, however
the amrmproxy has already started, and so the amrmproxy should respond by
redirecting the request to the Local RM
{code:java}
//Nodemanager logs
2019-07-26 17:32:25,480 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
container_1564162333195_0001_01_01's ip = 10.244.29.84, and hostname =
yarn-worker-2.yarn-service.yarn.svc.cluster.local
2019-07-26 17:32:26,962 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth
successful for appattempt_1564162333195_0001_01 (auth:SIMPLE)
2019-07-26 17:32:26,965 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyService:
Registering application master.
Host:yarn-worker-2.yarn-service.yarn.svc.cluster.local Port:42172 Tracking
Url:http://yarn-worker-2.yarn-service.yarn.svc.cluster.local:37110
2019-07-26 17:32:26,965 INFO
org.apache.hadoop.yarn.server.nodemanager.scheduler.DistributedScheduler:
Forwarding registration request to theDistributed Scheduler Service on YARN RM
2019-07-26 17:32:26,966 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.DefaultRequestInterceptor:
Forwarding registerApplicationMasterForDistributedSchedulingrequest to the real
YARN RM
2019-07-26 17:32:27,971 INFO
[jira] [Comment Edited] (YARN-9690) Invalid AMRM token when distributed scheduling is enabled.
[
https://issues.apache.org/jira/browse/YARN-9690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894019#comment-16894019
]
Babble Shack edited comment on YARN-9690 at 7/26/19 5:56 PM:
-
I also tried disabling _yarn.resourcemanager.scheduler_ and using it default
value at the resource manager.
{code:java}
{code}
I then create a new config directory with the following
{code:java}
yarn.resourcemanager.scheduler.address
localhost:8049
{code}
And launching the job with
{code:java}
yarn --config /debug/hadoop/etc/hadoop jar $MAPRED_EXAMPLES pi
-Dmapreduce.job.num-opportunistic-maps-percent="40" 10 10
{code}
The nodemanager then tries to connect with scheduler on 0.0.0.0:8030, however
the amrmproxy has already started, and so the amrmproxy should respond by
redirecting the request to the Local RM
{code:java}
//Nodemanager logs
2019-07-26 17:32:25,480 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
container_1564162333195_0001_01_01's ip = 10.244.29.84, and hostname =
yarn-worker-2.yarn-service.yarn.svc.cluster.local
2019-07-26 17:32:26,962 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth
successful for appattempt_1564162333195_0001_01 (auth:SIMPLE)
2019-07-26 17:32:26,965 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyService:
Registering application master.
Host:yarn-worker-2.yarn-service.yarn.svc.cluster.local Port:42172 Tracking
Url:http://yarn-worker-2.yarn-service.yarn.svc.cluster.local:37110
2019-07-26 17:32:26,965 INFO
org.apache.hadoop.yarn.server.nodemanager.scheduler.DistributedScheduler:
Forwarding registration request to theDistributed Scheduler Service on YARN RM
2019-07-26 17:32:26,966 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.DefaultRequestInterceptor:
Forwarding registerApplicationMasterForDistributedSchedulingrequest to the real
YARN RM
2019-07-26 17:32:27,971 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: 0.0.0.0/0.0.0.0:8030. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-07-26 17:32:28,485 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Skipping monitoring container container_1564162333195_0001_01_01 since CPU
usage is not yet available.
2019-07-26 17:32:28,971 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: 0.0.0.0/0.0.0.0:8030. Already tried 1 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
{code}
However there is no service listening on 0.0.0.0:8030
The application log, shows AM is connecting to AMRMproxy
{code:java}
// AM Logs
2019-07-26 17:50:17,238 INFO [main] org.apache.hadoop.yarn.client.RMProxy:
Connecting to ResourceManager at localhost/127.0.0.1:8049
End of LogType:syslog.This log file belongs to a running container
(container_1564163395606_0001_01_01) and so may not be complete.
***
{code}
was (Author: babbleshack):
I also tried disabling _yarn.resourcemanager.scheduler_ and using it default
value at the resource manager.
{code:java}
{code}
I then create a new config directory with the following
{code:java}
yarn.resourcemanager.scheduler.address
localhost:8049
{code}
And launching the job with
{code:java}
yarn --config /debug/hadoop/etc/hadoop jar $MAPRED_EXAMPLES pi
-Dmapreduce.job.num-opportunistic-maps-percent="40" 10 10
{code}
The nodemanager then tries to connect with scheduler on 0.0.0.0:8030, however
the amrmproxy has already started, and so the amrmproxy should respond by
redirecting the request to the Local RM
{code:java}
//Nodemanager logs
2019-07-26 17:32:25,480 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
container_1564162333195_0001_01_01's ip = 10.244.29.84, and hostname =
yarn-worker-2.yarn-service.yarn.svc.cluster.local
2019-07-26 17:32:26,962 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth
successful for appattempt_1564162333195_0001_01 (auth:SIMPLE)
2019-07-26 17:32:26,965 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyService:
Registering application master.
Host:yarn-worker-2.yarn-service.yarn.svc.cluster.local Port:42172 Tracking
Url:http://yarn-worker-2.yarn-service.yarn.svc.cluster.local:37110
2019-07-26 17:32:26,965 INFO
org.apache.hadoop.yarn.server.nodemanager.scheduler.DistributedScheduler:
Forwarding registration request to theDistributed Scheduler Service on YARN RM
2019-07-26 17:32:26,966 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.DefaultRequestInterceptor:
Forwarding registerApplicationMasterForDistributedSchedulingrequest to the real
YARN RM
2019-07-26 17:32:27,971 INFO
[jira] [Comment Edited] (YARN-9690) Invalid AMRM token when distributed scheduling is enabled.
[
https://issues.apache.org/jira/browse/YARN-9690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894019#comment-16894019
]
Babble Shack edited comment on YARN-9690 at 7/26/19 5:51 PM:
-
I also tried disabling _yarn.resourcemanager.scheduler_ and using it default
value at the resource manager.
{code:java}
{code}
I then create a new config directory with the following
{code:java}
yarn.resourcemanager.scheduler.address
localhost:8049
{code}
And launching the job with
{code:java}
yarn --config /debug/hadoop/etc/hadoop jar $MAPRED_EXAMPLES pi
-Dmapreduce.job.num-opportunistic-maps-percent="40" 10 10
{code}
The nodemanager then tries to connect with scheduler on 0.0.0.0:8030, however
the amrmproxy has already started, and so the amrmproxy should respond by
redirecting the request to the Local RM
{code:java}
//Nodemanager logs
2019-07-26 17:32:25,480 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
container_1564162333195_0001_01_01's ip = 10.244.29.84, and hostname =
yarn-worker-2.yarn-service.yarn.svc.cluster.local
2019-07-26 17:32:26,962 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth
successful for appattempt_1564162333195_0001_01 (auth:SIMPLE)
2019-07-26 17:32:26,965 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyService:
Registering application master.
Host:yarn-worker-2.yarn-service.yarn.svc.cluster.local Port:42172 Tracking
Url:http://yarn-worker-2.yarn-service.yarn.svc.cluster.local:37110
2019-07-26 17:32:26,965 INFO
org.apache.hadoop.yarn.server.nodemanager.scheduler.DistributedScheduler:
Forwarding registration request to theDistributed Scheduler Service on YARN RM
2019-07-26 17:32:26,966 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.DefaultRequestInterceptor:
Forwarding registerApplicationMasterForDistributedSchedulingrequest to the real
YARN RM
2019-07-26 17:32:27,971 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: 0.0.0.0/0.0.0.0:8030. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-07-26 17:32:28,485 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Skipping monitoring container container_1564162333195_0001_01_01 since CPU
usage is not yet available.
2019-07-26 17:32:28,971 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: 0.0.0.0/0.0.0.0:8030. Already tried 1 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
{code}
However there is no service listening on 0.0.0.0:8030
was (Author: babbleshack):
I also tried disabling _yarn.resourcemanager.scheduler_ and using it default
value at the resource manager.
{code:java}
{code}
I then create a new config directory with the following
{code:java}
yarn.resourcemanager.scheduler.address
localhost:8049
{code}
And launching the job with
{code:java}
yarn --config /debug/hadoop/etc/hadoop jar $MAPRED_EXAMPLES pi
-Dmapreduce.job.num-opportunistic-maps-percent="40" 10 10
{code}
The nodemanager then tries to connect with scheduler on 0.0.0.0:8030, however
the amrmproxy has already started, and so the amrmproxy should respond by
redirecting the request to the Local RM
{code:java}
2019-07-26 17:32:25,480 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
container_1564162333195_0001_01_01's ip = 10.244.29.84, and hostname =
yarn-worker-2.yarn-service.yarn.svc.cluster.local
2019-07-26 17:32:26,962 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth
successful for appattempt_1564162333195_0001_01 (auth:SIMPLE)
2019-07-26 17:32:26,965 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyService:
Registering application master.
Host:yarn-worker-2.yarn-service.yarn.svc.cluster.local Port:42172 Tracking
Url:http://yarn-worker-2.yarn-service.yarn.svc.cluster.local:37110
2019-07-26 17:32:26,965 INFO
org.apache.hadoop.yarn.server.nodemanager.scheduler.DistributedScheduler:
Forwarding registration request to theDistributed Scheduler Service on YARN RM
2019-07-26 17:32:26,966 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.DefaultRequestInterceptor:
Forwarding registerApplicationMasterForDistributedSchedulingrequest to the real
YARN RM
2019-07-26 17:32:27,971 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: 0.0.0.0/0.0.0.0:8030. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-07-26 17:32:28,485 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Skipping monitoring container container_1564162333195_0001_01_01 since CPU
usage is not yet available.
2019-07-26 17:32:28,971 INFO
[jira] [Commented] (YARN-8480) Add boolean option for resources
[ https://issues.apache.org/jira/browse/YARN-8480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894020#comment-16894020 ] Daniel Templeton commented on YARN-8480: I still hold that boolean resources are the way to go, but given that I'm no longer active in YARN, and that Yunicorn aspires to be the scheduler to rule them all, it's not worth the fight. Go ahead and close it. > Add boolean option for resources > > > Key: YARN-8480 > URL: https://issues.apache.org/jira/browse/YARN-8480 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Daniel Templeton >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-8480.001.patch, YARN-8480.002.patch > > > Make it possible to define a resource with a boolean value. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9690) Invalid AMRM token when distributed scheduling is enabled.
[
https://issues.apache.org/jira/browse/YARN-9690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894019#comment-16894019
]
Babble Shack edited comment on YARN-9690 at 7/26/19 5:43 PM:
-
I also tried disabling _yarn.resourcemanager.scheduler_ and using it default
value at the resource manager.
{code:java}
{code}
I then create a new config directory with the following
{code:java}
yarn.resourcemanager.scheduler.address
localhost:8049
{code}
And launching the job with
{code:java}
yarn --config /debug/hadoop/etc/hadoop jar $MAPRED_EXAMPLES pi
-Dmapreduce.job.num-opportunistic-maps-percent="40" 10 10
{code}
The nodemanager then tries to connect with scheduler on 0.0.0.0:8030, however
the amrmproxy has already started, and so the amrmproxy should respond by
redirecting the request to the Local RM
{code:java}
2019-07-26 17:32:25,480 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
container_1564162333195_0001_01_01's ip = 10.244.29.84, and hostname =
yarn-worker-2.yarn-service.yarn.svc.cluster.local
2019-07-26 17:32:26,962 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth
successful for appattempt_1564162333195_0001_01 (auth:SIMPLE)
2019-07-26 17:32:26,965 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyService:
Registering application master.
Host:yarn-worker-2.yarn-service.yarn.svc.cluster.local Port:42172 Tracking
Url:http://yarn-worker-2.yarn-service.yarn.svc.cluster.local:37110
2019-07-26 17:32:26,965 INFO
org.apache.hadoop.yarn.server.nodemanager.scheduler.DistributedScheduler:
Forwarding registration request to theDistributed Scheduler Service on YARN RM
2019-07-26 17:32:26,966 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.DefaultRequestInterceptor:
Forwarding registerApplicationMasterForDistributedSchedulingrequest to the real
YARN RM
2019-07-26 17:32:27,971 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: 0.0.0.0/0.0.0.0:8030. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-07-26 17:32:28,485 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Skipping monitoring container container_1564162333195_0001_01_01 since CPU
usage is not yet available.
2019-07-26 17:32:28,971 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: 0.0.0.0/0.0.0.0:8030. Already tried 1 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
{code}
However there is no service listening on 0.0.0.0:8030
was (Author: babbleshack):
I also tried disabling _yarn.resourcemanager.scheduler_ and using it default
value at the resource manager.
{code:java}
yarn.resourcemanager.scheduler.address
localhost:8049
-->
{code}
I then create a new config directory with the following
{code:java}
yarn.resourcemanager.scheduler.address
localhost:8049
{code}
And launching the job with
{code:java}
yarn --config /debug/hadoop/etc/hadoop jar $MAPRED_EXAMPLES pi
-Dmapreduce.job.num-opportunistic-maps-percent="40" 10 10
{code}
The nodemanager then tries to connect with scheduler on 0.0.0.0:8030, however
the amrmproxy has already started, and so the amrmproxy should respond by
redirecting the request to the Local RM
{code:java}
2019-07-26 17:32:25,480 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
container_1564162333195_0001_01_01's ip = 10.244.29.84, and hostname =
yarn-worker-2.yarn-service.yarn.svc.cluster.local
2019-07-26 17:32:26,962 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth
successful for appattempt_1564162333195_0001_01 (auth:SIMPLE)
2019-07-26 17:32:26,965 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyService:
Registering application master.
Host:yarn-worker-2.yarn-service.yarn.svc.cluster.local Port:42172 Tracking
Url:http://yarn-worker-2.yarn-service.yarn.svc.cluster.local:37110
2019-07-26 17:32:26,965 INFO
org.apache.hadoop.yarn.server.nodemanager.scheduler.DistributedScheduler:
Forwarding registration request to theDistributed Scheduler Service on YARN RM
2019-07-26 17:32:26,966 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.DefaultRequestInterceptor:
Forwarding registerApplicationMasterForDistributedSchedulingrequest to the real
YARN RM
2019-07-26 17:32:27,971 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: 0.0.0.0/0.0.0.0:8030. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-07-26 17:32:28,485 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Skipping monitoring container container_1564162333195_0001_01_01 since CPU
usage is
[jira] [Comment Edited] (YARN-9690) Invalid AMRM token when distributed scheduling is enabled.
[
https://issues.apache.org/jira/browse/YARN-9690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893981#comment-16893981
]
Babble Shack edited comment on YARN-9690 at 7/26/19 5:43 PM:
-
Setting `yarn.resourcemanager.scheduler.address` does not throw the Invalid
AMRMToken, however the application master is unable to register with the
resource manager.
I have configured my resource manager, which is also the machine I submit MR
jobs from with the following
{code:java}
yarn.resourcemanager.scheduler.address
localhost:8049
{code}
Whilst my nodemanager is configured as:
{code:java}
{code}
Doing so enable the AM to connect to RM at 127.0.0.1,
{code:java}
// Application Log
/
2019-07-26 16:47:46,928 INFO [main] org.apache.hadoop.security.SecurityUtil:
Updating Configuration
2019-07-26 16:47:46,993 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens: [Kind:
YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 1
cluster_timestamp: 1564159650361 } attemptId: 1 } keyId: -803833355)]
2019-07-26 16:47:47,021 INFO [main] org.apache.hadoop.conf.Configuration:
resource-types.xml not found
2019-07-26 16:47:47,021 INFO [main]
org.apache.hadoop.yarn.util.resource.ResourceUtils: Unable to find
'resource-types.xml'.
2019-07-26 16:47:47,028 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
2019-07-26 16:47:47,029 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config
null
2019-07-26 16:47:47,048 INFO [main]
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output
Committer Algorithm version is 2
2019-07-26 16:47:47,048 INFO [main]
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter
skip cleanup _temporary folders under output directory:false, ignore cleanup
failures: false
2019-07-26 16:47:47,389 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2019-07-26 16:47:47,490 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.jobhistory.EventType for class
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
2019-07-26 16:47:47,491 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
2019-07-26 16:47:47,491 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
2019-07-26 16:47:47,492 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
2019-07-26 16:47:47,492 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
2019-07-26 16:47:47,492 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
2019-07-26 16:47:47,492 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
2019-07-26 16:47:47,493 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for
class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
2019-07-26 16:47:47,507 INFO [main]
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system
[hdfs://hdfs-master-0.hdfs-service.hdfs:9000]
2019-07-26 16:47:47,514 INFO [main]
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system
[hdfs://hdfs-master-0.hdfs-service.hdfs:9000]
2019-07-26 16:47:47,521 INFO [main]
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system
[hdfs://hdfs-master-0.hdfs-service.hdfs:9000]
2019-07-26 16:47:47,526 INFO [main]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job
history data to the timeline server is not enabled
2019-07-26 16:47:47,550 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class
[jira] [Comment Edited] (YARN-9690) Invalid AMRM token when distributed scheduling is enabled.
[
https://issues.apache.org/jira/browse/YARN-9690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894019#comment-16894019
]
Babble Shack edited comment on YARN-9690 at 7/26/19 5:42 PM:
-
I also tried disabling _yarn.resourcemanager.scheduler_ and using it default
value at the resource manager.
{code:java}
yarn.resourcemanager.scheduler.address
localhost:8049
-->
{code}
I then create a new config directory with the following
{code:java}
yarn.resourcemanager.scheduler.address
localhost:8049
{code}
And launching the job with
{code:java}
yarn --config /debug/hadoop/etc/hadoop jar $MAPRED_EXAMPLES pi
-Dmapreduce.job.num-opportunistic-maps-percent="40" 10 10
{code}
The nodemanager then tries to connect with scheduler on 0.0.0.0:8030, however
the amrmproxy has already started, and so the amrmproxy should respond by
redirecting the request to the Local RM
{code:java}
2019-07-26 17:32:25,480 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
container_1564162333195_0001_01_01's ip = 10.244.29.84, and hostname =
yarn-worker-2.yarn-service.yarn.svc.cluster.local
2019-07-26 17:32:26,962 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth
successful for appattempt_1564162333195_0001_01 (auth:SIMPLE)
2019-07-26 17:32:26,965 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyService:
Registering application master.
Host:yarn-worker-2.yarn-service.yarn.svc.cluster.local Port:42172 Tracking
Url:http://yarn-worker-2.yarn-service.yarn.svc.cluster.local:37110
2019-07-26 17:32:26,965 INFO
org.apache.hadoop.yarn.server.nodemanager.scheduler.DistributedScheduler:
Forwarding registration request to theDistributed Scheduler Service on YARN RM
2019-07-26 17:32:26,966 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.DefaultRequestInterceptor:
Forwarding registerApplicationMasterForDistributedSchedulingrequest to the real
YARN RM
2019-07-26 17:32:27,971 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: 0.0.0.0/0.0.0.0:8030. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-07-26 17:32:28,485 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Skipping monitoring container container_1564162333195_0001_01_01 since CPU
usage is not yet available.
2019-07-26 17:32:28,971 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: 0.0.0.0/0.0.0.0:8030. Already tried 1 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
{code}
However there is no service listening on 0.0.0.0:8030
was (Author: babbleshack):
I also tried disabling _yarn.resourcemanager.scheduler_ and using it default
value at the resource manager.
{code:java}
yarn.resourcemanager.scheduler.address
localhost:8049
-->
{code}
I then create a new config directory with the following
yarn.resourcemanager.scheduler.address
localhost:8049
And launching the job with
{code:java}
yarn --config /debug/hadoop/etc/hadoop jar $MAPRED_EXAMPLES pi
-Dmapreduce.job.num-opportunistic-maps-percent="40" 10 10
{code}
The nodemanager then tries to connect with scheduler on 0.0.0.0:8030, however
the amrmproxy has already started, and so the amrmproxy should respond by
redirecting the request to the Local RM
{code:java}
2019-07-26 17:32:25,480 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
container_1564162333195_0001_01_01's ip = 10.244.29.84, and hostname =
yarn-worker-2.yarn-service.yarn.svc.cluster.local
2019-07-26 17:32:26,962 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth
successful for appattempt_1564162333195_0001_01 (auth:SIMPLE)
2019-07-26 17:32:26,965 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyService:
Registering application master.
Host:yarn-worker-2.yarn-service.yarn.svc.cluster.local Port:42172 Tracking
Url:http://yarn-worker-2.yarn-service.yarn.svc.cluster.local:37110
2019-07-26 17:32:26,965 INFO
org.apache.hadoop.yarn.server.nodemanager.scheduler.DistributedScheduler:
Forwarding registration request to theDistributed Scheduler Service on YARN RM
2019-07-26 17:32:26,966 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.DefaultRequestInterceptor:
Forwarding registerApplicationMasterForDistributedSchedulingrequest to the real
YARN RM
2019-07-26 17:32:27,971 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: 0.0.0.0/0.0.0.0:8030. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-07-26 17:32:28,485 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Skipping monitoring container
[jira] [Comment Edited] (YARN-9690) Invalid AMRM token when distributed scheduling is enabled.
[
https://issues.apache.org/jira/browse/YARN-9690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893981#comment-16893981
]
Babble Shack edited comment on YARN-9690 at 7/26/19 5:42 PM:
-
Setting `yarn.resourcemanager.scheduler.address` does not throw the Invalid
AMRMToken, however the application master is unable to register with the
resource manager.
I have configured my resource manager, which is also the machine I submit MR
jobs from with the following
{code:java}
{code}
Doing so enable the AM to connect to RM at 127.0.0.1,
{code:java}
// Application Log
/
2019-07-26 16:47:46,928 INFO [main] org.apache.hadoop.security.SecurityUtil:
Updating Configuration
2019-07-26 16:47:46,993 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens: [Kind:
YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 1
cluster_timestamp: 1564159650361 } attemptId: 1 } keyId: -803833355)]
2019-07-26 16:47:47,021 INFO [main] org.apache.hadoop.conf.Configuration:
resource-types.xml not found
2019-07-26 16:47:47,021 INFO [main]
org.apache.hadoop.yarn.util.resource.ResourceUtils: Unable to find
'resource-types.xml'.
2019-07-26 16:47:47,028 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
2019-07-26 16:47:47,029 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config
null
2019-07-26 16:47:47,048 INFO [main]
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output
Committer Algorithm version is 2
2019-07-26 16:47:47,048 INFO [main]
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter
skip cleanup _temporary folders under output directory:false, ignore cleanup
failures: false
2019-07-26 16:47:47,389 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2019-07-26 16:47:47,490 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.jobhistory.EventType for class
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
2019-07-26 16:47:47,491 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
2019-07-26 16:47:47,491 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
2019-07-26 16:47:47,492 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
2019-07-26 16:47:47,492 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
2019-07-26 16:47:47,492 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
2019-07-26 16:47:47,492 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
2019-07-26 16:47:47,493 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for
class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
2019-07-26 16:47:47,507 INFO [main]
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system
[hdfs://hdfs-master-0.hdfs-service.hdfs:9000]
2019-07-26 16:47:47,514 INFO [main]
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system
[hdfs://hdfs-master-0.hdfs-service.hdfs:9000]
2019-07-26 16:47:47,521 INFO [main]
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system
[hdfs://hdfs-master-0.hdfs-service.hdfs:9000]
2019-07-26 16:47:47,526 INFO [main]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job
history data to the timeline server is not enabled
2019-07-26 16:47:47,550 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
2019-07-26 16:47:47,677 INFO [main]
[jira] [Commented] (YARN-9690) Invalid AMRM token when distributed scheduling is enabled.
[
https://issues.apache.org/jira/browse/YARN-9690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894019#comment-16894019
]
Babble Shack commented on YARN-9690:
I also tried disabling _yarn.resourcemanager.scheduler_ and using it default
value at the resource manager.
{code:java}
yarn.resourcemanager.scheduler.address
localhost:8049
-->
{code}
I then create a new config directory with the following
yarn.resourcemanager.scheduler.address
localhost:8049
And launching the job with
{code:java}
yarn --config /debug/hadoop/etc/hadoop jar $MAPRED_EXAMPLES pi
-Dmapreduce.job.num-opportunistic-maps-percent="40" 10 10
{code}
The nodemanager then tries to connect with scheduler on 0.0.0.0:8030, however
the amrmproxy has already started, and so the amrmproxy should respond by
redirecting the request to the Local RM
{code:java}
2019-07-26 17:32:25,480 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
container_1564162333195_0001_01_01's ip = 10.244.29.84, and hostname =
yarn-worker-2.yarn-service.yarn.svc.cluster.local
2019-07-26 17:32:26,962 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth
successful for appattempt_1564162333195_0001_01 (auth:SIMPLE)
2019-07-26 17:32:26,965 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyService:
Registering application master.
Host:yarn-worker-2.yarn-service.yarn.svc.cluster.local Port:42172 Tracking
Url:http://yarn-worker-2.yarn-service.yarn.svc.cluster.local:37110
2019-07-26 17:32:26,965 INFO
org.apache.hadoop.yarn.server.nodemanager.scheduler.DistributedScheduler:
Forwarding registration request to theDistributed Scheduler Service on YARN RM
2019-07-26 17:32:26,966 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.DefaultRequestInterceptor:
Forwarding registerApplicationMasterForDistributedSchedulingrequest to the real
YARN RM
2019-07-26 17:32:27,971 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: 0.0.0.0/0.0.0.0:8030. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-07-26 17:32:28,485 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Skipping monitoring container container_1564162333195_0001_01_01 since CPU
usage is not yet available.
2019-07-26 17:32:28,971 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: 0.0.0.0/0.0.0.0:8030. Already tried 1 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
{code}
However there is no service listening on 0.0.0.0:8030
> Invalid AMRM token when distributed scheduling is enabled.
> --
>
> Key: YARN-9690
> URL: https://issues.apache.org/jira/browse/YARN-9690
> Project: Hadoop YARN
> Issue Type: Bug
> Components: distributed-scheduling, yarn
>Affects Versions: 2.9.2, 3.1.2
> Environment: OS: Ubuntu 18.04
> JVM: 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03
>Reporter: Babble Shack
>Priority: Major
> Attachments: applicationlog, distributed_log, ds_application.log,
> image-2019-07-26-18-00-14-980.png, nodemanager-yarn-site.xml,
> nodemanager.log, rm-yarn-site.xml, yarn-site.xml
>
>
> Applications fail to start due to invalild AMRM from application attempt.
> I have tested this with 0/100% opportunistic maps and the same issue occurs
> regardless.
> {code:java}
>
> -->
>
>
> mapreduceyarn.nodemanager.aux-services
> mapreduce_shuffle
>
>
> yarn.resourcemanager.address
> yarn-master-0.yarn-service.yarn:8032
>
>
> yarn.resourcemanager.scheduler.address
> 0.0.0.0:8049
>
>
>
> yarn.resourcemanager.opportunistic-container-allocation.enabled
> true
>
>
> yarn.nodemanager.opportunistic-containers-max-queue-length
> 10
>
>
> yarn.nodemanager.distributed-scheduling.enabled
> true
>
>
>
> yarn.webapp.ui2.enable
> true
>
>
> yarn.resourcemanager.resource-tracker.address
> yarn-master-0.yarn-service.yarn:8031
>
>
> yarn.log-aggregation-enable
> true
>
>
> yarn.nodemanager.aux-services
> mapreduce_shuffle
>
>
>
>
>
> yarn.nodemanager.resource.memory-mb
> 7168
>
>
> yarn.scheduler.minimum-allocation-mb
> 3584
>
>
> yarn.scheduler.maximum-allocation-mb
> 7168
>
>
> yarn.app.mapreduce.am.resource.mb
> 7168
>
>
>
> yarn.app.mapreduce.am.command-opts
> -Xmx5734m
>
>
>
> yarn.timeline-service.enabled
> true
>
>
> yarn.resourcemanager.system-metrics-publisher.enabled
> true
>
>
>
[jira] [Comment Edited] (YARN-9690) Invalid AMRM token when distributed scheduling is enabled.
[
https://issues.apache.org/jira/browse/YARN-9690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893981#comment-16893981
]
Babble Shack edited comment on YARN-9690 at 7/26/19 5:13 PM:
-
Setting `yarn.resourcemanager.scheduler.address` does not throw the Invalid
AMRMToken, however the application master is unable to register with the
resource manager.
I have configured my resource manager, which is also the machine I submit MR
jobs from with the following
{code:java}
yarn.resourcemanager.scheduler.address
localhost:8049
{code}
Whilst my nodemanager is configured as:
{code:java}
{code}
Doing so enable the AM to connect to RM at 127.0.0.1,
{code:java}
// Application Log
/
2019-07-26 16:47:46,928 INFO [main] org.apache.hadoop.security.SecurityUtil:
Updating Configuration
2019-07-26 16:47:46,993 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens: [Kind:
YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 1
cluster_timestamp: 1564159650361 } attemptId: 1 } keyId: -803833355)]
2019-07-26 16:47:47,021 INFO [main] org.apache.hadoop.conf.Configuration:
resource-types.xml not found
2019-07-26 16:47:47,021 INFO [main]
org.apache.hadoop.yarn.util.resource.ResourceUtils: Unable to find
'resource-types.xml'.
2019-07-26 16:47:47,028 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
2019-07-26 16:47:47,029 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config
null
2019-07-26 16:47:47,048 INFO [main]
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output
Committer Algorithm version is 2
2019-07-26 16:47:47,048 INFO [main]
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter
skip cleanup _temporary folders under output directory:false, ignore cleanup
failures: false
2019-07-26 16:47:47,389 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2019-07-26 16:47:47,490 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.jobhistory.EventType for class
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
2019-07-26 16:47:47,491 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
2019-07-26 16:47:47,491 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
2019-07-26 16:47:47,492 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
2019-07-26 16:47:47,492 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
2019-07-26 16:47:47,492 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
2019-07-26 16:47:47,492 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
2019-07-26 16:47:47,493 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for
class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
2019-07-26 16:47:47,507 INFO [main]
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system
[hdfs://hdfs-master-0.hdfs-service.hdfs:9000]
2019-07-26 16:47:47,514 INFO [main]
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system
[hdfs://hdfs-master-0.hdfs-service.hdfs:9000]
2019-07-26 16:47:47,521 INFO [main]
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system
[hdfs://hdfs-master-0.hdfs-service.hdfs:9000]
2019-07-26 16:47:47,526 INFO [main]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job
history data to the timeline server is not enabled
2019-07-26 16:47:47,550 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class
[jira] [Commented] (YARN-9479) Change String.equals to Objects.equals(String,String) to avoid possible NullPointerException
[
https://issues.apache.org/jira/browse/YARN-9479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893994#comment-16893994
]
Hadoop QA commented on YARN-9479:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 46s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
55s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 2 new + 15 unchanged - 0 fixed = 17 total (was 15) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 48s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 18s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}148m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
| |
hadoop.yarn.server.resourcemanager.reservation.TestCapacitySchedulerPlanFollower
|
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-738/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/738 |
| JIRA Issue | YARN-9479 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux b1db80e57343
[jira] [Comment Edited] (YARN-9690) Invalid AMRM token when distributed scheduling is enabled.
[
https://issues.apache.org/jira/browse/YARN-9690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893981#comment-16893981
]
Babble Shack edited comment on YARN-9690 at 7/26/19 5:12 PM:
-
Setting `yarn.resourcemanager.scheduler.address` does not throw the Invalid
AMRMToken, however the application master is unable to register with the
resource manager.
I have configured my resource manager, which is also the machine I submit MR
jobs from with the following
{code:java}
yarn.resourcemanager.scheduler.address
localhost:8049
{code}
Whilst my nodemanager is configured as:
{code:java}
{code}
Doing so enable the AM to connect to RM at 127.0.0.1,
{code:java}
// Application Log
/
2019-07-26 16:47:46,928 INFO [main] org.apache.hadoop.security.SecurityUtil:
Updating Configuration
2019-07-26 16:47:46,993 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens: [Kind:
YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 1
cluster_timestamp: 1564159650361 } attemptId: 1 } keyId: -803833355)]
2019-07-26 16:47:47,021 INFO [main] org.apache.hadoop.conf.Configuration:
resource-types.xml not found
2019-07-26 16:47:47,021 INFO [main]
org.apache.hadoop.yarn.util.resource.ResourceUtils: Unable to find
'resource-types.xml'.
2019-07-26 16:47:47,028 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
2019-07-26 16:47:47,029 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config
null
2019-07-26 16:47:47,048 INFO [main]
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output
Committer Algorithm version is 2
2019-07-26 16:47:47,048 INFO [main]
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter
skip cleanup _temporary folders under output directory:false, ignore cleanup
failures: false
2019-07-26 16:47:47,389 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2019-07-26 16:47:47,490 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.jobhistory.EventType for class
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
2019-07-26 16:47:47,491 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
2019-07-26 16:47:47,491 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
2019-07-26 16:47:47,492 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
2019-07-26 16:47:47,492 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
2019-07-26 16:47:47,492 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
2019-07-26 16:47:47,492 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
2019-07-26 16:47:47,493 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for
class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
2019-07-26 16:47:47,507 INFO [main]
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system
[hdfs://hdfs-master-0.hdfs-service.hdfs:9000]
2019-07-26 16:47:47,514 INFO [main]
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system
[hdfs://hdfs-master-0.hdfs-service.hdfs:9000]
2019-07-26 16:47:47,521 INFO [main]
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system
[hdfs://hdfs-master-0.hdfs-service.hdfs:9000]
2019-07-26 16:47:47,526 INFO [main]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job
history data to the timeline server is not enabled
2019-07-26 16:47:47,550 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class
[jira] [Comment Edited] (YARN-9690) Invalid AMRM token when distributed scheduling is enabled.
[
https://issues.apache.org/jira/browse/YARN-9690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893981#comment-16893981
]
Babble Shack edited comment on YARN-9690 at 7/26/19 5:10 PM:
-
Setting `yarn.resourcemanager.scheduler.address` does not throw the Invalid
AMRMToken, however the application master is unable to register with the
resource manager.
I have configured my resource manager, which is also the machine I submit MR
jobs from with the following
{code:java}
yarn.resourcemanager.scheduler.address
localhost:8049
{code}
Whilst my nodemanager is configured as:
{code:java}
{code}
Doing so enable the AM to connect to RM at 127.0.0.1,
{code:java}
// Application Log
/
2019-07-26 16:47:46,928 INFO [main] org.apache.hadoop.security.SecurityUtil:
Updating Configuration
2019-07-26 16:47:46,993 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens: [Kind:
YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 1
cluster_timestamp: 1564159650361 } attemptId: 1 } keyId: -803833355)]
2019-07-26 16:47:47,021 INFO [main] org.apache.hadoop.conf.Configuration:
resource-types.xml not found
2019-07-26 16:47:47,021 INFO [main]
org.apache.hadoop.yarn.util.resource.ResourceUtils: Unable to find
'resource-types.xml'.
2019-07-26 16:47:47,028 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
2019-07-26 16:47:47,029 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config
null
2019-07-26 16:47:47,048 INFO [main]
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output
Committer Algorithm version is 2
2019-07-26 16:47:47,048 INFO [main]
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter
skip cleanup _temporary folders under output directory:false, ignore cleanup
failures: false
2019-07-26 16:47:47,389 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2019-07-26 16:47:47,490 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.jobhistory.EventType for class
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
2019-07-26 16:47:47,491 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
2019-07-26 16:47:47,491 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
2019-07-26 16:47:47,492 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
2019-07-26 16:47:47,492 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
2019-07-26 16:47:47,492 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
2019-07-26 16:47:47,492 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
2019-07-26 16:47:47,493 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for
class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
2019-07-26 16:47:47,507 INFO [main]
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system
[hdfs://hdfs-master-0.hdfs-service.hdfs:9000]
2019-07-26 16:47:47,514 INFO [main]
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system
[hdfs://hdfs-master-0.hdfs-service.hdfs:9000]
2019-07-26 16:47:47,521 INFO [main]
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system
[hdfs://hdfs-master-0.hdfs-service.hdfs:9000]
2019-07-26 16:47:47,526 INFO [main]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job
history data to the timeline server is not enabled
2019-07-26 16:47:47,550 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class
[jira] [Commented] (YARN-9690) Invalid AMRM token when distributed scheduling is enabled.
[
https://issues.apache.org/jira/browse/YARN-9690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893981#comment-16893981
]
Babble Shack commented on YARN-9690:
Setting `yarn.resourcemanager.scheduler.address` does not throw the Invalid
AMRMToken, however the application master is unable to register with the
resource manager.
I have configured my resource manager, which is also the machine I submit MR
jobs from with the following
{code:java}
yarn.resourcemanager.scheduler.address
localhost:8049
{code}
{code:java}
{code}
> Invalid AMRM token when distributed scheduling is enabled.
> --
>
> Key: YARN-9690
> URL: https://issues.apache.org/jira/browse/YARN-9690
> Project: Hadoop YARN
> Issue Type: Bug
> Components: distributed-scheduling, yarn
>Affects Versions: 2.9.2, 3.1.2
> Environment: OS: Ubuntu 18.04
> JVM: 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03
>Reporter: Babble Shack
>Priority: Major
> Attachments: applicationlog, distributed_log, yarn-site.xml
>
>
> Applications fail to start due to invalild AMRM from application attempt.
> I have tested this with 0/100% opportunistic maps and the same issue occurs
> regardless.
> {code:java}
>
> -->
>
>
> mapreduceyarn.nodemanager.aux-services
> mapreduce_shuffle
>
>
> yarn.resourcemanager.address
> yarn-master-0.yarn-service.yarn:8032
>
>
> yarn.resourcemanager.scheduler.address
> 0.0.0.0:8049
>
>
>
> yarn.resourcemanager.opportunistic-container-allocation.enabled
> true
>
>
> yarn.nodemanager.opportunistic-containers-max-queue-length
> 10
>
>
> yarn.nodemanager.distributed-scheduling.enabled
> true
>
>
>
> yarn.webapp.ui2.enable
> true
>
>
> yarn.resourcemanager.resource-tracker.address
> yarn-master-0.yarn-service.yarn:8031
>
>
> yarn.log-aggregation-enable
> true
>
>
> yarn.nodemanager.aux-services
> mapreduce_shuffle
>
>
>
>
>
> yarn.nodemanager.resource.memory-mb
> 7168
>
>
> yarn.scheduler.minimum-allocation-mb
> 3584
>
>
> yarn.scheduler.maximum-allocation-mb
> 7168
>
>
> yarn.app.mapreduce.am.resource.mb
> 7168
>
>
>
> yarn.app.mapreduce.am.command-opts
> -Xmx5734m
>
>
>
> yarn.timeline-service.enabled
> true
>
>
> yarn.resourcemanager.system-metrics-publisher.enabled
> true
>
>
> yarn.timeline-service.generic-application-history.enabled
> true
>
>
> yarn.timeline-service.bind-host
> 0.0.0.0
>
>
> {code}
> Relevant logs:
> {code:java}
> 2019-07-22 14:56:37,104 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 100% of the
> mappers will be scheduled using OPPORTUNISTIC containers
> 2019-07-22 14:56:37,117 INFO [main] org.apache.hadoop.yarn.client.RMProxy:
> Connecting to ResourceManager at
> yarn-master-0.yarn-service.yarn/10.244.1.134:8030
> 2019-07-22 14:56:37,150 WARN [main] org.apache.hadoop.ipc.Client: Exception
> encountered while connecting to the server :
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> Invalid AMRMToken from appattempt_1563805140414_0002_02
> 2019-07-22 14:56:37,152 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Exception while
> registering
> org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> AMRMToken from appattempt_1563805140414_0002_02
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
>
[jira] [Commented] (YARN-9509) Capped cpu usage with cgroup strict-resource-usage based on a mulitplier
[
https://issues.apache.org/jira/browse/YARN-9509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893970#comment-16893970
]
Hadoop QA commented on YARN-9509:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
5s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
40s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
55s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 18s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 5 new + 219 unchanged - 0 fixed = 224 total (was 219) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 28s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
58s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
57s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m
31s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
45s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}126m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 base:
[jira] [Commented] (YARN-9009) Fix flaky test TestEntityGroupFSTimelineStore.testCleanLogs
[
https://issues.apache.org/jira/browse/YARN-9009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893933#comment-16893933
]
Hadoop QA commented on YARN-9009:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 14s{color}
| {color:red} https://github.com/apache/hadoop/pull/438 does not apply to
trunk. Rebase required? Wrong Branch? See
https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| GITHUB PR | https://github.com/apache/hadoop/pull/438 |
| JIRA Issue | YARN-9009 |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-438/2/console |
| versions | git=2.7.4 |
| Powered by | Apache Yetus 0.10.0 http://yetus.apache.org |
This message was automatically generated.
> Fix flaky test TestEntityGroupFSTimelineStore.testCleanLogs
> ---
>
> Key: YARN-9009
> URL: https://issues.apache.org/jira/browse/YARN-9009
> Project: Hadoop YARN
> Issue Type: Bug
> Environment: Ubuntu 18.04
> java version "1.8.0_181"
> Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
>
> Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
> 2018-06-17T13:33:14-05:00)
>Reporter: OrDTesters
>Assignee: OrDTesters
>Priority: Minor
> Fix For: 3.0.4, 3.1.2, 3.3.0, 3.2.1
>
> Attachments: YARN-9009-trunk-001.patch
>
>
> In TestEntityGroupFSTimelineStore, testCleanLogs fails when run after
> testMoveToDone.
> testCleanLogs fails because testMoveToDone moves a file into the same
> directory that testCleanLogs cleans, causing testCleanLogs to clean 3 files,
> instead of 2 as testCleanLogs expects.
> To fix the failure of testCleanLogs, we can delete the file after the file is
> moved by testMoveToDone.
> Pull request link: [https://github.com/apache/hadoop/pull/438]
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9468) Fix inaccurate documentations in Placement Constraints
[
https://issues.apache.org/jira/browse/YARN-9468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893920#comment-16893920
]
Hadoop QA commented on YARN-9468:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
35m 18s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 19s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-717/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/717 |
| JIRA Issue | YARN-9468 |
| Optional Tests | dupname asflicense mvnsite |
| uname | Linux 2bb965e54e85 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3
08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / c0a0c35 |
| Max. process+thread count | 341 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-717/2/console |
| versions | git=2.7.4 maven=3.3.9 |
| Powered by | Apache Yetus 0.10.0 http://yetus.apache.org |
This message was automatically generated.
> Fix inaccurate documentations in Placement Constraints
> --
>
> Key: YARN-9468
> URL: https://issues.apache.org/jira/browse/YARN-9468
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.2.0
>Reporter: hunshenshi
>Assignee: hunshenshi
>Priority: Major
>
> Document Placement Constraints
> *First*
> {code:java}
> zk=3,NOTIN,NODE,zk:hbase=5,IN,RACK,zk:spark=7,CARDINALITY,NODE,hbase,1,3{code}
> * place 5 containers with tag “hbase” with affinity to a rack on which
> containers with tag “zk” are running (i.e., an “hbase” container
> should{color:#ff} not{color} be placed at a rack where an “zk” container
> is running, given that “zk” is the TargetTag of the second constraint);
> The _*not*_ word in brackets should be delete.
>
> *Second*
> {code:java}
> PlacementSpec => "" | KeyVal;PlacementSpec
> {code}
> The semicolon should be replaced by colon
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9710) [UI2] Yarn Daemon Logs displays the URL instead of log name
Prabhu Joseph created YARN-9710: --- Summary: [UI2] Yarn Daemon Logs displays the URL instead of log name Key: YARN-9710 URL: https://issues.apache.org/jira/browse/YARN-9710 Project: Hadoop YARN Issue Type: Bug Components: yarn-ui-v2 Affects Versions: 3.2.0 Reporter: Prabhu Joseph Attachments: Screen Shot 2019-07-26 at 8.53.50 PM.png [UI2] Yarn Daemon Logs displays the URL instead of log name !Screen Shot 2019-07-26 at 8.53.50 PM.png|height=150|width=300! -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9710) [UI2] Yarn Daemon Logs displays the URL instead of log name
[ https://issues.apache.org/jira/browse/YARN-9710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated YARN-9710: Description: Yarn Daemon Logs displays the URL instead of log name. !Screen Shot 2019-07-26 at 8.53.50 PM.png|height=300! was: [UI2] Yarn Daemon Logs displays the URL instead of log name !Screen Shot 2019-07-26 at 8.53.50 PM.png|height=300|width=500! > [UI2] Yarn Daemon Logs displays the URL instead of log name > --- > > Key: YARN-9710 > URL: https://issues.apache.org/jira/browse/YARN-9710 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Priority: Minor > Attachments: Screen Shot 2019-07-26 at 8.53.50 PM.png > > > Yarn Daemon Logs displays the URL instead of log name. > !Screen Shot 2019-07-26 at 8.53.50 PM.png|height=300! -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9710) [UI2] Yarn Daemon Logs displays the URL instead of log name
[ https://issues.apache.org/jira/browse/YARN-9710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph reassigned YARN-9710: --- Assignee: Prabhu Joseph > [UI2] Yarn Daemon Logs displays the URL instead of log name > --- > > Key: YARN-9710 > URL: https://issues.apache.org/jira/browse/YARN-9710 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Minor > Attachments: Screen Shot 2019-07-26 at 8.53.50 PM.png > > > Yarn Daemon Logs displays the URL instead of log name. > !Screen Shot 2019-07-26 at 8.53.50 PM.png|height=300! -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9710) [UI2] Yarn Daemon Logs displays the URL instead of log name
[ https://issues.apache.org/jira/browse/YARN-9710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated YARN-9710: Description: [UI2] Yarn Daemon Logs displays the URL instead of log name !Screen Shot 2019-07-26 at 8.53.50 PM.png|height=300|width=500! was: [UI2] Yarn Daemon Logs displays the URL instead of log name !Screen Shot 2019-07-26 at 8.53.50 PM.png|height=150|width=300! > [UI2] Yarn Daemon Logs displays the URL instead of log name > --- > > Key: YARN-9710 > URL: https://issues.apache.org/jira/browse/YARN-9710 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Priority: Minor > Attachments: Screen Shot 2019-07-26 at 8.53.50 PM.png > > > [UI2] Yarn Daemon Logs displays the URL instead of log name > !Screen Shot 2019-07-26 at 8.53.50 PM.png|height=300|width=500! -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9579) the property of sharedcache in mapred-default.xml
[
https://issues.apache.org/jira/browse/YARN-9579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893873#comment-16893873
]
Hadoop QA commented on YARN-9579:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
41s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
28m 38s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 52s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m
18s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-848/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/848 |
| JIRA Issue | YARN-9579 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux 4d46d27e8af9 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / c0a0c35 |
| Default Java | 1.8.0_212 |
| Test Results |
https://builds.apache.org/job/hadoop-multibranch/job/PR-848/3/testReport/ |
| Max. process+thread count | 1620 (vs. ulimit of 5500) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-848/3/console |
| versions | git=2.7.4 maven=3.3.9 |
| Powered by | Apache Yetus 0.10.0 http://yetus.apache.org |
This message was automatically generated.
> the property of sharedcache in mapred-default.xml
> -
>
> Key: YARN-9579
> URL: https://issues.apache.org/jira/browse/YARN-9579
> Project: Hadoop
[jira] [Commented] (YARN-9601) Potential NPE in ZookeeperFederationStateStore#getPoliciesConfigurations
[
https://issues.apache.org/jira/browse/YARN-9601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893870#comment-16893870
]
Hadoop QA commented on YARN-9601:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
3s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
16s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 50s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
53s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 57m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-908/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/908 |
| JIRA Issue | YARN-9601 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 19d3ebdf9e02 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / c0a0c35 |
| Default Java | 1.8.0_212 |
| Test Results |
https://builds.apache.org/job/hadoop-multibranch/job/PR-908/3/testReport/ |
| Max. process+thread count | 412 (vs. ulimit of 5500) |
| modules | C:
[jira] [Comment Edited] (YARN-9690) Invalid AMRM token when distributed scheduling is enabled.
[
https://issues.apache.org/jira/browse/YARN-9690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16890883#comment-16890883
]
Babble Shack edited comment on YARN-9690 at 7/26/19 2:30 PM:
-
[~bibinchundatt]
I have tried with
{code:java}
yarn.resourcemanager.scheduler.address
0.0.0.0:8049
{code}
+I also tried with your suggestion (localhost instead of 0.0.0.0).+
but that did not work either. I have attached an additional log
The stack trace is different with the localhost modification, however the cause
is still "Invalid AMRMToken from appattempt_1563878096961_0001_01"
{code:java}
2019-07-23 10:35:21,517 INFO [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 40% of the mappers
will be scheduled using OPPORTUNISTIC containers
2019-07-23 10:35:21,530 INFO [main] org.apache.hadoop.yarn.client.RMProxy:
Connecting to ResourceManager at localhost/127.0.0.1:8049
2019-07-23 10:35:21,568 ERROR [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Exception while
registering
org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid AMRMToken
from appattempt_1563878096961_0001_02
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
at
org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
at
org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
at
org.apache.hadoop.yarn.server.api.impl.pb.client.DistributedSchedulingAMProtocolPBClientImpl.registerApplicationMasterForDistributedScheduling(DistributedSchedulingAMProtocolPBClientImpl.java:90)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at
com.sun.proxy.$Proxy79.registerApplicationMasterForDistributedScheduling(Unknown
Source)
at
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.DefaultRequestInterceptor.registerApplicationMasterForDistributedScheduling(DefaultRequestInterceptor.java:154)
at
org.apache.hadoop.yarn.server.nodemanager.scheduler.DistributedScheduler.registerApplicationMasterForDistributedScheduling(DistributedScheduler.java:214)
at
org.apache.hadoop.yarn.server.nodemanager.scheduler.DistributedScheduler.registerApplicationMaster(DistributedScheduler.java:133)
at
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyService.registerApplicationMaster(AMRMProxyService.java:295)
at
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.registerApplicationMaster(ApplicationMasterProtocolPBServiceImpl.java:90)
at
org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:95)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
Invalid AMRMToken from appattempt_1563878096961_0001_02
{code}
[^distributed_log]
was (Author: babbleshack):
[~bibinchundatt]
I have tried with
{code:java}
yarn.resourcemanager.scheduler.address
0.0.0.0:8049
{code}
I also tried with your suggestion (localhost instead of 0.0.0.0).
but that did not work either. I have attached an additional log
The stack trace is different with the
[jira] [Updated] (YARN-9690) Invalid AMRM token when distributed scheduling is enabled.
[
https://issues.apache.org/jira/browse/YARN-9690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Babble Shack updated YARN-9690:
---
Description:
Applications fail to start due to invalild AMRM from application attempt.
I have tested this with 0/100% opportunistic maps and the same issue occurs
regardless.
{code:java}
-->
mapreduceyarn.nodemanager.aux-services
mapreduce_shuffle
yarn.resourcemanager.address
yarn-master-0.yarn-service.yarn:8032
yarn.resourcemanager.scheduler.address
0.0.0.0:8049
yarn.resourcemanager.opportunistic-container-allocation.enabled
true
yarn.nodemanager.opportunistic-containers-max-queue-length
10
yarn.nodemanager.distributed-scheduling.enabled
true
yarn.webapp.ui2.enable
true
yarn.resourcemanager.resource-tracker.address
yarn-master-0.yarn-service.yarn:8031
yarn.log-aggregation-enable
true
yarn.nodemanager.aux-services
mapreduce_shuffle
yarn.nodemanager.resource.memory-mb
7168
yarn.scheduler.minimum-allocation-mb
3584
yarn.scheduler.maximum-allocation-mb
7168
yarn.app.mapreduce.am.resource.mb
7168
yarn.app.mapreduce.am.command-opts
-Xmx5734m
yarn.timeline-service.enabled
true
yarn.resourcemanager.system-metrics-publisher.enabled
true
yarn.timeline-service.generic-application-history.enabled
true
yarn.timeline-service.bind-host
0.0.0.0
{code}
Relevant logs:
{code:java}
2019-07-22 14:56:37,104 INFO [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 100% of the mappers
will be scheduled using OPPORTUNISTIC containers
2019-07-22 14:56:37,117 INFO [main] org.apache.hadoop.yarn.client.RMProxy:
Connecting to ResourceManager at
yarn-master-0.yarn-service.yarn/10.244.1.134:8030
2019-07-22 14:56:37,150 WARN [main] org.apache.hadoop.ipc.Client: Exception
encountered while connecting to the server :
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
Invalid AMRMToken from appattempt_1563805140414_0002_02
2019-07-22 14:56:37,152 ERROR [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Exception while
registering
org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid AMRMToken
from appattempt_1563805140414_0002_02
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
at
org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
at
org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy82.registerApplicationMaster(Unknown Source)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:160)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:121)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:274)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:959)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1272)
at
[jira] [Commented] (YARN-7621) Support submitting apps with queue path for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-7621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893856#comment-16893856 ] Weiwei Yang commented on YARN-7621: --- [~cane], could you pls help to review [~Tao Yang]'s patch? Just want to cross check. Thanks > Support submitting apps with queue path for CapacityScheduler > - > > Key: YARN-7621 > URL: https://issues.apache.org/jira/browse/YARN-7621 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Labels: fs2cs > Attachments: YARN-7621.001.patch, YARN-7621.002.patch > > > Currently there is a difference of queue definition in > ApplicationSubmissionContext between CapacityScheduler and FairScheduler. > FairScheduler needs queue path but CapacityScheduler needs queue name. There > is no doubt of the correction of queue definition for CapacityScheduler > because it does not allow duplicate leaf queue names, but it's hard to switch > between FairScheduler and CapacityScheduler. I propose to support submitting > apps with queue path for CapacityScheduler to make the interface clearer and > scheduler switch smoothly. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8045) Reduce log output from container status calls
[
https://issues.apache.org/jira/browse/YARN-8045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893841#comment-16893841
]
Jim Brennan commented on YARN-8045:
---
Would really like to see this pulled back to 2.8 - it looks like it will be
clean.
> Reduce log output from container status calls
> -
>
> Key: YARN-8045
> URL: https://issues.apache.org/jira/browse/YARN-8045
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Shane Kumpf
>Assignee: Craig Condit
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: YARN-8045.001.patch
>
>
> Each time a container's status is returned a log entry is produced in the NM
> from {{ContainerManagerImpl}}. The container status includes the diagnostics
> field for the container. If the diagnostics field contains an exception, it
> can appear as if the exception is logged repeatedly every second. The
> diagnostics message can also span many lines, which puts pressure on the logs
> and makes it harder to read.
> For example:
> {code}
> 2018-03-17 22:01:11,632 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Getting container-status for container_e01_1521323860653_0001_01_05
> 2018-03-17 22:01:11,632 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Returning ContainerStatus: [ContainerId:
> container_e01_1521323860653_0001_01_05, ExecutionType: GUARANTEED, State:
> RUNNING, Capability: , Diagnostics: [2018-03-17
> 22:01:00.675]Exception from container-launch.
> Container id: container_e01_1521323860653_0001_01_05
> Exit code: -1
> Exception message:
> Shell ouput:
> [2018-03-17 22:01:00.750]Diagnostic message from attempt :
> [2018-03-17 22:01:00.750]Container exited with a non-zero exit code -1.
> , ExitStatus: -1, IP: null, Host: null, ContainerSubState: SCHEDULED]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9698) [Umbrella] Tools to help migration from Fair Scheduler to Capacity Scheduler
[ https://issues.apache.org/jira/browse/YARN-9698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893792#comment-16893792 ] Tan, Wangda commented on YARN-9698: --- [~cane] , is the feature you mentioned supported by FairScheduler? Or it is just a new feature. > [Umbrella] Tools to help migration from Fair Scheduler to Capacity Scheduler > > > Key: YARN-9698 > URL: https://issues.apache.org/jira/browse/YARN-9698 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Reporter: Weiwei Yang >Priority: Major > Labels: fs2cs > > We see some users want to migrate from Fair Scheduler to Capacity Scheduler, > this Jira is created as an umbrella to track all related efforts for the > migration, the scope contains > * Bug fixes > * Add missing features > * Migration tools that help to generate CS configs based on FS, validate > configs etc > * Documents > this is part of CS component, the purpose is to make the migration process > smooth. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9698) [Umbrella] Tools to help migration from Fair Scheduler to Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-9698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893744#comment-16893744
]
zhoukang edited comment on YARN-9698 at 7/26/19 11:37 AM:
--
How about the idea that support ACL proxy user in CapacityScheduler.
Explanation: submit an app use command as below
{code:java}
./bin/yarn jar ~/workspace/yarn/Test3.1/wordcount-1.0-SNAPSHOT.jar
com.xxx.xxx.WordCount -Dmapred.job.queue.name=user@default /tmp/input.dat
/tmp/output
{code}
[~cheersyang]
was (Author: cane):
How about the idea that support ACL proxy user in CapacityScheduler.
Explanation: submit an app use command below
{code:java}
./bin/yarn jar ~/workspace/yarn/Test3.1/wordcount-1.0-SNAPSHOT.jar
com.xxx.xxx.WordCount -Dmapred.job.queue.name=user@default /tmp/input.dat
/tmp/output
{code}
[~cheersyang]
> [Umbrella] Tools to help migration from Fair Scheduler to Capacity Scheduler
>
>
> Key: YARN-9698
> URL: https://issues.apache.org/jira/browse/YARN-9698
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler
>Reporter: Weiwei Yang
>Priority: Major
> Labels: fs2cs
>
> We see some users want to migrate from Fair Scheduler to Capacity Scheduler,
> this Jira is created as an umbrella to track all related efforts for the
> migration, the scope contains
> * Bug fixes
> * Add missing features
> * Migration tools that help to generate CS configs based on FS, validate
> configs etc
> * Documents
> this is part of CS component, the purpose is to make the migration process
> smooth.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9698) [Umbrella] Tools to help migration from Fair Scheduler to Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-9698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893744#comment-16893744
]
zhoukang commented on YARN-9698:
How about the idea that support ACL proxy user in CapacityScheduler.
Explanation: submit an app use command below
{code:java}
./bin/yarn jar ~/workspace/yarn/Test3.1/wordcount-1.0-SNAPSHOT.jar
com.xxx.xxx.WordCount -Dmapred.job.queue.name=user@default /tmp/input.dat
/tmp/output
{code}
[~cheersyang]
> [Umbrella] Tools to help migration from Fair Scheduler to Capacity Scheduler
>
>
> Key: YARN-9698
> URL: https://issues.apache.org/jira/browse/YARN-9698
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler
>Reporter: Weiwei Yang
>Priority: Major
> Labels: fs2cs
>
> We see some users want to migrate from Fair Scheduler to Capacity Scheduler,
> this Jira is created as an umbrella to track all related efforts for the
> migration, the scope contains
> * Bug fixes
> * Add missing features
> * Migration tools that help to generate CS configs based on FS, validate
> configs etc
> * Documents
> this is part of CS component, the purpose is to make the migration process
> smooth.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9709) When we expanding queue list the scheduler page will not show any applications
[ https://issues.apache.org/jira/browse/YARN-9709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated YARN-9709: --- Attachment: (was: list2.png) > When we expanding queue list the scheduler page will not show any applications > -- > > Key: YARN-9709 > URL: https://issues.apache.org/jira/browse/YARN-9709 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Affects Versions: 3.1.2 >Reporter: zhoukang >Priority: Major > Attachments: list1.png, list3.png > > > When we expanding queue list the scheduler page will not show any > applications.But it works well in FairScheduler. > !list1.png! !list2.png! -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9709) When we expanding queue list the scheduler page will not show any applications
[ https://issues.apache.org/jira/browse/YARN-9709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated YARN-9709: --- Description: When we expanding queue list the scheduler page will not show any applications.But it works well in FairScheduler. !list1.png! !list3.png! was: When we expanding queue list the scheduler page will not show any applications.But it works well in FairScheduler. !list1.png! !list2.png! > When we expanding queue list the scheduler page will not show any applications > -- > > Key: YARN-9709 > URL: https://issues.apache.org/jira/browse/YARN-9709 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Affects Versions: 3.1.2 >Reporter: zhoukang >Priority: Major > Attachments: list1.png, list3.png > > > When we expanding queue list the scheduler page will not show any > applications.But it works well in FairScheduler. > !list1.png! > !list3.png! -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9709) When we expanding queue list the scheduler page will not show any applications
[ https://issues.apache.org/jira/browse/YARN-9709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated YARN-9709: --- Attachment: list3.png > When we expanding queue list the scheduler page will not show any applications > -- > > Key: YARN-9709 > URL: https://issues.apache.org/jira/browse/YARN-9709 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Affects Versions: 3.1.2 >Reporter: zhoukang >Priority: Major > Attachments: list1.png, list3.png > > > When we expanding queue list the scheduler page will not show any > applications.But it works well in FairScheduler. > !list1.png! !list2.png! -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9709) When we expanding queue list the scheduler page will not show any applications
zhoukang created YARN-9709: -- Summary: When we expanding queue list the scheduler page will not show any applications Key: YARN-9709 URL: https://issues.apache.org/jira/browse/YARN-9709 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Affects Versions: 3.1.2 Reporter: zhoukang Attachments: list1.png, list2.png When we expanding queue list the scheduler page will not show any applications.But it works well in FairScheduler. !list1.png! !list2.png! -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7621) Support submitting apps with queue path for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-7621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893728#comment-16893728 ] zhoukang commented on YARN-7621: Good job. We also encountered the same problem and have made some compatibility > Support submitting apps with queue path for CapacityScheduler > - > > Key: YARN-7621 > URL: https://issues.apache.org/jira/browse/YARN-7621 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Labels: fs2cs > Attachments: YARN-7621.001.patch, YARN-7621.002.patch > > > Currently there is a difference of queue definition in > ApplicationSubmissionContext between CapacityScheduler and FairScheduler. > FairScheduler needs queue path but CapacityScheduler needs queue name. There > is no doubt of the correction of queue definition for CapacityScheduler > because it does not allow duplicate leaf queue names, but it's hard to switch > between FairScheduler and CapacityScheduler. I propose to support submitting > apps with queue path for CapacityScheduler to make the interface clearer and > scheduler switch smoothly. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9708) Add Yarnclient#getDelegationToken API implementation and SecureLogin in router
Xie YiFan created YARN-9708: --- Summary: Add Yarnclient#getDelegationToken API implementation and SecureLogin in router Key: YARN-9708 URL: https://issues.apache.org/jira/browse/YARN-9708 Project: Hadoop YARN Issue Type: New Feature Components: router Affects Versions: 3.1.1 Reporter: Xie YiFan Attachments: Add_getDelegationToken_and_SecureLogin_in_router.patch 1.we use router as proxy to manage multiple cluster which be independent of each other in order to apply unified client. Thus, we implement our customized AMRMProxyPolicy that doesn't broadcast ResourceRequest to other cluster. 2.Our production environment need kerberos. But router doesn't support SecureLogin for now. https://issues.apache.org/jira/browse/YARN-6539 desn't work. So we improvement it. 3.Some framework like oozie would get Token via yarnclient#getDelegationToken which router doesn't support. Our solution is that adding homeCluster to ApplicationSubmissionContextProto & GetDelegationTokenRequestProto. Job would be submitted with specified clusterid so that router knows which cluster to submit this job. Router would get Token from one RM according to specified clusterid when client call getDelegation meanwhile apply some mechanism to save this token in memory. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9707) [UI2] App Attempt state data is missing
[
https://issues.apache.org/jira/browse/YARN-9707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth reassigned YARN-9707:
Assignee: Zoltan Siegl
> [UI2] App Attempt state data is missing
> ---
>
> Key: YARN-9707
> URL: https://issues.apache.org/jira/browse/YARN-9707
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Zoltan Siegl
>Priority: Critical
>
> Steps:
> 1) Launch a Dshell application or Yarn service application.
> 2) Go to app attempt page Grid view. State column shows N/A.
> Yarn UI1 shows app attempt state for Running and Finished application. This
> ability is missing from UI2.
> UI2 is using below rest call. This rest call does not show the app attempt
> state details.
> {code:title=ws/v1/cluster/apps/application_1563946396350_0002/appattempts?_=1564004553389}
> 115640045242901564004541852container_1563946396350_0002_01_01xx:yyxx:yyhttp://ixx:yy/node/containerlogs/container_1563946396350_0002_01_01/hrt_qaappattempt_1563946396350_0002_01{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9704) [UI2] Fix Pending, Allocated, Reserved Containers information for Fair Scheduler
[ https://issues.apache.org/jira/browse/YARN-9704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth reassigned YARN-9704: Assignee: Zoltan Siegl > [UI2] Fix Pending, Allocated, Reserved Containers information for Fair > Scheduler > > > Key: YARN-9704 > URL: https://issues.apache.org/jira/browse/YARN-9704 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Yesha Vora >Assignee: Zoltan Siegl >Priority: Critical > Attachments: Screenshot 2019-07-25 at 10.51.08 PM.png > > > YARN UI2 shows "Pending, Allocated, Reserved Containers" information for fair > scheduler. In here, pending container information is not printed. UI2 shows > ",0,0" instead "0,0,0". > In UI1, This same information is displayed as Num of active Application & > Pending applications. > Num Active Applications: 0 > Num Pending Applications: 0 > It's not clear from UI2 what do we intend to show in "Pending, Allocated, > Reserved Containers"? Is it really containers or apps? -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9706) [UI2] App Attempt state missing from Graph view
[ https://issues.apache.org/jira/browse/YARN-9706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth reassigned YARN-9706: Assignee: Zoltan Siegl > [UI2] App Attempt state missing from Graph view > --- > > Key: YARN-9706 > URL: https://issues.apache.org/jira/browse/YARN-9706 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Yesha Vora >Assignee: Zoltan Siegl >Priority: Major > > 1) Launch a Dshell application or Yarn service application. > 2) Go to app attempt page Grid view. State column shows N/A. > 3) Go to app attempt Graph view. State data is not present on this page. > Apparently, app attempt data is only shown in Grid view. Grid and Graph view > should show the same details. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9705) [UI2] AM Node Web UI should not display full link
[ https://issues.apache.org/jira/browse/YARN-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth reassigned YARN-9705: Assignee: Zoltan Siegl > [UI2] AM Node Web UI should not display full link > - > > Key: YARN-9705 > URL: https://issues.apache.org/jira/browse/YARN-9705 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Yesha Vora >Assignee: Zoltan Siegl >Priority: Major > > App Attempt page shows AM Node Web UI. It displays the full link. > It should not print out full link as display text. Rather, It should use > display AM Node name which links to the node. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9681) AM resource limit is incorrect for queue
[
https://issues.apache.org/jira/browse/YARN-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893408#comment-16893408
]
Hadoop QA commented on YARN-9681:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 30s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 43 unchanged - 2 fixed = 43 total (was 45) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 35s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 81m
39s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}132m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9681 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12975926/YARN-9681.0003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 3b7c3ab53f8f 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ce99cc3 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24429/testReport/ |
| Max. process+thread count | 926 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24429/console |
| Powered by | Apache
[jira] [Updated] (YARN-8468) Enable the use of queue based maximum container allocation limit and implement it in FairScheduler
[ https://issues.apache.org/jira/browse/YARN-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-8468: - Description: When using any scheduler, you can use "yarn.scheduler.maximum-allocation-mb" to limit the overall size of a container. This applies globally to all containers and cannot be limited by queue or and is not scheduler dependent. The goal of this ticket is to allow this value to be set on a per queue basis. The use case: User has two pools, one for ad hoc jobs and one for enterprise apps. User wants to limit ad hoc jobs to small containers but allow enterprise apps to request as many resources as needed. Setting yarn.scheduler.maximum-allocation-mb sets a default value for maximum container size for all queues and setting maximum resources per queue with “maxContainerResources” queue config value. NOTE: "maxContainerResources" has been changed to "maxContainerAllocation" as per the review comments. Suggested solution: All the infrastructure is already in the code. We need to do the following: * add the setting to the queue properties for all queue types (parent and leaf), this will cover dynamically created queues. * if we set it on the root we override the scheduler setting and we should not allow that. * make sure that queue resource cap can not be larger than scheduler max resource cap in the config. * implement getMaximumResourceCapability(String queueName) in the FairScheduler * implement getMaximumResourceCapability(String queueName) in both FSParentQueue and FSLeafQueue as follows * expose the setting in the queue information in the RM web UI. * expose the setting in the metrics etc for the queue. * Enforce the use of queue based maximum allocation limit if it is available, if not use the general scheduler level setting ** Use it during validation and normalization of requests in scheduler.allocate, app submit and resource request was: When using any scheduler, you can use "yarn.scheduler.maximum-allocation-mb" to limit the overall size of a container. This applies globally to all containers and cannot be limited by queue or and is not scheduler dependent. The goal of this ticket is to allow this value to be set on a per queue basis. The use case: User has two pools, one for ad hoc jobs and one for enterprise apps. User wants to limit ad hoc jobs to small containers but allow enterprise apps to request as many resources as needed. Setting yarn.scheduler.maximum-allocation-mb sets a default value for maximum container size for all queues and setting maximum resources per queue with “maxContainerResources” queue config value. Suggested solution: All the infrastructure is already in the code. We need to do the following: * add the setting to the queue properties for all queue types (parent and leaf), this will cover dynamically created queues. * if we set it on the root we override the scheduler setting and we should not allow that. * make sure that queue resource cap can not be larger than scheduler max resource cap in the config. * implement getMaximumResourceCapability(String queueName) in the FairScheduler * implement getMaximumResourceCapability(String queueName) in both FSParentQueue and FSLeafQueue as follows * expose the setting in the queue information in the RM web UI. * expose the setting in the metrics etc for the queue. * Enforce the use of queue based maximum allocation limit if it is available, if not use the general scheduler level setting ** Use it during validation and normalization of requests in scheduler.allocate, app submit and resource request > Enable the use of queue based maximum container allocation limit and > implement it in FairScheduler > -- > > Key: YARN-8468 > URL: https://issues.apache.org/jira/browse/YARN-8468 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler, scheduler >Affects Versions: 3.1.0 >Reporter: Antal Bálint Steinbach >Assignee: Antal Bálint Steinbach >Priority: Critical > Fix For: 3.2.0 > > Attachments: YARN-8468-branch-3.1.018.patch, > YARN-8468-branch-3.1.019.patch, YARN-8468-branch-3.1.020.patch, > YARN-8468-branch-3.1.021.patch, YARN-8468-branch-3.1.022.patch, > YARN-8468.000.patch, YARN-8468.001.patch, YARN-8468.002.patch, > YARN-8468.003.patch, YARN-8468.004.patch, YARN-8468.005.patch, > YARN-8468.006.patch, YARN-8468.007.patch, YARN-8468.008.patch, > YARN-8468.009.patch, YARN-8468.010.patch, YARN-8468.011.patch, > YARN-8468.012.patch, YARN-8468.013.patch, YARN-8468.014.patch, > YARN-8468.015.patch, YARN-8468.016.patch, YARN-8468.017.patch, > YARN-8468.018.patch > > > When using any scheduler, you can use
[jira] [Created] (YARN-9706) [UI2] App Attempt state missing from Graph view
Yesha Vora created YARN-9706: Summary: [UI2] App Attempt state missing from Graph view Key: YARN-9706 URL: https://issues.apache.org/jira/browse/YARN-9706 Project: Hadoop YARN Issue Type: Bug Reporter: Yesha Vora 1) Launch a Dshell application or Yarn service application. 2) Go to app attempt page Grid view. State column shows N/A. 3) Go to app attempt Graph view. State data is not present on this page. Apparently, app attempt data is only shown in Grid view. Grid and Graph view should show the same details. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9707) [UI2] App Attempt state data is missing
Yesha Vora created YARN-9707:
Summary: [UI2] App Attempt state data is missing
Key: YARN-9707
URL: https://issues.apache.org/jira/browse/YARN-9707
Project: Hadoop YARN
Issue Type: Bug
Reporter: Yesha Vora
Steps:
1) Launch a Dshell application or Yarn service application.
2) Go to app attempt page Grid view. State column shows N/A.
Yarn UI1 shows app attempt state for Running and Finished application. This
ability is missing from UI2.
UI2 is using below rest call. This rest call does not show the app attempt
state details.
{code:title=ws/v1/cluster/apps/application_1563946396350_0002/appattempts?_=1564004553389}
115640045242901564004541852container_1563946396350_0002_01_01xx:yyxx:yyhttp://ixx:yy/node/containerlogs/container_1563946396350_0002_01_01/hrt_qaappattempt_1563946396350_0002_01{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
