[jira] [Commented] (YARN-9690) Invalid AMRM token when distributed scheduling is enabled.
[ https://issues.apache.org/jira/browse/YARN-9690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16896678#comment-16896678 ] Abhishek Modi commented on YARN-9690: - Thanks [~Babbleshack]. I will help review that PR. > Invalid AMRM token when distributed scheduling is enabled. > -- > > Key: YARN-9690 > URL: https://issues.apache.org/jira/browse/YARN-9690 > Project: Hadoop YARN > Issue Type: Bug > Components: distributed-scheduling, yarn >Affects Versions: 2.9.2, 3.1.2 > Environment: OS: Ubuntu 18.04 > JVM: 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03 >Reporter: Babble Shack >Assignee: Abhishek Modi >Priority: Major > Attachments: applicationlog, distributed_log, ds_application.log, > image-2019-07-26-18-00-14-980.png, nodemanager-yarn-site.xml, > nodemanager.log, rm-yarn-site.xml, yarn-site.xml > > > Applications fail to start due to invalild AMRM from application attempt. > I have tested this with 0/100% opportunistic maps and the same issue occurs > regardless. > {code:java} > > --> > > > mapreduceyarn.nodemanager.aux-services > mapreduce_shuffle > > > yarn.resourcemanager.address > yarn-master-0.yarn-service.yarn:8032 > > > yarn.resourcemanager.scheduler.address > 0.0.0.0:8049 > > > > yarn.resourcemanager.opportunistic-container-allocation.enabled > true > > > yarn.nodemanager.opportunistic-containers-max-queue-length > 10 > > > yarn.nodemanager.distributed-scheduling.enabled > true > > > > yarn.webapp.ui2.enable > true > > > yarn.resourcemanager.resource-tracker.address > yarn-master-0.yarn-service.yarn:8031 > > > yarn.log-aggregation-enable > true > > > yarn.nodemanager.aux-services > mapreduce_shuffle > > > > > > yarn.nodemanager.resource.memory-mb > 7168 > > > yarn.scheduler.minimum-allocation-mb > 3584 > > > yarn.scheduler.maximum-allocation-mb > 7168 > > > yarn.app.mapreduce.am.resource.mb > 7168 > > > > yarn.app.mapreduce.am.command-opts > -Xmx5734m > > > > yarn.timeline-service.enabled > true > > > yarn.resourcemanager.system-metrics-publisher.enabled > true > > > yarn.timeline-service.generic-application-history.enabled > true > > > yarn.timeline-service.bind-host > 0.0.0.0 > > > {code} > Relevant logs: > {code:java} > 2019-07-22 14:56:37,104 INFO [main] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 100% of the > mappers will be scheduled using OPPORTUNISTIC containers > 2019-07-22 14:56:37,117 INFO [main] org.apache.hadoop.yarn.client.RMProxy: > Connecting to ResourceManager at > yarn-master-0.yarn-service.yarn/10.244.1.134:8030 > 2019-07-22 14:56:37,150 WARN [main] org.apache.hadoop.ipc.Client: Exception > encountered while connecting to the server : > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): > Invalid AMRMToken from appattempt_1563805140414_0002_02 > 2019-07-22 14:56:37,152 ERROR [main] > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Exception while > registering > org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid > AMRMToken from appattempt_1563805140414_0002_02 > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80) > at > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) > at > org.apache.hadoop.io.retry.RetryInvocati
[jira] [Commented] (YARN-9690) Invalid AMRM token when distributed scheduling is enabled.
[ https://issues.apache.org/jira/browse/YARN-9690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16896660#comment-16896660 ] Babble Shack commented on YARN-9690: Thank you [~abmodi] I found these setting earlier yesterday and was able to configure the cluster. I will make a PR for the documentation explicitly identifying the hostname parameter should be set in yarn-config.xml > Invalid AMRM token when distributed scheduling is enabled. > -- > > Key: YARN-9690 > URL: https://issues.apache.org/jira/browse/YARN-9690 > Project: Hadoop YARN > Issue Type: Bug > Components: distributed-scheduling, yarn >Affects Versions: 2.9.2, 3.1.2 > Environment: OS: Ubuntu 18.04 > JVM: 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03 >Reporter: Babble Shack >Assignee: Abhishek Modi >Priority: Major > Attachments: applicationlog, distributed_log, ds_application.log, > image-2019-07-26-18-00-14-980.png, nodemanager-yarn-site.xml, > nodemanager.log, rm-yarn-site.xml, yarn-site.xml > > > Applications fail to start due to invalild AMRM from application attempt. > I have tested this with 0/100% opportunistic maps and the same issue occurs > regardless. > {code:java} > > --> > > > mapreduceyarn.nodemanager.aux-services > mapreduce_shuffle > > > yarn.resourcemanager.address > yarn-master-0.yarn-service.yarn:8032 > > > yarn.resourcemanager.scheduler.address > 0.0.0.0:8049 > > > > yarn.resourcemanager.opportunistic-container-allocation.enabled > true > > > yarn.nodemanager.opportunistic-containers-max-queue-length > 10 > > > yarn.nodemanager.distributed-scheduling.enabled > true > > > > yarn.webapp.ui2.enable > true > > > yarn.resourcemanager.resource-tracker.address > yarn-master-0.yarn-service.yarn:8031 > > > yarn.log-aggregation-enable > true > > > yarn.nodemanager.aux-services > mapreduce_shuffle > > > > > > yarn.nodemanager.resource.memory-mb > 7168 > > > yarn.scheduler.minimum-allocation-mb > 3584 > > > yarn.scheduler.maximum-allocation-mb > 7168 > > > yarn.app.mapreduce.am.resource.mb > 7168 > > > > yarn.app.mapreduce.am.command-opts > -Xmx5734m > > > > yarn.timeline-service.enabled > true > > > yarn.resourcemanager.system-metrics-publisher.enabled > true > > > yarn.timeline-service.generic-application-history.enabled > true > > > yarn.timeline-service.bind-host > 0.0.0.0 > > > {code} > Relevant logs: > {code:java} > 2019-07-22 14:56:37,104 INFO [main] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 100% of the > mappers will be scheduled using OPPORTUNISTIC containers > 2019-07-22 14:56:37,117 INFO [main] org.apache.hadoop.yarn.client.RMProxy: > Connecting to ResourceManager at > yarn-master-0.yarn-service.yarn/10.244.1.134:8030 > 2019-07-22 14:56:37,150 WARN [main] org.apache.hadoop.ipc.Client: Exception > encountered while connecting to the server : > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): > Invalid AMRMToken from appattempt_1563805140414_0002_02 > 2019-07-22 14:56:37,152 ERROR [main] > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Exception while > registering > org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid > AMRMToken from appattempt_1563805140414_0002_02 > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80) > at > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422
[jira] [Commented] (YARN-9690) Invalid AMRM token when distributed scheduling is enabled.
[ https://issues.apache.org/jira/browse/YARN-9690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16896659#comment-16896659 ] Abhishek Modi commented on YARN-9690: - [~Babbleshack] thanks for filing the issue. Could you please try by setting this in yarn-site.xml of both RM and NM: yarn.resourcemanager.hostname to yarn-master-0.yarn-service.yarn and then you can remove following configs: yarn.resourcemanager.resource-tracker.address, yarn.resourcemanager.address. > Invalid AMRM token when distributed scheduling is enabled. > -- > > Key: YARN-9690 > URL: https://issues.apache.org/jira/browse/YARN-9690 > Project: Hadoop YARN > Issue Type: Bug > Components: distributed-scheduling, yarn >Affects Versions: 2.9.2, 3.1.2 > Environment: OS: Ubuntu 18.04 > JVM: 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03 >Reporter: Babble Shack >Priority: Major > Attachments: applicationlog, distributed_log, ds_application.log, > image-2019-07-26-18-00-14-980.png, nodemanager-yarn-site.xml, > nodemanager.log, rm-yarn-site.xml, yarn-site.xml > > > Applications fail to start due to invalild AMRM from application attempt. > I have tested this with 0/100% opportunistic maps and the same issue occurs > regardless. > {code:java} > > --> > > > mapreduceyarn.nodemanager.aux-services > mapreduce_shuffle > > > yarn.resourcemanager.address > yarn-master-0.yarn-service.yarn:8032 > > > yarn.resourcemanager.scheduler.address > 0.0.0.0:8049 > > > > yarn.resourcemanager.opportunistic-container-allocation.enabled > true > > > yarn.nodemanager.opportunistic-containers-max-queue-length > 10 > > > yarn.nodemanager.distributed-scheduling.enabled > true > > > > yarn.webapp.ui2.enable > true > > > yarn.resourcemanager.resource-tracker.address > yarn-master-0.yarn-service.yarn:8031 > > > yarn.log-aggregation-enable > true > > > yarn.nodemanager.aux-services > mapreduce_shuffle > > > > > > yarn.nodemanager.resource.memory-mb > 7168 > > > yarn.scheduler.minimum-allocation-mb > 3584 > > > yarn.scheduler.maximum-allocation-mb > 7168 > > > yarn.app.mapreduce.am.resource.mb > 7168 > > > > yarn.app.mapreduce.am.command-opts > -Xmx5734m > > > > yarn.timeline-service.enabled > true > > > yarn.resourcemanager.system-metrics-publisher.enabled > true > > > yarn.timeline-service.generic-application-history.enabled > true > > > yarn.timeline-service.bind-host > 0.0.0.0 > > > {code} > Relevant logs: > {code:java} > 2019-07-22 14:56:37,104 INFO [main] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 100% of the > mappers will be scheduled using OPPORTUNISTIC containers > 2019-07-22 14:56:37,117 INFO [main] org.apache.hadoop.yarn.client.RMProxy: > Connecting to ResourceManager at > yarn-master-0.yarn-service.yarn/10.244.1.134:8030 > 2019-07-22 14:56:37,150 WARN [main] org.apache.hadoop.ipc.Client: Exception > encountered while connecting to the server : > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): > Invalid AMRMToken from appattempt_1563805140414_0002_02 > 2019-07-22 14:56:37,152 ERROR [main] > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Exception while > registering > org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid > AMRMToken from appattempt_1563805140414_0002_02 > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80) > at > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.io.retry.RetryInvocationH
[jira] [Commented] (YARN-9690) Invalid AMRM token when distributed scheduling is enabled.
[ https://issues.apache.org/jira/browse/YARN-9690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16894019#comment-16894019 ] Babble Shack commented on YARN-9690: I also tried disabling _yarn.resourcemanager.scheduler_ and using it default value at the resource manager. {code:java} yarn.resourcemanager.scheduler.address localhost:8049 --> {code} I then create a new config directory with the following yarn.resourcemanager.scheduler.address localhost:8049 And launching the job with {code:java} yarn --config /debug/hadoop/etc/hadoop jar $MAPRED_EXAMPLES pi -Dmapreduce.job.num-opportunistic-maps-percent="40" 10 10 {code} The nodemanager then tries to connect with scheduler on 0.0.0.0:8030, however the amrmproxy has already started, and so the amrmproxy should respond by redirecting the request to the Local RM {code:java} 2019-07-26 17:32:25,480 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: container_1564162333195_0001_01_01's ip = 10.244.29.84, and hostname = yarn-worker-2.yarn-service.yarn.svc.cluster.local 2019-07-26 17:32:26,962 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1564162333195_0001_01 (auth:SIMPLE) 2019-07-26 17:32:26,965 INFO org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyService: Registering application master. Host:yarn-worker-2.yarn-service.yarn.svc.cluster.local Port:42172 Tracking Url:http://yarn-worker-2.yarn-service.yarn.svc.cluster.local:37110 2019-07-26 17:32:26,965 INFO org.apache.hadoop.yarn.server.nodemanager.scheduler.DistributedScheduler: Forwarding registration request to theDistributed Scheduler Service on YARN RM 2019-07-26 17:32:26,966 INFO org.apache.hadoop.yarn.server.nodemanager.amrmproxy.DefaultRequestInterceptor: Forwarding registerApplicationMasterForDistributedSchedulingrequest to the real YARN RM 2019-07-26 17:32:27,971 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2019-07-26 17:32:28,485 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Skipping monitoring container container_1564162333195_0001_01_01 since CPU usage is not yet available. 2019-07-26 17:32:28,971 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) {code} However there is no service listening on 0.0.0.0:8030 > Invalid AMRM token when distributed scheduling is enabled. > -- > > Key: YARN-9690 > URL: https://issues.apache.org/jira/browse/YARN-9690 > Project: Hadoop YARN > Issue Type: Bug > Components: distributed-scheduling, yarn >Affects Versions: 2.9.2, 3.1.2 > Environment: OS: Ubuntu 18.04 > JVM: 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03 >Reporter: Babble Shack >Priority: Major > Attachments: applicationlog, distributed_log, ds_application.log, > image-2019-07-26-18-00-14-980.png, nodemanager-yarn-site.xml, > nodemanager.log, rm-yarn-site.xml, yarn-site.xml > > > Applications fail to start due to invalild AMRM from application attempt. > I have tested this with 0/100% opportunistic maps and the same issue occurs > regardless. > {code:java} > > --> > > > mapreduceyarn.nodemanager.aux-services > mapreduce_shuffle > > > yarn.resourcemanager.address > yarn-master-0.yarn-service.yarn:8032 > > > yarn.resourcemanager.scheduler.address > 0.0.0.0:8049 > > > > yarn.resourcemanager.opportunistic-container-allocation.enabled > true > > > yarn.nodemanager.opportunistic-containers-max-queue-length > 10 > > > yarn.nodemanager.distributed-scheduling.enabled > true > > > > yarn.webapp.ui2.enable > true > > > yarn.resourcemanager.resource-tracker.address > yarn-master-0.yarn-service.yarn:8031 > > > yarn.log-aggregation-enable > true > > > yarn.nodemanager.aux-services > mapreduce_shuffle > > > > > > yarn.nodemanager.resource.memory-mb > 7168 > > > yarn.scheduler.minimum-allocation-mb > 3584 > > > yarn.scheduler.maximum-allocation-mb > 7168 > > > yarn.app.mapreduce.am.resource.mb > 7168 > > > > yarn.app.mapreduce.am.command-opts > -Xmx5734m > > > > yarn.timeline-service.enabled > true > > > yarn.resourcemanager.system-metrics-publisher.enabled > true > > > yarn.timeline-service.
[jira] [Commented] (YARN-9690) Invalid AMRM token when distributed scheduling is enabled.
[ https://issues.apache.org/jira/browse/YARN-9690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893981#comment-16893981 ] Babble Shack commented on YARN-9690: Setting `yarn.resourcemanager.scheduler.address` does not throw the Invalid AMRMToken, however the application master is unable to register with the resource manager. I have configured my resource manager, which is also the machine I submit MR jobs from with the following {code:java} yarn.resourcemanager.scheduler.address localhost:8049 {code} {code:java} {code} > Invalid AMRM token when distributed scheduling is enabled. > -- > > Key: YARN-9690 > URL: https://issues.apache.org/jira/browse/YARN-9690 > Project: Hadoop YARN > Issue Type: Bug > Components: distributed-scheduling, yarn >Affects Versions: 2.9.2, 3.1.2 > Environment: OS: Ubuntu 18.04 > JVM: 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03 >Reporter: Babble Shack >Priority: Major > Attachments: applicationlog, distributed_log, yarn-site.xml > > > Applications fail to start due to invalild AMRM from application attempt. > I have tested this with 0/100% opportunistic maps and the same issue occurs > regardless. > {code:java} > > --> > > > mapreduceyarn.nodemanager.aux-services > mapreduce_shuffle > > > yarn.resourcemanager.address > yarn-master-0.yarn-service.yarn:8032 > > > yarn.resourcemanager.scheduler.address > 0.0.0.0:8049 > > > > yarn.resourcemanager.opportunistic-container-allocation.enabled > true > > > yarn.nodemanager.opportunistic-containers-max-queue-length > 10 > > > yarn.nodemanager.distributed-scheduling.enabled > true > > > > yarn.webapp.ui2.enable > true > > > yarn.resourcemanager.resource-tracker.address > yarn-master-0.yarn-service.yarn:8031 > > > yarn.log-aggregation-enable > true > > > yarn.nodemanager.aux-services > mapreduce_shuffle > > > > > > yarn.nodemanager.resource.memory-mb > 7168 > > > yarn.scheduler.minimum-allocation-mb > 3584 > > > yarn.scheduler.maximum-allocation-mb > 7168 > > > yarn.app.mapreduce.am.resource.mb > 7168 > > > > yarn.app.mapreduce.am.command-opts > -Xmx5734m > > > > yarn.timeline-service.enabled > true > > > yarn.resourcemanager.system-metrics-publisher.enabled > true > > > yarn.timeline-service.generic-application-history.enabled > true > > > yarn.timeline-service.bind-host > 0.0.0.0 > > > {code} > Relevant logs: > {code:java} > 2019-07-22 14:56:37,104 INFO [main] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 100% of the > mappers will be scheduled using OPPORTUNISTIC containers > 2019-07-22 14:56:37,117 INFO [main] org.apache.hadoop.yarn.client.RMProxy: > Connecting to ResourceManager at > yarn-master-0.yarn-service.yarn/10.244.1.134:8030 > 2019-07-22 14:56:37,150 WARN [main] org.apache.hadoop.ipc.Client: Exception > encountered while connecting to the server : > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): > Invalid AMRMToken from appattempt_1563805140414_0002_02 > 2019-07-22 14:56:37,152 ERROR [main] > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Exception while > registering > org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid > AMRMToken from appattempt_1563805140414_0002_02 > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80) > at > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHan
[jira] [Commented] (YARN-9690) Invalid AMRM token when distributed scheduling is enabled.
[ https://issues.apache.org/jira/browse/YARN-9690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893032#comment-16893032 ] Babble Shack commented on YARN-9690: [~bibinchundatt] Would you be able to clarify what you mean by "This is client side propery in case of mapreduce application."? > Invalid AMRM token when distributed scheduling is enabled. > -- > > Key: YARN-9690 > URL: https://issues.apache.org/jira/browse/YARN-9690 > Project: Hadoop YARN > Issue Type: Bug > Components: distributed-scheduling, yarn >Affects Versions: 2.9.2, 3.1.2 > Environment: OS: Ubuntu 18.04 > JVM: 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03 >Reporter: Babble Shack >Priority: Major > Attachments: applicationlog, distributed_log, yarn-site.xml > > > Applications fail to start due to invalild AMRM from application attempt. > I have tested this with 0/100% opportunistic maps and the same issue occurs > regardless. > {code:java} > > --> > > > mapreduceyarn.nodemanager.aux-services > mapreduce_shuffle > > > yarn.resourcemanager.address > yarn-master-0.yarn-service.yarn:8032 > > > yarn.resourcemanager.scheduler.address > 0.0.0.0:8049 > > > > yarn.resourcemanager.opportunistic-container-allocation.enabled > true > > > yarn.nodemanager.opportunistic-containers-max-queue-length > 10 > > > yarn.nodemanager.distributed-scheduling.enabled > true > > > > yarn.webapp.ui2.enable > true > > > yarn.resourcemanager.resource-tracker.address > yarn-master-0.yarn-service.yarn:8031 > > > yarn.log-aggregation-enable > true > > > yarn.nodemanager.aux-services > mapreduce_shuffle > > > > > > yarn.nodemanager.resource.memory-mb > 7168 > > > yarn.scheduler.minimum-allocation-mb > 3584 > > > yarn.scheduler.maximum-allocation-mb > 7168 > > > yarn.app.mapreduce.am.resource.mb > 7168 > > > > yarn.app.mapreduce.am.command-opts > -Xmx5734m > > > > yarn.timeline-service.enabled > true > > > yarn.resourcemanager.system-metrics-publisher.enabled > true > > > yarn.timeline-service.generic-application-history.enabled > true > > > yarn.timeline-service.bind-host > 0.0.0.0 > > > {code} > Relevant logs: > {code:java} > 2019-07-22 14:56:37,104 INFO [main] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 100% of the > mappers will be scheduled using OPPORTUNISTIC containers > 2019-07-22 14:56:37,117 INFO [main] org.apache.hadoop.yarn.client.RMProxy: > Connecting to ResourceManager at > yarn-master-0.yarn-service.yarn/10.244.1.134:8030 > 2019-07-22 14:56:37,150 WARN [main] org.apache.hadoop.ipc.Client: Exception > encountered while connecting to the server : > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): > Invalid AMRMToken from appattempt_1563805140414_0002_02 > 2019-07-22 14:56:37,152 ERROR [main] > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Exception while > registering > org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid > AMRMToken from appattempt_1563805140414_0002_02 > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80) > at > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) > at > org.apache.hado
[jira] [Commented] (YARN-9690) Invalid AMRM token when distributed scheduling is enabled.
[ https://issues.apache.org/jira/browse/YARN-9690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890883#comment-16890883 ] Babble Shack commented on YARN-9690: [~bibinchundatt] [^distributed_log] I have tried with {code:java} yarn.resourcemanager.scheduler.address 0.0.0.0:8049 {code} As you can see in my config. I also tried with your suggestion (localhost instead of 0.0.0.0). but that did not work either. I have attached an additional log > Invalid AMRM token when distributed scheduling is enabled. > -- > > Key: YARN-9690 > URL: https://issues.apache.org/jira/browse/YARN-9690 > Project: Hadoop YARN > Issue Type: Bug > Components: distributed-scheduling, yarn >Affects Versions: 2.9.2, 3.1.2 > Environment: OS: Ubuntu 18.04 > JVM: 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03 >Reporter: Babble Shack >Priority: Major > Attachments: applicationlog, distributed_log, yarn-site.xml > > > Applications fail to start due to invalild AMRM from application attempt. > I have tested this with 0/100% opportunistic maps and the same issue occurs > regardless. > {code:java} > > --> > > > mapreduceyarn.nodemanager.aux-services > mapreduce_shuffle > > > yarn.resourcemanager.address > yarn-master-0.yarn-service.yarn:8032 > > > yarn.resourcemanager.scheduler.address > 0.0.0.0:8049 > > > > yarn.resourcemanager.opportunistic-container-allocation.enabled > true > > > yarn.nodemanager.opportunistic-containers-max-queue-length > 10 > > > yarn.nodemanager.distributed-scheduling.enabled > true > > > > yarn.webapp.ui2.enable > true > > > yarn.resourcemanager.resource-tracker.address > yarn-master-0.yarn-service.yarn:8031 > > > yarn.log-aggregation-enable > true > > > yarn.nodemanager.aux-services > mapreduce_shuffle > > > > > > yarn.nodemanager.resource.memory-mb > 7168 > > > yarn.scheduler.minimum-allocation-mb > 3584 > > > yarn.scheduler.maximum-allocation-mb > 7168 > > > yarn.app.mapreduce.am.resource.mb > 7168 > > > > yarn.app.mapreduce.am.command-opts > -Xmx5734m > > > > yarn.timeline-service.enabled > true > > > yarn.resourcemanager.system-metrics-publisher.enabled > true > > > yarn.timeline-service.generic-application-history.enabled > true > > > yarn.timeline-service.bind-host > 0.0.0.0 > > > {code} > Relevant logs: > {code:java} > 2019-07-22 14:56:37,104 INFO [main] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 100% of the > mappers will be scheduled using OPPORTUNISTIC containers > 2019-07-22 14:56:37,117 INFO [main] org.apache.hadoop.yarn.client.RMProxy: > Connecting to ResourceManager at > yarn-master-0.yarn-service.yarn/10.244.1.134:8030 > 2019-07-22 14:56:37,150 WARN [main] org.apache.hadoop.ipc.Client: Exception > encountered while connecting to the server : > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): > Invalid AMRMToken from appattempt_1563805140414_0002_02 > 2019-07-22 14:56:37,152 ERROR [main] > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Exception while > registering > org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid > AMRMToken from appattempt_1563805140414_0002_02 > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80) > at > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invok
[jira] [Commented] (YARN-9690) Invalid AMRM token when distributed scheduling is enabled.
[ https://issues.apache.org/jira/browse/YARN-9690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890681#comment-16890681 ] Bibin A Chundatt commented on YARN-9690: [~Babbleshack] Looks like the AM is trying to connect RM . As per the configuration mentioned in following document [Reference|https://hadoop.apache.org/docs/r3.2.0/hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html] AM should connect to *AMRMProxy* in nodemanager yarn.resourcemanager.scheduler.address localhost:8049 Redirects jobs to the Node Manager’s AMRMProxy port. This is client side propery in case of mapreduce application. > Invalid AMRM token when distributed scheduling is enabled. > -- > > Key: YARN-9690 > URL: https://issues.apache.org/jira/browse/YARN-9690 > Project: Hadoop YARN > Issue Type: Bug > Components: distributed-scheduling, yarn >Affects Versions: 2.9.2, 3.1.2 > Environment: OS: Ubuntu 18.04 > JVM: 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03 >Reporter: Babble Shack >Priority: Major > Attachments: applicationlog, yarn-site.xml > > > Applications fail to start due to invalild AMRM from application attempt. > I have tested this with 0/100% opportunistic maps and the same issue occurs > regardless. > {code:java} > > --> > > > mapreduceyarn.nodemanager.aux-services > mapreduce_shuffle > > > yarn.resourcemanager.address > yarn-master-0.yarn-service.yarn:8032 > > > yarn.resourcemanager.scheduler.address > 0.0.0.0:8049 > > > > yarn.resourcemanager.opportunistic-container-allocation.enabled > true > > > yarn.nodemanager.opportunistic-containers-max-queue-length > 10 > > > yarn.nodemanager.distributed-scheduling.enabled > true > > > > yarn.webapp.ui2.enable > true > > > yarn.resourcemanager.resource-tracker.address > yarn-master-0.yarn-service.yarn:8031 > > > yarn.log-aggregation-enable > true > > > yarn.nodemanager.aux-services > mapreduce_shuffle > > > > > > yarn.nodemanager.resource.memory-mb > 7168 > > > yarn.scheduler.minimum-allocation-mb > 3584 > > > yarn.scheduler.maximum-allocation-mb > 7168 > > > yarn.app.mapreduce.am.resource.mb > 7168 > > > > yarn.app.mapreduce.am.command-opts > -Xmx5734m > > > > yarn.timeline-service.enabled > true > > > yarn.resourcemanager.system-metrics-publisher.enabled > true > > > yarn.timeline-service.generic-application-history.enabled > true > > > yarn.timeline-service.bind-host > 0.0.0.0 > > > {code} > Relevant logs: > {code:java} > 2019-07-22 14:56:37,104 INFO [main] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 100% of the > mappers will be scheduled using OPPORTUNISTIC containers > 2019-07-22 14:56:37,117 INFO [main] org.apache.hadoop.yarn.client.RMProxy: > Connecting to ResourceManager at > yarn-master-0.yarn-service.yarn/10.244.1.134:8030 > 2019-07-22 14:56:37,150 WARN [main] org.apache.hadoop.ipc.Client: Exception > encountered while connecting to the server : > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): > Invalid AMRMToken from appattempt_1563805140414_0002_02 > 2019-07-22 14:56:37,152 ERROR [main] > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Exception while > registering > org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid > AMRMToken from appattempt_1563805140414_0002_02 > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80) > at > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.io.retry.Retry
[jira] [Commented] (YARN-9690) Invalid AMRM token when distributed scheduling is enabled.
[ https://issues.apache.org/jira/browse/YARN-9690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890254#comment-16890254 ] Babble Shack commented on YARN-9690: I enabled some additional configuration `yarn.nodemanager.amrmproxy.enabled` and set `yarn.resourcemanager.scheduler.address` to `0.0.0.0:8049` However I still get the same issue, in particular there is an excption whilst registering application master, because of an invalid AMRM-token. > Invalid AMRM token when distributed scheduling is enabled. > -- > > Key: YARN-9690 > URL: https://issues.apache.org/jira/browse/YARN-9690 > Project: Hadoop YARN > Issue Type: Bug > Components: distributed-scheduling, yarn >Affects Versions: 2.9.2, 3.1.2 > Environment: OS: Ubuntu 18.04 > JVM: 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03 >Reporter: Babble Shack >Priority: Major > Attachments: applicationlog, yarn-site.xml > > > Applications fail to start due to invalild AMRM from application attempt. > I have tested this with 0/100% opportunistic maps and the same issue occurs > regardless. > {code:java} > > --> > > > mapreduceyarn.nodemanager.aux-services > mapreduce_shuffle > > > yarn.resourcemanager.address > yarn-master-0.yarn-service.yarn:8032 > > > yarn.resourcemanager.scheduler.address > 0.0.0.0:8049 > > > > yarn.resourcemanager.opportunistic-container-allocation.enabled > true > > > yarn.nodemanager.opportunistic-containers-max-queue-length > 10 > > > yarn.nodemanager.distributed-scheduling.enabled > true > > > > yarn.webapp.ui2.enable > true > > > yarn.resourcemanager.resource-tracker.address > yarn-master-0.yarn-service.yarn:8031 > > > yarn.log-aggregation-enable > true > > > yarn.nodemanager.aux-services > mapreduce_shuffle > > > > > > yarn.nodemanager.resource.memory-mb > 7168 > > > yarn.scheduler.minimum-allocation-mb > 3584 > > > yarn.scheduler.maximum-allocation-mb > 7168 > > > yarn.app.mapreduce.am.resource.mb > 7168 > > > > yarn.app.mapreduce.am.command-opts > -Xmx5734m > > > > yarn.timeline-service.enabled > true > > > yarn.resourcemanager.system-metrics-publisher.enabled > true > > > yarn.timeline-service.generic-application-history.enabled > true > > > yarn.timeline-service.bind-host > 0.0.0.0 > > > {code} > Relevant logs: > {code:java} > 2019-07-22 14:56:37,104 INFO [main] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 100% of the > mappers will be scheduled using OPPORTUNISTIC containers > 2019-07-22 14:56:37,117 INFO [main] org.apache.hadoop.yarn.client.RMProxy: > Connecting to ResourceManager at > yarn-master-0.yarn-service.yarn/10.244.1.134:8030 > 2019-07-22 14:56:37,150 WARN [main] org.apache.hadoop.ipc.Client: Exception > encountered while connecting to the server : > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): > Invalid AMRMToken from appattempt_1563805140414_0002_02 > 2019-07-22 14:56:37,152 ERROR [main] > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Exception while > registering > org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid > AMRMToken from appattempt_1563805140414_0002_02 > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80) > at > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandl