[
https://issues.apache.org/jira/browse/YARN-9690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893981#comment-16893981
]
Babble Shack edited comment on YARN-9690 at 7/26/19 5:13 PM:
-------------------------------------------------------------
Setting `yarn.resourcemanager.scheduler.address` does not throw the Invalid
AMRMToken, however the application master is unable to register with the
resource manager.
I have configured my resource manager, which is also the machine I submit MR
jobs from with the following
{code:java}
<!-- ResouceManager & job submitting client yarn-site.xml -->
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>localhost:8049</value>
</property>
{code}
Whilst my nodemanager is configured as:
{code:java}
<!-- Nodemanager yarn-site.xml -->
<!-- comment out yarn.resourcemanager.scheduler.address
defaults to 0.0.0.0:8030
-->
<!--
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>localhost:8049</value>
</property>
-->
{code}
Doing so enable the AM to connect to RM at 127.0.0.1,
{code:java}
// Application Log
************************************************************/
2019-07-26 16:47:46,928 INFO [main] org.apache.hadoop.security.SecurityUtil:
Updating Configuration
2019-07-26 16:47:46,993 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens: [Kind:
YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 1
cluster_timestamp: 1564159650361 } attemptId: 1 } keyId: -803833355)]
2019-07-26 16:47:47,021 INFO [main] org.apache.hadoop.conf.Configuration:
resource-types.xml not found
2019-07-26 16:47:47,021 INFO [main]
org.apache.hadoop.yarn.util.resource.ResourceUtils: Unable to find
'resource-types.xml'.
2019-07-26 16:47:47,028 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
2019-07-26 16:47:47,029 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config
null
2019-07-26 16:47:47,048 INFO [main]
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output
Committer Algorithm version is 2
2019-07-26 16:47:47,048 INFO [main]
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter
skip cleanup _temporary folders under output directory:false, ignore cleanup
failures: false
2019-07-26 16:47:47,389 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2019-07-26 16:47:47,490 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.jobhistory.EventType for class
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
2019-07-26 16:47:47,491 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
2019-07-26 16:47:47,491 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
2019-07-26 16:47:47,492 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
2019-07-26 16:47:47,492 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
2019-07-26 16:47:47,492 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
2019-07-26 16:47:47,492 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
2019-07-26 16:47:47,493 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for
class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
2019-07-26 16:47:47,507 INFO [main]
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system
[hdfs://hdfs-master-0.hdfs-service.hdfs:9000]
2019-07-26 16:47:47,514 INFO [main]
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system
[hdfs://hdfs-master-0.hdfs-service.hdfs:9000]
2019-07-26 16:47:47,521 INFO [main]
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system
[hdfs://hdfs-master-0.hdfs-service.hdfs:9000]
2019-07-26 16:47:47,526 INFO [main]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job
history data to the timeline server is not enabled
2019-07-26 16:47:47,550 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
2019-07-26 16:47:47,677 INFO [main]
org.apache.commons.beanutils.FluentPropertyBeanIntrospector: Error when
creating PropertyDescriptor for public final void
org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)!
Ignoring this property.
2019-07-26 16:47:47,696 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
hadoop-metrics2.properties
2019-07-26 16:47:47,729 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled Metric snapshot
period at 10 second(s).
2019-07-26 16:47:47,729 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system
started
2019-07-26 16:47:47,733 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for
job_1564159650361_0001 to jobTokenSecretManager
2019-07-26 16:47:47,816 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing
job_1564159650361_0001 because: not enabled; too many maps;
2019-07-26 16:47:47,826 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job
job_1564159650361_0001 = 1180. Number of splits = 10
2019-07-26 16:47:47,826 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job
job_1564159650361_0001 = 1
2019-07-26 16:47:47,826 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1564159650361_0001Job
Transitioned from NEW to INITED
2019-07-26 16:47:47,827 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal,
non-uberized, multi-container job job_1564159650361_0001.
2019-07-26 16:47:47,841 INFO [main] org.apache.hadoop.ipc.CallQueueManager:
Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity:
100 scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler
2019-07-26 16:47:47,846 INFO [Socket Reader #1 for port 43689]
org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 43689
2019-07-26 16:47:47,941 INFO [main]
org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
2019-07-26 16:47:47,942 INFO [IPC Server Responder]
org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2019-07-26 16:47:47,942 INFO [IPC Server listener on 43689]
org.apache.hadoop.ipc.Server: IPC Server listener on 43689: starting
2019-07-26 16:47:47,942 INFO [main]
org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated
MRClientService at
yarn-worker-0.yarn-service.yarn.svc.cluster.local/10.244.27.80:43689
2019-07-26 16:47:47,959 INFO [main] org.eclipse.jetty.util.log: Logging
initialized @1509ms
2019-07-26 16:47:48,002 INFO [main]
org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable
to initialize FileSignerSecretProvider, falling back to use random secrets.
2019-07-26 16:47:48,003 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http
request log for http.requests.mapreduce is not defined
2019-07-26 16:47:48,005 INFO [main] org.apache.hadoop.http.HttpServer2: Added
global filter 'safety'
(class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2019-07-26 16:47:48,019 INFO [main] org.apache.hadoop.http.HttpServer2: Added
filter AM_PROXY_FILTER
(class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context
mapreduce
2019-07-26 16:47:48,019 INFO [main] org.apache.hadoop.http.HttpServer2: Added
filter AM_PROXY_FILTER
(class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context
static
2019-07-26 16:47:48,020 INFO [main] org.apache.hadoop.http.HttpServer2: adding
path spec: /mapreduce/*
2019-07-26 16:47:48,020 INFO [main] org.apache.hadoop.http.HttpServer2: adding
path spec: /ws/*
2019-07-26 16:47:48,245 INFO [main] org.apache.hadoop.yarn.webapp.WebApps:
Registered webapp guice modules
2019-07-26 16:47:48,245 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty
bound to port 42493
2019-07-26 16:47:48,246 INFO [main] org.eclipse.jetty.server.Server:
jetty-9.3.24.v20180605, build timestamp: 2018-06-05T17:11:56Z, git hash:
84205aa28f11a4f31f2a3b86d1bba2cc8ab69827
2019-07-26 16:47:48,263 INFO [main]
org.eclipse.jetty.server.handler.ContextHandler: Started
o.e.j.s.ServletContextHandler@178270b2{/static,jar:file:/opt/hadoop/share/hadoop/yarn/hadoop-yarn-common-3.1.2.jar!/webapps/static,AVAILABLE}
2019-07-26 16:47:48,725 INFO [main]
org.eclipse.jetty.server.handler.ContextHandler: Started
o.e.j.w.WebAppContext@30ec7d21{/,file:///opt/hadoop/hadooptmpdata/nm-local-dir/usercache/root/appcache/application_1564159650361_0001/container_1564159650361_0001_01_000001/tmp/jetty-0.0.0.0-42493-mapreduce-_-any-6621652234045247964.dir/webapp/,AVAILABLE}{/mapreduce}
2019-07-26 16:47:48,727 INFO [main] org.eclipse.jetty.server.AbstractConnector:
Started ServerConnector@1642eeae{HTTP/1.1,[http/1.1]}{0.0.0.0:42493}
2019-07-26 16:47:48,728 INFO [main] org.eclipse.jetty.server.Server: Started
@2277ms
2019-07-26 16:47:48,728 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web
app mapreduce started at 42493
2019-07-26 16:47:48,730 INFO [main] org.apache.hadoop.ipc.CallQueueManager:
Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity:
3000 scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler
2019-07-26 16:47:48,730 INFO [Socket Reader #1 for port 46011]
org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 46011
2019-07-26 16:47:48,733 INFO [IPC Server Responder]
org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2019-07-26 16:47:48,733 INFO [IPC Server listener on 46011]
org.apache.hadoop.ipc.Server: IPC Server listener on 46011: starting
2019-07-26 16:47:48,743 INFO [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
nodeBlacklistingEnabled:true
2019-07-26 16:47:48,743 INFO [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
maxTaskFailuresPerNode is 3
2019-07-26 16:47:48,743 INFO [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
blacklistDisablePercent is 33
2019-07-26 16:47:48,745 INFO [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 40% of the mappers
will be scheduled using OPPORTUNISTIC containers
2019-07-26 16:47:48,757 INFO [main] org.apache.hadoop.yarn.client.RMProxy:
Connecting to ResourceManager at localhost/127.0.0.1:8049
End of LogType:syslog.This log file belongs to a running container
(container_1564159650361_0001_01_000001) and so may not be complete.
***********************************************************************{code}
however the AM fails to register with the RM. Which can be observed from the
webui:
!image-2019-07-26-18-00-14-980.png!
Furthermore the node manager log shows
{code:java}
//Nodemanager Log
2019-07-26 15:48:40,274 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth
successful for appattempt_1564156105896_0001_000001 (auth:SIMPLE)
2019-07-26 15:48:40,276 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyService:
Registering application master.
Host:yarn-worker-0.yarn-service.yarn.svc.cluster.local Port:44235 Tracking
Url:http://yarn-worker-0.yarn-service.yarn.svc.cluster.local:43404
2019-07-26 15:48:40,277 INFO
org.apache.hadoop.yarn.server.nodemanager.scheduler.DistributedScheduler:
Forwarding registration request to theDistributed Scheduler Service on YARN RM
2019-07-26 15:48:40,277 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.DefaultRequestInterceptor:
Forwarding registerApplicationMasterForDistributedSchedulingrequest to the real
YARN RM
2019-07-26 15:48:41,282 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 0
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:48:42,033 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Skipping monitoring container container_1564156105896_0001_01_000001 since CPU
usage is not yet available.
2019-07-26 15:48:42,283 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 1
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:48:43,284 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 2
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:48:44,284 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 3
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:48:45,286 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 4
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:48:46,287 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 5
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:48:47,288 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 6
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:48:48,288 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 7
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:48:49,289 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 8
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:48:50,290 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 9
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:49:21,296 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 0
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:49:22,297 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 1
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:49:23,298 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 2
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:49:24,299 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 3
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:49:25,300 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 4
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:49:26,301 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 5
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:49:27,302 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 6
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:49:28,302 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 7
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:49:29,303 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 8
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:49:30,304 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 9
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:01,308 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 0
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:02,309 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 1
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:03,310 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 2
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:04,311 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 3
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:05,312 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 4
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:06,313 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 5
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:07,314 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 6
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:08,315 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 7
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:09,317 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 8
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:10,317 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 9
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:41,321 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 0
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:42,322 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 1
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:43,323 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 2
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:44,324 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 3
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:45,325 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 4
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:46,326 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 5
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:47,327 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 6
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:48,328 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 7
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:49,329 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 8
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:50,329 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 9
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:51:21,333 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 0
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:51:22,334 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 1
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:51:23,335 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 2
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:51:24,336 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 3
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:51:25,337 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 4
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:51:26,337 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 5
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:51:27,338 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 6
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:51:28,339 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 7
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:51:29,340 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 8
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:51:30,341 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 9
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:51:53,666 ERROR
org.apache.hadoop.yarn.server.nodemanager.NodeManager: RECEIVED SIGNAL 15:
SIGTERM
2019-07-26 15:51:53,678 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Successfully
Unregistered the Node yarn-worker-0.yarn-service.yarn.svc.cluster.local:42600
with ResourceManager.
2019-07-26 15:51:53,680 INFO org.eclipse.jetty.server.handler.ContextHandler:
Stopped o.e.j.w.WebAppContext@3c782d8e{/,null,UNAVAILABLE}{/node}
2019-07-26 15:51:53,681 INFO org.eclipse.jetty.server.AbstractConnector:
Stopped ServerConnector@41ffaeb8{HTTP/1.1,[http/1.1]}{0.0.0.0:8042}
2019-07-26 15:51:53,681 INFO org.eclipse.jetty.server.handler.ContextHandler:
Stopped
o.e.j.s.ServletContextHandler@22b53226{/static,jar:file:/opt/hadoop/share/hadoop/yarn/hadoop-yarn-common-3.1.2.jar!/webapps/static,UNAVAILABLE}
2019-07-26 15:51:53,681 INFO org.eclipse.jetty.server.handler.ContextHandler:
Stopped
o.e.j.s.ServletContextHandler@75d0911a{/logs,file:///opt/hadoop/logs/,UNAVAILABLE}
2019-07-26 15:51:53,682 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Applications still running : [application_1564156105896_0001]
2019-07-26 15:51:53,683 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Waiting for Applications to be Finished
2019-07-26 15:51:53,684 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Application application_1564156105896_0001 transitioned from RUNNING to
FINISHING_CONTAINERS_WAIT
2019-07-26 15:51:53,684 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Container container_1564156105896_0001_01_000001 transitioned from RUNNING to
KILLING
2019-07-26 15:51:53,684 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
Cleaning up container container_1564156105896_0001_01_000001
2019-07-26 15:51:53,690 WARN
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code
from container container_1564156105896_0001_01_000001 is : 143
2019-07-26 15:51:53,698 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
absolute path :
/opt/hadoop/hadooptmpdata/nm-local-dir/usercache/root/appcache/application_1564156105896_0001/container_1564156105896_0001_01_000001/launch_container.sh
2019-07-26 15:51:53,698 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
absolute path :
/opt/hadoop/hadooptmpdata/nm-local-dir/usercache/root/appcache/application_1564156105896_0001/container_1564156105896_0001_01_000001/container_tokens
2019-07-26 15:51:53,704 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Container container_1564156105896_0001_01_000001 transitioned from KILLING to
CONTAINER_CLEANEDUP_AFTER_KILL
2019-07-26 15:51:53,704 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
absolute path :
/opt/hadoop/hadooptmpdata/nm-local-dir/usercache/root/appcache/application_1564156105896_0001/container_1564156105896_0001_01_000001
2019-07-26 15:51:53,704 INFO
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=root
OPERATION=Container Finished - Killed TARGET=ContainerImpl RESULT=SUCCESS
APPID=application_1564156105896_0001
CONTAINERID=container_1564156105896_0001_01_000001
2019-07-26 15:51:53,705 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Container container_1564156105896_0001_01_000001 transitioned from
CONTAINER_CLEANEDUP_AFTER_KILL to DONE
2019-07-26 15:51:53,706 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Removing container_1564156105896_0001_01_000001 from application
application_1564156105896_0001
2019-07-26 15:51:53,706 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Application application_1564156105896_0001 transitioned from
FINISHING_CONTAINERS_WAIT to APPLICATION_RESOURCES_CLEANINGUP
2019-07-26 15:51:53,706 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Stopping resource-monitoring for container_1564156105896_0001_01_000001
2019-07-26 15:51:53,706 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl:
Considering container container_1564156105896_0001_01_000001 for
log-aggregation
2019-07-26 15:51:53,706 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
event CONTAINER_STOP for appId application_1564156105896_0001
2019-07-26 15:51:53,707 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
absolute path :
/opt/hadoop/hadooptmpdata/nm-local-dir/usercache/root/appcache/application_1564156105896_0001
2019-07-26 15:51:53,707 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
event APPLICATION_STOP for appId application_1564156105896_0001
2019-07-26 15:51:53,708 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Application application_1564156105896_0001 transitioned from
APPLICATION_RESOURCES_CLEANINGUP to FINISHED
2019-07-26 15:51:53,709 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyService:
Application stop event received for stopping
AppId:application_1564156105896_0001
2019-07-26 15:51:53,709 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyTokenSecretManager:
Application finished, removing password for
appattempt_1564156105896_0001_000001
2019-07-26 15:51:53,709 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyService: Stopping
the request processing pipeline for application: application_1564156105896_0001
2019-07-26 15:51:53,709 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl:
Application just finished : application_1564156105896_0001
2019-07-26 15:51:53,738 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl:
Uploading logs for container container_1564156105896_0001_01_000001. Current
good log dirs are /opt/hadoop/logs/userlogs
2019-07-26 15:51:53,742 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
path :
/opt/hadoop/logs/userlogs/application_1564156105896_0001/container_1564156105896_0001_01_000001/launch_container.sh
2019-07-26 15:51:53,742 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
path :
/opt/hadoop/logs/userlogs/application_1564156105896_0001/container_1564156105896_0001_01_000001/prelaunch.out
2019-07-26 15:51:53,742 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
path :
/opt/hadoop/logs/userlogs/application_1564156105896_0001/container_1564156105896_0001_01_000001/stdout
2019-07-26 15:51:53,742 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
path :
/opt/hadoop/logs/userlogs/application_1564156105896_0001/container_1564156105896_0001_01_000001/directory.info
2019-07-26 15:51:53,742 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
path :
/opt/hadoop/logs/userlogs/application_1564156105896_0001/container_1564156105896_0001_01_000001/syslog
2019-07-26 15:51:53,742 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
path :
/opt/hadoop/logs/userlogs/application_1564156105896_0001/container_1564156105896_0001_01_000001/stderr
2019-07-26 15:51:53,742 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
path :
/opt/hadoop/logs/userlogs/application_1564156105896_0001/container_1564156105896_0001_01_000001/prelaunch.err
2019-07-26 15:51:53,799 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
path : /opt/hadoop/logs/userlogs/application_1564156105896_0001
2019-07-26 15:51:53,799 WARN
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyService: Event
EventType: APPLICATION_LOG_HANDLING_FINISHED sent to absent application
application_1564156105896_0001
2019-07-26 15:51:54,683 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
All applications in FINISHED state
2019-07-26 15:51:54,684 INFO org.apache.hadoop.ipc.Server: Stopping server on
42600
2019-07-26 15:51:54,684 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server
listener on 42600
2019-07-26 15:51:54,684 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyService: Stopping
AMRMProxyService
2019-07-26 15:51:54,684 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server
Responder
2019-07-26 15:51:54,684 INFO org.apache.hadoop.ipc.Server: Stopping server on
8049
2019-07-26 15:51:54,685 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server
listener on 8049
2019-07-26 15:51:54,685 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService:
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService
waiting for pending aggregation during exit
2019-07-26 15:51:54,685 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server
Responder
2019-07-26 15:51:54,686 INFO org.apache.hadoop.ipc.Server: IPC Server handler
15 on 8049, call Call#10 Retry#0
org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB.registerApplicationMaster
from 127.0.0.1:52366
java.io.InterruptedIOException: Retry interrupted
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.processWaitTimeAndRetryInfo(RetryInvocationHandler.java:136)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:107)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at
com.sun.proxy.$Proxy79.registerApplicationMasterForDistributedScheduling(Unknown
Source)
at
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.DefaultRequestInterceptor.registerApplicationMasterForDistributedScheduling(DefaultRequestInterceptor.java:154)
at
org.apache.hadoop.yarn.server.nodemanager.scheduler.DistributedScheduler.registerApplicationMasterForDistributedScheduling(DistributedScheduler.java:214)
at
org.apache.hadoop.yarn.server.nodemanager.scheduler.DistributedScheduler.registerApplicationMaster(DistributedScheduler.java:133)
at
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyService.registerApplicationMaster(AMRMProxyService.java:295)
at
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.registerApplicationMaster(ApplicationMasterProtocolPBServiceImpl.java:90)
at
org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:95)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)
Caused by: java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.processWaitTimeAndRetryInfo(RetryInvocationHandler.java:130)
... 17 more
2019-07-26 15:51:54,686 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
is interrupted. Exiting.
2019-07-26 15:51:54,687 WARN org.apache.hadoop.ipc.Server: IPC Server handler
15 on 8049, call Call#10 Retry#0
org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB.registerApplicationMaster
from 127.0.0.1:52366: output error
2019-07-26 15:51:54,687 INFO org.apache.hadoop.ipc.Server: IPC Server handler
15 on 8049 caught an exception
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:270)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:461)
at org.apache.hadoop.ipc.Server.channelWrite(Server.java:3245)
at org.apache.hadoop.ipc.Server.access$1700(Server.java:137)
at org.apache.hadoop.ipc.Server$Responder.processResponse(Server.java:1469)
at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:1539)
at org.apache.hadoop.ipc.Server$Connection.sendResponse(Server.java:2589)
at org.apache.hadoop.ipc.Server$Connection.access$300(Server.java:1611)
at org.apache.hadoop.ipc.Server$RpcCall.doResponse(Server.java:936)
at org.apache.hadoop.ipc.Server$Call.sendResponse(Server.java:770)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:881)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)
2019-07-26 15:51:54,690 INFO org.apache.hadoop.ipc.Server: Stopping server on
8040
{code}
Here we can observe the container is started and the token exchange occurs,
however teh amrmproxy is unable to connect to the central RM at
_yarn-master-0.yarn-service.yarn:8030_.
{code:java}
// AMRMProxy connecting to yarn-master-0.yarn-service.yarn:8030
2019-07-26 15:48:40,274 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth
successful for appattempt_1564156105896_0001_000001 (auth:SIMPLE)
2019-07-26 15:48:40,276 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyService:
Registering application master.
Host:yarn-worker-0.yarn-service.yarn.svc.cluster.local Port:44235 Tracking
Url:http://yarn-worker-0.yarn-service.yarn.svc.cluster.local:43404
2019-07-26 15:48:40,277 INFO
org.apache.hadoop.yarn.server.nodemanager.scheduler.DistributedScheduler:
Forwarding registration request to theDistributed Scheduler Service on YARN RM
2019-07-26 15:48:40,277 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.DefaultRequestInterceptor:
Forwarding registerApplicationMasterForDistributedSchedulingrequest to the real
YARN RM
2019-07-26 15:48:41,282 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 0
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:48:42,033 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Skipping monitoring container container_1564156105896_0001_01_000001 since CPU
usage is not yet available.
2019-07-26 15:48:42,283 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 1
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:48:43,284 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 2
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS){code}
The container does not have a open port on 8030, which is why it cannot connect.
[yarn-default.xml|https://hadoop.apache.org/docs/r3.1.2/hadoop-yarn/hadoop-yarn-common/yarn-default.xml]
identifies port 8030, as _yarn.resourcemanager.scheduler.address_. This is the
config I set at the masternode/resource manager as:
{code:java}
<!-- ResouceManager & job submitting client yarn-site.xml -->
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>localhost:8049</value>
</property>
{code}
I am unsure how to continue, as the submitting client is configured to instruct
AM to register with the RM on localhost. The AMRMproxy should receive the
request and forward it to central resource manager.
It seems before this can happen the nodemanager must connect with RM, however
teh RM node is configure to hosts scheduler on localhost:8049, not 0.0.0.0:8030.
It seems these configs clash.
Here are my configs and logs:
[^ds_application.log]
[^nodemanager.log]
[^rm-yarn-site.xml]
[^nodemanager-yarn-site.xml]
was (Author: babbleshack):
Setting `yarn.resourcemanager.scheduler.address` does not throw the Invalid
AMRMToken, however the application master is unable to register with the
resource manager.
I have configured my resource manager, which is also the machine I submit MR
jobs from with the following
{code:java}
<!-- ResouceManager & job submitting client yarn-site.xml -->
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>localhost:8049</value>
</property>
{code}
Whilst my nodemanager is configured as:
{code:java}
<!-- Nodemanager yarn-site.xml -->
<!-- comment out yarn.resourcemanager.scheduler.address
defaults to 0.0.0.0:8030
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>localhost:8049</value>
</property>
-->
{code}
Doing so enable the AM to connect to RM at 127.0.0.1,
{code:java}
// Application Log
************************************************************/
2019-07-26 16:47:46,928 INFO [main] org.apache.hadoop.security.SecurityUtil:
Updating Configuration
2019-07-26 16:47:46,993 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens: [Kind:
YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 1
cluster_timestamp: 1564159650361 } attemptId: 1 } keyId: -803833355)]
2019-07-26 16:47:47,021 INFO [main] org.apache.hadoop.conf.Configuration:
resource-types.xml not found
2019-07-26 16:47:47,021 INFO [main]
org.apache.hadoop.yarn.util.resource.ResourceUtils: Unable to find
'resource-types.xml'.
2019-07-26 16:47:47,028 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
2019-07-26 16:47:47,029 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config
null
2019-07-26 16:47:47,048 INFO [main]
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output
Committer Algorithm version is 2
2019-07-26 16:47:47,048 INFO [main]
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter
skip cleanup _temporary folders under output directory:false, ignore cleanup
failures: false
2019-07-26 16:47:47,389 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2019-07-26 16:47:47,490 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.jobhistory.EventType for class
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
2019-07-26 16:47:47,491 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
2019-07-26 16:47:47,491 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
2019-07-26 16:47:47,492 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
2019-07-26 16:47:47,492 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
2019-07-26 16:47:47,492 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
2019-07-26 16:47:47,492 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
2019-07-26 16:47:47,493 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for
class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
2019-07-26 16:47:47,507 INFO [main]
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system
[hdfs://hdfs-master-0.hdfs-service.hdfs:9000]
2019-07-26 16:47:47,514 INFO [main]
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system
[hdfs://hdfs-master-0.hdfs-service.hdfs:9000]
2019-07-26 16:47:47,521 INFO [main]
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system
[hdfs://hdfs-master-0.hdfs-service.hdfs:9000]
2019-07-26 16:47:47,526 INFO [main]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job
history data to the timeline server is not enabled
2019-07-26 16:47:47,550 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
2019-07-26 16:47:47,677 INFO [main]
org.apache.commons.beanutils.FluentPropertyBeanIntrospector: Error when
creating PropertyDescriptor for public final void
org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)!
Ignoring this property.
2019-07-26 16:47:47,696 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
hadoop-metrics2.properties
2019-07-26 16:47:47,729 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled Metric snapshot
period at 10 second(s).
2019-07-26 16:47:47,729 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system
started
2019-07-26 16:47:47,733 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for
job_1564159650361_0001 to jobTokenSecretManager
2019-07-26 16:47:47,816 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing
job_1564159650361_0001 because: not enabled; too many maps;
2019-07-26 16:47:47,826 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job
job_1564159650361_0001 = 1180. Number of splits = 10
2019-07-26 16:47:47,826 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job
job_1564159650361_0001 = 1
2019-07-26 16:47:47,826 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1564159650361_0001Job
Transitioned from NEW to INITED
2019-07-26 16:47:47,827 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal,
non-uberized, multi-container job job_1564159650361_0001.
2019-07-26 16:47:47,841 INFO [main] org.apache.hadoop.ipc.CallQueueManager:
Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity:
100 scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler
2019-07-26 16:47:47,846 INFO [Socket Reader #1 for port 43689]
org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 43689
2019-07-26 16:47:47,941 INFO [main]
org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
2019-07-26 16:47:47,942 INFO [IPC Server Responder]
org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2019-07-26 16:47:47,942 INFO [IPC Server listener on 43689]
org.apache.hadoop.ipc.Server: IPC Server listener on 43689: starting
2019-07-26 16:47:47,942 INFO [main]
org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated
MRClientService at
yarn-worker-0.yarn-service.yarn.svc.cluster.local/10.244.27.80:43689
2019-07-26 16:47:47,959 INFO [main] org.eclipse.jetty.util.log: Logging
initialized @1509ms
2019-07-26 16:47:48,002 INFO [main]
org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable
to initialize FileSignerSecretProvider, falling back to use random secrets.
2019-07-26 16:47:48,003 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http
request log for http.requests.mapreduce is not defined
2019-07-26 16:47:48,005 INFO [main] org.apache.hadoop.http.HttpServer2: Added
global filter 'safety'
(class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2019-07-26 16:47:48,019 INFO [main] org.apache.hadoop.http.HttpServer2: Added
filter AM_PROXY_FILTER
(class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context
mapreduce
2019-07-26 16:47:48,019 INFO [main] org.apache.hadoop.http.HttpServer2: Added
filter AM_PROXY_FILTER
(class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context
static
2019-07-26 16:47:48,020 INFO [main] org.apache.hadoop.http.HttpServer2: adding
path spec: /mapreduce/*
2019-07-26 16:47:48,020 INFO [main] org.apache.hadoop.http.HttpServer2: adding
path spec: /ws/*
2019-07-26 16:47:48,245 INFO [main] org.apache.hadoop.yarn.webapp.WebApps:
Registered webapp guice modules
2019-07-26 16:47:48,245 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty
bound to port 42493
2019-07-26 16:47:48,246 INFO [main] org.eclipse.jetty.server.Server:
jetty-9.3.24.v20180605, build timestamp: 2018-06-05T17:11:56Z, git hash:
84205aa28f11a4f31f2a3b86d1bba2cc8ab69827
2019-07-26 16:47:48,263 INFO [main]
org.eclipse.jetty.server.handler.ContextHandler: Started
o.e.j.s.ServletContextHandler@178270b2{/static,jar:file:/opt/hadoop/share/hadoop/yarn/hadoop-yarn-common-3.1.2.jar!/webapps/static,AVAILABLE}
2019-07-26 16:47:48,725 INFO [main]
org.eclipse.jetty.server.handler.ContextHandler: Started
o.e.j.w.WebAppContext@30ec7d21{/,file:///opt/hadoop/hadooptmpdata/nm-local-dir/usercache/root/appcache/application_1564159650361_0001/container_1564159650361_0001_01_000001/tmp/jetty-0.0.0.0-42493-mapreduce-_-any-6621652234045247964.dir/webapp/,AVAILABLE}{/mapreduce}
2019-07-26 16:47:48,727 INFO [main] org.eclipse.jetty.server.AbstractConnector:
Started ServerConnector@1642eeae{HTTP/1.1,[http/1.1]}{0.0.0.0:42493}
2019-07-26 16:47:48,728 INFO [main] org.eclipse.jetty.server.Server: Started
@2277ms
2019-07-26 16:47:48,728 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web
app mapreduce started at 42493
2019-07-26 16:47:48,730 INFO [main] org.apache.hadoop.ipc.CallQueueManager:
Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity:
3000 scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler
2019-07-26 16:47:48,730 INFO [Socket Reader #1 for port 46011]
org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 46011
2019-07-26 16:47:48,733 INFO [IPC Server Responder]
org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2019-07-26 16:47:48,733 INFO [IPC Server listener on 46011]
org.apache.hadoop.ipc.Server: IPC Server listener on 46011: starting
2019-07-26 16:47:48,743 INFO [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
nodeBlacklistingEnabled:true
2019-07-26 16:47:48,743 INFO [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
maxTaskFailuresPerNode is 3
2019-07-26 16:47:48,743 INFO [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
blacklistDisablePercent is 33
2019-07-26 16:47:48,745 INFO [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 40% of the mappers
will be scheduled using OPPORTUNISTIC containers
2019-07-26 16:47:48,757 INFO [main] org.apache.hadoop.yarn.client.RMProxy:
Connecting to ResourceManager at localhost/127.0.0.1:8049
End of LogType:syslog.This log file belongs to a running container
(container_1564159650361_0001_01_000001) and so may not be complete.
***********************************************************************{code}
however the AM fails to register with the RM. Which can be observed from the
webui:
!image-2019-07-26-18-00-14-980.png!
Furthermore the node manager log shows
{code:java}
//Nodemanager Log
2019-07-26 15:48:40,274 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth
successful for appattempt_1564156105896_0001_000001 (auth:SIMPLE)
2019-07-26 15:48:40,276 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyService:
Registering application master.
Host:yarn-worker-0.yarn-service.yarn.svc.cluster.local Port:44235 Tracking
Url:http://yarn-worker-0.yarn-service.yarn.svc.cluster.local:43404
2019-07-26 15:48:40,277 INFO
org.apache.hadoop.yarn.server.nodemanager.scheduler.DistributedScheduler:
Forwarding registration request to theDistributed Scheduler Service on YARN RM
2019-07-26 15:48:40,277 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.DefaultRequestInterceptor:
Forwarding registerApplicationMasterForDistributedSchedulingrequest to the real
YARN RM
2019-07-26 15:48:41,282 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 0
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:48:42,033 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Skipping monitoring container container_1564156105896_0001_01_000001 since CPU
usage is not yet available.
2019-07-26 15:48:42,283 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 1
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:48:43,284 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 2
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:48:44,284 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 3
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:48:45,286 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 4
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:48:46,287 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 5
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:48:47,288 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 6
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:48:48,288 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 7
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:48:49,289 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 8
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:48:50,290 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 9
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:49:21,296 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 0
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:49:22,297 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 1
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:49:23,298 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 2
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:49:24,299 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 3
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:49:25,300 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 4
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:49:26,301 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 5
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:49:27,302 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 6
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:49:28,302 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 7
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:49:29,303 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 8
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:49:30,304 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 9
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:01,308 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 0
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:02,309 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 1
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:03,310 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 2
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:04,311 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 3
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:05,312 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 4
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:06,313 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 5
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:07,314 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 6
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:08,315 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 7
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:09,317 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 8
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:10,317 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 9
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:41,321 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 0
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:42,322 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 1
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:43,323 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 2
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:44,324 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 3
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:45,325 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 4
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:46,326 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 5
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:47,327 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 6
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:48,328 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 7
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:49,329 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 8
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:50:50,329 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 9
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:51:21,333 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 0
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:51:22,334 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 1
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:51:23,335 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 2
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:51:24,336 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 3
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:51:25,337 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 4
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:51:26,337 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 5
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:51:27,338 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 6
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:51:28,339 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 7
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:51:29,340 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 8
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:51:30,341 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 9
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:51:53,666 ERROR
org.apache.hadoop.yarn.server.nodemanager.NodeManager: RECEIVED SIGNAL 15:
SIGTERM
2019-07-26 15:51:53,678 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Successfully
Unregistered the Node yarn-worker-0.yarn-service.yarn.svc.cluster.local:42600
with ResourceManager.
2019-07-26 15:51:53,680 INFO org.eclipse.jetty.server.handler.ContextHandler:
Stopped o.e.j.w.WebAppContext@3c782d8e{/,null,UNAVAILABLE}{/node}
2019-07-26 15:51:53,681 INFO org.eclipse.jetty.server.AbstractConnector:
Stopped ServerConnector@41ffaeb8{HTTP/1.1,[http/1.1]}{0.0.0.0:8042}
2019-07-26 15:51:53,681 INFO org.eclipse.jetty.server.handler.ContextHandler:
Stopped
o.e.j.s.ServletContextHandler@22b53226{/static,jar:file:/opt/hadoop/share/hadoop/yarn/hadoop-yarn-common-3.1.2.jar!/webapps/static,UNAVAILABLE}
2019-07-26 15:51:53,681 INFO org.eclipse.jetty.server.handler.ContextHandler:
Stopped
o.e.j.s.ServletContextHandler@75d0911a{/logs,file:///opt/hadoop/logs/,UNAVAILABLE}
2019-07-26 15:51:53,682 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Applications still running : [application_1564156105896_0001]
2019-07-26 15:51:53,683 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Waiting for Applications to be Finished
2019-07-26 15:51:53,684 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Application application_1564156105896_0001 transitioned from RUNNING to
FINISHING_CONTAINERS_WAIT
2019-07-26 15:51:53,684 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Container container_1564156105896_0001_01_000001 transitioned from RUNNING to
KILLING
2019-07-26 15:51:53,684 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
Cleaning up container container_1564156105896_0001_01_000001
2019-07-26 15:51:53,690 WARN
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code
from container container_1564156105896_0001_01_000001 is : 143
2019-07-26 15:51:53,698 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
absolute path :
/opt/hadoop/hadooptmpdata/nm-local-dir/usercache/root/appcache/application_1564156105896_0001/container_1564156105896_0001_01_000001/launch_container.sh
2019-07-26 15:51:53,698 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
absolute path :
/opt/hadoop/hadooptmpdata/nm-local-dir/usercache/root/appcache/application_1564156105896_0001/container_1564156105896_0001_01_000001/container_tokens
2019-07-26 15:51:53,704 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Container container_1564156105896_0001_01_000001 transitioned from KILLING to
CONTAINER_CLEANEDUP_AFTER_KILL
2019-07-26 15:51:53,704 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
absolute path :
/opt/hadoop/hadooptmpdata/nm-local-dir/usercache/root/appcache/application_1564156105896_0001/container_1564156105896_0001_01_000001
2019-07-26 15:51:53,704 INFO
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=root
OPERATION=Container Finished - Killed TARGET=ContainerImpl RESULT=SUCCESS
APPID=application_1564156105896_0001
CONTAINERID=container_1564156105896_0001_01_000001
2019-07-26 15:51:53,705 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Container container_1564156105896_0001_01_000001 transitioned from
CONTAINER_CLEANEDUP_AFTER_KILL to DONE
2019-07-26 15:51:53,706 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Removing container_1564156105896_0001_01_000001 from application
application_1564156105896_0001
2019-07-26 15:51:53,706 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Application application_1564156105896_0001 transitioned from
FINISHING_CONTAINERS_WAIT to APPLICATION_RESOURCES_CLEANINGUP
2019-07-26 15:51:53,706 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Stopping resource-monitoring for container_1564156105896_0001_01_000001
2019-07-26 15:51:53,706 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl:
Considering container container_1564156105896_0001_01_000001 for
log-aggregation
2019-07-26 15:51:53,706 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
event CONTAINER_STOP for appId application_1564156105896_0001
2019-07-26 15:51:53,707 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
absolute path :
/opt/hadoop/hadooptmpdata/nm-local-dir/usercache/root/appcache/application_1564156105896_0001
2019-07-26 15:51:53,707 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
event APPLICATION_STOP for appId application_1564156105896_0001
2019-07-26 15:51:53,708 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Application application_1564156105896_0001 transitioned from
APPLICATION_RESOURCES_CLEANINGUP to FINISHED
2019-07-26 15:51:53,709 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyService:
Application stop event received for stopping
AppId:application_1564156105896_0001
2019-07-26 15:51:53,709 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyTokenSecretManager:
Application finished, removing password for
appattempt_1564156105896_0001_000001
2019-07-26 15:51:53,709 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyService: Stopping
the request processing pipeline for application: application_1564156105896_0001
2019-07-26 15:51:53,709 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl:
Application just finished : application_1564156105896_0001
2019-07-26 15:51:53,738 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl:
Uploading logs for container container_1564156105896_0001_01_000001. Current
good log dirs are /opt/hadoop/logs/userlogs
2019-07-26 15:51:53,742 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
path :
/opt/hadoop/logs/userlogs/application_1564156105896_0001/container_1564156105896_0001_01_000001/launch_container.sh
2019-07-26 15:51:53,742 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
path :
/opt/hadoop/logs/userlogs/application_1564156105896_0001/container_1564156105896_0001_01_000001/prelaunch.out
2019-07-26 15:51:53,742 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
path :
/opt/hadoop/logs/userlogs/application_1564156105896_0001/container_1564156105896_0001_01_000001/stdout
2019-07-26 15:51:53,742 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
path :
/opt/hadoop/logs/userlogs/application_1564156105896_0001/container_1564156105896_0001_01_000001/directory.info
2019-07-26 15:51:53,742 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
path :
/opt/hadoop/logs/userlogs/application_1564156105896_0001/container_1564156105896_0001_01_000001/syslog
2019-07-26 15:51:53,742 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
path :
/opt/hadoop/logs/userlogs/application_1564156105896_0001/container_1564156105896_0001_01_000001/stderr
2019-07-26 15:51:53,742 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
path :
/opt/hadoop/logs/userlogs/application_1564156105896_0001/container_1564156105896_0001_01_000001/prelaunch.err
2019-07-26 15:51:53,799 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
path : /opt/hadoop/logs/userlogs/application_1564156105896_0001
2019-07-26 15:51:53,799 WARN
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyService: Event
EventType: APPLICATION_LOG_HANDLING_FINISHED sent to absent application
application_1564156105896_0001
2019-07-26 15:51:54,683 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
All applications in FINISHED state
2019-07-26 15:51:54,684 INFO org.apache.hadoop.ipc.Server: Stopping server on
42600
2019-07-26 15:51:54,684 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server
listener on 42600
2019-07-26 15:51:54,684 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyService: Stopping
AMRMProxyService
2019-07-26 15:51:54,684 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server
Responder
2019-07-26 15:51:54,684 INFO org.apache.hadoop.ipc.Server: Stopping server on
8049
2019-07-26 15:51:54,685 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server
listener on 8049
2019-07-26 15:51:54,685 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService:
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService
waiting for pending aggregation during exit
2019-07-26 15:51:54,685 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server
Responder
2019-07-26 15:51:54,686 INFO org.apache.hadoop.ipc.Server: IPC Server handler
15 on 8049, call Call#10 Retry#0
org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB.registerApplicationMaster
from 127.0.0.1:52366
java.io.InterruptedIOException: Retry interrupted
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.processWaitTimeAndRetryInfo(RetryInvocationHandler.java:136)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:107)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at
com.sun.proxy.$Proxy79.registerApplicationMasterForDistributedScheduling(Unknown
Source)
at
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.DefaultRequestInterceptor.registerApplicationMasterForDistributedScheduling(DefaultRequestInterceptor.java:154)
at
org.apache.hadoop.yarn.server.nodemanager.scheduler.DistributedScheduler.registerApplicationMasterForDistributedScheduling(DistributedScheduler.java:214)
at
org.apache.hadoop.yarn.server.nodemanager.scheduler.DistributedScheduler.registerApplicationMaster(DistributedScheduler.java:133)
at
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyService.registerApplicationMaster(AMRMProxyService.java:295)
at
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.registerApplicationMaster(ApplicationMasterProtocolPBServiceImpl.java:90)
at
org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:95)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)
Caused by: java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.processWaitTimeAndRetryInfo(RetryInvocationHandler.java:130)
... 17 more
2019-07-26 15:51:54,686 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
is interrupted. Exiting.
2019-07-26 15:51:54,687 WARN org.apache.hadoop.ipc.Server: IPC Server handler
15 on 8049, call Call#10 Retry#0
org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB.registerApplicationMaster
from 127.0.0.1:52366: output error
2019-07-26 15:51:54,687 INFO org.apache.hadoop.ipc.Server: IPC Server handler
15 on 8049 caught an exception
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:270)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:461)
at org.apache.hadoop.ipc.Server.channelWrite(Server.java:3245)
at org.apache.hadoop.ipc.Server.access$1700(Server.java:137)
at org.apache.hadoop.ipc.Server$Responder.processResponse(Server.java:1469)
at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:1539)
at org.apache.hadoop.ipc.Server$Connection.sendResponse(Server.java:2589)
at org.apache.hadoop.ipc.Server$Connection.access$300(Server.java:1611)
at org.apache.hadoop.ipc.Server$RpcCall.doResponse(Server.java:936)
at org.apache.hadoop.ipc.Server$Call.sendResponse(Server.java:770)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:881)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)
2019-07-26 15:51:54,690 INFO org.apache.hadoop.ipc.Server: Stopping server on
8040
{code}
Here we can observe the container is started and the token exchange occurs,
however teh amrmproxy is unable to connect to the central RM at
_yarn-master-0.yarn-service.yarn:8030_.
{code:java}
// AMRMProxy connecting to yarn-master-0.yarn-service.yarn:8030
2019-07-26 15:48:40,274 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth
successful for appattempt_1564156105896_0001_000001 (auth:SIMPLE)
2019-07-26 15:48:40,276 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.AMRMProxyService:
Registering application master.
Host:yarn-worker-0.yarn-service.yarn.svc.cluster.local Port:44235 Tracking
Url:http://yarn-worker-0.yarn-service.yarn.svc.cluster.local:43404
2019-07-26 15:48:40,277 INFO
org.apache.hadoop.yarn.server.nodemanager.scheduler.DistributedScheduler:
Forwarding registration request to theDistributed Scheduler Service on YARN RM
2019-07-26 15:48:40,277 INFO
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.DefaultRequestInterceptor:
Forwarding registerApplicationMasterForDistributedSchedulingrequest to the real
YARN RM
2019-07-26 15:48:41,282 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 0
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:48:42,033 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Skipping monitoring container container_1564156105896_0001_01_000001 since CPU
usage is not yet available.
2019-07-26 15:48:42,283 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 1
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2019-07-26 15:48:43,284 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: yarn-master-0.yarn-service.yarn/10.244.1.139:8030. Already tried 2
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS){code}
The container does not have a open port on 8030, which is why it cannot connect.
[yarn-default.xml|https://hadoop.apache.org/docs/r3.1.2/hadoop-yarn/hadoop-yarn-common/yarn-default.xml]
identifies port 8030, as _yarn.resourcemanager.scheduler.address_. This is the
config I set at the masternode/resource manager as:
{code:java}
<!-- ResouceManager & job submitting client yarn-site.xml -->
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>localhost:8049</value>
</property>
{code}
I am unsure how to continue, as the submitting client is configured to instruct
AM to register with the RM on localhost. The AMRMproxy should receive the
request and forward it to central resource manager.
It seems before this can happen the nodemanager must connect with RM, however
teh RM node is configure to hosts scheduler on localhost:8049, not 0.0.0.0:8030.
It seems these configs clash.
Here are my configs and logs:
[^ds_application.log]
[^nodemanager.log]
[^rm-yarn-site.xml]
[^nodemanager-yarn-site.xml]
> Invalid AMRM token when distributed scheduling is enabled.
> ----------------------------------------------------------
>
> Key: YARN-9690
> URL: https://issues.apache.org/jira/browse/YARN-9690
> Project: Hadoop YARN
> Issue Type: Bug
> Components: distributed-scheduling, yarn
> Affects Versions: 2.9.2, 3.1.2
> Environment: OS: Ubuntu 18.04
> JVM: 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03
> Reporter: Babble Shack
> Priority: Major
> Attachments: applicationlog, distributed_log, ds_application.log,
> image-2019-07-26-18-00-14-980.png, nodemanager-yarn-site.xml,
> nodemanager.log, rm-yarn-site.xml, yarn-site.xml
>
>
> Applications fail to start due to invalild AMRM from application attempt.
> I have tested this with 0/100% opportunistic maps and the same issue occurs
> regardless.
> {code:java}
> <!-- yarn-site.xml -->
> -->
> <configuration>
> <property>
> <name>mapreduceyarn.nodemanager.aux-services</name>
> <value>mapreduce_shuffle</value>
> </property>
> <property>
> <name>yarn.resourcemanager.address</name>
> <value>yarn-master-0.yarn-service.yarn:8032</value>
> </property>
> <property>
> <name>yarn.resourcemanager.scheduler.address</name>
> <value>0.0.0.0:8049</value>
> </property>
> <property>
>
> <name>yarn.resourcemanager.opportunistic-container-allocation.enabled</name>
> <value>true</value>
> </property>
> <property>
> <name>yarn.nodemanager.opportunistic-containers-max-queue-length</name>
> <value>10</value>
> </property>
> <property>
> <name>yarn.nodemanager.distributed-scheduling.enabled</name>
> <value>true</value>
> </property>
>
> <property>
> <name>yarn.webapp.ui2.enable</name>
> <value>true</value>
> </property>
> <property>
> <name>yarn.resourcemanager.resource-tracker.address</name>
> <value>yarn-master-0.yarn-service.yarn:8031</value>
> </property>
> <property>
> <name>yarn.log-aggregation-enable</name>
> <value>true</value>
> </property>
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce_shuffle</value>
> </property>
> <!--disable virtual memory-->
> <!--
> <property>
> <name>yarn.nodemanager.vmem-pmem-ratio</name>
> <value>1.1</value>
> </property>
> -->
> <!-- Worker(memory)=8GB, Node(mem)=6GB, Container(mem)=[1,2]GB-->
> <property>
> <name>yarn.nodemanager.resource.memory-mb</name>
> <value>7168</value>
> </property>
> <property>
> <name>yarn.scheduler.minimum-allocation-mb</name>
> <value>3584</value>
> </property>
> <property>
> <name>yarn.scheduler.maximum-allocation-mb</name>
> <value>7168</value>
> </property>
> <property>
> <name>yarn.app.mapreduce.am.resource.mb</name>
> <value>7168</value>
> </property>
> <!--AM 80% of mapred container-->
> <property>
> <name>yarn.app.mapreduce.am.command-opts</name>
> <value>-Xmx5734m</value>
> </property>
> <!--Timeline server -->
> <property>
> <name>yarn.timeline-service.enabled</name>
> <value>true</value>
> </property>
> <property>
> <name>yarn.resourcemanager.system-metrics-publisher.enabled</name>
> <value>true</value>
> </property>
> <property>
> <name>yarn.timeline-service.generic-application-history.enabled</name>
> <value>true</value>
> </property>
> <property>
> <name>yarn.timeline-service.bind-host</name>
> <value>0.0.0.0</value>
> </property>
> </configuration>
> {code}
> Relevant logs:
> {code:java}
> 2019-07-22 14:56:37,104 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 100% of the
> mappers will be scheduled using OPPORTUNISTIC containers
> 2019-07-22 14:56:37,117 INFO [main] org.apache.hadoop.yarn.client.RMProxy:
> Connecting to ResourceManager at
> yarn-master-0.yarn-service.yarn/10.244.1.134:8030
> 2019-07-22 14:56:37,150 WARN [main] org.apache.hadoop.ipc.Client: Exception
> encountered while connecting to the server :
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> Invalid AMRMToken from appattempt_1563805140414_0002_000002
> 2019-07-22 14:56:37,152 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Exception while
> registering
> org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> AMRMToken from appattempt_1563805140414_0002_000002
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy82.registerApplicationMaster(Unknown Source)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:160)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:121)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:274)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:959)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1272)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1723)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1719)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1650)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> Invalid AMRMToken from appattempt_1563805140414_0002_000002
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1507)
> at org.apache.hadoop.ipc.Client.call(Client.java:1453)
> at org.apache.hadoop.ipc.Client.call(Client.java:1363)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy81.registerApplicationMaster(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:106)
> ... 25 more
> 2019-07-22 14:56:37,153 INFO [main]
> org.apache.hadoop.service.AbstractService: Service RMCommunicator failed in
> state STARTED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> AMRMToken from appattempt_1563805140414_0002_000002
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> AMRMToken from appattempt_1563805140414_0002_000002
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:177)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:121)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:274)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:959)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1272)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1723)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1719)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1650)
> Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken:
> Invalid AMRMToken from appattempt_1563805140414_0002_000002
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy82.registerApplicationMaster(Unknown Source)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:160)
> ... 14 more
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> Invalid AMRMToken from appattempt_1563805140414_0002_000002
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1507)
> at org.apache.hadoop.ipc.Client.call(Client.java:1453)
> at org.apache.hadoop.ipc.Client.call(Client.java:1363)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy81.registerApplicationMaster(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:106)
> ... 25 more
> 2019-07-22 14:56:37,154 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats:
> PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0
> CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0
> 2019-07-22 14:56:37,154 INFO [main]
> org.apache.hadoop.service.AbstractService: Service
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
> failed in state STARTED; cause:
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> AMRMToken from appattempt_1563805140414_0002_000002
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> AMRMToken from appattempt_1563805140414_0002_000002
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:177)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:121)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:274)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:959)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1272)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1723)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1719)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1650)
> Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken:
> Invalid AMRMToken from appattempt_1563805140414_0002_000002
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy82.registerApplicationMaster(Unknown Source)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:160)
> ... 14 more
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> Invalid AMRMToken from appattempt_1563805140414_0002_000002
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1507)
> at org.apache.hadoop.ipc.Client.call(Client.java:1453)
> at org.apache.hadoop.ipc.Client.call(Client.java:1363)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy81.registerApplicationMaster(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:106)
> ... 25 more
> 2019-07-22 14:56:37,154 INFO [main]
> org.apache.hadoop.service.AbstractService: Service
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state STARTED;
> cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> AMRMToken from appattempt_1563805140414_0002_000002
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> AMRMToken from appattempt_1563805140414_0002_000002
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:177)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:121)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:274)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:959)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1272)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1723)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1719)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1650)
> Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken:
> Invalid AMRMToken from appattempt_1563805140414_0002_000002
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy82.registerApplicationMaster(Unknown Source)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:160)
> ... 14 more
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> Invalid AMRMToken from appattempt_1563805140414_0002_000002
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1507)
> at org.apache.hadoop.ipc.Client.call(Client.java:1453)
> at org.apache.hadoop.ipc.Client.call(Client.java:1363)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy81.registerApplicationMaster(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:106)
> ... 25 more
> 2019-07-22 14:56:37,155 INFO [main]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping
> JobHistoryEventHandler. Size of the outstanding queue size is 3
> 2019-07-22 14:56:37,155 INFO [main]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop,
> writing event AM_STARTED
> 2019-07-22 14:56:37,179 INFO [main]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer
> setup for JobId: job_1563805140414_0002, File:
> hdfs://hdfs-master-0.hdfs-service.hdfs:9000/tmp/hadoop-yarn/staging/root/.staging/job_1563805140414_0002/job_1563805140414_0002_2.jhist
> 2019-07-22 14:56:37,283 INFO [main]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop,
> writing event AM_STARTED
> 2019-07-22 14:56:37,284 INFO [main]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop,
> writing event JOB_SUBMITTED
> 2019-07-22 14:56:37,300 INFO [main]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped
> JobHistoryEventHandler. super.stop()
> 2019-07-22 14:56:37,300 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Skipping cleaning up the
> staging dir. assuming AM will be retried.
> 2019-07-22 14:56:37,300 INFO [main] org.apache.hadoop.ipc.Server: Stopping
> server on 42017
> 2019-07-22 14:56:37,301 INFO [IPC Server listener on 42017]
> org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 42017
> 2019-07-22 14:56:37,302 INFO [IPC Server Responder]
> org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
> 2019-07-22 14:56:37,302 INFO [TaskHeartbeatHandler PingChecker]
> org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler
> thread interrupted
> 2019-07-22 14:56:37,308 INFO [Ping Checker]
> org.apache.hadoop.yarn.util.AbstractLivelinessMonitor:
> TaskAttemptFinishingMonitor thread interrupted
> 2019-07-22 14:56:37,308 FATAL [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> AMRMToken from appattempt_1563805140414_0002_000002
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:177)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:121)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:274)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:959)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1272)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1723)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1719)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1650)
> Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken:
> Invalid AMRMToken from appattempt_1563805140414_0002_000002
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy82.registerApplicationMaster(Unknown Source)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:160)
> ... 14 more
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> Invalid AMRMToken from appattempt_1563805140414_0002_000002
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1507)
> at org.apache.hadoop.ipc.Client.call(Client.java:1453)
> at org.apache.hadoop.ipc.Client.call(Client.java:1363)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy81.registerApplicationMaster(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:106)
> ... 25 more
> 2019-07-22 14:56:37,309 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1: org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> AMRMToken from appattempt_1563805140414_0002_000002
> {code}
> A similar issue can be found at
> https://issues.apache.org/jira/browse/YARN-6514
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
