Hey, Roland:
Could you also check the RM logs for this application, please ? Maybe we
could find something there.
Thanks
Xuan Gong
From: Roland DePratti <[email protected]<mailto:[email protected]>>
Reply-To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Date: Thursday, February 19, 2015 at 5:11 PM
To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Subject: RE: Yarn AM is abending job when submitting a remote job to cluster
No, I hear you.
I was just stating that the fact that hdfs works, there is something right
about the connectivity, that’s all, i.e. Server is reachable, hadoop was able
to process the request – but like you said, doesn’t mean yarn works.
I tried both your solution and Alex’s solution unfortunately without any
improvement.
Here is the command I am executing:
hadoop jar avgWordlength.jar solution.AvgWordLength -conf
~/conf/hadoop-cluster.xml /user/cloudera/shakespeare wordlength4
Here is the new hadoop-cluseter.xml
<?xml version="1.0" encoding="UTF-8"?>
<!--generated by Roland-->
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop0.rdpratti.com:8020</value>
</property>
<property>
<name>mapreduce.jobtracker.address</name>
<value>hadoop0.rdpratti.com:8032</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>hadoop0.rdpratti.com:8032</value>
</property>
I also deleted the .staging directory under the submitting user. Plus restarted
Job History Server.
Resubmitted the job with the same result. Here is the log:
2015-02-19 19:56:05,061 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for
application appattempt_1424003606313_0012_000002
2015-02-19 19:56:05,468 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.require.client.cert;
Ignoring.
2015-02-19 19:56:05,471 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-19 19:56:05,471 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.client.conf;
Ignoring.
2015-02-19 19:56:05,473 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
hadoop.ssl.keystores.factory.class; Ignoring.
2015-02-19 19:56:05,476 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.server.conf;
Ignoring.
2015-02-19 19:56:05,490 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-19 19:56:05,621 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2015-02-19 19:56:05,621 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN,
Service: , Ident:
(org.apache.hadoop.yarn.security.AMRMTokenIdentifier@3909f88f<mailto:org.apache.hadoop.yarn.security.AMRMTokenIdentifier@3909f88f>)
2015-02-19 19:56:05,684 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
2015-02-19 19:56:05,923 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.require.client.cert;
Ignoring.
2015-02-19 19:56:05,925 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-19 19:56:05,929 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.client.conf;
Ignoring.
2015-02-19 19:56:05,930 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
hadoop.ssl.keystores.factory.class; Ignoring.
2015-02-19 19:56:05,934 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.server.conf;
Ignoring.
2015-02-19 19:56:05,958 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-19 19:56:06,529 WARN [main] org.apache.hadoop.util.NativeCodeLoader:
Unable to load native-hadoop library for your platform... using builtin-java
classes where applicable
2015-02-19 19:56:06,719 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config
null
2015-02-19 19:56:06,837 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2015-02-19 19:56:06,881 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.jobhistory.EventType for class
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
2015-02-19 19:56:06,882 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
2015-02-19 19:56:06,882 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
2015-02-19 19:56:06,883 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
2015-02-19 19:56:06,884 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
2015-02-19 19:56:06,885 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
2015-02-19 19:56:06,885 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
2015-02-19 19:56:06,886 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for
class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
2015-02-19 19:56:06,899 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Recovery is enabled. Will try
to recover from previous life on best effort basis.
2015-02-19 19:56:06,918 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at
hdfs://hadoop0.rdpratti.com:8020/user/cloudera/.staging/job_1424003606313_0012/job_1424003606313_0012_1.jhist
2015-02-19 19:56:07,377 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Read completed tasks from
history 0
2015-02-19 19:56:07,423 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
2015-02-19 19:56:07,453 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
hadoop-metrics2.properties
2015-02-19 19:56:07,507 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at
10 second(s).
2015-02-19 19:56:07,507 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system
started
2015-02-19 19:56:07,515 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for
job_1424003606313_0012 to jobTokenSecretManager
2015-02-19 19:56:07,536 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing
job_1424003606313_0012 because: not enabled; too much RAM;
2015-02-19 19:56:07,555 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job
job_1424003606313_0012 = 5343207. Number of splits = 5
2015-02-19 19:56:07,557 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job
job_1424003606313_0012 = 1
2015-02-19 19:56:07,557 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1424003606313_0012Job
Transitioned from NEW to INITED
2015-02-19 19:56:07,558 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal,
non-uberized, multi-container job job_1424003606313_0012.
2015-02-19 19:56:07,618 INFO [main] org.apache.hadoop.ipc.CallQueueManager:
Using callQueue class java.util.concurrent.LinkedBlockingQueue
2015-02-19 19:56:07,630 INFO [Socket Reader #1 for port 46841]
org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 46841
2015-02-19 19:56:07,648 INFO [main]
org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
2015-02-19 19:56:07,648 INFO [IPC Server Responder]
org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2015-02-19 19:56:07,649 INFO [main]
org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated
MRClientService at hadoop0.rdpratti.com/192.168.2.253:46841
2015-02-19 19:56:07,650 INFO [IPC Server listener on 46841]
org.apache.hadoop.ipc.Server: IPC Server listener on 46841: starting
2015-02-19 19:56:07,721 INFO [main] org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2015-02-19 19:56:07,727 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http
request log for http.requests.mapreduce is not defined
2015-02-19 19:56:07,739 INFO [main] org.apache.hadoop.http.HttpServer2: Added
global filter 'safety'
(class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2015-02-19 19:56:07,745 INFO [main] org.apache.hadoop.http.HttpServer2: Added
filter AM_PROXY_FILTER
(class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context
mapreduce
2015-02-19 19:56:07,745 INFO [main] org.apache.hadoop.http.HttpServer2: Added
filter AM_PROXY_FILTER
(class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context
static
2015-02-19 19:56:07,749 INFO [main] org.apache.hadoop.http.HttpServer2: adding
path spec: /mapreduce/*
2015-02-19 19:56:07,749 INFO [main] org.apache.hadoop.http.HttpServer2: adding
path spec: /ws/*
2015-02-19 19:56:07,760 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty
bound to port 39939
2015-02-19 19:56:07,760 INFO [main] org.mortbay.log: jetty-6.1.26.cloudera.4
2015-02-19 19:56:07,789 INFO [main] org.mortbay.log: Extract
jar:file:/opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/jars/hadoop-yarn-common-2.5.0-cdh5.3.0.jar!/webapps/mapreduce
to /tmp/Jetty_0_0_0_0_39939_mapreduce____.o5qk0w/webapp
2015-02-19 19:56:08,156 INFO [main] org.mortbay.log: Started
[email protected]:39939<mailto:[email protected]:39939>
2015-02-19 19:56:08,157 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web
app /mapreduce started at 39939
2015-02-19 19:56:08,629 INFO [main] org.apache.hadoop.yarn.webapp.WebApps:
Registered webapp guice modules
2015-02-19 19:56:08,634 INFO [main] org.apache.hadoop.ipc.CallQueueManager:
Using callQueue class java.util.concurrent.LinkedBlockingQueue
2015-02-19 19:56:08,635 INFO [Socket Reader #1 for port 43858]
org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 43858
2015-02-19 19:56:08,639 INFO [IPC Server Responder]
org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2015-02-19 19:56:08,642 INFO [IPC Server listener on 43858]
org.apache.hadoop.ipc.Server: IPC Server listener on 43858: starting
2015-02-19 19:56:08,663 INFO [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
nodeBlacklistingEnabled:true
2015-02-19 19:56:08,663 INFO [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
maxTaskFailuresPerNode is 3
2015-02-19 19:56:08,663 INFO [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
blacklistDisablePercent is 33
2015-02-19 19:56:08,797 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.require.client.cert;
Ignoring.
2015-02-19 19:56:08,798 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-19 19:56:08,798 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.client.conf;
Ignoring.
2015-02-19 19:56:08,798 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
hadoop.ssl.keystores.factory.class; Ignoring.
2015-02-19 19:56:08,799 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.server.conf;
Ignoring.
2015-02-19 19:56:08,809 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-19 19:56:08,821 INFO [main] org.apache.hadoop.yarn.client.RMProxy:
Connecting to ResourceManager at quickstart.cloudera/192.168.2.185:8030
2015-02-19 19:56:08,975 WARN [main]
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:cloudera (auth:SIMPLE)
cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
appattempt_1424003606313_0012_000002 not found in AMRMTokenSecretManager.
2015-02-19 19:56:08,976 WARN [main] org.apache.hadoop.ipc.Client: Exception
encountered while connecting to the server :
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
appattempt_1424003606313_0012_000002 not found in AMRMTokenSecretManager.
2015-02-19 19:56:08,976 WARN [main]
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:cloudera (auth:SIMPLE)
cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
appattempt_1424003606313_0012_000002 not found in AMRMTokenSecretManager.
2015-02-19 19:56:08,981 ERROR [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Exception while
registering
org.apache.hadoop.security.token.SecretManager$InvalidToken:
appattempt_1424003606313_0012_000002 not found in AMRMTokenSecretManager.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
at
org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy36.registerApplicationMaster(Unknown Source)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:161)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:122)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:238)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:807)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1075)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1478)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1474)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1407)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
appattempt_1424003606313_0012_000002 not found in AMRMTokenSecretManager.
at org.apache.hadoop.ipc.Client.call(Client.java:1411)
at org.apache.hadoop.ipc.Client.call(Client.java:1364)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy35.registerApplicationMaster(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:106)
... 22 more
2015-02-19 19:56:08,983 INFO [main] org.apache.hadoop.service.AbstractService:
Service RMCommunicator failed in state STARTED; cause:
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
org.apache.hadoop.security.token.SecretManager$InvalidToken:
appattempt_1424003606313_0012_000002 not found in AMRMTokenSecretManager.
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
org.apache.hadoop.security.token.SecretManager$InvalidToken:
appattempt_1424003606313_0012_000002 not found in AMRMTokenSecretManager.
From: Ulul [mailto:[email protected]]
Sent: Thursday, February 19, 2015 5:08 PM
To: [email protected]<mailto:[email protected]>
Subject: Re: Yarn AM is abending job when submitting a remote job to cluster
Is your point is that using the hdfs:// prefix is valid since our hdfs client
works ?
fs.defaultFS defines the namenode address and the filesystem type. It doen't
imply that the prefix should be used for yarn and mapreduce options that are
not directly linked to hdfs
Le 19/02/2015 22:56, Ulul a écrit :
In that case it's just between your hdfs client, the NN and the DNs, no YARN or
MR component involved.
The fact that this works is not related to your MR job not succeeding.
Le 19/02/2015 22:45, roland.depratti a écrit :
Thanks for looking at my problem.
I can run an hdfs command from the client, with the config file listed, that
does a cat on a file in hdfs on the remote cluster and returns the contents of
that file to the client.
- rd
Sent from my Verizon Wireless 4G LTE smartphone
-------- Original message --------
From: Ulul <[email protected]><mailto:[email protected]>
Date:02/19/2015 4:03 PM (GMT-05:00)
To: [email protected]<mailto:[email protected]>
Subject: Re: Yarn AM is abending job when submitting a remote job to cluster
Hi
Doesn't seem like an ssl error to me (the log states that attempts to
override final properties are ignored)
On the other hand the configuration seems wrong
:mapreduce.jobtracker.address and yarn.resourcemanager.address should
only contain an IP or a hostname. You should remove 'hdfs://' though the
log doesn't suggest it has anything to do with your problem....
And what do you mean by an "HDFS job" ?
Ulul
Le 19/02/2015 04:22, daemeon reiydelle a écrit :
> I would guess you do not have your ssl certs set up, client or server,
> based on the error.
>
> ***
> .......
> ***“Life should not be a journey to the grave with the intention of
> arriving safely in a
> pretty and well preserved body, but rather to skid in broadside in a
> cloud of smoke,
> thoroughly used up, totally worn out, and loudly proclaiming “Wow!
> What a Ride!”*
> - Hunter Thompson
>
> Daemeon C.M. Reiydelle
> USA (+1) 415.501.0198
> London (+44) (0) 20 8144 9872*/
> /
>
> On Wed, Feb 18, 2015 at 5:19 PM, Roland DePratti
> <[email protected]<mailto:[email protected]>
> <mailto:[email protected]><mailto:[email protected]>> wrote:
>
> I have been searching for a handle on a problem without very
> little clues. Any help pointing me to the right direction will be
> huge.
>
> I have not received any input form the Cloudera google groups.
> Perhaps this is more Yarn based and I am hoping I have more luck here.
>
> Any help is greatly appreciated.
>
> I am running a Hadoop cluster using CDH5.3. I also have a client
> machine with a standalone one node setup (VM).
>
> All environments are running CentOS 6.6.
>
> I have submitted some Java mapreduce jobs locally on both the
> cluster and the standalone environment with successfully completions.
>
> I can submit a remote HDFS job from client to cluster using -conf
> hadoop-cluster.xml (see below) and get data back from the cluster
> with no problem.
>
> When submitted remotely the mapreduce jobs remotely, I get an AM
> error:
>
> AM fails the job with the error:
>
>
> SecretManager$InvalidToken:
> appattempt_1424003606313_0001_000002 not found in
> AMRMTokenSecretManager
>
>
> I searched /var/log/secure on the client and cluster with no
> unusual messages.
>
> Here is the contents of hadoop-cluster.xml:
>
> <?xml version="1.0" encoding="UTF-8"?>
>
> <!--generated by Roland-->
> <configuration>
> <property>
> <name>fs.defaultFS</name>
> <value>hdfs://mycluser:8020</value>
> </property>
> <property>
> <name>mapreduce.jobtracker.address</name>
> <value>hdfs://mycluster:8032</value>
> </property>
> <property>
> <name>yarn.resourcemanager.address</name>
> <value>hdfs://mycluster:8032</value>
> </property>
>
> Here is the output from the job log on the cluster:
>
> 2015-02-15 07:51:06,544 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created
> MRAppMaster for application appattempt_1424003606313_0001_000002
>
> 2015-02-15 07:51:06,949 WARN [main]
> org.apache.hadoop.conf.Configuration: job.xml:an attempt to
> override final parameter: hadoop.ssl.require.client.cert; Ignoring.
>
> 2015-02-15 07:51:06,952 WARN [main]
> org.apache.hadoop.conf.Configuration: job.xml:an attempt to
> override final parameter:
> mapreduce.job.end-notification.max.retry.interval; Ignoring.
>
> 2015-02-15 07:51:06,952 WARN [main]
> org.apache.hadoop.conf.Configuration: job.xml:an attempt to
> override final parameter: hadoop.ssl.client.conf; Ignoring.
>
> 2015-02-15 07:51:06,954 WARN [main]
> org.apache.hadoop.conf.Configuration: job.xml:an attempt to
> override final parameter: hadoop.ssl.keystores.factory.class;
> Ignoring.
>
> 2015-02-15 07:51:06,957 WARN [main]
> org.apache.hadoop.conf.Configuration: job.xml:an attempt to
> override final parameter: hadoop.ssl.server.conf; Ignoring.
>
> 2015-02-15 07:51:06,973 WARN [main]
> org.apache.hadoop.conf.Configuration: job.xml:an attempt to
> override final parameter:
> mapreduce.job.end-notification.max.attempts; Ignoring.
>
> 2015-02-15 07:51:07,241 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
>
> 2015-02-15 07:51:07,241 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind:
> YARN_AM_RM_TOKEN, Service: , Ident:
>
> (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@33be1aa0<mailto:org.apache.hadoop.yarn.security.AMRMTokenIdentifier@33be1aa0>)
>
> 2015-02-15 07:51:07,332 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred
> newApiCommitter.
>
> 2015-02-15 07:51:07,627 WARN [main]
> org.apache.hadoop.conf.Configuration: job.xml:an attempt to
> override final parameter: hadoop.ssl.require.client.cert; Ignoring.
>
> 2015-02-15 07:51:07,632 WARN [main]
> org.apache.hadoop.conf.Configuration: job.xml:an attempt to
> override final parameter:
> mapreduce.job.end-notification.max.retry.interval; Ignoring.
>
> 2015-02-15 07:51:07,632 WARN [main]
> org.apache.hadoop.conf.Configuration: job.xml:an attempt to
> override final parameter: hadoop.ssl.client.conf; Ignoring.
>
> 2015-02-15 07:51:07,639 WARN [main]
> org.apache.hadoop.conf.Configuration: job.xml:an attempt to
> override final parameter: hadoop.ssl.keystores.factory.class;
> Ignoring.
>
> 2015-02-15 07:51:07,645 WARN [main]
> org.apache.hadoop.conf.Configuration: job.xml:an attempt to
> override final parameter: hadoop.ssl.server.conf; Ignoring.
>
> 2015-02-15 07:51:07,663 WARN [main]
> org.apache.hadoop.conf.Configuration: job.xml:an attempt to
> override final parameter:
> mapreduce.job.end-notification.max.attempts; Ignoring.
>
> 2015-02-15 07:51:08,237 WARN [main]
> org.apache.hadoop.util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java
> classes where applicable
>
> 2015-02-15 07:51:08,429 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter
> set in config null
>
> 2015-02-15 07:51:08,499 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
>
> 2015-02-15 07:51:08,526 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.hadoop.mapreduce.jobhistory.EventType for class
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
>
> 2015-02-15 07:51:08,527 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for
> class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
>
> 2015-02-15 07:51:08,561 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for
> class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
>
> 2015-02-15 07:51:08,562 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType
> for class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
>
> 2015-02-15 07:51:08,566 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for
> class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
>
> 2015-02-15 07:51:08,568 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType
> for class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
>
> 2015-02-15 07:51:08,568 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType
> for class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
>
> 2015-02-15 07:51:08,570 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType
> for class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
>
> 2015-02-15 07:51:08,599 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Recovery is
> enabled. Will try to recover from previous life on best effort basis.
>
> 2015-02-15 07:51:08,642 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history
> file is at
>
> hdfs://mycluster.com:8020/user/cloudera/.staging/job_1424003606313_0001/job_1424003606313_0001_1.jhist
>
> <http://mycluster.com:8020/user/cloudera/.staging/job_1424003606313_0001/job_1424003606313_0001_1.jhist2015-02-15><http://mycluster.com:8020/user/cloudera/.staging/job_1424003606313_0001/job_1424003606313_0001_1.jhist2015-02-15>
>
> _2015-02-15
>
> <http://mycluster.com:8020/user/cloudera/.staging/job_1424003606313_0001/job_1424003606313_0001_1.jhist2015-02-15><http://mycluster.com:8020/user/cloudera/.staging/job_1424003606313_0001/job_1424003606313_0001_1.jhist2015-02-15>_07:51:09,147
> INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Read
> completed tasks from history 0
>
> 2015-02-15 07:51:09,193 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type
> for class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
>
> 2015-02-15 07:51:09,222 INFO [main]
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties
> from hadoop-metrics2.properties
>
> 2015-02-15 07:51:09,277 INFO [main]
