[
https://issues.apache.org/jira/browse/MAPREDUCE-3056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14530788#comment-14530788
]
Sergey commented on MAPREDUCE-3056:
-----------------------------------
Hi, not sure, but I hit described problem:
CDH-5.3.2-1.cdh5.3.2.p0.10
```
2015-05-06 18:56:35,606 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for
application appattempt_1430916274046_0418_000001
2015-05-06 18:56:35,905 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.require.client.cert;
Ignoring.
2015-05-06 18:56:35,908 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-05-06 18:56:35,909 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.client.conf;
Ignoring.
2015-05-06 18:56:35,911 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
hadoop.ssl.keystores.factory.class; Ignoring.
2015-05-06 18:56:35,915 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.server.conf;
Ignoring.
2015-05-06 18:56:35,932 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.attempts; Ignoring.
2015-05-06 18:56:36,033 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2015-05-06 18:56:36,033 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN,
Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@1c667739)
2015-05-06 18:56:36,068 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: RM_DELEGATION_TOKEN,
Service: 10.66.62.141:8032,10.66.62.146:8032, Ident: (owner=devops,
renewer=oozie mr token, realUser=oozie, issueDate=1430927768380,
maxDate=1431532568380, sequenceNumber=70067, masterKeyId=110)
2015-05-06 18:56:36,083 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
2015-05-06 18:56:36,181 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.require.client.cert;
Ignoring.
2015-05-06 18:56:36,183 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-05-06 18:56:36,183 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.client.conf;
Ignoring.
2015-05-06 18:56:36,184 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
hadoop.ssl.keystores.factory.class; Ignoring.
2015-05-06 18:56:36,187 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.server.conf;
Ignoring.
2015-05-06 18:56:36,197 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.attempts; Ignoring.
2015-05-06 18:56:36,925 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config
null
2015-05-06 18:56:37,054 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is
com.linkedin.camus.etl.kafka.mapred.EtlMultiOutputCommitter
2015-05-06 18:56:37,080 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.jobhistory.EventType for class
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
2015-05-06 18:56:37,081 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
2015-05-06 18:56:37,082 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
2015-05-06 18:56:37,083 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
2015-05-06 18:56:37,083 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
2015-05-06 18:56:37,089 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
2015-05-06 18:56:37,090 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
2015-05-06 18:56:37,091 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for
class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
2015-05-06 18:56:37,107 INFO [main]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Perms after
creating 448, Expected: 448
2015-05-06 18:56:37,156 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
2015-05-06 18:56:37,444 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
hadoop-metrics2.properties
2015-05-06 18:56:37,504 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at
10 second(s).
2015-05-06 18:56:37,504 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system
started
2015-05-06 18:56:37,513 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for
job_1430916274046_0418 to jobTokenSecretManager
2015-05-06 18:56:37,535 WARN [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Job init failed
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.io.FileNotFoundException: File does not exist:
hdfs://nameservice1/user/devops/.staging/job_1430916274046_0418/job.splitmetainfo
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1566)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1430)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1388)
at
org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:996)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:138)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1272)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1045)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1478)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1474)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1407)
Caused by: java.io.FileNotFoundException: File does not exist:
hdfs://nameservice1/user/devops/.staging/job_1430916274046_0418/job.splitmetainfo
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1093)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085)
at
org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(SplitMetaInfoReader.java:51)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1561)
... 17 more
2015-05-06 18:56:37,542 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal,
non-uberized, multi-container job job_1430916274046_0418.
2015-05-06 18:56:37,578 INFO [main] org.apache.hadoop.ipc.CallQueueManager:
Using callQueue class java.util.concurrent.LinkedBlockingQueue
2015-05-06 18:56:37,588 INFO [Socket Reader #1 for port 52894]
org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 52894
2015-05-06 18:56:37,609 INFO [main]
org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
2015-05-06 18:56:37,609 INFO [IPC Server Responder]
org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2015-05-06 18:56:37,609 INFO [IPC Server listener on 52894]
org.apache.hadoop.ipc.Server: IPC Server listener on 52894: starting
2015-05-06 18:56:37,610 INFO [main]
org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated
MRClientService at prod-node0149.kyc.myhost.ru/10.66.62.122:52894
2015-05-06 18:56:37,675 INFO [main] org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2015-05-06 18:56:37,679 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http
request log for http.requests.mapreduce is not defined
2015-05-06 18:56:37,703 INFO [main] org.apache.hadoop.http.HttpServer2: Added
global filter 'safety'
(class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2015-05-06 18:56:37,767 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.require.client.cert;
Ignoring.
2015-05-06 18:56:37,769 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-05-06 18:56:37,769 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.client.conf;
Ignoring.
2015-05-06 18:56:37,770 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
hadoop.ssl.keystores.factory.class; Ignoring.
2015-05-06 18:56:37,771 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.server.conf;
Ignoring.
2015-05-06 18:56:37,779 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.attempts; Ignoring.
2015-05-06 18:56:37,784 INFO [main] org.apache.hadoop.http.HttpServer2: Added
filter AM_PROXY_FILTER
(class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context
mapreduce
2015-05-06 18:56:37,784 INFO [main] org.apache.hadoop.http.HttpServer2: Added
filter AM_PROXY_FILTER
(class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context
static
2015-05-06 18:56:37,788 INFO [main] org.apache.hadoop.http.HttpServer2: adding
path spec: /mapreduce/*
2015-05-06 18:56:37,788 INFO [main] org.apache.hadoop.http.HttpServer2: adding
path spec: /ws/*
2015-05-06 18:56:37,800 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty
bound to port 43170
2015-05-06 18:56:37,800 INFO [main] org.mortbay.log: jetty-6.1.26.cloudera.4
2015-05-06 18:56:37,827 INFO [main] org.mortbay.log: Extract
jar:file:/opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/jars/hadoop-yarn-common-2.5.0-cdh5.3.2.jar!/webapps/mapreduce
to /tmp/Jetty_0_0_0_0_43170_mapreduce____.3fyrma/webapp
2015-05-06 18:56:38,170 INFO [main] org.mortbay.log: Started
[email protected]:43170
2015-05-06 18:56:38,170 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web
app /mapreduce started at 43170
2015-05-06 18:56:38,496 INFO [main] org.apache.hadoop.yarn.webapp.WebApps:
Registered webapp guice modules
2015-05-06 18:56:38,499 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: JOB_CREATE
job_1430916274046_0418
2015-05-06 18:56:38,501 INFO [main] org.apache.hadoop.ipc.CallQueueManager:
Using callQueue class java.util.concurrent.LinkedBlockingQueue
2015-05-06 18:56:38,501 INFO [Socket Reader #1 for port 34861]
org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 34861
2015-05-06 18:56:38,506 INFO [IPC Server Responder]
org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2015-05-06 18:56:38,506 INFO [IPC Server listener on 34861]
org.apache.hadoop.ipc.Server: IPC Server listener on 34861: starting
2015-05-06 18:56:38,525 INFO [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
nodeBlacklistingEnabled:true
2015-05-06 18:56:38,526 INFO [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
maxTaskFailuresPerNode is 3
2015-05-06 18:56:38,526 INFO [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
blacklistDisablePercent is 33
2015-05-06 18:56:38,596 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.require.client.cert;
Ignoring.
2015-05-06 18:56:38,596 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-05-06 18:56:38,597 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.client.conf;
Ignoring.
2015-05-06 18:56:38,597 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
hadoop.ssl.keystores.factory.class; Ignoring.
2015-05-06 18:56:38,598 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.server.conf;
Ignoring.
2015-05-06 18:56:38,602 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.attempts; Ignoring.
2015-05-06 18:56:38,663 INFO [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
maxContainerCapability: <memory:202752, vCores:34>
2015-05-06 18:56:38,664 INFO [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue:
root.masterdata
2015-05-06 18:56:38,667 INFO [main]
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit
on the thread pool size is 500
2015-05-06 18:56:38,669 INFO [main]
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
yarn.client.max-cached-nodemanagers-proxies : 0
2015-05-06 18:56:38,698 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1430916274046_0418Job
Transitioned from NEW to FAIL_ABORT
2015-05-06 18:56:38,700 INFO [CommitterEvent Processor #0]
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the
event EventType: JOB_ABORT
2015-05-06 18:56:38,709 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1430916274046_0418Job
Transitioned from FAIL_ABORT to FAILED
2015-05-06 18:56:38,709 INFO [Thread-55]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: We are finishing cleanly so
this is the last retry
2015-05-06 18:56:38,709 INFO [Thread-55]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator
isAMLastRetry: true
2015-05-06 18:56:38,709 INFO [Thread-55]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: RMCommunicator
notified that shouldUnregistered is: true
2015-05-06 18:56:38,709 INFO [Thread-55]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true
2015-05-06 18:56:38,709 INFO [Thread-55]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
JobHistoryEventHandler notified that forceJobCompletion is true
2015-05-06 18:56:38,709 INFO [Thread-55]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the
services
2015-05-06 18:56:38,710 INFO [Thread-55]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping
JobHistoryEventHandler. Size of the outstanding queue size is 3
2015-05-06 18:56:38,807 INFO [eventHandlingThread]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer
setup for JobId: job_1430916274046_0418, File:
hdfs://nameservice1/user/devops/.staging/job_1430916274046_0418/job_1430916274046_0418_1.jhist
2015-05-06 18:56:39,144 INFO [Thread-55]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing
event JOB_SUBMITTED
2015-05-06 18:56:39,157 INFO [Thread-55]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing
event JOB_QUEUE_CHANGED
2015-05-06 18:56:39,157 INFO [Thread-55]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing
event JOB_FAILED
2015-05-06 18:56:39,215 INFO [Thread-55]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying
hdfs://nameservice1/user/devops/.staging/job_1430916274046_0418/job_1430916274046_0418_1.jhist
to
hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418-1430927787783-devops-Camus+job+%2D+cisgw%2Dmonitoring-1430927798706-0-0-FAILED-root.masterdata-1430927798706.jhist_tmp
2015-05-06 18:56:39,299 INFO [Thread-55]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done
location:
hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418-1430927787783-devops-Camus+job+%2D+cisgw%2Dmonitoring-1430927798706-0-0-FAILED-root.masterdata-1430927798706.jhist_tmp
2015-05-06 18:56:39,302 INFO [Thread-55]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying
hdfs://nameservice1/user/devops/.staging/job_1430916274046_0418/job_1430916274046_0418_1_conf.xml
to
hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418_conf.xml_tmp
2015-05-06 18:56:39,319 INFO [Thread-55]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done
location:
hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418_conf.xml_tmp
2015-05-06 18:56:39,325 INFO [Thread-55]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to
done:
hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418.summary_tmp
to
hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418.summary
2015-05-06 18:56:39,327 INFO [Thread-55]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to
done:
hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418_conf.xml_tmp
to
hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418_conf.xml
2015-05-06 18:56:39,328 INFO [Thread-55]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to
done:
hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418-1430927787783-devops-Camus+job+%2D+cisgw%2Dmonitoring-1430927798706-0-0-FAILED-root.masterdata-1430927798706.jhist_tmp
to
hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418-1430927787783-devops-Camus+job+%2D+cisgw%2Dmonitoring-1430927798706-0-0-FAILED-root.masterdata-1430927798706.jhist
2015-05-06 18:56:39,328 INFO [Thread-55]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped
JobHistoryEventHandler. super.stop()
2015-05-06 18:56:39,329 INFO [Thread-55]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Setting job
diagnostics to Job init failed :
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.io.FileNotFoundException: File does not exist:
hdfs://nameservice1/user/devops/.staging/job_1430916274046_0418/job.splitmetainfo
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1566)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1430)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1388)
at
org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:996)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:138)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1272)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1045)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1478)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1474)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1407)
Caused by: java.io.FileNotFoundException: File does not exist:
hdfs://nameservice1/user/devops/.staging/job_1430916274046_0418/job.splitmetainfo
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1093)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085)
at
org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(SplitMetaInfoReader.java:51)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1561)
... 17 more
2015-05-06 18:56:39,331 INFO [Thread-55]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: History url is
http://prod-node0138.kyc.myhost.ru:19888/jobhistory/job/job_1430916274046_0418
2015-05-06 18:56:39,339 INFO [Thread-55]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for
application to be successfully unregistered.
2015-05-06 18:56:40,340 INFO [Thread-55]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats:
PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0
CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0
2015-05-06 18:56:40,341 INFO [Thread-55]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory
hdfs://nameservice1 /user/yarn/.staging/job_1430916274046_0418
2015-05-06 18:56:40,343 INFO [Thread-55] org.apache.hadoop.ipc.Server: Stopping
server on 34861
2015-05-06 18:56:40,343 INFO [IPC Server listener on 34861]
org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 34861
2015-05-06 18:56:40,346 INFO [TaskHeartbeatHandler PingChecker]
org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler
thread interrupted
2015-05-06 18:56:40,346 INFO [IPC Server Responder]
org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
```
> Jobs are failing when those are submitted by other users
> --------------------------------------------------------
>
> Key: MAPREDUCE-3056
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3056
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2
> Affects Versions: 0.23.0, 2.0.0-alpha
> Reporter: Devaraj K
> Assignee: Devaraj K
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3056-1.patch, MAPREDUCE-3056-2.patch,
> MAPREDUCE-3056.patch
>
>
> MR cluster is started by the user 'root'. If any other users other than
> 'root' submit a job, it is failing always.
> Find the conatiner logs in the comments section.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
