Hi Folks
I'm running into FileNotFoundExceptions when using using Pseudo Distributed
Single Node YARN using the Local FileSystem. I'd greatly appreciate any
insights/solutions.
To level set, I'm using RHEL 6.2 and I've successfully setup a single node
pseudo-distributed YARN on HDFS 2.0 using the HDP 2.0.2 Alpha Release (tarball
extract to /opt). All the processes were started and the jobs submitted as
root. I ran some smoke tests with TeraGen and TeraSort and it works great.
The next step was to leave YARN in pseudo-distributed mode and stop HDFS and
change the Hadoop FileSystem from HDFS to the Local FileSystem. I stopped all
the daemons, changed the core-site.xml to use the Local FileSystem as
demonstrated below, and then restarted the resourcemanager, nodemanager and
historyserver. Still running as root, everything started just fine. I ran
TeraGen (params: 1000 data/in-dir) it worked fine. I then ran TeraSort (params:
data/in-dir data/out-dir) and the Job Failed with a FileNotFoundException. I've
provided my core-site and mapred-site below.
-- core-site.xml --
<configuration>
<property>
<name>fs.default.name</name>
<value>file:///</value>
</property>
</configuration>
-- mapred-site.xml --
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
-- Stack Trace Exception --
2013-06-18 23:06:40,876 INFO [AsyncDispatcher event handler]
org.apache.hadoop.yarn.util.RackResolver: Resolved yarn-1 to /default-rack
2013-06-18 23:06:40,881 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container
container_1371596024885_0003_01_000002 to attempt_1371596024885_0003_m_000000_0
2013-06-18 23:06:40,881 INFO [RMCommunicator Allocator]
org.apache.hadoop.yarn.util.RackResolver: Resolved yarn-1 to /default-rack
2013-06-18 23:06:40,882 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container
container_1371596024885_0003_01_000003 to attempt_1371596024885_0003_m_000001_0
2013-06-18 23:06:40,882 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating
schedule, headroom=4096
2013-06-18 23:06:40,882 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start
threshold not met. completedMapsForReduceSlowstart 1
2013-06-18 23:06:40,882 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling:
PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:2 AssignedReds:0
CompletedMaps:0 CompletedReds:0 ContAlloc:2 ContRel:0 HostLocal:0 RackLocal:2
2013-06-18 23:06:40,896 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-jar file
on the remote FS is
file:///tmp/hadoop-yarn/staging/root/.staging/job_1371596024885_0003/job.jar
2013-06-18 23:06:40,901 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-conf file
on the remote FS is
/tmp/hadoop-yarn/staging/root/.staging/job_1371596024885_0003/job.xml
2013-06-18 23:06:40,902 FATAL [AsyncDispatcher event handler]
org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
org.apache.hadoop.yarn.YarnException: java.io.FileNotFoundException: File
file:/opt/hadoop-2.0.3.22-alpha-hdp/nm-local-dir/usercache/root/appcache/application_1371596024885_0003/container_1371596024885_0003_01_000001/data/out-dir/_partition.lst#_partition.lst
does not exist
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createCommonContainerLaunchContext(TaskAttemptImpl.java:723)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createContainerLaunchContext(TaskAttemptImpl.java:771)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1352)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1310)
at
org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:359)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:299)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:445)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1018)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:142)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1116)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1108)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:130)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: File
file:/opt/hadoop-2.0.3.22-alpha-hdp/nm-local-dir/usercache/root/appcache/application_1371596024885_0003/container_1371596024885_0003_01_000001/data/out-dir/_partition.lst#_partition.lst
does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:492)
at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:697)
at
org.apache.hadoop.fs.FilterFileSystem.resolvePath(FilterFileSystem.java:144)
at
org.apache.hadoop.mapreduce.v2.util.MRApps.parseDistributedCacheArtifacts(MRApps.java:417)
at
org.apache.hadoop.mapreduce.v2.util.MRApps.setupDistributedCache(MRApps.java:365)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createCommonContainerLaunchContext(TaskAttemptImpl.java:686)
... 14 more
2013-06-18 23:06:40,906 INFO [AsyncDispatcher event handler]
org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..
