I resolved this. The issue is that I was using relative paths (i.e "teragen 1000 data/in-dir") as the params for TeraGen and TeraSort. When I changed it to use absolute paths, (i.e. "teragen 1000 /data/in-dir") it works.
----- Original Message ----- From: "Stephen Watt" <[email protected]> To: [email protected] Sent: Thursday, June 20, 2013 12:25:17 PM Subject: FileNotFoundExceptions with Pseudo Distributed YARN MR using the Local FileSystem Hi Folks I'm running into FileNotFoundExceptions when using using Pseudo Distributed Single Node YARN using the Local FileSystem. I'd greatly appreciate any insights/solutions. To level set, I'm using RHEL 6.2 and I've successfully setup a single node pseudo-distributed YARN on HDFS 2.0 using the HDP 2.0.2 Alpha Release (tarball extract to /opt). All the processes were started and the jobs submitted as root. I ran some smoke tests with TeraGen and TeraSort and it works great. The next step was to leave YARN in pseudo-distributed mode and stop HDFS and change the Hadoop FileSystem from HDFS to the Local FileSystem. I stopped all the daemons, changed the core-site.xml to use the Local FileSystem as demonstrated below, and then restarted the resourcemanager, nodemanager and historyserver. Still running as root, everything started just fine. I ran TeraGen (params: 1000 data/in-dir) it worked fine. I then ran TeraSort (params: data/in-dir data/out-dir) and the Job Failed with a FileNotFoundException. I've provided my core-site and mapred-site below. -- core-site.xml -- <configuration> <property> <name>fs.default.name</name> <value>file:///</value> </property> </configuration> -- mapred-site.xml -- <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration> -- Stack Trace Exception -- 2013-06-18 23:06:40,876 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved yarn-1 to /default-rack 2013-06-18 23:06:40,881 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1371596024885_0003_01_000002 to attempt_1371596024885_0003_m_000000_0 2013-06-18 23:06:40,881 INFO [RMCommunicator Allocator] org.apache.hadoop.yarn.util.RackResolver: Resolved yarn-1 to /default-rack 2013-06-18 23:06:40,882 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1371596024885_0003_01_000003 to attempt_1371596024885_0003_m_000001_0 2013-06-18 23:06:40,882 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=4096 2013-06-18 23:06:40,882 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 1 2013-06-18 23:06:40,882 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:2 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:2 ContRel:0 HostLocal:0 RackLocal:2 2013-06-18 23:06:40,896 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-jar file on the remote FS is file:///tmp/hadoop-yarn/staging/root/.staging/job_1371596024885_0003/job.jar 2013-06-18 23:06:40,901 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-conf file on the remote FS is /tmp/hadoop-yarn/staging/root/.staging/job_1371596024885_0003/job.xml 2013-06-18 23:06:40,902 FATAL [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread org.apache.hadoop.yarn.YarnException: java.io.FileNotFoundException: File file:/opt/hadoop-2.0.3.22-alpha-hdp/nm-local-dir/usercache/root/appcache/application_1371596024885_0003/container_1371596024885_0003_01_000001/data/out-dir/_partition.lst#_partition.lst does not exist at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createCommonContainerLaunchContext(TaskAttemptImpl.java:723) at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createContainerLaunchContext(TaskAttemptImpl.java:771) at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1352) at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1310) at org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:359) at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:299) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:445) at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1018) at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:142) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1116) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1108) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:130) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.FileNotFoundException: File file:/opt/hadoop-2.0.3.22-alpha-hdp/nm-local-dir/usercache/root/appcache/application_1371596024885_0003/container_1371596024885_0003_01_000001/data/out-dir/_partition.lst#_partition.lst does not exist at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:492) at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:697) at org.apache.hadoop.fs.FilterFileSystem.resolvePath(FilterFileSystem.java:144) at org.apache.hadoop.mapreduce.v2.util.MRApps.parseDistributedCacheArtifacts(MRApps.java:417) at org.apache.hadoop.mapreduce.v2.util.MRApps.setupDistributedCache(MRApps.java:365) at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createCommonContainerLaunchContext(TaskAttemptImpl.java:686) ... 14 more 2013-06-18 23:06:40,906 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..
