Please let us know your final results. Interesting to see YARN+MR directly working on local-file-system.
Thanks, +Vinod On Jun 20, 2013, at 2:27 PM, Stephen Watt wrote: > I resolved this. The issue is that I was using relative paths (i.e "teragen > 1000 data/in-dir") as the params for TeraGen and TeraSort. When I changed it > to use absolute paths, (i.e. "teragen 1000 /data/in-dir") it works. > > ----- Original Message ----- > From: "Stephen Watt" <[email protected]> > To: [email protected] > Sent: Thursday, June 20, 2013 12:25:17 PM > Subject: FileNotFoundExceptions with Pseudo Distributed YARN MR using the > Local FileSystem > > Hi Folks > > I'm running into FileNotFoundExceptions when using using Pseudo Distributed > Single Node YARN using the Local FileSystem. I'd greatly appreciate any > insights/solutions. > > To level set, I'm using RHEL 6.2 and I've successfully setup a single node > pseudo-distributed YARN on HDFS 2.0 using the HDP 2.0.2 Alpha Release > (tarball extract to /opt). All the processes were started and the jobs > submitted as root. I ran some smoke tests with TeraGen and TeraSort and it > works great. > > The next step was to leave YARN in pseudo-distributed mode and stop HDFS and > change the Hadoop FileSystem from HDFS to the Local FileSystem. I stopped all > the daemons, changed the core-site.xml to use the Local FileSystem as > demonstrated below, and then restarted the resourcemanager, nodemanager and > historyserver. Still running as root, everything started just fine. I ran > TeraGen (params: 1000 data/in-dir) it worked fine. I then ran TeraSort > (params: data/in-dir data/out-dir) and the Job Failed with a > FileNotFoundException. I've provided my core-site and mapred-site below. > > -- core-site.xml -- > > <configuration> > > <property> > <name>fs.default.name</name> > <value>file:///</value> > </property> > > </configuration> > > -- mapred-site.xml -- > > <configuration> > > <property> > <name>mapreduce.framework.name</name> > <value>yarn</value> > </property> > > </configuration> > > -- Stack Trace Exception -- > > 2013-06-18 23:06:40,876 INFO [AsyncDispatcher event handler] > org.apache.hadoop.yarn.util.RackResolver: Resolved yarn-1 to /default-rack > 2013-06-18 23:06:40,881 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned > container container_1371596024885_0003_01_000002 to > attempt_1371596024885_0003_m_000000_0 > 2013-06-18 23:06:40,881 INFO [RMCommunicator Allocator] > org.apache.hadoop.yarn.util.RackResolver: Resolved yarn-1 to /default-rack > 2013-06-18 23:06:40,882 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned > container container_1371596024885_0003_01_000003 to > attempt_1371596024885_0003_m_000001_0 > 2013-06-18 23:06:40,882 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating > schedule, headroom=4096 > 2013-06-18 23:06:40,882 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start > threshold not met. completedMapsForReduceSlowstart 1 > 2013-06-18 23:06:40,882 INFO [RMCommunicator Allocator] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: > PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:2 AssignedReds:0 > CompletedMaps:0 CompletedReds:0 ContAlloc:2 ContRel:0 HostLocal:0 RackLocal:2 > 2013-06-18 23:06:40,896 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-jar file > on the remote FS is > file:///tmp/hadoop-yarn/staging/root/.staging/job_1371596024885_0003/job.jar > 2013-06-18 23:06:40,901 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-conf > file on the remote FS is > /tmp/hadoop-yarn/staging/root/.staging/job_1371596024885_0003/job.xml > 2013-06-18 23:06:40,902 FATAL [AsyncDispatcher event handler] > org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread > org.apache.hadoop.yarn.YarnException: java.io.FileNotFoundException: File > file:/opt/hadoop-2.0.3.22-alpha-hdp/nm-local-dir/usercache/root/appcache/application_1371596024885_0003/container_1371596024885_0003_01_000001/data/out-dir/_partition.lst#_partition.lst > does not exist > at > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createCommonContainerLaunchContext(TaskAttemptImpl.java:723) > at > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createContainerLaunchContext(TaskAttemptImpl.java:771) > at > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1352) > at > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1310) > at > org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:359) > at > org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:299) > at > org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43) > at > org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:445) > at > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1018) > at > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:142) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1116) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1108) > at > org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:130) > at > org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.io.FileNotFoundException: File > file:/opt/hadoop-2.0.3.22-alpha-hdp/nm-local-dir/usercache/root/appcache/application_1371596024885_0003/container_1371596024885_0003_01_000001/data/out-dir/_partition.lst#_partition.lst > does not exist > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:492) > at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:697) > at > org.apache.hadoop.fs.FilterFileSystem.resolvePath(FilterFileSystem.java:144) > at > org.apache.hadoop.mapreduce.v2.util.MRApps.parseDistributedCacheArtifacts(MRApps.java:417) > at > org.apache.hadoop.mapreduce.v2.util.MRApps.setupDistributedCache(MRApps.java:365) > at > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createCommonContainerLaunchContext(TaskAttemptImpl.java:686) > ... 14 more > 2013-06-18 23:06:40,906 INFO [AsyncDispatcher event handler] > org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..
