I have a small hadoop cluster running 1.0.4 and I'm trying to have it setup
so that I can run jobs remotely from a computer on the same network that is
not a part of the cluster. I've got a main java class that
implements org.apache.hadoop.util.Tool and I'm able to run this job from
the NameNode using ToolRunner.run(), setting up the JobConf, and submitting
with JobClient.submitJob().
When I try to run the same class remotely from any machine that is not the
NameNode the job is submitted and it appears that the Map tasks
successfully complete, but I get the following exception for all of the
reduce tasks:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
output/map_0.out in any of the configured local directories
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
at
org.apache.hadoop.mapred.MapOutputFile.getInputFile(MapOutputFile.java:161)
at org.apache.hadoop.mapred.ReduceTask.getMapFiles(ReduceTask.java:220)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:398)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
I'm not sure how to interpret that error message. I think I'm missing
some config files that are not present on the remote machine but I
can't pin down exactly what I need. Does anyone have any guidance on
what the above error means or how to submit jobs remotely?
Thanks,
--Oren
