I have seen this error from time to time and it has been either due to space or
missing directories or disk errors.
Space issue was caused by the fact that the I had mounted /de/sdc on
/hadoop-dsk
and the mount had failed. And in another case I had
accidentally deleted hadoop.tmp.dir in a node and whenever the reduce job was
scheduled on that node that attempt would fail.
I usually look for where attempt_id ( in your case
attempt_201008101445_0212_r_000002_0) scheduled in the jobtracker , look in
that node and see if space and directories are Ok and sufficient and check for
disk errors.
Raj
________________________________
From: Rares Vernica <rvern...@gmail.com>
To: common-user@hadoop.apache.org
Sent: Thu, August 12, 2010 10:38:06 AM
Subject: DiskChecker$DiskErrorException: Could not find any valid local
directory -- lots of free disk space
Hello,
>From time to time I get the following error:
Error initializing attempt_201008101445_0212_r_000002_0:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
any valid local directory for
taskTracker/jobcache/job_201008101445_0212/job.xml
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:343)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:750)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1664)
at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:97)
at
org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1629)
If I restart the job without making any changes to the cluster or the
disks it eventually works. After a while I get this error again for a
different job. Restarting the job always seems to work but this is
very annoying.
I searched online and it seems that this error is triggered when there
is not enough space on any of the disks. This is not the case for me
as each node has 200GB of free space.
Is there anything else I can check besides the free space on the disks?
Thanks!
Rares
