[
https://issues.apache.org/jira/browse/HADOOP-5146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated HADOOP-5146:
--------------------------------
Attachment: localdirallocator.patch
I think I found the cause of this. Basically there is a race condition in
TaskRunner where the distributed cache files are localized. Multiple TaskRunner
threads may end up trying to localize the same files. The attached patch
prevents this from happening. Arun, could you please take a shot at this?
Thanks!
> LocalDirAllocator misses files on the local filesystem
> ------------------------------------------------------
>
> Key: HADOOP-5146
> URL: https://issues.apache.org/jira/browse/HADOOP-5146
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Arun C Murthy
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: localdirallocator.patch, localdirallocator.patch
>
>
> For some reason the LocalDirAllocator.getLocaPathToRead doesn't find files
> which are present, extra logging shows:
> {noformat}
> 2009-01-30 06:43:32,312 INFO
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: in ifExists,
> /grid/2/arunc/mapred-local/taskTracker/archive/xxx.yyy.com/tera/in/_partition.lst
> exists
> 2009-01-30 06:43:32,389 WARN
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: in
> getLocalPathToRead, taskTracker/archive/xxx.yyy.com/tera/in/_partition.lst
> doesn't exist
> 2009-01-30 06:43:32,390 WARN org.apache.hadoop.mapred.TaskRunner:
> attempt_200901300512_0007_m_000055_0 Child Error
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
> taskTracker/archive/xx.yyy.com/tera/in/_partition.lst in any of the
> configured local directories
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:388)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:172)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
