[
https://issues.apache.org/jira/browse/MAPREDUCE-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated MAPREDUCE-3988:
----------------------------------
Attachment: MAPREDUCE-3988-1.txt
This patch uses the LocalDirAllocator.getLocalPathToRead() API to first check
for existence of a scratch directory before using the
LocalDirAllocator.getLocalPathForWrite() API if the directory does not exist.
There are a couple of race conditions with this implementation because of the
fact that each YarnChild task attempt is running in its own JVM. This patch
addresses one of the race conditions.
The race addressed by this patch is when 2 or more tasks at the same time
detect there is no mapreduce.job.local.dir and try to create it in the same
directory at the same time. Whichever one loses the race will get a
FileAlreadyExistsException, which is ignored.
There is still a race condition wherein 2 tasks running on the same node could
both see that there is no mapreduce.job.local.dir and then try to create it in
2 different directories; for eg, one in /grid/0 and one in /grid/1.
We could go a step further in addressing the second race and try to hash the
attempt ID and map it to the same directory, but that seems overly complicated.
> mapreduce.job.local.dir doesn't point to a single directory on a node.
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-3988
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3988
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Eric Payne
> Attachments: MAPREDUCE-3988-1.txt
>
>
> After MAPREDUCE-3975, mapreduce.job.local.dir is set correctly for the tasks
> but it doesn't point to the same directory for all tasks running on the node.
> It is a public API. Either we should point to a single directory or point it
> to all directories and change the documentation to say that it points to all
> dirs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
