[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309767#comment-16309767
 ] 

Jason Lowe commented on MAPREDUCE-7015:
---------------------------------------

IIRC one reason the moveToDoneExecutor was added was to avoid spending a long 
time processing every move necessary during the scan.  By adding the sync 
parameter or removing the executor completely, that makes the scan move 
everything it finds inline. This can take quite a long time since it not only 
has to wait for RPC calls but also can spend a long time waiting to acquire the 
lock because another thread could be calling loadJob() at the time and waiting 
on a very slow datanode.  That makes the scan single-threaded since it can't 
make progress on other intermediate files until it finishes moving each one in 
order.  So I don't think making this sync is the ideal solution.

It may make more sense to have the RPC call wait for out-of-band intermediate 
scan results it will return to be moved rather than forcing the entire scan to 
be single-threaded or always waiting for all intermediates to be moved.  For 
example, HistoryFileManager#getFileInfo could explicitly wait on the move to 
complete for the one job it is interested in if that job was found in the 
intermediate scan.  Then we don't have to wait for _every_ intermediate job to 
be moved, just the one we care about.  getAllFileInfo would need to wait for 
all of them, but they would be processed in parallel while we're waiting for 
each in turn.

> Possible race condition in JHS if the job is not loaded
> -------------------------------------------------------
>
>                 Key: MAPREDUCE-7015
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7015
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: jobhistoryserver
>            Reporter: Peter Bacsko
>            Assignee: Peter Bacsko
>         Attachments: MAPREDUCE-7015-POC01.patch
>
>
> There could be a race condition inside JHS. In our build environment, 
> {{TestMRJobClient.testJobClient()}} failed with this exception:
> {noformat}
> ava.io.FileNotFoundException: File does not exist: 
> hdfs://localhost:32836/tmp/hadoop-yarn/staging/history/done_intermediate/jenkins/job_1509975084722_0001_conf.xml
>       at 
> org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1266)
>       at 
> org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1258)
>       at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>       at 
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1258)
>       at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
>       at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
>       at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2123)
>       at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2092)
>       at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2068)
>       at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:460)
>       at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>       at 
> org.apache.hadoop.mapreduce.TestMRJobClient.runTool(TestMRJobClient.java:94)
>       at 
> org.apache.hadoop.mapreduce.TestMRJobClient.testConfig(TestMRJobClient.java:551)
>       at 
> org.apache.hadoop.mapreduce.TestMRJobClient.testJobClient(TestMRJobClient.java:167)
> {noformat}
> Root cause:
> 1. MapReduce job completes
> 2. CLI calls {{cluster.getJob(jobid)}}
> 3. The job is finished and the client side gets redirected to JHS
> 4. The job data is missing from {{CachedHistoryStorage}} so JHS tries to find 
> the job
> 5. First it scans the intermediate directory and finds the job
> 6. The call {{moveToDone()}} is scheduled for execution on a separate thread 
> inside {{moveToDoneExecutor}} and it starts to run immediately
> 7. RPC invocation returns with the path pointing to 
> {{/tmp/hadoop-yarn/staging/history/done_intermediate}}
> 8. The call to {{moveToDone()}} completes which moves the contents of 
> {{done_intermediate}} to {{done}}
> 9. Hadoop CLI tries to download the config file from done_intermediate but 
> it's no longer there
> Usually step #6 is slow enough to complete after #7, but sometimes it's 
> faster, causing this race condition.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to