[
https://issues.apache.org/jira/browse/HADOOP-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12604411#action_12604411
]
Arun C Murthy commented on HADOOP-3517:
---------------------------------------
bq. -1 findbugs. The patch appears to introduce 1 new Findbugs warnings.
Findbugs seems confused...
http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2642/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
says:
{quote}
IS Inconsistent synchronization of
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$ShuffleRamManager.numPendingRequests;
locked 66% of time
Bug type IS2_INCONSISTENT_SYNC (click for details)
In class org.apache.hadoop.mapred.ReduceTask$ReduceCopier$ShuffleRamManager
Field
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$ShuffleRamManager.numPendingRequests
Synchronized 66% of the time
Unsynchronized access at ReduceTask.java:[line 793]
Unsynchronized access at ReduceTask.java:[line 793]
Synchronized access at ReduceTask.java:[line 723]
Synchronized access at ReduceTask.java:[line 753]
Synchronized access at ReduceTask.java:[line 753]
Synchronized access at ReduceTask.java:[line 762]
Synchronized access at ReduceTask.java:[line 762]
{quote}
which isn't true! numPendingRequests is always guarded by 'dataAvailable'.
> The last InMemory merge may be missed
> -------------------------------------
>
> Key: HADOOP-3517
> URL: https://issues.apache.org/jira/browse/HADOOP-3517
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Devaraj Das
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: HADOOP-3517_1_20080610.patch,
> HADOOP-3517_2_20080611.patch, HADOOP-3517_2_20080611.patch
>
>
> This is post HADOOP-3366. The inmem merge thread has the loop:
> {code}
> while (!exitInMemMerge) {
> ramManager.waitForDataToMerge();
> doInMemMerge();
> }
> {code}
> The fetchOutputs, at the end of copying everything, does the following:
> {code}
> exitInMemMerge = true;
> ramManager.close();
> {code}
> Now if the merge thread is doing a merge (inside the doInMemMerge method)
> when the exitInMemMerge is set to true, the loop will break and the last
> merge of the files that got shuffled recently will be skipped.
> ramManager.close(), that internally does a final notify to the merge thread
> also won't have any effect in this case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
