[
https://issues.apache.org/jira/browse/MAPREDUCE-2541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044395#comment-13044395
]
Hadoop QA commented on MAPREDUCE-2541:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12481461/MAPREDUCE-2541.v2.patch
against trunk revision 1131265.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.cli.TestMRCLI
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results:
https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/349//testReport/
Findbugs warnings:
https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/349//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/349//console
This message is automatically generated.
> Race Condition in IndexCache(readIndexFileToCache,removeMap) causes value of
> totalMemoryUsed corrupt, which may cause TaskTracker continue throw Exception
> ----------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2541
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2541
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 0.20.1, 0.21.0, 0.22.0, 0.23.0
> Environment: all
> Reporter: Binglin Chang
> Priority: Critical
> Attachments: MAPREDUCE-2541.patch, MAPREDUCE-2541.v2.patch
>
>
> The race condition goes like this:
> Thread1: readIndexFileToCache() totalMemoryUsed.addAndGet(newInd.getSize())
> Thread2: removeMap() totalMemoryUsed.addAndGet(-info.getSize());
> When SpillRecord is being read from fileSystem, client kills the job,
> info.getSize() equals 0, so in fact totalMemoryUsed is not reduced, but after
> thread1 finished reading SpillRecord, it adds the real index size to
> totalMemoryUsed, which makes the value of totalMemoryUsed wrong(larger).
> When this value(totalMemoryUsed) exceeds totalMemoryAllowed (this usually
> happens when a vary large job with vary large reduce number is killed by the
> user, probably because the user sets a wrong reduce number by mistake), and
> actually indexCache has not cache anything, freeIndexInformation() will throw
> exception constantly.
> A quick fix for this issue is to make removeMap() do nothing, let
> freeIndexInformation() do this job only.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
