[
https://issues.apache.org/jira/browse/HADOOP-3326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12598906#action_12598906
]
Amar Kamat commented on HADOOP-3326:
------------------------------------
Some comments
1) Make {{copiersDone}} volatile.
2) I am not sure if changing the ordering of triggering InMem merge and
removing it from the neededOutputs is safe. Sharad, can you plz verify this?
3) Remove extra newlines.
4) Since there is only one thread we can use {{notify()}} instead of
{{notifyAll()}}
5) The In-Mem merge trigger condition was
{code:title=Condition #1 (Before)|borderStyle=solid}
!mergeInProgress &&
(inMemFileSys.getPercentUsed() >= MAX_INMEM_FILESYS_USE ||
(mergeThreshold > 0 &&
inMemFileSys.getNumFiles(MAP_OUTPUT_FILTER) >=
mergeThreshold))&&
mergeThrowable == null
{code}
was changed to
{code:title=Condition #2 (After)|borderStyle=solid}
mergeThreshold<0 && inMemFileSys.getNumFiles(MAP_OUTPUT_FILTER) < mergeThreshold
{code}
Since condition #1 was the triggering condition and condition #2 is used to
wait for the triggering condition, condition #1 = ~ condition #2. Hence it
should be
{code}
inMemFileSys.getPercentUsed() < MAX_INMEM_FILESYS_USE && (mergeThreshold <= 0
|| inMemFileSys.getNumFiles(MAP_OUTPUT_FILTER) < mergeThreshold)
{code}
Or am I missing something?
BTW the approach seems fine.
> ReduceTask should not sleep for 200 ms while waiting for merge to finish
> ------------------------------------------------------------------------
>
> Key: HADOOP-3326
> URL: https://issues.apache.org/jira/browse/HADOOP-3326
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Sharad Agarwal
> Attachments: 3326_1.patch
>
>
> Currently the merge code in Reduce task does:
> {code}
> // Wait for the on-disk merge to complete
> while (localFSMergeInProgress) {
> Thread.sleep(200);
> }
>
> //wait for an ongoing merge (if it is in flight) to complete
> while (mergeInProgress) {
> Thread.sleep(200);
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
