[
https://issues.apache.org/jira/browse/HADOOP-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12603622#action_12603622
]
Arun C Murthy commented on HADOOP-3517:
---------------------------------------
bq. would just calling doInMemMerge after the while (!exitInMemMerge) loop in
InMemFSMergeThread work?
It probably would work, but as with race-conditions it's hard to _prove_ it
works. Either way, this is hacky at best. *smile*
bq.1. The condition in ShuffleRamManager.reserve is not related to condition in
ShuffleRamManager.waitForDataToMerge(). So there could be situations (depends
on the various thresholds set) when the merge thread is waiting for thresholds
to be met; and at the same time shuffle is stalled waiting for merge to happen.
This definitely should not happen - the idea is that amount of data in-memory
and no. of in-memory file are the only thresholds. 'dataAvailable' and 'close'
are the variables tracking these.
bq. 2. i think the RamManger.reserve(int requestedSize, InputStream in)
interface could be simplied to RamManger.reserve(int requestedSize). There is
no real need to close the connection in reserve method. The connection could be
created after the call to reserve in getMapOutput method.
No. We need to open to connection to 'figure' _requestedSize_ and we absolutely
need to close the connection if we are waiting for requestedSize to be
available...
> The last InMemory merge may be missed
> -------------------------------------
>
> Key: HADOOP-3517
> URL: https://issues.apache.org/jira/browse/HADOOP-3517
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Devaraj Das
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.18.0
>
>
> This is post HADOOP-3366. The inmem merge thread has the loop:
> {code}
> while (!exitInMemMerge) {
> ramManager.waitForDataToMerge();
> doInMemMerge();
> }
> {code}
> The fetchOutputs, at the end of copying everything, does the following:
> {code}
> exitInMemMerge = true;
> ramManager.close();
> {code}
> Now if the merge thread is doing a merge (inside the doInMemMerge method)
> when the exitInMemMerge is set to true, the loop will break and the last
> merge of the files that got shuffled recently will be skipped.
> ramManager.close(), that internally does a final notify to the merge thread
> also won't have any effect in this case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
