[
https://issues.apache.org/jira/browse/HADOOP-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12603788#action_12603788
]
Sharad Agarwal commented on HADOOP-3517:
----------------------------------------
{quote} It probably would work, but as with race-conditions it's hard to prove
it works. Either way, this is hacky at best. {quote}
not sure I understand the possible race condition here. with which
thread/threads do you see the race condition? exitInMemMerge is set to true
only after all copier threads are done.
{quote} This definitely should not happen - the idea is that amount of data
in-memory and no. of in-memory file are the only thresholds. 'dataAvailable'
and 'close' are the variables tracking these. {quote}
Thread waiting conditions are based on thresholds (MAX_INMEM_FILESYS_USE,
MAX_SINGLE_SHUFFLE_SEGMENT_PERCENT etc) If you independently tweak these nos,
there could be situation for both threads waiting.
consider this case:
MAX_INMEM_FILESYS_USE = 0.66f
MAX_SINGLE_SHUFFLE_SEGMENT_PERCENT = 0.4f
mergeThreshold = 0
maxSize = 100
fullSize = size = 65
requestedSize = 36
Wont the shuffle threads and the merge thread, both be waiting for the above
case? OR am I missing anything?
{quote}No. We need to open to connection to 'figure' requestedSize and we
absolutely need to close the connection if we are waiting for requestedSize to
be available...{quote}
Agreed..it seems i missed the requestedSize part.
> The last InMemory merge may be missed
> -------------------------------------
>
> Key: HADOOP-3517
> URL: https://issues.apache.org/jira/browse/HADOOP-3517
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Devaraj Das
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.18.0
>
>
> This is post HADOOP-3366. The inmem merge thread has the loop:
> {code}
> while (!exitInMemMerge) {
> ramManager.waitForDataToMerge();
> doInMemMerge();
> }
> {code}
> The fetchOutputs, at the end of copying everything, does the following:
> {code}
> exitInMemMerge = true;
> ramManager.close();
> {code}
> Now if the merge thread is doing a merge (inside the doInMemMerge method)
> when the exitInMemMerge is set to true, the loop will break and the last
> merge of the files that got shuffled recently will be skipped.
> ramManager.close(), that internally does a final notify to the merge thread
> also won't have any effect in this case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
