I also noticed this in 0.20.2:
conf.getFloat("mapred.job.
shuffle.merge.percent", 0.66f);
And this in 0.21.0:
jobConf.getFloat(JobContext.SHUFFLE_MERGE_EPRCENT,
0.90f));
I suggest changing the default limit back to 0.66 - 0.9 is too high.
Also the spelling should be SHUFFLE_MERGE_P*E*RCENT
On Thu, Mar 11, 2010 at 10:49 AM, Ted Yu <[email protected]> wrote:
> I checked out code from git://github.com/apache/hadoop-mapreduce.git which
> I assume is 0.21.0
>
> The new reserve() method at line 216 has this code:
> if (usedMemory > memoryLimit) {
> LOG.debug(mapId + ": Stalling shuffle since usedMemory (" +
> usedMemory +
> ") is greater than memoryLimit (" + memoryLimit + ")");
>
> return stallShuffle;
> }
> The above check isn't enough because of the following scenario:
> requestedSize < maxSingleShuffleLimit
> usedMemory < memoryLimit
> usedMemory + requestedSize > memoryLimit
>
> Meaning we shouldn't call unconditionalReserve() after this check:
> if (usedMemory > memoryLimit) {
>
> What do you think ?
