Spill size underestimated when using a combiner, causing map tasks to fail
--------------------------------------------------------------------------
Key: MAPREDUCE-1363
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1363
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task
Affects Versions: 0.20.1
Reporter: Miklos Erdelyi
Spill size could get underestimated when using certain combiners, causing map
tasks to fail when disk space is really low in some mapred.local.dir.
When doing sortAndSpill(), MapOutputBuffer gets an output path through
LocalDirAllocator which checks if the estimated size of the spill is available
on any paths specified for intermediate data storage. In case a combiner is
specified which emits key-value pairs having serialized size larger than the
input key-value pairs' size, the size of the spill file is underestimated. If
LocalDirAllocator selects a path for intermediate data storage which does not
have enough space to hold the spilled records, an IOException is thrown and the
map task fails.
This could be avoided by either improving the estimation of the size of the
spill (increasing it by a constant amount or by constant percentage), or
LocalDirAllocator could take into consideration a configuration parameter
specifying how much extra unused space should be on the path returned by
getLocalPathForWrite (similarly to dfs.datanode.du.reserved). In case there is
no space left on a device designated for writing intermediate data on, the
spill could be retried on a different device (without the failure of the map
task).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
