[
https://issues.apache.org/jira/browse/MAPREDUCE-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797827#action_12797827
]
Chris Douglas commented on MAPREDUCE-1363:
------------------------------------------
bq. This could be avoided by either improving the estimation of the size of the
spill (increasing it by a constant amount or by constant percentage), or
LocalDirAllocator could take into consideration a configuration parameter
specifying how much extra unused space should be on the path returned by
getLocalPathForWrite
Yet another esoteric, static configuration parameter would be an unfortunate
approach and a fixed percentage is patently wrong (wordcount-like applications
will overestimate, not underestimate space requirements). A decent fix could
estimate and adapt requests for space based on the expansion/contraction
observed in prior spills. Most combiners' ratio won't be a simple scalar, but a
function of key distribution during both the collection and merge phases; this
ratio may not be stable between those phases, either. That said, it's not a bad
place to start.
One approach could initialize a float using a configurable ratio (defaulting to
1.0) adjusted at each spill. This could naively update a global average for
each spill, use an exponential moving average to avoid radical shifts due to
abnormally large records, odd skews, etc. Whether the ratio is tracked for each
invocation of the combiner and not over the entire spill doesn't seem too
important if the ratio is stable, which this implicitly assumes. Tracking
per-partition expansion/contraction is probably more correct for some
applications, but of dubious value generally.
bq. In case there is no space left on a device designated for writing
intermediate data on, the spill could be retried on a different device (without
the failure of the map task).
This is slightly tricky, as one would want to copy the data already spilled to
the end of the prior partition and re-run the combiner on the partition that
caused the error (since partially serialized data causes all sorts of problems)
before continuing the spill.
> Spill size underestimated when using certain combiners, causing map tasks to
> fail
> ---------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1363
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1363
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task
> Affects Versions: 0.20.1
> Reporter: Miklos Erdelyi
>
> Spill size could get underestimated when using certain combiners, causing map
> tasks to fail when disk space is really low in some mapred.local.dir.
> When doing sortAndSpill(), MapOutputBuffer gets an output path through
> LocalDirAllocator which checks if the estimated size of the spill is
> available on any paths specified for intermediate data storage. In case a
> combiner is specified which emits key-value pairs having serialized size
> larger than the input key-value pairs' size, the size of the spill file is
> underestimated. If LocalDirAllocator selects a path for intermediate data
> storage which does not have enough space to hold the spilled records, an
> IOException is thrown and the map task fails.
> This could be avoided by either improving the estimation of the size of the
> spill (increasing it by a constant amount or by constant percentage), or
> LocalDirAllocator could take into consideration a configuration parameter
> specifying how much extra unused space should be on the path returned by
> getLocalPathForWrite (similarly to dfs.datanode.du.reserved). In case there
> is no space left on a device designated for writing intermediate data on, the
> spill could be retried on a different device (without the failure of the map
> task).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
