mapred should provide an optional upper-bound for size of map outputs
---------------------------------------------------------------------
Key: HADOOP-2117
URL: https://issues.apache.org/jira/browse/HADOOP-2117
Project: Hadoop
Issue Type: Improvement
Components: mapred
Affects Versions: 0.14.3
Environment: All
Reporter: Milind Bhandarkar
Fix For: 0.16.0
AFAIK, currently, there is no explicit limit on the amount of disk space
available for map tasks to emit map-outputs. They can use whatever space is
available (This corresponds to total disk space minus disk space currently used
by data-nodes to store dfs blocks.) Thus, some jobs that run fine when DFS is
less full, stop working when dfs becomes more full.
If mapred had an (optional) upper-limit on the amount of local disk space used
for map outputs, these jobs will run more predictably.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
