[
https://issues.apache.org/jira/browse/HADOOP-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635922#action_12635922
]
Christian Kunz commented on HADOOP-3841:
----------------------------------------
To get beyond this bottleneck, for such reduces requiring a lot disk space for
merging, we deleted all map outputs on such nodes, getting back a lot of space.
On one of these nodes with about 280GB reduce input we observed that one of the
merged files was 75GB (a single file!)
> merge phase runs out of disk space
> ----------------------------------
>
> Key: HADOOP-3841
> URL: https://issues.apache.org/jira/browse/HADOOP-3841
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.2
> Reporter: Christian Kunz
>
> We observe that reduce tasks run out of disk space during merging (after
> fetching all map output) although there would be enough space if the
> framework did not try to generate too large merge files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
