[
https://issues.apache.org/jira/browse/HADOOP-4714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12651400#action_12651400
]
Christian Kunz commented on HADOOP-4714:
----------------------------------------
io.sort.factor is 100, number of reducers 18,000.
>From your comment I would conclude that recordCtr is reset for every reducer,
>making progress reporting dependent on the number of reducers. That would not
>be good.
In my opiniion progress reporting should be configured to be at the minimum
once per a certain time period (e.g. 1 minute). Using number of processed
records could be used for cnvenience (although decompression/compression slows
processing down a lot), but it would have to be a counter independent of the
number of reducers.
> map tasks timing out during merge phase
> ---------------------------------------
>
> Key: HADOOP-4714
> URL: https://issues.apache.org/jira/browse/HADOOP-4714
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.1
> Reporter: Christian Kunz
>
> With compression of transient data turned on some parts of the merge phase
> seem to not report progress enough.
> We see a lot of task failures during the merge phase, most of them timing out
> (even with a 20 min timeout)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
