[
https://issues.apache.org/jira/browse/HADOOP-4714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12651298#action_12651298
]
Jothi Padmanabhan commented on HADOOP-4714:
-------------------------------------------
bq. does progress() in writeFile() just set a flag with maybe no consequences
Yes, progress does set a flag, but there is a separate thread that looks at
this flag value and the progress is propogated.
Could you let us also know what io.sort.factor is and the total number of
reduces for this application?
>From the above logs, the total number of spills is 13 and it is also apparent
>that your io.sort.factor is > 13 and there is no multi-level merge; there is
>only one merge of all these 13 files.
In the above log, reducer 0's total intermediate output size is 111126 bytes,
which implies 111126/256 = 434 records. Since this number is < 10,000, there
will not be any progress reported for this reducer.
If all the reducers have similar number of record counts, there will no
progress information at all.
A simple fix to try would be to send a progress right at the beginning.
{code}
if ((++recordCtr % PROGRESS_BAR) == 0) {
progressable.progress();
}
{code}
to
{code}
if ((recordCtr++ % PROGRESS_BAR) ==0) {
progressable.progress();
}
{code}
> map tasks timing out during merge phase
> ---------------------------------------
>
> Key: HADOOP-4714
> URL: https://issues.apache.org/jira/browse/HADOOP-4714
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.1
> Reporter: Christian Kunz
>
> With compression of transient data turned on some parts of the merge phase
> seem to not report progress enough.
> We see a lot of task failures during the merge phase, most of them timing out
> (even with a 20 min timeout)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
