[
https://issues.apache.org/jira/browse/MAPREDUCE-3895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13213946#comment-13213946
]
Nathan Roberts commented on MAPREDUCE-3895:
-------------------------------------------
Mahadev, here's some data I gathered for a 24 hour period from 3 different
grids (good sized grids, consistent job mix from day to day).
Took all successful jobs which had a run_time greater than 60 seconds and then
calculated failed_maps/total_maps and failed_reduces/total_reduces under the
assumption that almost all failed tasks are speculative tasks. (let me know if
that doesn't make sense to do).
{code}
Grid Map Reduce
==== === ======
1 19% 20%
2 8% 16%
3 17% 23%
{code}
> Speculative execution algorithm in 1.0 is too pessimistic in many cases
> -----------------------------------------------------------------------
>
> Key: MAPREDUCE-3895
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3895
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker, performance
> Affects Versions: 1.0.0
> Reporter: Nathan Roberts
>
> We are seeing many instances where largish jobs are ending up with 30-50% of
> reduce tasks being speculatively re-executed. This can be a significant drain
> on cluster resources.
> The primary reason is due to the way progress in the reduce phase can make
> huge jumps in a very short amount of time. This fact leads the speculative
> execution code to think lots of tasks have fallen way behind the average when
> in fact they haven't
> The important piece of the algorithm is essentially:
> * Am I more than 20% behind the average progress?
> * Have I been running for at least a minute?
> * Have any tasks completed yet?
> Unfortunately, a set of reduce tasks which spend a couple of minutes in the
> Copy phase, and very little time in the Sort phase, will trigger all these
> conditions for a large percentage of the reduce tasks. (the tasks' progress
> jump from 33% to 66% almost instantly which then triggers the speculation).
> I've seen this on several very large jobs which spend about 2 minutes in
> Copy, a few seconds in Sort, and 40 minutes in Reduce. These jobs launch
> about 30-40% additional reduce tasks which then run for almost the full 40
> minutes.
> This area becomes more plugable in MRv2 but for 1.0 it would be good if some
> portion of this algorithm could be configurable so that a job could have some
> degree of control (just disabling speculative execution is not really an
> option).
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
