[
https://issues.apache.org/jira/browse/HADOOP-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joydeep Sen Sarma updated HADOOP-2752:
--------------------------------------
Summary: ability to configure gap/lag parameters for speculative execution
for maps/reduces (was: Ability to turn on/off speculative execution for
maps/reduces separately)
thanks for pointing out the dup.
also looking for a way to configure these parameters - just going to repurpose
this bug for that. the issue is that we want to run with very conservative
settings for speculative execution - but we do want it on (to take care of
those laggard tasks that never want to come out of initialization mode ..). i
can think there is going to difference in how people with busy clusters want to
tune spec. exec. vs. those that have idle clusters.
might become moot if spec. exec. is load aware and based on completion etc. -
but until then - the fine grained control would be helpful.
> ability to configure gap/lag parameters for speculative execution for
> maps/reduces
> ----------------------------------------------------------------------------------
>
> Key: HADOOP-2752
> URL: https://issues.apache.org/jira/browse/HADOOP-2752
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Fix For: 0.16.0
>
>
> Motivation is obvious:
> - sometimes reduces have side-effects - but maps don't (for example reduces
> compute something and write it somewhere)
> - want to turn ON speculative for maps (because maps are usually way less
> expensive. map progress counters are reliable etc.) - but OFF for reduces
> (very expensive to run two reducers instead of one).
> It's also likely that in future you may want different setting for
> speculative triggers for maps and reduces ... (but that's a different issue).
> I will submit a patch (since we are putting this into effect on our cluster -
> and it seems simple enough :-)).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
