[
https://issues.apache.org/jira/browse/MAPREDUCE-1266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12786219#action_12786219
]
Todd Lipcon commented on MAPREDUCE-1266:
----------------------------------------
Well, actually, in trunk there's mapreduce.jobtracker.heartbeats.in.second
which sets the individual trackers such that that number of heartbeats arrive
every second. The default is 100, which would be a 10ms interval for a
pseudo-distributed cluster, which is silly. So there's a minimum as well,
hardcoded. Here's the relevant code:
{code}
int heartbeatInterval = Math.max(
(int)(1000 * HEARTBEATS_SCALING_FACTOR *
Math.ceil((double)clusterSize /
NUM_HEARTBEATS_IN_SECOND)),
HEARTBEAT_INTERVAL_MIN) ;
{code}
HEARTBEAT_INTERVAL_MIN is hard coded to 3 seconds in MRConstants.java.
Maybe I'm misunderstanding your question - are you in support of lowering the
minimum and just asking why make it undocumented-configurable instead of
hardcoded? I was offering the undocumented configuration option just in case
someone had an argument against this change. If everyone's for it, happy to
just change the constant.
> Allow heartbeat interval smaller than 3 seconds for tiny clusters
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1266
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1266
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker, task, tasktracker
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Priority: Minor
>
> For small clusters, the heartbeat interval has a large effect on job latency.
> This is especially true on pseudo-distributed or other "tiny" (<5 nodes)
> clusters. It's not a big deal for production, but new users would have a
> happier first experience if Hadoop seemed snappier.
> I'd like to change the minimum heartbeat interval from 3.0 seconds to perhaps
> 0.5 seconds (but have it governed by an undocumented config parameter in case
> people don't like this change). The cluster size-based ramp up of interval
> will maintain the current scalable behavior for large clusters with no
> negative effect.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
