[
https://issues.apache.org/jira/browse/MAPREDUCE-1906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Scott Carey updated MAPREDUCE-1906:
-----------------------------------
Status: Patch Available (was: Open)
MAPREDUCE-1906-0.21-v2.patch
changes ping to a smooth function from a step function and lowers the minimum
to 300ms. Clusters larger than 300 nodes only see the step-function > smooth
function change. Clusters between 30 and 300 nodes smoothly increase their
ping interval. Clusters with 30 nodes or less have 300ms ping intervals when
the TT has nothing to do. This improves scheduling latency on small clusters
significantly.
The cluster wide ping interval is roughly proportional to how fast the cluster
can schedule a job.
|| cluster size || current ping interval (ms) || current ping rate at JT ||
patched ping interval (ms) || patched ping rate at JT ||
| 10 | 3000 | 3.33 /sec | 300 | 33.3 /sec |
| 30 | 3000 | 10 /sec | 300 | 100 /sec |
| 100 | 3000 | 33.3 /sec | 1000 | 100 /sec |
| 300 | 3000 | 100 /sec | 3000 | 100 /sec |
| 301 | 4000 | 75 /sec | 3010 | 100 /sec |
| 1000 | 10000 | 100 /sec | 10000 | 100 /sec |
| 1001 | 11000 | 91 /sec | 10010 | 100 /sec |
> Lower minimum heartbeat interval for tasktracker > Jobtracker
> -------------------------------------------------------------
>
> Key: MAPREDUCE-1906
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1906
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.20.2, 0.20.1
> Reporter: Scott Carey
> Attachments: MAPREDUCE-1906-0.21-v2.patch, MAPREDUCE-1906-0.21.patch
>
>
> I get a 0% to 15% performance increase for smaller clusters by making the
> heartbeat throttle stop penalizing clusters with less than 300 nodes.
> Between 0.19 and 0.20, the default minimum heartbeat interval increased from
> 2s to 3s. If a JobTracker is throttled at 100 heartbeats / sec for large
> clusters, why should a cluster with 10 nodes be throttled to 3.3 heartbeats
> per second?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
