GitHub user revans2 opened a pull request:
https://github.com/apache/storm/pull/847
YSTORM-162: Load Aware Shuffle Grouping
I am recreating this pull request. Originally there was discussion about
using the velocity of a bolt in addition to the queue size in determining the
routing, but I didn't have time to implement that, and we have been running
with this code in production for quite a while, so I decided to just put it up,
as it has a really positive impact. This is most pronounced on heterogeneous
clusters where not all nodes have the same CPU. But also this improves the
performance of the groupings implementations significantly dropping the 99th
and 99.9th %-lie latencies significantly for the Latency vs Throughput test
case.
I would still like to explore using velocity and network distance in the
routing calculations, but in a follow on JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/revans2/incubator-storm STORM-162
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/847.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #847
----
commit 579ec324fc6172b61ef7fd3d14a532c1642f99fb
Author: Bobby Evans <[email protected]>
Date: 2015-02-10T15:07:21Z
YSTORM-162: Load Aware Shuffle Grouping
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
