GitHub user revans2 opened a pull request:
https://github.com/apache/storm/pull/704
Disruptor Queue Batching (DON'T MERGE YET)
This is mostly a proof of concept, as a comparison to
https://github.com/apache/storm/pull/694. They may complement each other. Of
there may be a lot of overlap between them. I want to run some more tests.
Currently I have a few benchmarks based on this branch showing the CPU savings,
and increased performance by doing some batching.
The following were all using the modified version of wordcount on my
branch. They were run on my Mac Book Pro with 8 cores. The first number is
the max spout pending. The second number is the batch size, and the third
number is the disruptor queue timeout.
Also before merging anything I want to explore reducing the number of
threads and queues. Because if context switching is killing us having less
threads should make it a lot better, and may reduce the need for this a lot.
wc-1-1-100 8 cores System - 30% User - 60% Idle - 10%
TIME | EMITTED | TRANS | COMP LAT | ACKED | FAILED
---|---|---|---|---|---
2m 41s | 64940 | 64940 | 11.852 | 64240 | 0
5m 7s | 145780 | 145780 | 10.254 | 144840 | 0
7m 4s | 209680 | 209680 | 9.932 | 208640 | 0
wc-1-100-100 8 cores System - 4% User - 5% Idle - 91%
TIME | EMITTED | TRANS | COMP LAT | ACKED | FAILED
---|---|---|---|---|---
7m 0s | 1540 | 1540 | 1138.412 | 1700 | 0
wc-10-1-100 8 cores System - 30% User - 65% Idle - 6%
TIME | EMITTED | TRANS | COMP LAT | ACKED | FAILED
---|---|---|---|---|---
2m 46s | 66540 | 66540 | 99.480 | 65400 | 0
5m 6s | 142300 | 142300 | 95.101 | 141540 | 0
7m 1s | 206660 | 206660 | 93.951 | 206600 | 0
wc-10-2-100 8 cores System - 30% User - 65% Idle - 6%
TIME | EMITTED | TRANS | COMP LAT | ACKED | FAILED
---|---|---|---|---|---
3m 9s | 94340 | 94340 | 90.511 | 93940 | 0
5m 0s | 155420 | 155420 | 90.281 | 154700 | 0
7m 2s | 223020 | 223020 | 90.047 | 221800 | 0
wc-10-3-100 8 cores System - 30% User - 65% Idle - 6%
TIME | EMITTED | TRANS | COMP LAT | ACKED | FAILED
---|---|---|---|---|---
2m 42s | 70560 | 70560 | 98.507 | 70120 | 0
5m 2s | 150040 | 150040 | 92.656 | 150000 | 0
7m 3s | 217400 | 217400 | 90.777 | 218920 | 0
wc-200-1-100 8 cores System - 50% User - 50% Idle - 0%
TIME | EMITTED | TRANS | COMP LAT | ACKED | FAILED
---|---|---|---|---|---
2m 49s | 46440 | 46440 | 3083.811 | 45160 | 0
5m 6s | 94920 | 94920 | 2990.687 | 93340 | 0
7m 3s | 133920 | 133920 | 3008.993 | 132600 | 0
wc-200-100-100 8 cores System - 20% User - 30-40% Idle - 40-50%
TIME | EMITTED | TRANS | COMP LAT | ACKED | FAILED
---|---|---|---|---|---
2m 48s | 48680 | 48680 | 2962.734 | 47480 | 0
5m 3s | 99840 | 99840 | 2885.372 | 97640 | 0
7m 3s | 142960 | 142960 | 2845.131 | 140860 | 0 |
wc-400-1-100 8 cores System - 50% User - 50% Idle - 0%
TIME | EMITTED | TRANS | COMP LAT | ACKED | FAILED
---|---|---|---|---|---
5m 6s | 100880 | 100880 | 5725.489 | 97880 | 0
7m 13s | 141400 | 141400 | 5780.984 | 138160 | 0
wc-400-100-100 8 cores System - 25% User - 40-50% Idle - 20-30%
TIME | EMITTED | TRANS | COMP LAT | ACKED | FAILED
---|---|---|---|---|---
2m 46s | 63960 | 63960 | 4206.964 | 62700 | 0
5m 0s | 136720 | 136720 | 4011.031 | 135440 | 0
7m 1s | 197640 | 197640 | 3982.946 | 196520 | 0
wc-1000-200-100 8 cores System - 28% User - 68% Idle - 3%
TIME | EMITTED | TRANS | COMP LAT | ACKED | FAILED
---|---|---|---|---|---
2m 43s | 77000 | 77000 | 8417.592 | 73480 | 0
5m 4s | 158560 | 158560 | 8439.014 | 155800 | 0
7m 4s | 230240 | 230240 | 8456.893 | 226580 | 0
wc-1000-1-100 8 cores System - 50% User - 50% Idle - 0%
TIME | EMITTED | TRANS | COMP LAT | ACKED | FAILED
---|---|---|---|---|---
2m 43s | 39980 | 39980 | 18441.455 | 35140 | 20
5m 5s | 84520 | 84520 | 16939.596 | 79600 | 20
7m 6s | 121720 | 121720 | 16697.250 | 116620 | 20
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/revans2/incubator-storm disruptor-batching
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/704.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #704
----
commit 329155a61a3581cd1a3ce3aebdb8dcb3a64e0cb3
Author: Robert (Bobby) Evans <[email protected]>
Date: 2015-08-28T01:39:59Z
Disruptor Queue Batching.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
