Github user Gvain commented on the pull request:
https://github.com/apache/incubator-storm/pull/103#issuecomment-44372134
@clockfly
I used latest storm to reproduce some of your test. I found that adding a
few more workers will increase throughput, as well as CPU usage. I also get a
max doubled throughput.
Im my test, i used 4 nodes(24 cores each, not power enough than yours), 48
spouts, 48 bolts, 48 ackers, and 4 to 64 workers, SOL benchmark, message size
is 100 Bytes. Here is the results:
workers | Throughput | CPU usage | NET usage (only IN Bytes)
---------------------------------------------------------------------
4 | 320,000 tps | 56% | 14MB/s
8 | 656,000 tps | 89% | 28MB/s
16 | 560,000 tps | 92% | 26MB/s
32 | 353,000 tps | 90% | 20MB/s
64 | 208,000 tps | 90% | 16MB/s
Using 8 workers, the throughput doubled.
As we already discussed:
(a) Increasing worker count, not too much, may not cause netty context
switching problem. As using a SHARED threadpool(its default size is 1) among
all netty client within a worker, the netty threads number do not increase as
total worker numbers increase. Check
https://github.com/apache/incubator-storm/pull/57
(b) Increasing worker count will increase netty threads. Using more netty
threads working in sync and non-batch mode may have some what the same effect
with using less netty threads working in async and batch mode which is your way.
From the test result, it seems like the above point of view (b) is true.
So, maybe, what we need to to is just pulling the
https://github.com/apache/incubator-storm/pull/57 and adding a few workers.
But i am still curious about if i adding more nodes, should this still work
?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
