Github user clockfly commented on the pull request:
https://github.com/apache/incubator-storm/pull/103#issuecomment-43627624
Gvain,
>What if we use MORE workers per node than just only ONE worker per node
without changing the total number of executors ? By doing so, we will have MORE
received threads, transfer threads and netty i/o threads for the total 144
executors. Should this increase total CPU usage and network bandwidth usage?
We tried this approach before, but it won't give us the performance data we
want. There were inherit bottleneck there. Besides, receiver.count is not the
biggest bottleneck here, netty performance matter more.
The logic behind make "receiver.count" configurable is that, since we allow
user to configure spout/bolt executor count per worker, we should also allow
user to configure the receiver thread count to make it consistent with the
parallism settings.
Besides, increasing worker count will,
1. add more cross process or cross machine communication. For tasks inside
same worker process, the message will be **local** dispatched to target task
if possible.
2. More Netty context switch and contention. Check
http://yahooeng.tumblr.com/post/64758709722/making-storm-fly-with-netty
3. More outbound acker message count. Usually we will allocate one acker to
one worker.
So in common practice, each worker will have a moderate size of executors,
neither too small, nor too big.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
