[
https://issues.apache.org/jira/browse/STORM-297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14004413#comment-14004413
]
ASF GitHub Bot commented on STORM-297:
--------------------------------------
Github user clockfly commented on the pull request:
https://github.com/apache/incubator-storm/pull/103#issuecomment-43716283
> Do you mean that allocating one worker per node is better than several
workers per node as the netty threads from different worker process will
compete with each other ?
It depends. Executor is the basic unit of parallism, suppose executor
number is the same, adding a worker will add more traffic, but if it is
co-located with other worker, it may also increase the bandwith of intra-worker
communication(as we do intra-process communication instead of intra-machine).
For example, 1, 1, 1 (means 3 machine, one worker on each) may be faster
than co-located 4, 4, 4
But co-located 3, 0, 0 may be faster than distributed 1, 1, 1. This
requires experiment to verify it is better or worse.
>And i think using several netty threads working in sync and non-batch mode
may have some what the same effect with using only one netty threads working in
async and batch mode. Maybe i should test this out. By the way, what storm
version do you use in the test ?
Yes, sync-async can only give us pointers about what could be the possible
bottleneck, these need continous profile-tune experiments to prove our guess.
By the way, I use storm-0.9 release version.
> Storm Performance cannot be scaled up by adding more CPU cores
> --------------------------------------------------------------
>
> Key: STORM-297
> URL: https://issues.apache.org/jira/browse/STORM-297
> Project: Apache Storm (Incubating)
> Issue Type: Bug
> Reporter: Sean Zhong
> Labels: Performance, netty
> Fix For: 0.9.2-incubating
>
> Attachments: Storm_performance_fix.pdf,
> storm_Netty_receiver_diagram.png, storm_performance_fix.patch
>
>
> We cannot scale up the performance by adding more CPU cores and increasing
> parallelism.
> For a 2 layer topology Spout ---shuffle grouping--> bolt, when message size
> is small (around 100 bytes), we can find in the below picture that neither
> the CPU nor the network is saturated. When message size is 100 bytes, only
> 40% of CPU is used, only 18% of network is used, although we have a high
> parallelism (overall we have 144 executors)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
