[
https://issues.apache.org/jira/browse/STORM-297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14013447#comment-14013447
]
ASF GitHub Bot commented on STORM-297:
--------------------------------------
Github user Gvain commented on the pull request:
https://github.com/apache/incubator-storm/pull/103#issuecomment-44629767
@clockfly , @revans2
I added more nodes, from 4 nodes to 15 nodes, as well as spouts, bolts,
ackers and workers, and keep their parallelism count in the same ratio, eg, for
every other 4 nodes, add 48 spouts, 48 bolts, 48 ackers and 8 workers. Here is
the test result:
nodes | workers | Throughput | CPU usage | NET usage (only IN Bytes)
4 | 8 | 656,000 tps | 89% | 28MB/s
8 | 16 | 1,004,000 tps | 82% | 28MB/s
12 | 24 | 1,133,000 tps | 72% | 25MB/s
15 | 30 | 1,235,000 tps | 69% | 24MB/s
for last two rows, the CPU usage decreased, and the throughput increasing
speed slows down. I added a few more worker to re-run the test, the result is:
nodes | workers | Throughput | CPU usage | NET usage (only IN Bytes)
12 | 48 | 1,444,000 tps | 88% | 30MB/s
15 | 30 | 1,735,000 tps | 88% | 30MB/s
From the result, We can see adding some more workers DO help to scale up
performance as adding more CPUs.
Finally, maybe we should re-considerate the root cause of this issue
"STORM-297 Storm Performance cannot be scaled up by adding more CPU cores"
> Storm Performance cannot be scaled up by adding more CPU cores
> --------------------------------------------------------------
>
> Key: STORM-297
> URL: https://issues.apache.org/jira/browse/STORM-297
> Project: Apache Storm (Incubating)
> Issue Type: Bug
> Reporter: Sean Zhong
> Labels: Performance, netty
> Fix For: 0.9.2-incubating
>
> Attachments: Storm_performance_fix.pdf,
> storm_Netty_receiver_diagram.png, storm_performance_fix.patch
>
>
> We cannot scale up the performance by adding more CPU cores and increasing
> parallelism.
> For a 2 layer topology Spout ---shuffle grouping--> bolt, when message size
> is small (around 100 bytes), we can find in the below picture that neither
> the CPU nor the network is saturated. When message size is 100 bytes, only
> 40% of CPU is used, only 18% of network is used, although we have a high
> parallelism (overall we have 144 executors)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
