[
https://issues.apache.org/jira/browse/STORM-376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14062272#comment-14062272
]
ASF GitHub Bot commented on STORM-376:
--------------------------------------
Github user revans2 commented on the pull request:
https://github.com/apache/incubator-storm/pull/168#issuecomment-49057769
I don't know hard limits when force sync is off. It depends on the
operating system, file system, disk RPM/type, the amount of free memory, and
what else is going on in the system.
When force sync is off ZK will not wait for the edit to hit the platter, it
just waits for the data to hit the page cache in the OS before going on. The
OS decides when to write the dirty pages back to disk. Linux typically is
configured to make sure dirty pages are not around for more than a couple of
mins. But even a few seconds worth of edits can become a rather large batch
that can usually be written in a single large extent to disk, so seeks are
greatly reduced. For us we regularly see disk utilization hovering around 5%
and occasionally spiking to 20% for a 400 node cluster. It was very similar on
the 800 simulated nodes with disk utilization about 7%. I did not save the
simulated 2000 node metrics so I don't feel comfortable giving hard numbers.
But off the top of my head I seem to remember it being 10 to 15%, but I really
don't know for sure.
Even if utilization doubled to 10% going from 400 to 800 nodes and it grew
linearly there after it would become a bottleneck again at 6500+ nodes (It is
really hard to actually hit 100% disk utilization in the real world. I picked
80% for this number).
> Add compression to data stored in ZK
> ------------------------------------
>
> Key: STORM-376
> URL: https://issues.apache.org/jira/browse/STORM-376
> Project: Apache Storm (Incubating)
> Issue Type: Improvement
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: storm-2000.png
>
>
> If you run zookeeper with -Dzookeeper.forceSync=no the zookeeper disk no
> longer is the bottleneck for scaling storm. For us on a Gigabit Ethernet
> (scale test cluster) it becomes the aggregate reads by all of the supervisors
> and workers trying to download the compiled topology assignments.
> To reduce this load we took two approaches. First we compressed the data
> being stored in zookeeper (this JIRA) which also has the added benefit of
> increasing the size of the topology you can store in ZK. Second we used the
> ZK version number to see if the data had changed and avoid downloading it
> again needlessly (STORM-375).
> With these changes we were able to scale to a simulated 1965 nodes (5
> supervisors running on each of 393 real nodes, with each supervisor
> configured to have 10 slots). We also filled the cluster with 131 topologies
> of 100 workers each. (we are going to 200 topos, and may try to scale the
> cluster even larger, but it takes forever to launch topologies once the
> cluster is under load. We may try to address that shortly too)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
