[
https://issues.apache.org/jira/browse/CASSANDRA-14747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16612856#comment-16612856
]
Joseph Lynch commented on CASSANDRA-14747:
------------------------------------------
*Setup:*
* Cassandra: 192 (2*96) node i3.xlarge AWS instance (4 cpu cores, 30GB ram)
running cassandra trunk f25a765b.
* Two datacenters with 100ms latency between them
* No compression, encryption, or coalescing turned on
*Test:*
ndbench sent 30k QPS at a coordinator level to one datacenter (RF=3*2 = 6 so
180k global replica QPS) of 40kb and then 4kb single partition BATCH mutations.
This represents about 300 QPS per coordinator in the first datacenter or 75 per
core.
*Result:*
We quickly overwhelmed the 4.0 cluster which started having high latencies and
throwing errors while the 3.0 cluster remained healthy. 4.0 nodes were running
out of heap within minutes during the 40kb test and a few minutes with the 4kb
test. I've attached flamegraphs showing 4.0 spending half its time garbage
collecting and logs indicating large on heap usage.
On the bright side the thread count was _way down:_ the 3.0 cluster had 1.2k
threads and the 4.0 cluster only had 220 threads (and almost all of that
reduction was the messaging thread reduction). Also the startup time was super
fast (as in less than one second to handshake the entire cluster, vs 3.0 which
took minutes. We didn't feel that proceeding with the test made sense given the
instability until follow ups could be committed. We used heap dumps and
{{jmap}} to determine the issue was the outgoing message queue retaining large
numbers of mutations on heap rather than dropping them.
*Follow Ups:*
The outgoing queue holding mutations on heap appears to be the problem.
Specifically the 3.x code would police the internode queues and ensure they did
not get too large at enqueue and dequeue time (expiring messages and turning
them into hints as needed), the 4.0 code took out the enqueue policing
complexity in the hope that we wouldn't need it. It appears it is necessary.
[~jasobrown] is including fixes to the queue policing in CASSANDRA-14503 and
CASSANDRA-13630 and we will re-execute this test once those are merged to
ensure that they fix the issue with large volume mutations.
> Evaluate 200 node, compression=none, encryption=none, coalescing=off
> ---------------------------------------------------------------------
>
> Key: CASSANDRA-14747
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14747
> Project: Cassandra
> Issue Type: Sub-task
> Reporter: Joseph Lynch
> Priority: Major
>
> Tracks evaluating a 200 node cluster with all internode settings off (no
> compression, no encryption, no coalescing).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
