I’d also take a look at the O/S level. You might be queued up on flushing of dirty pages, which would also throttle your ability to write mempages. Once the I/O gets throttled badly, I’ve seen it push back into what you see in C*. To Aaron’s point, you want a balance in memory between C* and O/S buffer cache, because to write to disk you pass through buffer cache first.
From: Aaron Ploetz <aaronplo...@gmail.com> Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> Date: Tuesday, June 2, 2020 at 9:38 AM To: "user@cassandra.apache.org" <user@cassandra.apache.org> Subject: Re: Cassandra crashes when using offheap_objects for memtable_allocation_type Message from External Sender I would try running it with memtable_offheap_space_in_mb at the default for sure, but definitely lower than 8GB. With 32GB of RAM, you're already allocating half of that for your heap, and then halving the remainder for off heap memtables. What's left may not be enough for the OS, etc. Giving some of that back, will allow more to be used for page cache, which always helps. "JVM heap size: 16GB, CMS, 1GB newgen" For CMS GC with a 16GB heap, 1GB is way too small for new gen. You're going to want that to be at least 40% of the max heap size. Some folks here even advocate for setting Xmn as high as 50% of Xmx/s. If you want to stick with CMS GC, take a look at https://issues.apache.org/jira/browse/CASSANDRA-8150<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_CASSANDRA-2D8150&d=DwMFaQ&c=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA&r=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc&m=eX5wEX3VXggU-3Y51C3hDF0XBO9xUZQZd12jW_Da8Qw&s=M9v_2_0-S3OhfynpBRwgk6dnPh_pJiNC9Fps-9Qu2S8&e=>. There's plenty of good info in there on CMS GC tuning. Make sure to read through the whole ticket, so that you understand what each setting does. You can't just pick-and-choose. Regards, Aaron On Tue, Jun 2, 2020 at 1:31 AM onmstester onmstester <onmstes...@zoho.com.invalid> wrote: I just changed these properties to increase flushed file size (decrease number of compactions): * memtable_allocation_type from heap_buffers to offheap_objects * memtable_offheap_space_in_mb: from default (2048) to 8192 Using default value for other memtable/compaction/commitlog configurations . After a few hours some of nodes stopped to do any mutations (dropped mutaion increased) and also pending flushes increased, they were just up and running and there was only a single CPU core with 100% usage(other cores was 0%). other nodes on the cluster determines the node as DN. Could not access 7199 and also could not create thread dump even with jstack -F. Restarting Cassandra service fixes the problem but after a while some other node would be DN. Am i missing some configurations? What should i change in cassandra default configuration to maximize write throughput in single node/cluster in write-heavy scenario for the data model: Data mode is a single table: create table test( text partition_key, text clustering_key, set<text> rows, primary key ((partition_key, clustering_key)) vCPU: 12 Memory: 32GB Node data size: 2TB Apache cassandra 3.11.2 JVM heap size: 16GB, CMS, 1GB newgen Sent using Zoho Mail<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.zoho.com_mail_&d=DwMFaQ&c=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA&r=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc&m=eX5wEX3VXggU-3Y51C3hDF0XBO9xUZQZd12jW_Da8Qw&s=OpD4crlAdz_RSLV3zi8jcRzgpKzGjKn2up0lzd8mDlE&e=>
