Thank you for your reply. I have run the same test but writing into a
non-counter table equivalent to the counter one. The rate in this case is
around 160k writes/second. I am not sure if I should be expecting much more
in a 3 nodes cluster. In terms of I/O is not much really and the system is
not reaching its limit in any are but am still to look closer.
I will monitor the CPU, Disk and Network usage when I come back to the
system to try and find the weakest link in the chain.
Regarding the configurations of the nodes regarding counters, here are a
few settings in case anyone can see something wrong:
+ counter_cache_size_in_mb: [LEFT EMPTY] => It will use 50MB in my case
+ counter_cache_save_period: 7200
+ concurrent_reads: 112
+ concurrent_writes: 128
+ concurrent_counter_writes: 112
I found the following using nodetool info. Looks like the counter cache is
full so I will try increasing it.
Gossip active : true
Thrift active : false
Native Transport active: true
Load : 69.02 GiB
Generation No : 1519044361
Uptime (seconds) : 987326
Heap Memory (MB) : 4653.43 / 8032.00
Off Heap Memory (MB) : 361.52
Data Center : dc1
Rack : rack1
Exceptions : 634
Key Cache : entries 1297379, size 100 MiB, capacity 100 MiB,
3123293769 hits, 5261224916 requests, 0.594 recent hit rate, 14400 save
period in seconds
Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits,
0 requests, NaN recent hit rate, 0 save period in seconds
Counter Cache : entries 436906, size 50 MiB, capacity 50 MiB,
478644319 hits, 7183133047 requests, 0.067 recent hit rate, 7200 save
period in seconds
Chunk Cache : entries 7680, size 480 MiB, capacity 480 MiB,
4635329524 misses, 6673522516 requests, 0.305 recent hit rate, NaN
microseconds miss latency
Percent Repaired : 0.0%
F Javier Pareja
On Fri, Mar 2, 2018 at 7:01 PM, Alain RODRIGUEZ <arodr...@gmail.com> wrote:
> Hi Javier,
> The only bottleneck in the writes as far as I understand it is the commit
> Sadly this is somewhat wrong, specially in your case. CPU, network limits
> can be reached, and other issues, can happen. Plus in your case, using
> counters, there is way more things involved.
> The 2 main things that I saw from your comment are:
> - You are using counters. When writing a counter, Apache Cassandra
> performs a read-before-write. In 3.11.1 there should be a counter cache
> that you can play with to alleviate impacts of this, but what you want to
> do with counters generally is to put a buffer somewhere that will count for
> and send one request of '+5000' instead of 5000 thousands request of +1.
> The difference should be substencial.
> - Given this first consideration, and in general, using HDD is not the
> best to have good throughput, and makes it almost impossible to reach
> something close to the ms latency. Having 9 disks will not make each of
> them faster, but allow more concurrency, so latency will never be 'the
> Before making changes in the hardware it is important to understand where
> the bottleneck comes from:
> Ideally, I often recommend dashboards, they allow to spot this kind of
> things very well.
> If no dashboards are available, maybe the logs (specially warn / error /
> gc) could help, or commands such as 'nodetool cfstats' or 'nodetool
> tpstats' to build a better understanding.
> If the machines and disks can handle it and you want to try it as it is,
> maybe try to increase the amount of 'concurrent_counter' in cassandra.yaml
> or increase the cache size, but I am really guessing here.
> - I have configured all 3 nodes to act as seeds but I don't think this
>> affects write performance.
> No problem
> - The hints_directory and the saved_caches_directory use the same drive as
>> the commitlog_directory. The data is in the other 7 drives as I explained
> It should be good, did you check the disks performances / usage?
> Could the saved_cached, specially because of the counters, have a
>> meaningful impact on the write performance?
> It depends on the frequency of the cache being written to the disk is and
> the size of it.
> Alain Rodriguez - @arodream - al...@thelastpickle.com
> France / Spain
> The Last Pickle - Apache Cassandra Consulting
> 2018-03-02 17:36 GMT+00:00 Javier Pareja <pareja.jav...@gmail.com>:
>> Hi again,
>> Two more thoughts with respect to my question:
>> - I have configured all 3 nodes to act as seeds but I don't think this
>> affects write performance.
>> - The hints_directory and the saved_caches_directory use the same drive
>> as the commitlog_directory. The data is in the other 7 drives as I
>> explained earlier. Could the saved_cached, specially because of the
>> counters, have a meaningful impact on the write performance?
>> - If more nodes are needed for whatever reason, would a layer of
>> virtualization on top of each machine help. Each virtual machine will have
>> assigned dedicated drives (there are plenty of them) and only share the CPU
>> and RAM.
>> The only bottleneck in the writes as far as I understand it is the commit
>> log. Shall I create RAID0 (for speed) or install an SSD just for the
>> F Javier Pareja
>> On Fri, Mar 2, 2018 at 12:21 PM, Javier Pareja <pareja.jav...@gmail.com>
>>> Hello everyone,
>>> I have configured a Cassandra cluster with 3 nodes, however I am not
>>> getting the write speed that I was expecting. I have tested against a
>>> counter table because it is the bottleneck of the system.
>>> So with the system iddle I run the attached sample code (very simple
>>> async writes with a throttle) against an schema with RF=2 and a table with
>>> The speeds that I get are around 65k updates-writes/second and I was
>>> hoping for at least 150k updates-writes/second. Even if I run the test
>>> in 2 machines in parallel, the execution is 35k updates-writes/second
>>> in each. I have executed the test in the nodes themselves (1 and 2 of the 3
>>> The nodes are fairly powerful. Each has the following configuration
>>> running Cassandra 3.11.1
>>> - RAM: 256GB
>>> - HDD Disks: 9 (7 configured for cassandra data, 1 for the OS and 1
>>> configured for cassandra commits)
>>> - CPU: 8 processors with hyperthreading => 16 processors
>>> The RAM, CPU and HDDs are far from being maxed out when running the
>>> The test command line class uses two parameters: max executions and
>>> parallelism. Parallelism is the max number of AsyncExecutions running in
>>> parallel. Any other execution will have to wait for available slots.
>>> I tried increasing the parallelism (64, 128, 256...) but the results
>>> are the same, 128 seems enough.
>>> Table definition:
>>> CREATE TABLE counttest (
>>> key_column bigint,
>>> cluster_column int,
>>> count1_column counter,
>>> count2_column counter,
>>> count3_column counter,
>>> count4_column counter,
>>> count5_column counter,
>>> PRIMARY KEY ((key_column),cluster_column)
>>> Write test data generation (from the class attached). Each insert is
>>> prepared with uniform random values from below:
>>> long key_column = getRandom(0, 5000000);
>>> int cluster_column = getRandom(0, 4096);
>>> long count1_column = getRandom(0, 10);
>>> long count2_column = getRandom(0, 10);
>>> long count3_column = getRandom(0, 10);
>>> long count4_column = getRandom(0, 10);
>>> long count5_column = getRandom(0, 10);
>>> *I suspect that we took the wrong approach when designing the hardware:
>>> Should we have used more nodes and less drives per node? If this is the
>>> case, I am trying to understand why or if there is any change that we could
>>> do to the configuration (other than getting more nodes) to improve that.*
>>> Will an SSD dedicated for the commit log improve things dramatically?
>>> Best Regards,