Hello!
I am using Cassandra 3.10, on Ubuntu 14.04 and I have a counter
table(RF=1), with the following schema:
CREATE TABLE edges (
src_id text,
src_type text,
source text
weight counter,
PRIMARY KEY ((src_id, src_type), source)
) WITH
compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '4'}
SELECT vs UPDATE requests ratio is 0.001. ( Read Count: 3771000, Write
Count: 3401236000, in one month)
We have Counter Cache enabled:
Counter Cache : entries 1018782, size 256 MiB, capacity 256
MiB, 2799913189 hits, 3469459479 requests, 0.807 recent hit rate, 7200
save period in seconds
The problem is that our read rate limit on our hard-disk is always
near 30MBps and our write rate limit is near 500KBps.
One example of output of "iostat -x" is
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.06 1.04 263.65 2.04 28832.42 572.53
146.07 0.36 1.35 0.74 81.16 1.27 33.81
Also with iotop, we saw that are about 8 threads that each goes around
3MB/s read rate.
Total DISK READ : 22.73 M/s | Total DISK WRITE : 494.35 K/s
Actual DISK READ: 22.62 M/s | Actual DISK WRITE: 528.57 K/s
TID PRIO USER DISK READ> DISK WRITE SWAPIN IO COMMAND
14793 be/4 cassandra 3.061 M/s 0.0010 B/s 0.00 % 93.27 % java
-Dcassandra.fd_max_interval_ms=400
The output of strace on these threads is :
strace -cp 14793
Process 14793 attached
^CProcess 14793 detached
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
99.85 32.118518 57 567288 256251 futex
0.15 0.048822 3 15339 write
0.00 0.000000 0 1 rt_sigreturn
------ ----------- ----------- --------- --------- ----------------
100.00 32.167340 582628 256251 total
Despite that iotop shows that this thread is reading with 3MB/s, there
is no read syscall in strace.
I want to ask if actually the futex is responsible for the read rate
and how can we debug this problem further ?
Btw, there are no compaction tasks in progress and there are no SELECT
queries in progress.
Also, I know that for each update, a lock is obtained[1]
Thank you,
[1]https://apache.googlesource.com/cassandra/+/refs/heads/trunk/src/java/org/apache/cassandra/db/CounterMutation.java#121
--
Octavian Rinciog
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org
