Cassandra read requests not getting timeout
Hi Community, I am using Cassanadra 3.0.13 . 5 node cluster simple topology. Following are the timeout parameters in yaml file: # grep timeout /etc/cassandra/conf/cassandra.yaml cas_contention_timeout_in_ms: 1000 counter_write_request_timeout_in_ms: 5000 cross_node_timeout: false range_request_timeout_in_ms: 1 read_request_timeout_in_ms: 1 request_timeout_in_ms: 1 truncate_request_timeout_in_ms: 6 write_request_timeout_in_ms: 2000 i'm trying a cassandra query using cqlsh and it is not getting timeout. #time cqlsh 10.50.11.11 -e "CONSISTENCY QUORUM; select asset_name,profile_name,job_index,active,last_valid_op,last_valid_op_ts,status,status_description,live_depth,asset_type,dest_path,source_docroot_name,source_asset_name,start_time,end_time,iptv,drm,geo,last_gc from cdvr.jobs where model_type ='asset' AND docroot_name='vx030' LIMIT 10 ALLOW FILTERING;" Consistency level set to QUORUM. () () (79024 rows) real16m30.488s user0m39.761s sys 0m3.896s The query took 16.5 minutes to display the output. But my read_request_timeout is 10 seconds. why the query doesn't got timeout after 10 s ?? Regards, Renoy
Cassandra repair in different version
Hi , i have 5 node cassandra cluster. 3 nodes are in 3.0.13 version and 2 nodes are in 3.0.6. Is it safe to do a node repair on the cluster?? Regards, Renoy
Re: Too many Cassandra threads waiting!!!
Hi Joshua, # uname -a Linux cn6.chn6us1c1.cdn 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux # cat /etc/redhat-release CentOS Linux release 7.4.1708 (Core) On Fri, Aug 3, 2018 at 8:27 AM, Joshua Galbraith < jgalbra...@newrelic.com.invalid> wrote: > Renoy, > > Out of curiosity, which kernel version are your nodes running? > > You may find this old message on the mailing list helpful: > http://mail-archives.apache.org/mod_mbox/cassandra-user/ > 201602.mbox/%3CCAA=6J0-0VabfAn3DJfatOxyJwwEHpdiE67v2w > m_u5kaqoro...@mail.gmail.com%3E > > On Wed, Aug 1, 2018 at 5:38 PM, Elliott Sims > wrote: > >> You might have more luck trying to analyze at the Java level, either via >> a (Java) stack dump and the "ttop" tool from Swiss Java Knife, or Cassandra >> tools like "nodetool tpstats" >> >> On Wed, Aug 1, 2018 at 2:08 AM, nokia ceph >> wrote: >> >>> Hi, >>> >>> i'm having a 5 node cluster with cassandra 3.0.13. >>> >>> i could see the cassandra process has too many threads. >>> >>> *# pstree -p `pgrep java` | wc -l* >>> *453* >>> >>> And almost all of those threads are in *sleeping* state and wait at >>> *# cat /proc/166022/task/1698913/wchan* >>> *futex_wait_queue_me* >>> >>> Some more info: >>> *# strace -e trace=all -p 166022* >>> *strace: Process 166022 attached* >>> *futex(0x7efc24aeb9d0, FUTEX_WAIT, 166023, NULL* >>> >>> # cat /proc/166022/stack >>> [] futex_wait_queue_me+0xc6/0x130 >>> [] futex_wait+0x17b/0x280 >>> [] do_futex+0x106/0x5a0 >>> [] SyS_futex+0x80/0x180 >>> [] system_call_fastpath+0x16/0x1b >>> [] 0x >>> >>> >>> What is the reason cassandra is having these many threads? is it the >>> normal behavior of cassandra? Is there a way to reduce this thread count? >>> will there be any performance impact because of this (our platform experts >>> suspects so)? >>> >>> Regards, >>> Renoy Paulose >>> >>> >> > > > -- > *Joshua Galbraith *| Lead Software Engineer | New Relic >
Too many Cassandra threads waiting!!!
Hi, i'm having a 5 node cluster with cassandra 3.0.13. i could see the cassandra process has too many threads. *# pstree -p `pgrep java` | wc -l* *453* And almost all of those threads are in *sleeping* state and wait at *# cat /proc/166022/task/1698913/wchan* *futex_wait_queue_me* Some more info: *# strace -e trace=all -p 166022* *strace: Process 166022 attached* *futex(0x7efc24aeb9d0, FUTEX_WAIT, 166023, NULL* # cat /proc/166022/stack [] futex_wait_queue_me+0xc6/0x130 [] futex_wait+0x17b/0x280 [] do_futex+0x106/0x5a0 [] SyS_futex+0x80/0x180 [] system_call_fastpath+0x16/0x1b [] 0x What is the reason cassandra is having these many threads? is it the normal behavior of cassandra? Is there a way to reduce this thread count? will there be any performance impact because of this (our platform experts suspects so)? Regards, Renoy Paulose
Re: Cassandra + NUMA integration
Hello, In addition to my previous comment. Regarding the NUMA integration with Cassadra, we configured NODE-1 only utilize the resources. However after enabling this functionality, we noticed on NODE-1 Private allocated to 4G . What we did << #tail -n1 /etc/cassandra/conf/cassandra-env.sh JVM_OPTS="$JVM_OPTS -XX:+UseNUMA" #grep -A3 numactl /usr/sbin/cassandra # If numactl is available, use it. For Cassandra, the priority is to # avoid disk I/O. Even for the purpose of CPU efficiency, we don't # really have CPU<->data affinity anyway. Also, empirically test that numactl # works before trying to use it (CASSANDRA-3245). #NUMACTL_ARGS="--interleave=all" ===>>> Here we commented this parameter. After triggering below command #/usr/bin/numactl --cpunodebind=1 -- /usr/sbin/cassandra -p /run/cassandra/cassandra.pid We noticed on NODE-1 Stack,Heap,Huge values changed to null . We need to know why Private on NODE-0 utilizing memory even after we routed traffic to NODE-1? Please let us know do we miss anything here? Can anyone please explain what below parameters will do as per numactl output. --- *Private* *Stack* *Heap* *Huge* *--* // # numastat -s cassandra Per-node process memory usage (in MBs) for PID 5732 (java) Node 0 Node 1 Total --- --- --- Private 4205.57 4570.72 8776.30 Stack0.00 210.09 210.09 Heap 0.00 32.59 32.59 Huge 0.000.000.00 --- --- --- Total 4205.57 4813.40 9018.98 Env, RHEL7.3 3.10.0-514.2.2.el7.x86_64 cassandra30-3.0.6-1.noarch Awaiting for comments Thanks On Thu, Mar 16, 2017 at 5:07 PM, nokia ceph wrote: > Hello, > > Recently we tried top integrate cassandra with NUMA > > == > commented out the numactl --interleave in bin/cassandra and add > -XX:+UseNUMA to cassandra-env.sh. > == > > But after inplementing, we saw HEAP and STACK values were changed to null. > > #/usr/bin/numactl --cpunodebind=1 -- /usr/sbin/cassandra -p > /run/cassandra/cassandra.pid > #numastat -s cassandra > > Per-node process memory usage (in MBs) for PID 13258 (java) >Node 0 Node 1 Total > --- --- --- > *Private 4205.18 4508.09* 8713.28 > Stack0.00 219.61 219.61 > Heap 0.00 30.52 30.52 > Huge 0.000.000.00 > --- --- --- > Total 4205.18 4758.23 8963.41 > > > Here you can see both Node 0 and Node 1 utilising 4GB. As per our > expectation NODE 0 Private should be zero. I > > Is there any way to specify NODE-0 Private value change to 0? > > Env, > > RHEL7.3 > 3.10.0-514.2.2.el7.x86_64 > cassandra30-3.0.6-1.noarch > > Thanks > >
Cassandra + NUMA integration
Hello, Recently we tried top integrate cassandra with NUMA == commented out the numactl --interleave in bin/cassandra and add -XX:+UseNUMA to cassandra-env.sh. == But after inplementing, we saw HEAP and STACK values were changed to null. #/usr/bin/numactl --cpunodebind=1 -- /usr/sbin/cassandra -p /run/cassandra/cassandra.pid #numastat -s cassandra Per-node process memory usage (in MBs) for PID 13258 (java) Node 0 Node 1 Total --- --- --- *Private 4205.18 4508.09* 8713.28 Stack0.00 219.61 219.61 Heap 0.00 30.52 30.52 Huge 0.000.000.00 --- --- --- Total 4205.18 4758.23 8963.41 Here you can see both Node 0 and Node 1 utilising 4GB. As per our expectation NODE 0 Private should be zero. I Is there any way to specify NODE-0 Private value change to 0? Env, RHEL7.3 3.10.0-514.2.2.el7.x86_64 cassandra30-3.0.6-1.noarch Thanks
