Re: Too many Cassandra threads waiting!!!
Renoy, Thanks. That kernel is new enough to have the patch for the infamous Linux kernel futex bug detailed here: https://groups.google.com/d/topic/mechanical-sympathy/QbmpZxp6C64 To answer your questions above: What you're seeing is likely just normal behavior for Cassandra and is an artifact of its staged event driven architecture (SEDA). You can read more about that if you follow the links in the post above. There is work to move from SEDA to a thread-per-core (TPC) architecture, which you can read about in https://issues.apache.org/jira/browse/CASSANDRA-10989. There are a number of tuning parameters you can tune to adjust the number of threads working on a few of the various stages within Cassandra (e.g. memtable_flush_writers, native_transport_max_threads, and max_hints_delivery_threads). There will of course be performance impacts for tuning these parameters and the right values will depend on your data model, hardware, and workload (among other things). On Thu, Aug 2, 2018 at 10:44 PM, nokia ceph wrote: > Hi Joshua, > > # uname -a > Linux cn6.chn6us1c1.cdn 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 > UTC 2017 x86_64 x86_64 x86_64 GNU/Linux > # cat /etc/redhat-release > CentOS Linux release 7.4.1708 (Core) > > On Fri, Aug 3, 2018 at 8:27 AM, Joshua Galbraith invalid> wrote: > >> Renoy, >> >> Out of curiosity, which kernel version are your nodes running? >> >> You may find this old message on the mailing list helpful: >> http://mail-archives.apache.org/mod_mbox/cassandra-user/2016 >> 02.mbox/%3CCAA=6J0-0VabfAn3DJfatOxyJwwEHpdiE67v2wm_ >> u5kaqoro...@mail.gmail.com%3E >> >> On Wed, Aug 1, 2018 at 5:38 PM, Elliott Sims >> wrote: >> >>> You might have more luck trying to analyze at the Java level, either via >>> a (Java) stack dump and the "ttop" tool from Swiss Java Knife, or Cassandra >>> tools like "nodetool tpstats" >>> >>> On Wed, Aug 1, 2018 at 2:08 AM, nokia ceph >>> wrote: >>> Hi, i'm having a 5 node cluster with cassandra 3.0.13. i could see the cassandra process has too many threads. *# pstree -p `pgrep java` | wc -l* *453* And almost all of those threads are in *sleeping* state and wait at *# cat /proc/166022/task/1698913/wchan* *futex_wait_queue_me* Some more info: *# strace -e trace=all -p 166022* *strace: Process 166022 attached* *futex(0x7efc24aeb9d0, FUTEX_WAIT, 166023, NULL* # cat /proc/166022/stack [] futex_wait_queue_me+0xc6/0x130 [] futex_wait+0x17b/0x280 [] do_futex+0x106/0x5a0 [] SyS_futex+0x80/0x180 [] system_call_fastpath+0x16/0x1b [] 0x What is the reason cassandra is having these many threads? is it the normal behavior of cassandra? Is there a way to reduce this thread count? will there be any performance impact because of this (our platform experts suspects so)? Regards, Renoy Paulose >>> >> >> >> -- >> *Joshua Galbraith *| Lead Software Engineer | New Relic >> > > -- *Joshua Galbraith *| Lead Software Engineer | New Relic
Re: Too many Cassandra threads waiting!!!
Hi Joshua, # uname -a Linux cn6.chn6us1c1.cdn 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux # cat /etc/redhat-release CentOS Linux release 7.4.1708 (Core) On Fri, Aug 3, 2018 at 8:27 AM, Joshua Galbraith < jgalbra...@newrelic.com.invalid> wrote: > Renoy, > > Out of curiosity, which kernel version are your nodes running? > > You may find this old message on the mailing list helpful: > http://mail-archives.apache.org/mod_mbox/cassandra-user/ > 201602.mbox/%3CCAA=6J0-0VabfAn3DJfatOxyJwwEHpdiE67v2w > m_u5kaqoro...@mail.gmail.com%3E > > On Wed, Aug 1, 2018 at 5:38 PM, Elliott Sims > wrote: > >> You might have more luck trying to analyze at the Java level, either via >> a (Java) stack dump and the "ttop" tool from Swiss Java Knife, or Cassandra >> tools like "nodetool tpstats" >> >> On Wed, Aug 1, 2018 at 2:08 AM, nokia ceph >> wrote: >> >>> Hi, >>> >>> i'm having a 5 node cluster with cassandra 3.0.13. >>> >>> i could see the cassandra process has too many threads. >>> >>> *# pstree -p `pgrep java` | wc -l* >>> *453* >>> >>> And almost all of those threads are in *sleeping* state and wait at >>> *# cat /proc/166022/task/1698913/wchan* >>> *futex_wait_queue_me* >>> >>> Some more info: >>> *# strace -e trace=all -p 166022* >>> *strace: Process 166022 attached* >>> *futex(0x7efc24aeb9d0, FUTEX_WAIT, 166023, NULL* >>> >>> # cat /proc/166022/stack >>> [] futex_wait_queue_me+0xc6/0x130 >>> [] futex_wait+0x17b/0x280 >>> [] do_futex+0x106/0x5a0 >>> [] SyS_futex+0x80/0x180 >>> [] system_call_fastpath+0x16/0x1b >>> [] 0x >>> >>> >>> What is the reason cassandra is having these many threads? is it the >>> normal behavior of cassandra? Is there a way to reduce this thread count? >>> will there be any performance impact because of this (our platform experts >>> suspects so)? >>> >>> Regards, >>> Renoy Paulose >>> >>> >> > > > -- > *Joshua Galbraith *| Lead Software Engineer | New Relic >
Re: Too many Cassandra threads waiting!!!
Renoy, Out of curiosity, which kernel version are your nodes running? You may find this old message on the mailing list helpful: http://mail-archives.apache.org/mod_mbox/cassandra-user/201602.mbox/%3CCAA=6j0-0vabfan3djfatoxyjwwehpdie67v2wm_u5kaqoro...@mail.gmail.com%3E On Wed, Aug 1, 2018 at 5:38 PM, Elliott Sims wrote: > You might have more luck trying to analyze at the Java level, either via a > (Java) stack dump and the "ttop" tool from Swiss Java Knife, or Cassandra > tools like "nodetool tpstats" > > On Wed, Aug 1, 2018 at 2:08 AM, nokia ceph > wrote: > >> Hi, >> >> i'm having a 5 node cluster with cassandra 3.0.13. >> >> i could see the cassandra process has too many threads. >> >> *# pstree -p `pgrep java` | wc -l* >> *453* >> >> And almost all of those threads are in *sleeping* state and wait at >> *# cat /proc/166022/task/1698913/wchan* >> *futex_wait_queue_me* >> >> Some more info: >> *# strace -e trace=all -p 166022* >> *strace: Process 166022 attached* >> *futex(0x7efc24aeb9d0, FUTEX_WAIT, 166023, NULL* >> >> # cat /proc/166022/stack >> [] futex_wait_queue_me+0xc6/0x130 >> [] futex_wait+0x17b/0x280 >> [] do_futex+0x106/0x5a0 >> [] SyS_futex+0x80/0x180 >> [] system_call_fastpath+0x16/0x1b >> [] 0x >> >> >> What is the reason cassandra is having these many threads? is it the >> normal behavior of cassandra? Is there a way to reduce this thread count? >> will there be any performance impact because of this (our platform experts >> suspects so)? >> >> Regards, >> Renoy Paulose >> >> > -- *Joshua Galbraith *| Lead Software Engineer | New Relic
Re: Too many Cassandra threads waiting!!!
You might have more luck trying to analyze at the Java level, either via a (Java) stack dump and the "ttop" tool from Swiss Java Knife, or Cassandra tools like "nodetool tpstats" On Wed, Aug 1, 2018 at 2:08 AM, nokia ceph wrote: > Hi, > > i'm having a 5 node cluster with cassandra 3.0.13. > > i could see the cassandra process has too many threads. > > *# pstree -p `pgrep java` | wc -l* > *453* > > And almost all of those threads are in *sleeping* state and wait at > *# cat /proc/166022/task/1698913/wchan* > *futex_wait_queue_me* > > Some more info: > *# strace -e trace=all -p 166022* > *strace: Process 166022 attached* > *futex(0x7efc24aeb9d0, FUTEX_WAIT, 166023, NULL* > > # cat /proc/166022/stack > [] futex_wait_queue_me+0xc6/0x130 > [] futex_wait+0x17b/0x280 > [] do_futex+0x106/0x5a0 > [] SyS_futex+0x80/0x180 > [] system_call_fastpath+0x16/0x1b > [] 0x > > > What is the reason cassandra is having these many threads? is it the > normal behavior of cassandra? Is there a way to reduce this thread count? > will there be any performance impact because of this (our platform experts > suspects so)? > > Regards, > Renoy Paulose > >
Too many Cassandra threads waiting!!!
Hi, i'm having a 5 node cluster with cassandra 3.0.13. i could see the cassandra process has too many threads. *# pstree -p `pgrep java` | wc -l* *453* And almost all of those threads are in *sleeping* state and wait at *# cat /proc/166022/task/1698913/wchan* *futex_wait_queue_me* Some more info: *# strace -e trace=all -p 166022* *strace: Process 166022 attached* *futex(0x7efc24aeb9d0, FUTEX_WAIT, 166023, NULL* # cat /proc/166022/stack [] futex_wait_queue_me+0xc6/0x130 [] futex_wait+0x17b/0x280 [] do_futex+0x106/0x5a0 [] SyS_futex+0x80/0x180 [] system_call_fastpath+0x16/0x1b [] 0x What is the reason cassandra is having these many threads? is it the normal behavior of cassandra? Is there a way to reduce this thread count? will there be any performance impact because of this (our platform experts suspects so)? Regards, Renoy Paulose
