Re: High frequency garbage collection leading to High load average
All these symptoms indicate a potential hotspot in this replica, which can be caused by one or likely multiple "hot" partitions. Finding out which particular partition(s) is responsible for this is tricky, but good candidates are the ones mentioned in the log warning. Ideally you should fix your data model to avoid large partitions and hotspots, keep your partitions under 100MB. There are some bucketing techniques available to reduce partition sizes. Em ter., 8 de mar. de 2022 às 18:44, Inquistive allen escreveu: > Hello team, > > On a given day , a node in 27 node cluster observed higher frequency of > garbage collection. Mostly young gc. > > I have found below issues: > 1. Higher number of slow queries being observed on that particular node > for that particular day compared to other days > > 2. Higher outgoing traffic observed from the node , 10 times the average > outbound traffic on that particular day > > 3. Higher number of cache requests hitting the key cache and chunk cache > that other days on the particular node > > The cluster has large partition warning as well. > > My query is, which of the above is a likely cause of higher frequency of > GC leading to High load average on the system. >
High frequency garbage collection leading to High load average
Hello team, On a given day , a node in 27 node cluster observed higher frequency of garbage collection. Mostly young gc. I have found below issues: 1. Higher number of slow queries being observed on that particular node for that particular day compared to other days 2. Higher outgoing traffic observed from the node , 10 times the average outbound traffic on that particular day 3. Higher number of cache requests hitting the key cache and chunk cache that other days on the particular node The cluster has large partition warning as well. My query is, which of the above is a likely cause of higher frequency of GC leading to High load average on the system.
Re: Cassandra Management tools?
Install cluster and manage all files: Puppet Alerts: Sensu-Go Metrics: Prometheus + Grafana Scheduled jobs or manually tasks like rolling upgrade/restart/cleanup: Jenkins + Cstar Best, Yakir Gibraltar
Re: Cassandra Management tools?
Hi all. Just a plug for a great community resource that would love some updated new tools. Also a good resource for old ones as well. https://github.com/Anant/awesome-cassandra Thanks! Patrick On Tue, Mar 8, 2022 at 8:43 AM wrote: > I use an Ansible Collection for Cassandra - > https://galaxy.ansible.com/community/cassandra > > > > Full disclosure: I am the author and I eat my own dogfood. > > > > I've automated rolling restarts well using the cassandra_status module. > > > > Cheers, > > > Rhys > > > > *Von:* Adam Scott > *Gesendet:* Dienstag, 1. März 2022 00:32 > *An:* user@cassandra.apache.org > *Betreff:* Re: Cassandra Management tools? > > > > I use pssh -i -h hosts nodetool for one-offs. > > > > Rolling restart is tricky to automate, but haven't had to yet. If I were > to, I would be sure to do a test connect and query to confirm the node is > up before going to the next one. > > > > For automation I use python fabric. > > > > I too, would be curious what others use. > > > > Thanks, > > Adam > > > > > > On Mon, Feb 28, 2022 at 1:59 PM Joe Obernberger < > joseph.obernber...@gmail.com> wrote: > > Hi all - curious what tools are folks using to manage large Cassandra > clusters? For example, to do tasks such as nodetool cleanup after a > node or nodes are added to the cluster, or simply rolling start/stops > after an update to the config or a new version? > We've used puppet before; is that what other folks are using? > Thanks for any suggestions. > > -Joe > >
AW: Cassandra Management tools?
I use an Ansible Collection for Cassandra - https://galaxy.ansible.com/community/cassandra Full disclosure: I am the author and I eat my own dogfood. I've automated rolling restarts well using the cassandra_status module. Cheers, Rhys Von: Adam Scott Gesendet: Dienstag, 1. März 2022 00:32 An: user@cassandra.apache.org Betreff: Re: Cassandra Management tools? I use pssh -i -h hosts nodetool for one-offs. Rolling restart is tricky to automate, but haven't had to yet. If I were to, I would be sure to do a test connect and query to confirm the node is up before going to the next one. For automation I use python fabric. I too, would be curious what others use. Thanks, Adam On Mon, Feb 28, 2022 at 1:59 PM Joe Obernberger mailto:joseph.obernber...@gmail.com>> wrote: Hi all - curious what tools are folks using to manage large Cassandra clusters? For example, to do tasks such as nodetool cleanup after a node or nodes are added to the cluster, or simply rolling start/stops after an update to the config or a new version? We've used puppet before; is that what other folks are using? Thanks for any suggestions. -Joe
