Re: Cassandra eats all cpu cores, high load average

2016-02-13 Thread Skvazh Roman
s every node eventually hit this problem? > > After a restart, how long before the problem recurs for that node? > > -- Jack Krupansky > > On Fri, Feb 12, 2016 at 4:06 AM, Skvazh Roman <r...@skvazh.com > <mailto:r...@skvazh.com>> wrote: > Hello! > We have a clu

Cassandra eats all cpu cores, high load average

2016-02-12 Thread Skvazh Roman
Hello! We have a cluster of 25 c3.4xlarge nodes (16 cores, 32 GiB) with attached 1.5 TB 4000 PIOPS EBS drive. Sometimes one or two nodes user cpu spikes to 100%, load average to 20-30 - read requests drops of. Only restart of this cassandra services helps. Please advice. One big table with wide

Re: Cassandra eats all cpu cores, high load average

2016-02-12 Thread Skvazh Roman
0 _TRACE 0 MUTATION 0 COUNTER_MUTATION 0 BINARY 0 REQUEST_RESPONSE 0 PAGED_RANGE 0 READ_REPAIR 0 > On 12 Feb 2016, at 17:45, Skvazh Roman <r...@skvazh.com&

Re: Cassandra eats all cpu cores, high load average

2016-02-12 Thread Skvazh Roman
s happening? > > J. > > >> On Feb 12, 2016, at 3:06 AM, Skvazh Roman <r...@skvazh.com> wrote: >> >> Hello! >> We have a cluster of 25 c3.4xlarge nodes (16 cores, 32 GiB) with attached >> 1.5 TB 4000 PIOPS EBS drive. >> Sometimes one or two nod

Re: Cassandra eats all cpu cores, high load average

2016-02-12 Thread Skvazh Roman
compactions. > On 12 Feb 2016, at 18:44, Julien Anguenot <jul...@anguenot.org> wrote: > >> >> On Feb 12, 2016, at 9:24 AM, Skvazh Roman <r...@skvazh.com >> <mailto:r...@skvazh.com>> wrote: >> >> I have disabled autocompaction and stop it

Re: Cassandra eats all cpu cores, high load average

2016-02-12 Thread Skvazh Roman
e_seconds so > they get cleared out earlier during compactions. > > J. > >> On Feb 12, 2016, at 8:45 AM, Skvazh Roman <r...@skvazh.com> wrote: >> >> There is 1-4 compactions at that moment. >> We have many tombstones, which does not removed. >> Drop