Re: Cassandra 3.11 is compacting forever

[Ioannis Zafiropoulos]

A maybe long shot, have you deleted the cassandra-topology.properties file
since you are using GossipingPropertyFileSnitch?
I am sure I have seen a ticket about problems caused in some cases if that
file stays around.
I removed it from all the nodes and non-stop compaction stopped (after a
proper restart - not rolling).


On Fri, Sep 8, 2017 at 4:24 PM, Romain Hardouin  wrote:

> Hi,
>
> It might be useful to enable compaction logging with log_all subproperties.
>
> Best,
>
> Romain
>
> Le vendredi 8 septembre 2017 à 00:15:19 UTC+2, kurt greaves <
> k...@instaclustr.com> a écrit :
>
>
> Might be worth turning on debug logging for that node and when the
> compaction kicks off and CPU skyrockets send through the logs.​
>

Re: Cassandra 3.11 is compacting forever

[Romain Hardouin]

Hi,
It might be useful to enable compaction logging with log_all subproperties.
Best,
Romain 
Le vendredi 8 septembre 2017 à 00:15:19 UTC+2, kurt greaves 
 a écrit :  
 
 Might be worth turning on debug logging for that node and when the compaction 
kicks off and CPU skyrockets send through the logs.​

Re: Cassandra 3.11 is compacting forever

[kurt greaves]

Might be worth turning on debug logging for that node and when the
compaction kicks off and CPU skyrockets send through the logs.​

Re: Cassandra 3.11 is compacting forever

[Igor Leão]

Last week I moved all nodes back to cassandra 3.9. Everything worked fine
since then.
Yesterday I tried to upgrade again, running a rolling restart after the
upgrade.Nodes were just fine. Today one node started consuming 94.6% of its
CPU. Compacting is running all the time for this node.
I'm afraid to have the remaining nodes increasing their CPUs over the next
couple of days, as happened last week.

2017-09-03 22:28 GMT-03:00 kurt greaves :

> Can't say that message explains why the compaction would be stuck.
> Generally not a good sign and you might need to investigate more but
> hopefully shouldn't be related. Has that stuck compaction moved since last
> week?
>
>
> On 1 September 2017 at 22:54, Fay Hou [Storage Service] ­ <
> fay...@coupang.com> wrote:
>
>> try to do a rolling restart for the cluster before doing a compation
>>
>> On Fri, Sep 1, 2017 at 3:09 PM, Igor Leão  wrote:
>>
>>> Some generic errors:
>>>
>>> *[aladdin@ip-172-16-1-10 cassandra]$ tail cassandra.log | grep -i error*
>>> *[aladdin@ip-172-16-1-10 cassandra]$ tail cassandra.log | grep -i excep*
>>> *[aladdin@ip-172-16-1-10 cassandra]$ tail cassandra.log | grep -i fail*
>>> *[aladdin@ip-172-16-1-10 cassandra]$ tail debug.log | grep -i error*
>>> *[aladdin@ip-172-16-1-10 cassandra]$ tail debug.log | grep -i exce*
>>> *[aladdin@ip-172-16-1-10 cassandra]$ tail debug.log | grep -i fail*
>>> *DEBUG [GossipStage:1] 2017-09-01 15:33:27,046 FailureDetector.java:457
>>> - Ignoring interval time of 2108299431 <(210)%20829-9431> for /172.16.1.112
>>> *
>>> *DEBUG [GossipStage:1] 2017-09-01 15:33:29,051 FailureDetector.java:457
>>> - Ignoring interval time of 2005507384 for /172.16.1.74
>>> *
>>> *DEBUG [GossipStage:1] 2017-09-01 15:33:45,968 FailureDetector.java:457
>>> - Ignoring interval time of 2003371497 for /172.16.1.74
>>> *
>>> *DEBUG [GossipStage:1] 2017-09-01 15:33:51,133 FailureDetector.java:457
>>> - Ignoring interval time of 2013260173 <(201)%20326-0173> for /172.16.1.74
>>> *
>>> *DEBUG [GossipStage:1] 2017-09-01 15:33:58,981 FailureDetector.java:457
>>> - Ignoring interval time of 2009620081 for /172.16.1.112
>>> *
>>> *DEBUG [GossipStage:1] 2017-09-01 15:34:19,235 FailureDetector.java:457
>>> - Ignoring interval time of 2010956256 for /172.16.1.74
>>> *
>>> *DEBUG [GossipStage:1] 2017-09-01 15:34:19,235 FailureDetector.java:457
>>> - Ignoring interval time of 2011127930 for /10.0.1.122 *
>>> *[aladdin@ip-172-16-1-10 cassandra]$ tail system.log | grep -i error*
>>> *io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)()
>>> failed: Connection reset by peer*
>>> *[aladdin@ip-172-16-1-10 cassandra]$ tail system.log | grep -i exce*
>>> *INFO  [Native-Transport-Requests-5] 2017-09-01 15:22:58,806
>>> Message.java:619 - Unexpected exception during request; channel = [id:
>>> 0xdd63db2f, L:/10.0.1.47:9042  !
>>> R:/10.0.44.196:41422 ]*
>>> *io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)()
>>> failed: Connection reset by peer*
>>> *[aladdin@ip-172-16-1-10 cassandra]$ tail system.log | grep -i fail*
>>> *io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)()
>>> failed: Connection reset by peer*
>>>
>>>
>>> Some interesting errors:
>>>
>>> 1.
>>> *DEBUG [ReadRepairStage:1] 2017-09-01 15:34:58,485 ReadCallback.java:242
>>> - Digest mismatch:*
>>> *org.apache.cassandra.service.DigestMismatchException: Mismatch for key
>>> DecoratedKey(5988282114260523734,
>>> 32623331326162652d63352d343237632d626334322d306466643762653836343830)
>>> (023d99bbcf2263f0fa450c2312fdce88 vs a60ba37a46e0a61227a8b560fa4e0dfb)*
>>> * at
>>> org.apache.cassandra.service.DigestResolver.compareResponses(DigestResolver.java:92)
>>> ~[apache-cassandra-3.11.0.jar:3.11.0]*
>>> * at
>>> org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:233)
>>> ~[apache-cassandra-3.11.0.jar:3.11.0]*
>>> * at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> [na:1.8.0_112]*
>>> * at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> [na:1.8.0_112]*
>>> * at
>>> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
>>> [apache-cassandra-3.11.0.jar:3.11.0]*
>>> * at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_112]*
>>>
>>> 2.
>>> *INFO  [Native-Transport-Requests-5] 2017-09-01 15:22:58,806
>>> Message.java:619 - Unexpected exception during request; channel = [id:
>>> 0xdd63db2f, L:/10.0.1.47:9042  !
>>> R:/10.0.44.196:41422 ]*
>>> *io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)()
>>> failed: Connection reset by peer*
>>> * at 

Re: Cassandra 3.11 is compacting forever

[kurt greaves]

Can't say that message explains why the compaction would be stuck.
Generally not a good sign and you might need to investigate more but
hopefully shouldn't be related. Has that stuck compaction moved since last
week?

On 1 September 2017 at 22:54, Fay Hou [Storage Service] ­ <
fay...@coupang.com> wrote:

> try to do a rolling restart for the cluster before doing a compation
>
> On Fri, Sep 1, 2017 at 3:09 PM, Igor Leão  wrote:
>
>> Some generic errors:
>>
>> *[aladdin@ip-172-16-1-10 cassandra]$ tail cassandra.log | grep -i error*
>> *[aladdin@ip-172-16-1-10 cassandra]$ tail cassandra.log | grep -i excep*
>> *[aladdin@ip-172-16-1-10 cassandra]$ tail cassandra.log | grep -i fail*
>> *[aladdin@ip-172-16-1-10 cassandra]$ tail debug.log | grep -i error*
>> *[aladdin@ip-172-16-1-10 cassandra]$ tail debug.log | grep -i exce*
>> *[aladdin@ip-172-16-1-10 cassandra]$ tail debug.log | grep -i fail*
>> *DEBUG [GossipStage:1] 2017-09-01 15:33:27,046 FailureDetector.java:457 -
>> Ignoring interval time of 2108299431 <(210)%20829-9431> for /172.16.1.112
>> *
>> *DEBUG [GossipStage:1] 2017-09-01 15:33:29,051 FailureDetector.java:457 -
>> Ignoring interval time of 2005507384 for /172.16.1.74 *
>> *DEBUG [GossipStage:1] 2017-09-01 15:33:45,968 FailureDetector.java:457 -
>> Ignoring interval time of 2003371497 for /172.16.1.74 *
>> *DEBUG [GossipStage:1] 2017-09-01 15:33:51,133 FailureDetector.java:457 -
>> Ignoring interval time of 2013260173 <(201)%20326-0173> for /172.16.1.74
>> *
>> *DEBUG [GossipStage:1] 2017-09-01 15:33:58,981 FailureDetector.java:457 -
>> Ignoring interval time of 2009620081 for /172.16.1.112
>> *
>> *DEBUG [GossipStage:1] 2017-09-01 15:34:19,235 FailureDetector.java:457 -
>> Ignoring interval time of 2010956256 for /172.16.1.74 *
>> *DEBUG [GossipStage:1] 2017-09-01 15:34:19,235 FailureDetector.java:457 -
>> Ignoring interval time of 2011127930 for /10.0.1.122 *
>> *[aladdin@ip-172-16-1-10 cassandra]$ tail system.log | grep -i error*
>> *io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)()
>> failed: Connection reset by peer*
>> *[aladdin@ip-172-16-1-10 cassandra]$ tail system.log | grep -i exce*
>> *INFO  [Native-Transport-Requests-5] 2017-09-01 15:22:58,806
>> Message.java:619 - Unexpected exception during request; channel = [id:
>> 0xdd63db2f, L:/10.0.1.47:9042  !
>> R:/10.0.44.196:41422 ]*
>> *io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)()
>> failed: Connection reset by peer*
>> *[aladdin@ip-172-16-1-10 cassandra]$ tail system.log | grep -i fail*
>> *io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)()
>> failed: Connection reset by peer*
>>
>>
>> Some interesting errors:
>>
>> 1.
>> *DEBUG [ReadRepairStage:1] 2017-09-01 15:34:58,485 ReadCallback.java:242
>> - Digest mismatch:*
>> *org.apache.cassandra.service.DigestMismatchException: Mismatch for key
>> DecoratedKey(5988282114260523734,
>> 32623331326162652d63352d343237632d626334322d306466643762653836343830)
>> (023d99bbcf2263f0fa450c2312fdce88 vs a60ba37a46e0a61227a8b560fa4e0dfb)*
>> * at
>> org.apache.cassandra.service.DigestResolver.compareResponses(DigestResolver.java:92)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]*
>> * at
>> org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:233)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]*
>> * at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> [na:1.8.0_112]*
>> * at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [na:1.8.0_112]*
>> * at
>> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
>> [apache-cassandra-3.11.0.jar:3.11.0]*
>> * at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_112]*
>>
>> 2.
>> *INFO  [Native-Transport-Requests-5] 2017-09-01 15:22:58,806
>> Message.java:619 - Unexpected exception during request; channel = [id:
>> 0xdd63db2f, L:/10.0.1.47:9042  !
>> R:/10.0.44.196:41422 ]*
>> *io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)()
>> failed: Connection reset by peer*
>> * at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown
>> Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]*
>> *INFO  [Native-Transport-Requests-11] 2017-09-01 15:31:42,722
>> NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot
>> allocate chunk of 1.000MiB*
>>
>> *INFO  [CompactionExecutor:470] 2017-09-01 10:16:42,026
>> NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot
>> allocate chunk of 1.000MiB*
>> *INFO  [CompactionExecutor:475] 2017-09-01 10:31:42,032
>> NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot
>> allocate chunk of 1.000MiB*
>> 

Re: Cassandra 3.11 is compacting forever

[Fay Hou [Storage Service] ­]

try to do a rolling restart for the cluster before doing a compation

On Fri, Sep 1, 2017 at 3:09 PM, Igor Leão  wrote:

> Some generic errors:
>
> *[aladdin@ip-172-16-1-10 cassandra]$ tail cassandra.log | grep -i error*
> *[aladdin@ip-172-16-1-10 cassandra]$ tail cassandra.log | grep -i excep*
> *[aladdin@ip-172-16-1-10 cassandra]$ tail cassandra.log | grep -i fail*
> *[aladdin@ip-172-16-1-10 cassandra]$ tail debug.log | grep -i error*
> *[aladdin@ip-172-16-1-10 cassandra]$ tail debug.log | grep -i exce*
> *[aladdin@ip-172-16-1-10 cassandra]$ tail debug.log | grep -i fail*
> *DEBUG [GossipStage:1] 2017-09-01 15:33:27,046 FailureDetector.java:457 -
> Ignoring interval time of 2108299431 <(210)%20829-9431> for /172.16.1.112
> *
> *DEBUG [GossipStage:1] 2017-09-01 15:33:29,051 FailureDetector.java:457 -
> Ignoring interval time of 2005507384 for /172.16.1.74 *
> *DEBUG [GossipStage:1] 2017-09-01 15:33:45,968 FailureDetector.java:457 -
> Ignoring interval time of 2003371497 for /172.16.1.74 *
> *DEBUG [GossipStage:1] 2017-09-01 15:33:51,133 FailureDetector.java:457 -
> Ignoring interval time of 2013260173 <(201)%20326-0173> for /172.16.1.74
> *
> *DEBUG [GossipStage:1] 2017-09-01 15:33:58,981 FailureDetector.java:457 -
> Ignoring interval time of 2009620081 for /172.16.1.112
> *
> *DEBUG [GossipStage:1] 2017-09-01 15:34:19,235 FailureDetector.java:457 -
> Ignoring interval time of 2010956256 for /172.16.1.74 *
> *DEBUG [GossipStage:1] 2017-09-01 15:34:19,235 FailureDetector.java:457 -
> Ignoring interval time of 2011127930 for /10.0.1.122 *
> *[aladdin@ip-172-16-1-10 cassandra]$ tail system.log | grep -i error*
> *io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)()
> failed: Connection reset by peer*
> *[aladdin@ip-172-16-1-10 cassandra]$ tail system.log | grep -i exce*
> *INFO  [Native-Transport-Requests-5] 2017-09-01 15:22:58,806
> Message.java:619 - Unexpected exception during request; channel = [id:
> 0xdd63db2f, L:/10.0.1.47:9042  !
> R:/10.0.44.196:41422 ]*
> *io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)()
> failed: Connection reset by peer*
> *[aladdin@ip-172-16-1-10 cassandra]$ tail system.log | grep -i fail*
> *io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)()
> failed: Connection reset by peer*
>
>
> Some interesting errors:
>
> 1.
> *DEBUG [ReadRepairStage:1] 2017-09-01 15:34:58,485 ReadCallback.java:242 -
> Digest mismatch:*
> *org.apache.cassandra.service.DigestMismatchException: Mismatch for key
> DecoratedKey(5988282114260523734,
> 32623331326162652d63352d343237632d626334322d306466643762653836343830)
> (023d99bbcf2263f0fa450c2312fdce88 vs a60ba37a46e0a61227a8b560fa4e0dfb)*
> * at
> org.apache.cassandra.service.DigestResolver.compareResponses(DigestResolver.java:92)
> ~[apache-cassandra-3.11.0.jar:3.11.0]*
> * at
> org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:233)
> ~[apache-cassandra-3.11.0.jar:3.11.0]*
> * at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_112]*
> * at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_112]*
> * at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> [apache-cassandra-3.11.0.jar:3.11.0]*
> * at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_112]*
>
> 2.
> *INFO  [Native-Transport-Requests-5] 2017-09-01 15:22:58,806
> Message.java:619 - Unexpected exception during request; channel = [id:
> 0xdd63db2f, L:/10.0.1.47:9042  !
> R:/10.0.44.196:41422 ]*
> *io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)()
> failed: Connection reset by peer*
> * at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source)
> ~[netty-all-4.0.44.Final.jar:4.0.44.Final]*
> *INFO  [Native-Transport-Requests-11] 2017-09-01 15:31:42,722
> NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot
> allocate chunk of 1.000MiB*
>
> *INFO  [CompactionExecutor:470] 2017-09-01 10:16:42,026
> NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot
> allocate chunk of 1.000MiB*
> *INFO  [CompactionExecutor:475] 2017-09-01 10:31:42,032
> NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot
> allocate chunk of 1.000MiB*
> *INFO  [CompactionExecutor:478] 2017-09-01 10:46:42,108
> NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot
> allocate chunk of 1.000MiB*
> *INFO  [CompactionExecutor:482] 2017-09-01 11:01:42,131
> NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot
> allocate chunk of 1.000MiB*
>
> About this last error, I tried to increase `file_cache_size_in_mb` of

Re: Cassandra 3.11 is compacting forever

[Igor Leão]

Some generic errors:

*[aladdin@ip-172-16-1-10 cassandra]$ tail cassandra.log | grep -i error*
*[aladdin@ip-172-16-1-10 cassandra]$ tail cassandra.log | grep -i excep*
*[aladdin@ip-172-16-1-10 cassandra]$ tail cassandra.log | grep -i fail*
*[aladdin@ip-172-16-1-10 cassandra]$ tail debug.log | grep -i error*
*[aladdin@ip-172-16-1-10 cassandra]$ tail debug.log | grep -i exce*
*[aladdin@ip-172-16-1-10 cassandra]$ tail debug.log | grep -i fail*
*DEBUG [GossipStage:1] 2017-09-01 15:33:27,046 FailureDetector.java:457 -
Ignoring interval time of 2108299431 <(210)%20829-9431> for /172.16.1.112
*
*DEBUG [GossipStage:1] 2017-09-01 15:33:29,051 FailureDetector.java:457 -
Ignoring interval time of 2005507384 for /172.16.1.74 *
*DEBUG [GossipStage:1] 2017-09-01 15:33:45,968 FailureDetector.java:457 -
Ignoring interval time of 2003371497 for /172.16.1.74 *
*DEBUG [GossipStage:1] 2017-09-01 15:33:51,133 FailureDetector.java:457 -
Ignoring interval time of 2013260173 <(201)%20326-0173> for /172.16.1.74
*
*DEBUG [GossipStage:1] 2017-09-01 15:33:58,981 FailureDetector.java:457 -
Ignoring interval time of 2009620081 for /172.16.1.112
*
*DEBUG [GossipStage:1] 2017-09-01 15:34:19,235 FailureDetector.java:457 -
Ignoring interval time of 2010956256 for /172.16.1.74 *
*DEBUG [GossipStage:1] 2017-09-01 15:34:19,235 FailureDetector.java:457 -
Ignoring interval time of 2011127930 for /10.0.1.122 *
*[aladdin@ip-172-16-1-10 cassandra]$ tail system.log | grep -i error*
*io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)()
failed: Connection reset by peer*
*[aladdin@ip-172-16-1-10 cassandra]$ tail system.log | grep -i exce*
*INFO  [Native-Transport-Requests-5] 2017-09-01 15:22:58,806
Message.java:619 - Unexpected exception during request; channel = [id:
0xdd63db2f, L:/10.0.1.47:9042  !
R:/10.0.44.196:41422 ]*
*io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)()
failed: Connection reset by peer*
*[aladdin@ip-172-16-1-10 cassandra]$ tail system.log | grep -i fail*
*io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)()
failed: Connection reset by peer*


Some interesting errors:

1.
*DEBUG [ReadRepairStage:1] 2017-09-01 15:34:58,485 ReadCallback.java:242 -
Digest mismatch:*
*org.apache.cassandra.service.DigestMismatchException: Mismatch for key
DecoratedKey(5988282114260523734,
32623331326162652d63352d343237632d626334322d306466643762653836343830)
(023d99bbcf2263f0fa450c2312fdce88 vs a60ba37a46e0a61227a8b560fa4e0dfb)*
* at
org.apache.cassandra.service.DigestResolver.compareResponses(DigestResolver.java:92)
~[apache-cassandra-3.11.0.jar:3.11.0]*
* at
org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:233)
~[apache-cassandra-3.11.0.jar:3.11.0]*
* at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_112]*
* at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_112]*
* at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.0.jar:3.11.0]*
* at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_112]*

2.
*INFO  [Native-Transport-Requests-5] 2017-09-01 15:22:58,806
Message.java:619 - Unexpected exception during request; channel = [id:
0xdd63db2f, L:/10.0.1.47:9042  !
R:/10.0.44.196:41422 ]*
*io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)()
failed: Connection reset by peer*
* at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source)
~[netty-all-4.0.44.Final.jar:4.0.44.Final]*
*INFO  [Native-Transport-Requests-11] 2017-09-01 15:31:42,722
NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot
allocate chunk of 1.000MiB*

*INFO  [CompactionExecutor:470] 2017-09-01 10:16:42,026
NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot
allocate chunk of 1.000MiB*
*INFO  [CompactionExecutor:475] 2017-09-01 10:31:42,032
NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot
allocate chunk of 1.000MiB*
*INFO  [CompactionExecutor:478] 2017-09-01 10:46:42,108
NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot
allocate chunk of 1.000MiB*
*INFO  [CompactionExecutor:482] 2017-09-01 11:01:42,131
NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot
allocate chunk of 1.000MiB*

About this last error, I tried to increase `file_cache_size_in_mb` of this
node to 2048, but the error only changed to
*INFO  [ReadStage-2] 2017-09-01 16:18:38,657 NoSpamLogger.java:91 - Maximum
memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB*

2017-09-01 9:07 GMT-03:00 kurt greaves :

> are you seeing any errors in the logs? Is that one 

Re: Cassandra 3.11 is compacting forever

[kurt greaves]

are you seeing any errors in the logs? Is that one compaction still getting
stuck?

Re: Cassandra 3.11 is compacting forever

[Igor Leão]

Hey Kurt,

Thanks for your reply.
Soon as the whole cluster was upgraded (using existing nodes) it worked
pretty well. After a while, the high cpu usage/ pending compactions was
back affecting all cluster.
It's still an open problem.

2017-08-21 20:24 GMT-03:00 kurt greaves :

> Why are you adding new nodes? If you're upgrading you should upgrade the
> existing nodes first and then add nodes. ​
>

Re: Cassandra 3.11 is compacting forever

[kurt greaves]

Why are you adding new nodes? If you're upgrading you should upgrade the
existing nodes first and then add nodes. ​