Re: Bootstrapping new node throwing error - Mutation too large

[Surbhi Gupta]

Thanks Scott

On Wed, Mar 1, 2023 at 4:00 PM C. Scott Andreas 
wrote:

> The performance implications would primarily be due to the challenge of
> handling mutations this large themselves rather than the commitlog segment
> size. These would occupy large, contiguous areas of heap and increase
> memory pressure in the process.
>
> Increasing commit_log_segment_size_in_mb is likely the best / only
> approach, along with addressing mutation size for future writes from the
> application.
>
> I'd also strongly recommend upgrading from 3.11.5 to 3.11.15. 3.11.5 was
> released about 3.5 years ago, with a large number of bugfixes available in
> 3.11.15. That release is also drop-in, so you can upgrade simply by rev'ing
> the version and performing a rolling restart of the instances.
>
>
> – Scott
>
> On Mar 1, 2023, at 2:43 PM, Surbhi Gupta  wrote:
>
>
> Hi Cassandra Community,
>
> We have to expand our cluster and I tried to add the first node to the
> cluster and when the new node was bootstrapping , I noticed the error like
> below in the system.log, but the bootstrap process was successful .
>
> We are on 3.11.5 .
>
> ERROR [MutationStage-7] 2023-03-01 07:01:40,026
> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[MutationStage-7,5,main]
>
> java.lang.IllegalArgumentException: Mutation of 24.510MiB is too large for
> the maximum size of 16.000MiB
>
> I tried to add another node after 5 days of adding 1st node and the 2nd
> node also gave an error but this time the mutation size was bigger .
>
> ERROR [BatchlogTasks:1] 2023-03-01 01:39:59,357 CassandraDaemon.java:228
> - Exception in thread Thread[BatchlogTasks:1,5,main]
>
> java.lang.IllegalArgumentException: Mutation of 40.717MiB is too large for
> the maximum size of 16.000MiB
>
> And on the first node( which was added 5 days ago), we saw messages like
> below till the 2nd node bootstrap was completed.
>  "Mutation of 24.510MiB is too large for the maximum size of 16.000MiB"
> but after the 2nd node bootstarp process was completed, we started seeing
> the message about hint byte is too large but there are no hint files in any
> of the nodes on hints directory . And the message about mutation too large
> has stopped popping in system.log on the first added node. Now we are
> seeing hints error as below on the first newly added node.
>
> ERROR [BatchlogTasks:1] 2023-03-01 07:48:32,091 CassandraDaemon.java:228
> - Exception in thread Thread[BatchlogTasks:1,5,main]
>
> java.lang.IllegalArgumentException: Hint of 25700336 bytes is too large -
> the maximum size is 16777216
>
> As per the application team , nothing changed .
>
> As per the workaround commitlog_segment_size_in_mb can be increased to
> accommodate the increased size but that doesnt seem to be a concrete
> solution and it can have performance impact ,  because by design intent
> the maximum allowed segment size is 50% of the configured
> commit_log_segment_size_in_mb. This is so Cassandra avoids writing segments
> with large amounts of empty space.
>
> Looks like i am hitting
> https://issues.apache.org/jira/browse/CASSANDRA-15152
> 
>
> Anyone have any suggestions?
>
> Thanks
> Surbhi
>
>
>
>
>
>
>
>
>

Re: Bootstrapping new node throwing error - Mutation too large

[C. Scott Andreas]

The performance implications would primarily be due to the challenge of handling mutations this 
large themselves rather than the commitlog segment size. These would occupy large, contiguous 
areas of heap and increase memory pressure in the process.Increasing 
commit_log_segment_size_in_mb is likely the best / only approach, along with addressing 
mutation size for future writes from the application.I'd also strongly recommend upgrading from 
3.11.5 to 3.11.15. 3.11.5 was released about 3.5 years ago, with a large number of bugfixes 
available in 3.11.15. That release is also drop-in, so you can upgrade simply by rev'ing the 
version and performing a rolling restart of the instances.– ScottOn Mar 1, 2023, at 2:43 PM, 
Surbhi Gupta  wrote:Hi Cassandra Community,We have to expand 
our cluster and I tried to add the first node to the cluster and when the new node was 
bootstrapping , I noticed the error like below in the system.log, but the bootstrap process was 
successful . We are on 3.11.5 .ERROR [MutationStage-7] 2023-03-01 07:01:40,026 
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread 
Thread[MutationStage-7,5,main]java.lang.IllegalArgumentException: Mutation of 24.510MiB is too 
large for the maximum size of 16.000MiBI tried to add another node after 5 days of adding 1st 
node and the 2nd node also gave an error but this time the mutation size was bigger .ERROR 
[BatchlogTasks:1] 2023-03-01 01:39:59,357 CassandraDaemon.java:228 - Exception in thread 
Thread[BatchlogTasks:1,5,main]java.lang.IllegalArgumentException: Mutation of 40.717MiB is too 
large for the maximum size of 16.000MiBAnd on the first node( which was added 5 days ago), we 
saw messages like below till the 2nd node bootstrap was completed. "Mutation of 24.510MiB 
is too large for the maximum size of 16.000MiB"but after the 2nd node bootstarp process 
was completed, we started seeing the message about hint byte is too large but there are no hint 
files in any of the nodes on hints directory . And the message about mutation too large has 
stopped popping in system.log on the first added node. Now we are seeing hints error as below 
on the first newly added node. ERROR [BatchlogTasks:1] 2023-03-01 07:48:32,091 
CassandraDaemon.java:228 - Exception in thread 
Thread[BatchlogTasks:1,5,main]java.lang.IllegalArgumentException: Hint of 25700336 bytes is too 
large - the maximum size is 16777216As per the application team , nothing changed . As per the 
workaround commitlog_segment_size_in_mb can be increased to accommodate the increased size but 
that doesnt seem to be a concrete solution and it can have performance impact ,  because by 
design intent the maximum allowed segment size is 50% of the configured 
commit_log_segment_size_in_mb. This is so Cassandra avoids writing segments with large amounts 
of empty space.  Looks like i am hitting https://issues.apache.org/jira/browse/CASSANDRA-15152 
Anyone have any suggestions?ThanksSurbhi 

Bootstrapping new node throwing error - Mutation too large

[Surbhi Gupta]

Hi Cassandra Community,

We have to expand our cluster and I tried to add the first node to the
cluster and when the new node was bootstrapping , I noticed the error like
below in the system.log, but the bootstrap process was successful .

We are on 3.11.5 .

ERROR [MutationStage-7] 2023-03-01 07:01:40,026
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
Thread[MutationStage-7,5,main]

java.lang.IllegalArgumentException: Mutation of 24.510MiB is too large for
the maximum size of 16.000MiB

I tried to add another node after 5 days of adding 1st node and the 2nd
node also gave an error but this time the mutation size was bigger .

ERROR [BatchlogTasks:1] 2023-03-01 01:39:59,357 CassandraDaemon.java:228 -
Exception in thread Thread[BatchlogTasks:1,5,main]

java.lang.IllegalArgumentException: Mutation of 40.717MiB is too large for
the maximum size of 16.000MiB

And on the first node( which was added 5 days ago), we saw messages like
below till the 2nd node bootstrap was completed.
 "Mutation of 24.510MiB is too large for the maximum size of 16.000MiB"
but after the 2nd node bootstarp process was completed, we started seeing
the message about hint byte is too large but there are no hint files in any
of the nodes on hints directory . And the message about mutation too large
has stopped popping in system.log on the first added node. Now we are
seeing hints error as below on the first newly added node.

ERROR [BatchlogTasks:1] 2023-03-01 07:48:32,091 CassandraDaemon.java:228 -
Exception in thread Thread[BatchlogTasks:1,5,main]

java.lang.IllegalArgumentException: Hint of 25700336 bytes is too large -
the maximum size is 16777216

As per the application team , nothing changed .

As per the workaround commitlog_segment_size_in_mb can be increased to
accommodate the increased size but that doesnt seem to be a concrete
solution and it can have performance impact ,  because by design intent the
maximum allowed segment size is 50% of the configured
commit_log_segment_size_in_mb. This is so Cassandra avoids writing segments
with large amounts of empty space.

Looks like i am hitting
https://issues.apache.org/jira/browse/CASSANDRA-15152


Anyone have any suggestions?

Thanks
Surbhi