subject:"RE\: MV's stuck in build state"

RE: MV's stuck in build state

2019-03-04 Thread Kenneth Brotman

Hi Dipan,

 

What is the state of the dev and production clusters now?  

How big is the production cluster?

How many nodes on each cluster spin out of control?

 

On the production cluster, the data is 67 MB, you'd have use a value at
least twice that size as the commitlog_segment_size.  Of course you don't
want to leave it really high if you do change it.  

On the dev cluster the data is 18 MB, you used a value way over twice size
when you bumped the commitlog_sement_size to 128 MB, but in doing so you are
wasting a lot of memory capacity of course, but you say it didn't fix the
problem on that cluster, so. is the message you are getting reflecting this
change by showing  to be 128 MB now or is it still a different value?

 

Is there another problem too perhaps you were running low on capacity on the
node?  Could low capacity be a problem on either cluster? 

 

 

From: Dipan Shah [mailto:dipan@hotmail.com] 
Sent: Monday, March 04, 2019 12:52 AM
To: Kenneth Brotman; user@cassandra.apache.org
Subject: Re: MV's stuck in build state

 

Hello Kenneth,

 

Apologies for the late reply.

 

1) On production the value of x was 67 MB and y was 16 MV as value of
commitlog_segment_size_in_mb is 32.

2) On Dev the value of x was 18 MB and y was 16 MV as value of
commitlog_segment_size_in_mb was 32 initially. I had bumped up the value of
commitlog_segment_size_in_mb to 128 when the node eventually crashed.

3) No I did not try org.apache.cassandra.db:type=CompactionManager but I did
try "nodetool stop" and "nodetool stop VIEW_BUILD".

 

Thanks,

Dipan Shah

  _  

From: Kenneth Brotman 
Sent: Friday, March 1, 2019 8:19 PM
To: user@cassandra.apache.org
Subject: RE: MV's stuck in build state 

 

Dipan,

 

On your production cluster, when you were first getting the "Mutation of 
bytes ." message, what was the value of x and y?

How about when you got the message on the Dev Cluster, what was the value of
x and y in that message?

On the Dev cluster, did you try going into JMX and directly hitting the
org.apache.cassandra.db:type=CompactionManager mbean's stopCompaction
operation?

 

 

From: Dipan Shah [mailto:dipan@hotmail.com] 
Sent: Friday, March 01, 2019 12:56 AM
To: Kenneth Brotman; user@cassandra.apache.org
Subject: Re: MV's stuck in build state

 

Hello Kenneth,

 

Thanks for replying.

 

I had actually tried this on a Dev environment earlier and it caused the
node to spin out of control. I'll explain what I did over there:

 

1) Found "Mutation of  bytes is too large for the maxiumum size of "
and thus increased the value of "commitlog_segment_size_in_mb" to 64

2) This worked for a few minutes and again the view started failing when it
hit the new limits and the messages now were "Mutation of  bytes is too
large for the maxiumum size of 2*"

3) So just to try I increased the value to 128

4) Now after this change the node started crashing as soon as I brought the
service online. I was not able to recover even after restoring the value of
"commitlog_segment_size_in_mb" to 32

 

Now there is a key differences to that issue and what I am facing currently:

 

The views were not dropped on the earlier environment whereas I have already
dropped the view on the current environment (and cant experiment much as the
current environment is in production).

 

I know this is a bit tricky but I'm pretty much stuck over here and thinking
of finding a non-problem creating solution over here.

 

Thanks,

Dipan Shah

  _  

From: Kenneth Brotman 
Sent: Friday, March 1, 2019 12:26 AM
To: user@cassandra.apache.org
Subject: RE: MV's stuck in build state 

 

Hi Dipan,

 

Did you try following the advice in the referenced DataStax article called
Mutation of
<https://support.datastax.com/hc/en-us/articles/207267063-Mutation-of-x-byte
s-is-too-large-for-the-maxiumum-size-of-y->  bytes is too large for the
maximum size of  as suggested in the stackoverflow.com post you cited?

 

Kenneth Brotman

 

From: Dipan Shah [mailto:dipan@hotmail.com] 
Sent: Thursday, February 28, 2019 2:23 AM
To: Dipan Shah; user@cassandra.apache.org
Subject: Re: MV's stuck in build state

 

Forgot to add version info. This is on 37.

 

[cqlsh 5.0.1 | Cassandra 3.7 | CQL spec 34.2 | Native protocol v4]

 

Thanks,

Dipan Shah

  _  

From: Dipan Shah 
Sent: Thursday, February 28, 2019 3:38 PM
To: user@cassandra.apache.org
Subject: MV's stuck in build state 

 

Hello All,

 

I have a few MV's that are stuck in build state because of a bad schema
design and thus getting a lot of messages like this "Mutation xxx is too
large for maximum size of 16.000MiB".

 



 

I have dropped those MV's and I can no longer see their schema in the
keyspace. But they are visible under "system.views_build_in_progress" and
"nodetool viewbuildstatus"

 

I have tried "nodetool stop VIEW_BUILD" as suggested here:
h

Re: MV's stuck in build state

2019-03-04 Thread Dipan Shah

Hello Kenneth,

Apologies for the late reply.

1) On production the value of x was 67 MB and y was 16 MV as value of 
commitlog_segment_size_in_mb is 32.
2) On Dev the value of x was 18 MB and y was 16 MV as value of 
commitlog_segment_size_in_mb was 32 initially. I had bumped up the value of 
commitlog_segment_size_in_mb to 128 when the node eventually crashed.
3) No I did not try org.apache.cassandra.db:type=CompactionManager but I did 
try "nodetool stop" and "nodetool stop VIEW_BUILD".


Thanks,

Dipan Shah


From: Kenneth Brotman 
Sent: Friday, March 1, 2019 8:19 PM
To: user@cassandra.apache.org
Subject: RE: MV's stuck in build state


Dipan,



On your production cluster, when you were first getting the “Mutation of  
bytes …” message, what was the value of x and y?

How about when you got the message on the Dev Cluster, what was the value of x 
and y in that message?

On the Dev cluster, did you try going into JMX and directly hitting the 
org.apache.cassandra.db:type=CompactionManager mbean's stopCompaction operation?





From: Dipan Shah [mailto:dipan@hotmail.com]
Sent: Friday, March 01, 2019 12:56 AM
To: Kenneth Brotman; user@cassandra.apache.org
Subject: Re: MV's stuck in build state



Hello Kenneth,



Thanks for replying.



I had actually tried this on a Dev environment earlier and it caused the node 
to spin out of control. I'll explain what I did over there:



1) Found "Mutation of  bytes is too large for the maxiumum size of " and 
thus increased the value of "commitlog_segment_size_in_mb" to 64

2) This worked for a few minutes and again the view started failing when it hit 
the new limits and the messages now were "Mutation of  bytes is too large 
for the maxiumum size of 2*"

3) So just to try I increased the value to 128

4) Now after this change the node started crashing as soon as I brought the 
service online. I was not able to recover even after restoring the value of 
"commitlog_segment_size_in_mb" to 32



Now there is a key differences to that issue and what I am facing currently:



The views were not dropped on the earlier environment whereas I have already 
dropped the view on the current environment (and cant experiment much as the 
current environment is in production).



I know this is a bit tricky but I'm pretty much stuck over here and thinking of 
finding a non-problem creating solution over here.



Thanks,

Dipan Shah



From: Kenneth Brotman 
Sent: Friday, March 1, 2019 12:26 AM
To: user@cassandra.apache.org
Subject: RE: MV's stuck in build state



Hi Dipan,



Did you try following the advice in the referenced DataStax article called 
Mutation of  bytes is too large for the maximum size of 
<https://support.datastax.com/hc/en-us/articles/207267063-Mutation-of-x-bytes-is-too-large-for-the-maxiumum-size-of-y->
 as suggested in the stackoverflow.com post you cited?



Kenneth Brotman



From: Dipan Shah [mailto:dipan@hotmail.com]
Sent: Thursday, February 28, 2019 2:23 AM
To: Dipan Shah; user@cassandra.apache.org
Subject: Re: MV's stuck in build state



Forgot to add version info. This is on 3.7.



[cqlsh 5.0.1 | Cassandra 3.7 | CQL spec 34.2 | Native protocol v4]



Thanks,

Dipan Shah



From: Dipan Shah 
Sent: Thursday, February 28, 2019 3:38 PM
To: user@cassandra.apache.org
Subject: MV's stuck in build state



Hello All,



I have a few MV's that are stuck in build state because of a bad schema design 
and thus getting a lot of messages like this "Mutation xxx is too large for 
maximum size of 16.000MiB".



[cid:image001.png@01D4CFFA.539C41B0]



I have dropped those MV's and I can no longer see their schema in the keyspace. 
But they are visible under "system.views_build_in_progress" and "nodetool 
viewbuildstatus".



I have tried "nodetool stop VIEW_BUILD" as suggested here: 
https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-view-build
 and have also reboot a few nodes in the cluster. This has also not helped.



Is there anything else that can be done over here?

[https://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch-i...@2.png?v=73d79a89bded]<https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-view-build>


Stop Cassandra Materialized View Build - Stack 
Overflow<https://stackoverflowcom/questions/40553499/stop-cassandra-materialized-view-build>

Its not documented, but nodetool stop actually takes any compaction type, not 
just the ones listed (which the view build is one of). So you can simply: 
nodetool stop VIEW_BUILD Or you can hit JMX directly with the 
org.apache.cassandra.db:type=CompactionManager mbean's stopCompaction 
operation.. All thats really gonna do is set a flag for the view builder to 
stop on its next loop.

stackoverflow.com






Thanks,

Dipan Shah

RE: MV's stuck in build state

2019-03-01 Thread Kenneth Brotman

Dipan,

 

On your production cluster, when you were first getting the "Mutation of 
bytes ." message, what was the value of x and y?

How about when you got the message on the Dev Cluster, what was the value of
x and y in that message?

On the Dev cluster, did you try going into JMX and directly hitting the
org.apache.cassandra.db:type=CompactionManager mbean's stopCompaction
operation?

 

 

From: Dipan Shah [mailto:dipan@hotmail.com] 
Sent: Friday, March 01, 2019 12:56 AM
To: Kenneth Brotman; user@cassandra.apache.org
Subject: Re: MV's stuck in build state

 

Hello Kenneth,

 

Thanks for replying.

 

I had actually tried this on a Dev environment earlier and it caused the
node to spin out of control. I'll explain what I did over there:

 

1) Found "Mutation of  bytes is too large for the maxiumum size of "
and thus increased the value of "commitlog_segment_size_in_mb" to 64

2) This worked for a few minutes and again the view started failing when it
hit the new limits and the messages now were "Mutation of  bytes is too
large for the maxiumum size of 2*"

3) So just to try I increased the value to 128

4) Now after this change the node started crashing as soon as I brought the
service online. I was not able to recover even after restoring the value of
"commitlog_segment_size_in_mb" to 32

 

Now there is a key differences to that issue and what I am facing currently:

 

The views were not dropped on the earlier environment whereas I have already
dropped the view on the current environment (and cant experiment much as the
current environment is in production).

 

I know this is a bit tricky but I'm pretty much stuck over here and thinking
of finding a non-problem creating solution over here.

 

Thanks,

Dipan Shah

  _  

From: Kenneth Brotman 
Sent: Friday, March 1, 2019 12:26 AM
To: user@cassandra.apache.org
Subject: RE: MV's stuck in build state 

 

Hi Dipan,

 

Did you try following the advice in the referenced DataStax article called
Mutation of
<https://support.datastax.com/hc/en-us/articles/207267063-Mutation-of-x-byte
s-is-too-large-for-the-maxiumum-size-of-y->  bytes is too large for the
maximum size of  as suggested in the stackoverflow.com post you cited?

 

Kenneth Brotman

 

From: Dipan Shah [mailto:dipan@hotmail.com] 
Sent: Thursday, February 28, 2019 2:23 AM
To: Dipan Shah; user@cassandra.apache.org
Subject: Re: MV's stuck in build state

 

Forgot to add version info. This is on 3.7.

 

[cqlsh 5.0.1 | Cassandra 3.7 | CQL spec 34.2 | Native protocol v4]

 

Thanks,

Dipan Shah

  _  

From: Dipan Shah 
Sent: Thursday, February 28, 2019 3:38 PM
To: user@cassandra.apache.org
Subject: MV's stuck in build state 

 

Hello All,

 

I have a few MV's that are stuck in build state because of a bad schema
design and thus getting a lot of messages like this "Mutation xxx is too
large for maximum size of 16.000MiB".

 



 

I have dropped those MV's and I can no longer see their schema in the
keyspace. But they are visible under "system.views_build_in_progress" and
"nodetool viewbuildstatus".

 

I have tried "nodetool stop VIEW_BUILD" as suggested here:
https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-vie
w-build and have also reboot a few nodes in the cluster. This has also not
helped.

 

Is there anything else that can be done over here?


 
<https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-vi
ew-build> 

 
<https://stackoverflowcom/questions/40553499/stop-cassandra-materialized-vie
w-build> Stop Cassandra Materialized View Build - Stack Overflow

Its not documented, but nodetool stop actually takes any compaction type,
not just the ones listed (which the view build is one of). So you can
simply: nodetool stop VIEW_BUILD Or you can hit JMX directly with the
org.apache.cassandra.db:type=CompactionManager mbean's stopCompaction
operation.. All thats really gonna do is set a flag for the view builder to
stop on its next loop.

stackoverflow.com

 

 

Thanks,

Dipan Shah

Re: MV's stuck in build state

2019-03-01 Thread Dipan Shah

Hello Kenneth,

Thanks for replying.

I had actually tried this on a Dev environment earlier and it caused the node 
to spin out of control. I'll explain what I did over there:

1) Found "Mutation of  bytes is too large for the maxiumum size of " and 
thus increased the value of "commitlog_segment_size_in_mb" to 64
2) This worked for a few minutes and again the view started failing when it hit 
the new limits and the messages now were "Mutation of  bytes is too large 
for the maxiumum size of 2*"
3) So just to try I increased the value to 128
4) Now after this change the node started crashing as soon as I brought the 
service online. I was not able to recover even after restoring the value of 
"commitlog_segment_size_in_mb" to 32

Now there is a key differences to that issue and what I am facing currently:

The views were not dropped on the earlier environment whereas I have already 
dropped the view on the current environment (and cant experiment much as the 
current environment is in production).

I know this is a bit tricky but I'm pretty much stuck over here and thinking of 
finding a non-problem creating solution over here.


Thanks,

Dipan Shah


From: Kenneth Brotman 
Sent: Friday, March 1, 2019 12:26 AM
To: user@cassandra.apache.org
Subject: RE: MV's stuck in build state


Hi Dipan,



Did you try following the advice in the referenced DataStax article called 
Mutation of  bytes is too large for the maximum size of 
<https://support.datastax.com/hc/en-us/articles/207267063-Mutation-of-x-bytes-is-too-large-for-the-maxiumum-size-of-y->
 as suggested in the stackoverflow.com post you cited?



Kenneth Brotman



From: Dipan Shah [mailto:dipan@hotmail.com]
Sent: Thursday, February 28, 2019 2:23 AM
To: Dipan Shah; user@cassandra.apache.org
Subject: Re: MV's stuck in build state



Forgot to add version info. This is on 3.7.



[cqlsh 5.0.1 | Cassandra 3.7 | CQL spec 3.4.2 | Native protocol v4]



Thanks,

Dipan Shah



From: Dipan Shah 
Sent: Thursday, February 28, 2019 3:38 PM
To: user@cassandra.apache.org
Subject: MV's stuck in build state



Hello All,



I have a few MV's that are stuck in build state because of a bad schema design 
and thus getting a lot of messages like this "Mutation xxx is too large for 
maximum size of 16.000MiB".



[cid:image001.png@01D4CF54.4F7AFE10]



I have dropped those MV's and I can no longer see their schema in the keyspace. 
But they are visible under "system.views_build_in_progress" and "nodetool 
viewbuildstatus".



I have tried "nodetool stop VIEW_BUILD" as suggested here: 
https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-view-build
 and have also reboot a few nodes in the cluster. This has also not helped.



Is there anything else that can be done over here?

[https://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch-i...@2.png?v=73d79a89bded]<https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-view-build>


Stop Cassandra Materialized View Build - Stack 
Overflow<https://stackoverflowcom/questions/40553499/stop-cassandra-materialized-view-build>

Its not documented, but nodetool stop actually takes any compaction type, not 
just the ones listed (which the view build is one of). So you can simply: 
nodetool stop VIEW_BUILD Or you can hit JMX directly with the 
org.apache.cassandra.db:type=CompactionManager mbean's stopCompaction 
operation.. All thats really gonna do is set a flag for the view builder to 
stop on its next loop.

stackoverflow.com






Thanks,

Dipan Shah

RE: MV's stuck in build state

2019-02-28 Thread Kenneth Brotman

Hi Dipan,

 

Did you try following the advice in the referenced DataStax article called
Mutation of
<https://support.datastax.com/hc/en-us/articles/207267063-Mutation-of-x-byte
s-is-too-large-for-the-maxiumum-size-of-y->  bytes is too large for the
maximum size of  as suggested in the stackoverflow.com post you cited?

 

Kenneth Brotman

 

From: Dipan Shah [mailto:dipan@hotmail.com] 
Sent: Thursday, February 28, 2019 2:23 AM
To: Dipan Shah; user@cassandra.apache.org
Subject: Re: MV's stuck in build state

 

Forgot to add version info. This is on 3.7.

 

[cqlsh 5.0.1 | Cassandra 3.7 | CQL spec 3.4.2 | Native protocol v4]

 

Thanks,

Dipan Shah

  _  

From: Dipan Shah 
Sent: Thursday, February 28, 2019 3:38 PM
To: user@cassandra.apache.org
Subject: MV's stuck in build state 

 

Hello All,

 

I have a few MV's that are stuck in build state because of a bad schema
design and thus getting a lot of messages like this "Mutation xxx is too
large for maximum size of 16.000MiB".

 



 

I have dropped those MV's and I can no longer see their schema in the
keyspace. But they are visible under "system.views_build_in_progress" and
"nodetool viewbuildstatus".

 

I have tried "nodetool stop VIEW_BUILD" as suggested here:
https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-vie
w-build and have also reboot a few nodes in the cluster. This has also not
helped.

 

Is there anything else that can be done over here?


 
<https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-vi
ew-build> 

 
<https://stackoverflowcom/questions/40553499/stop-cassandra-materialized-vie
w-build> Stop Cassandra Materialized View Build - Stack Overflow

Its not documented, but nodetool stop actually takes any compaction type,
not just the ones listed (which the view build is one of). So you can
simply: nodetool stop VIEW_BUILD Or you can hit JMX directly with the
org.apache.cassandra.db:type=CompactionManager mbean's stopCompaction
operation.. All thats really gonna do is set a flag for the view builder to
stop on its next loop.

stackoverflow.com

 

 

Thanks,

Dipan Shah

Re: MV's stuck in build state

2019-02-28 Thread Dipan Shah

Forgot to add version info. This is on 3.7.

[cqlsh 5.0.1 | Cassandra 3.7 | CQL spec 3.4.2 | Native protocol v4]


Thanks,

Dipan Shah


From: Dipan Shah 
Sent: Thursday, February 28, 2019 3:38 PM
To: user@cassandra.apache.org
Subject: MV's stuck in build state

Hello All,

I have a few MV's that are stuck in build state because of a bad schema design 
and thus getting a lot of messages like this "Mutation xxx is too large for 
maximum size of 16.000MiB".

[cid:cde7867a-aa71-4046-97f8-7a16b1a4f3c9]

I have dropped those MV's and I can no longer see their schema in the keyspace. 
But they are visible under "system.views_build_in_progress" and "nodetool 
viewbuildstatus".

I have tried "nodetool stop VIEW_BUILD" as suggested here: 
https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-view-build
 and have also reboot a few nodes in the cluster. This has also not helped.

Is there anything else that can be done over here?
[https://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch-i...@2.png?v=73d79a89bded]

Stop Cassandra Materialized View Build - Stack 
Overflow
Its not documented, but nodetool stop actually takes any compaction type, not 
just the ones listed (which the view build is one of). So you can simply: 
nodetool stop VIEW_BUILD Or you can hit JMX directly with the 
org.apache.cassandra.db:type=CompactionManager mbean's stopCompaction 
operation.. All thats really gonna do is set a flag for the view builder to 
stop on its next loop.
stackoverflow.com




Thanks,

Dipan Shah

RE: MV's stuck in build state

Re: MV's stuck in build state

RE: MV's stuck in build state

Re: MV's stuck in build state

RE: MV's stuck in build state

Re: MV's stuck in build state

6 matches

Site Navigation

Mail list logo

Footer information