Re: Consequences of dropping Materialized views

[Surbhi Gupta]

Application team confirmed that they are * not* referencing the dropped MVs
anywhere for reading or writing


On Tue, 18 Feb 2020 at 22:25, Surbhi Gupta  wrote:

> So should upgrading to 3.11.1 will solve this issue?
>
> On Tue, 18 Feb 2020 at 22:18, Surbhi Gupta 
> wrote:
>
>> Thanks Eric...
>>
>> On Tue, 18 Feb 2020 at 22:06, Erick Ramirez 
>> wrote:
>>
>>> Just to add to my above point because here we are dropping MV not a
 regular table.
 And MV does read before write , Is this the reason we are seeing the
 below message? Trying to understand

 WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,932 HintsReader.java:237 
 - Failed to read a hint for /10.X.X.X: 
 f75e58e8-c255-4318-a553-06487b6bbe8c - table with id 
 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 is unknown in file 
 f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints


>>> That warning message means that the hint for host /10.X.X.X (the IP you
>>> obfuscated) could not be read from the hint file (file
>>> f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints) because the
>>> table (or MV) it belongs to with CF ID
>>> 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 doesn't exist anymore.
>>>
>>>

Re: Consequences of dropping Materialized views

[Erick Ramirez]

>
> So should upgrading to 3.11.1 will solve this issue?
>

Upgrading off 3.11.0 will prevent nodes going down as a result of the hint
replay bug in CASSANDRA-13696, yes. But I'd recommend upgrading to the
latest C* 3.11.6 unless you have a very specific reason for upgrading to
3.11.1 which was released nearly two and half years ago. Cheers!

Re: Consequences of dropping Materialized views

[Surbhi Gupta]

So should upgrading to 3.11.1 will solve this issue?

On Tue, 18 Feb 2020 at 22:18, Surbhi Gupta  wrote:

> Thanks Eric...
>
> On Tue, 18 Feb 2020 at 22:06, Erick Ramirez 
> wrote:
>
>> Just to add to my above point because here we are dropping MV not a
>>> regular table.
>>> And MV does read before write , Is this the reason we are seeing the
>>> below message? Trying to understand
>>>
>>> WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,932 HintsReader.java:237 - 
>>> Failed to read a hint for /10.X.X.X: f75e58e8-c255-4318-a553-06487b6bbe8c - 
>>> table with id 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 is unknown in file 
>>> f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints
>>>
>>>
>> That warning message means that the hint for host /10.X.X.X (the IP you
>> obfuscated) could not be read from the hint file (file
>> f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints) because the
>> table (or MV) it belongs to with CF ID
>> 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 doesn't exist anymore.
>>
>>

Re: Consequences of dropping Materialized views

[Surbhi Gupta]

Thanks Eric...

On Tue, 18 Feb 2020 at 22:06, Erick Ramirez 
wrote:

> Just to add to my above point because here we are dropping MV not a
>> regular table.
>> And MV does read before write , Is this the reason we are seeing the
>> below message? Trying to understand
>>
>> WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,932 HintsReader.java:237 - 
>> Failed to read a hint for /10.X.X.X: f75e58e8-c255-4318-a553-06487b6bbe8c - 
>> table with id 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 is unknown in file 
>> f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints
>>
>>
> That warning message means that the hint for host /10.X.X.X (the IP you
> obfuscated) could not be read from the hint file (file
> f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints) because the
> table (or MV) it belongs to with CF ID
> 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 doesn't exist anymore.
>
>

Re: Consequences of dropping Materialized views

[Erick Ramirez]

>
> Just to add to my above point because here we are dropping MV not a
> regular table.
> And MV does read before write , Is this the reason we are seeing the below
> message? Trying to understand
>
> WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,932 HintsReader.java:237 - 
> Failed to read a hint for /10.X.X.X: f75e58e8-c255-4318-a553-06487b6bbe8c - 
> table with id 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 is unknown in file 
> f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints
>
>
That warning message means that the hint for host /10.X.X.X (the IP you
obfuscated) could not be read from the hint file (file
f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints) because the
table (or MV) it belongs to with CF ID 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1
doesn't exist anymore.

Re: Consequences of dropping Materialized views

[Surbhi Gupta]

Just to add to my above point because here we are dropping MV not a regular
table.
And MV does read before write , Is this the reason we are seeing the below
message? Trying to understand

WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,932
HintsReader.java:237 - Failed to read a hint for /10.X.X.X:
f75e58e8-c255-4318-a553-06487b6bbe8c - table with id
7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 is unknown in file
f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints


On Tue, 18 Feb 2020 at 21:41, Erick Ramirez 
wrote:

> I'm not sure I understand your last response. Was there a question in
> there somewhere? Cheers!
>
>>

Re: Consequences of dropping Materialized views

[Erick Ramirez]

I'm not sure I understand your last response. Was there a question in there
somewhere? Cheers!

>

Re: Consequences of dropping Materialized views

[Surbhi Gupta]

Hi Eric,
As per  https://issues.apache.org/jira/browse/CASSANDRA-13696 , this issue
happens even with write traffic

"I did more investigation today. Seems it's more serious than I thought.
Even there's no down node, "drop table" + write traffic, will trigger the
problem."

Thanks
Surbhi

On Tue, 18 Feb 2020 at 20:04, Erick Ramirez 
wrote:

> We are Cassandra 3.11.0 unfortunately :(
>>
>
> Oh, right. That's why the hint read failure is causing the node to
> shutdown. We've at least identified that. I was worried that there was
> another bug we didn't know about. Cheers!
>
>>

Re: Consequences of dropping Materialized views

[Erick Ramirez]

>
> We are Cassandra 3.11.0 unfortunately :(
>

Oh, right. That's why the hint read failure is causing the node to
shutdown. We've at least identified that. I was worried that there was
another bug we didn't know about. Cheers!

>

Re: Consequences of dropping Materialized views

[Surbhi Gupta]

We tested this is dev and test before dropping in production but did not
see this issue in dev/test .
I am yet to hear from application team.
However if it was application read was happening from the dropped MV then
we would have caught this error in dev/test itself.
And we are running Cassandra 3.11.0 in dev/test and prod .

Just a thought

On Tue, 18 Feb 2020 at 19:49, Surbhi Gupta  wrote:

> We are Cassandra 3.11.0 unfortunately :(
>
> On Tue, 18 Feb 2020 at 19:41, Erick Ramirez 
> wrote:
>
>> Clearly the hint error invoked the fs error handler - probably
>>> incorrectly - which shut down the db. That’s not ok and deserves a JIRA.
>>>
>>
>> It's supposed to have been fixed by CASSANDRA-13696 in 3.0.15/3.11.1 but
>> I'm waiting for Surbhi to confirm the exact C* version. Cheers!
>>
>

Re: Consequences of dropping Materialized views

[Surbhi Gupta]

We are Cassandra 3.11.0 unfortunately :(

On Tue, 18 Feb 2020 at 19:41, Erick Ramirez 
wrote:

> Clearly the hint error invoked the fs error handler - probably incorrectly
>> - which shut down the db. That’s not ok and deserves a JIRA.
>>
>
> It's supposed to have been fixed by CASSANDRA-13696 in 3.0.15/3.11.1 but
> I'm waiting for Surbhi to confirm the exact C* version. Cheers!
>

Re: Consequences of dropping Materialized views

[Erick Ramirez]

>
> Clearly the hint error invoked the fs error handler - probably incorrectly
> - which shut down the db. That’s not ok and deserves a JIRA.
>

It's supposed to have been fixed by CASSANDRA-13696 in 3.0.15/3.11.1 but
I'm waiting for Surbhi to confirm the exact C* version. Cheers!

Re: Consequences of dropping Materialized views

[Jeff Jirsa]

Reading from a non-existent table shouldn’t crash the database

Clearly the hint error invoked the fs error handler - probably incorrectly - 
which shut down the db. That’s not ok and deserves a JIRA.

If dropping a table causes a hint checksum error then it needs to be fixed.



>> On Feb 18, 2020, at 6:49 PM, Erick Ramirez  
>> wrote:
> 
>> We are on cassandra 3.11 , we are using G1GC and using 16GB of heap.
> 
> Which exact version of C* is it again?
>> WARN  [MessagingService-Incoming-/10.X.X.X] 2020-02-18 14:21:47,115 
>> IncomingTcpConnection.java:103 - UnknownColumnFamilyException reading from 
>> socket; closing
> 
> This is expected because your app is trying to read the MV you just dropped.
> 
>> ERROR [MutationStage-9] 2020-02-18 14:21:47,267 Keyspace.java:593 - 
>> Attempting to mutate non-existant table 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 
> 
> Any view update transactions which are already in-flight will fail. Again, 
> this is expected since the MV updates are not synchronous with the base table 
> updates.
> 
>> WARN  [BatchlogTasks:1] 2020-02-18 14:21:53,786 BatchlogManager.java:252 - 
>> Skipped batch replay of a19c6480-5294-11ea-9e09-3948c59ad0f5 due to {}
> 
> As above, this is expected since the MV updates which are already in the 
> queue will fail to apply because the MV got dropped.
> 
>> WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,932 HintsReader.java:237 - 
>> Failed to read a hint for /10.X.X.X: f75e58e8-c255-4318-a553-06487b6bbe8c - 
>> table with id 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 is unknown in file 
>> f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints
> 
> This is also expected. It won't be able to read the hint for a non-existent 
> MV.
> 
> So where to from here? The symptoms you described indicate that you haven't 
> updated your app before you dropped the MVs. That is key here -- your app 
> shouldn't be issuing reads for the MV since your intention is to drop it. 
> What I think happened is that the nodes were backed up with read requests for 
> the MV which no longer exists.
> 
> Can you confirm that you have changed your application so it is not supposed 
> to issue reads to the MV? You shouldn't drop MVs if you're still going to 
> issue read requests from them just like you shouldn't drop tables your app is 
> still expected to access. I hope that makes sense. Cheers!

Re: Consequences of dropping Materialized views

[Surbhi Gupta]

Thanks Eric, Let me go back to the app team

On Tue, Feb 18, 2020 at 6:49 PM Erick Ramirez 
wrote:

> We are on cassandra 3.11 , we are using G1GC and using 16GB of heap.
>>
>
> Which exact version of C* is it again?
>
>> WARN [MessagingService-Incoming-/10.X.X.X] 2020-02-18 14:21:47,115
>> IncomingTcpConnection.java:103 - UnknownColumnFamilyException reading from
>> socket; closing
>>
>
> This is expected because your app is trying to read the MV you just
> dropped.
>
> ERROR [MutationStage-9] 2020-02-18 14:21:47,267 Keyspace.java:593 - 
> Attempting to mutate non-existant table 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1
>>
>>
> Any view update transactions which are already in-flight will fail. Again,
> this is expected since the MV updates are not synchronous with the base
> table updates.
>
> WARN  [BatchlogTasks:1] 2020-02-18 14:21:53,786 BatchlogManager.java:252 - 
> Skipped batch replay of a19c6480-5294-11ea-9e09-3948c59ad0f5 due to {}
>>
>>
> As above, this is expected since the MV updates which are already in the
> queue will fail to apply because the MV got dropped.
>
> WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,932 HintsReader.java:237 - 
> Failed to read a hint for /10.X.X.X: f75e58e8-c255-4318-a553-06487b6bbe8c - 
> table with id 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 is unknown in file 
> f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints
>>
>>
> This is also expected. It won't be able to read the hint for a
> non-existent MV.
>
> So where to from here? The symptoms you described indicate that you
> haven't updated your app *before* you dropped the MVs. That is key here
> -- your app shouldn't be issuing reads for the MV since your intention is
> to drop it. What I think happened is that the nodes were backed up with
> read requests for the MV which no longer exists.
>
> Can you confirm that you have changed your application so it is not
> supposed to issue reads to the MV? You shouldn't drop MVs if you're still
> going to issue read requests from them just like you shouldn't drop tables
> your app is still expected to access. I hope that makes sense. Cheers!
>

Re: Consequences of dropping Materialized views

[Erick Ramirez]

>
> We are on cassandra 3.11 , we are using G1GC and using 16GB of heap.
>

Which exact version of C* is it again?

> WARN [MessagingService-Incoming-/10.X.X.X] 2020-02-18 14:21:47,115
> IncomingTcpConnection.java:103 - UnknownColumnFamilyException reading from
> socket; closing
>

This is expected because your app is trying to read the MV you just dropped.

ERROR [MutationStage-9] 2020-02-18 14:21:47,267 Keyspace.java:593 -
Attempting to mutate non-existant table
7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1
>
>
Any view update transactions which are already in-flight will fail. Again,
this is expected since the MV updates are not synchronous with the base
table updates.

WARN  [BatchlogTasks:1] 2020-02-18 14:21:53,786
BatchlogManager.java:252 - Skipped batch replay of
a19c6480-5294-11ea-9e09-3948c59ad0f5 due to {}
>
>
As above, this is expected since the MV updates which are already in the
queue will fail to apply because the MV got dropped.

WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,932
HintsReader.java:237 - Failed to read a hint for /10.X.X.X:
f75e58e8-c255-4318-a553-06487b6bbe8c - table with id
7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 is unknown in file
f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints
>
>
This is also expected. It won't be able to read the hint for a non-existent
MV.

So where to from here? The symptoms you described indicate that you haven't
updated your app *before* you dropped the MVs. That is key here -- your app
shouldn't be issuing reads for the MV since your intention is to drop it.
What I think happened is that the nodes were backed up with read requests
for the MV which no longer exists.

Can you confirm that you have changed your application so it is not
supposed to issue reads to the MV? You shouldn't drop MVs if you're still
going to issue read requests from them just like you shouldn't drop tables
your app is still expected to access. I hope that makes sense. Cheers!

Re: Consequences of dropping Materialized views

[Surbhi Gupta]

Thanks Jonathan, it still helps...

Anyone knows the solution?

On Tue, 18 Feb 2020 at 18:08, Jonathan Koppenhofer 
wrote:

> Forensics are gone at this point, so I can't verify exact errors, but
> wanted to mention we had seen something similar to corroborate your
> experience and warn others.
>
> The version would have been 3.0.15 or 3.11.3 as that is what we were
> deploying on our clusters at the time. I think it was more likely 3.0.15.
>
> So sorry for the "vagueness" :(
>
> On Tue, Feb 18, 2020, 8:54 PM Surbhi Gupta 
> wrote:
>
>> Jonathan,
>> As per https://issues.apache.org/jira/browse/CASSANDRA-13696 the issue,
>> Digest mismatch Exception if hints file has UnknownColumnFamily,  is fixed
>> for 3.0.15 , did you still faced this issue on 3.0.15 ?
>>
>> Thanks
>> Surbhi
>>
>> On Tue, 18 Feb 2020 at 17:40, Jonathan Koppenhofer 
>> wrote:
>>
>>> I believe we had something similar happen on 3.0.15 a while back. We had
>>> a cluster that created mass havoc by creating MVs on a large existing
>>> dataset. We thought we had stabilized the cluster, but saw similar issues
>>> as you when we dropped the MVs.
>>>
>>> We interpreted our errors to mean that we should not attempt to write to
>>> base tables while also dropping downstream materialized views. We
>>> essentially had the app stop their app, then drop the views 1 by 1 with
>>> some pause between. That then seemed to work fine, but yes, be careful with
>>> everything MVs.
>>>
>>> We now disallow the use of MVs globally.
>>>
>>> On Tue, Feb 18, 2020, 8:27 PM Surbhi Gupta 
>>> wrote:
>>>
 We are on cassandra 3.11 , we are using G1GC and using 16GB of heap.

 So we had to drop 7 MVs in production, as soon as we dropped the first
 Materialized View, our cluster became unstable and app started giving 100%
 error, what we noticed:
 1. As soon as MV was dropped , cluster became unstable and nodes were
 showing down from each other.
 2. We saw below warnings in system.log which is understood,
  WARN [MessagingService-Incoming-/10.X.X.X] 2020-02-18 14:21:47,115
 IncomingTcpConnection.java:103 - UnknownColumnFamilyException reading from
 socket; closing

 org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table 
 for cfId 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1. If a table was just 
 created, this is likely due to the schema not being fully propagated.  
 Please wait for schema agreement on table creation.

 3. We noticed below errors as well:

 ERROR [MutationStage-9] 2020-02-18 14:21:47,267 Keyspace.java:593 - 
 Attempting to mutate non-existant table 
 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1

 4. We noticed messages like below:

 WARN  [BatchlogTasks:1] 2020-02-18 14:21:53,786 BatchlogManager.java:252 - 
 Skipped batch replay of a19c6480-5294-11ea-9e09-3948c59ad0f5 due to {}

 5. Hints file corrupted:

 WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,932 HintsReader.java:237 
 - Failed to read a hint for /10.X.X.X: 
 f75e58e8-c255-4318-a553-06487b6bbe8c - table with id 
 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 is unknown in file 
 f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints
 ERROR [HintsDispatcher:6737] 2020-02-18 14:22:24,933 
 HintsDispatchExecutor.java:243 - Failed to dispatch hints file 
 f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints: file is 
 corrupted ({})
 org.apache.cassandra.io.FSReadError: java.io.IOException: Digest mismatch 
 exception


 5. And then Cassandra shut down:

 ERROR [HintsDispatcher:6737] 2020-02-18 14:22:24,937 
 StorageService.java:430 - Stopping gossiper
 WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,937 
 StorageService.java:321 - Stopping gossip by operator request
 INFO  [HintsDispatcher:6737] 2020-02-18 14:22:24,937 Gossiper.java:1530 - 
 Announcing shutdown


 Any views? Below are the issues I

 https://support.datastax.com/hc/en-us/articles/36368126-Hints-file-with-unknown-CFID-can-cause-nodes-to-fail

 https://issues.apache.org/jira/browse/CASSANDRA-13696

 https://issues.apache.org/jira/browse/CASSANDRA-6822

 https://support.datastax.com/hc/en-us/articles/36368126-Hints-file-with-unknown-CFID-can-cause-nodes-to-fail





 On Wed, 12 Feb 2020 at 19:10, Surbhi Gupta 
 wrote:

> Thanks Eric ...
> This is helpful...
>
>
> On Wed, 12 Feb 2020 at 17:46, Erick Ramirez <
> erick.rami...@datastax.com> wrote:
>
>> There shouldn't be any negative impact from dropping MVs and there's
>> certainly no risk to the base table if that is your concern. All it will 
>> do
>> is remove all the data in the respective views plus drop any pending view
>> mutations from the batch log. If anything, you should see some 
>> performance
>> gain since updates to the base 

Re: Consequences of dropping Materialized views

[Jonathan Koppenhofer]

Forensics are gone at this point, so I can't verify exact errors, but
wanted to mention we had seen something similar to corroborate your
experience and warn others.

The version would have been 3.0.15 or 3.11.3 as that is what we were
deploying on our clusters at the time. I think it was more likely 3.0.15.

So sorry for the "vagueness" :(

On Tue, Feb 18, 2020, 8:54 PM Surbhi Gupta  wrote:

> Jonathan,
> As per https://issues.apache.org/jira/browse/CASSANDRA-13696 the issue,
> Digest mismatch Exception if hints file has UnknownColumnFamily,  is fixed
> for 3.0.15 , did you still faced this issue on 3.0.15 ?
>
> Thanks
> Surbhi
>
> On Tue, 18 Feb 2020 at 17:40, Jonathan Koppenhofer 
> wrote:
>
>> I believe we had something similar happen on 3.0.15 a while back. We had
>> a cluster that created mass havoc by creating MVs on a large existing
>> dataset. We thought we had stabilized the cluster, but saw similar issues
>> as you when we dropped the MVs.
>>
>> We interpreted our errors to mean that we should not attempt to write to
>> base tables while also dropping downstream materialized views. We
>> essentially had the app stop their app, then drop the views 1 by 1 with
>> some pause between. That then seemed to work fine, but yes, be careful with
>> everything MVs.
>>
>> We now disallow the use of MVs globally.
>>
>> On Tue, Feb 18, 2020, 8:27 PM Surbhi Gupta 
>> wrote:
>>
>>> We are on cassandra 3.11 , we are using G1GC and using 16GB of heap.
>>>
>>> So we had to drop 7 MVs in production, as soon as we dropped the first
>>> Materialized View, our cluster became unstable and app started giving 100%
>>> error, what we noticed:
>>> 1. As soon as MV was dropped , cluster became unstable and nodes were
>>> showing down from each other.
>>> 2. We saw below warnings in system.log which is understood,
>>>  WARN [MessagingService-Incoming-/10.X.X.X] 2020-02-18 14:21:47,115
>>> IncomingTcpConnection.java:103 - UnknownColumnFamilyException reading from
>>> socket; closing
>>>
>>> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table 
>>> for cfId 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1. If a table was just created, 
>>> this is likely due to the schema not being fully propagated.  Please wait 
>>> for schema agreement on table creation.
>>>
>>> 3. We noticed below errors as well:
>>>
>>> ERROR [MutationStage-9] 2020-02-18 14:21:47,267 Keyspace.java:593 - 
>>> Attempting to mutate non-existant table 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1
>>>
>>> 4. We noticed messages like below:
>>>
>>> WARN  [BatchlogTasks:1] 2020-02-18 14:21:53,786 BatchlogManager.java:252 - 
>>> Skipped batch replay of a19c6480-5294-11ea-9e09-3948c59ad0f5 due to {}
>>>
>>> 5. Hints file corrupted:
>>>
>>> WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,932 HintsReader.java:237 - 
>>> Failed to read a hint for /10.X.X.X: f75e58e8-c255-4318-a553-06487b6bbe8c - 
>>> table with id 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 is unknown in file 
>>> f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints
>>> ERROR [HintsDispatcher:6737] 2020-02-18 14:22:24,933 
>>> HintsDispatchExecutor.java:243 - Failed to dispatch hints file 
>>> f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints: file is 
>>> corrupted ({})
>>> org.apache.cassandra.io.FSReadError: java.io.IOException: Digest mismatch 
>>> exception
>>>
>>>
>>> 5. And then Cassandra shut down:
>>>
>>> ERROR [HintsDispatcher:6737] 2020-02-18 14:22:24,937 
>>> StorageService.java:430 - Stopping gossiper
>>> WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,937 
>>> StorageService.java:321 - Stopping gossip by operator request
>>> INFO  [HintsDispatcher:6737] 2020-02-18 14:22:24,937 Gossiper.java:1530 - 
>>> Announcing shutdown
>>>
>>>
>>> Any views? Below are the issues I
>>>
>>> https://support.datastax.com/hc/en-us/articles/36368126-Hints-file-with-unknown-CFID-can-cause-nodes-to-fail
>>>
>>> https://issues.apache.org/jira/browse/CASSANDRA-13696
>>>
>>> https://issues.apache.org/jira/browse/CASSANDRA-6822
>>>
>>> https://support.datastax.com/hc/en-us/articles/36368126-Hints-file-with-unknown-CFID-can-cause-nodes-to-fail
>>>
>>>
>>>
>>>
>>>
>>> On Wed, 12 Feb 2020 at 19:10, Surbhi Gupta 
>>> wrote:
>>>
 Thanks Eric ...
 This is helpful...


 On Wed, 12 Feb 2020 at 17:46, Erick Ramirez 
 wrote:

> There shouldn't be any negative impact from dropping MVs and there's
> certainly no risk to the base table if that is your concern. All it will 
> do
> is remove all the data in the respective views plus drop any pending view
> mutations from the batch log. If anything, you should see some performance
> gain since updates to the base table will only trigger 4 view updates
> instead of the previous 11. Cheers!
>
> Erick Ramirez  |  Developer Relations
>
> erick.rami...@datastax.com | datastax.com 
> 
> 

Re: Consequences of dropping Materialized views

[Surbhi Gupta]

Jonathan,
As per https://issues.apache.org/jira/browse/CASSANDRA-13696 the issue,
Digest mismatch Exception if hints file has UnknownColumnFamily,  is fixed
for 3.0.15 , did you still faced this issue on 3.0.15 ?

Thanks
Surbhi

On Tue, 18 Feb 2020 at 17:40, Jonathan Koppenhofer 
wrote:

> I believe we had something similar happen on 3.0.15 a while back. We had a
> cluster that created mass havoc by creating MVs on a large existing
> dataset. We thought we had stabilized the cluster, but saw similar issues
> as you when we dropped the MVs.
>
> We interpreted our errors to mean that we should not attempt to write to
> base tables while also dropping downstream materialized views. We
> essentially had the app stop their app, then drop the views 1 by 1 with
> some pause between. That then seemed to work fine, but yes, be careful with
> everything MVs.
>
> We now disallow the use of MVs globally.
>
> On Tue, Feb 18, 2020, 8:27 PM Surbhi Gupta 
> wrote:
>
>> We are on cassandra 3.11 , we are using G1GC and using 16GB of heap.
>>
>> So we had to drop 7 MVs in production, as soon as we dropped the first
>> Materialized View, our cluster became unstable and app started giving 100%
>> error, what we noticed:
>> 1. As soon as MV was dropped , cluster became unstable and nodes were
>> showing down from each other.
>> 2. We saw below warnings in system.log which is understood,
>>  WARN [MessagingService-Incoming-/10.X.X.X] 2020-02-18 14:21:47,115
>> IncomingTcpConnection.java:103 - UnknownColumnFamilyException reading from
>> socket; closing
>>
>> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table 
>> for cfId 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1. If a table was just created, 
>> this is likely due to the schema not being fully propagated.  Please wait 
>> for schema agreement on table creation.
>>
>> 3. We noticed below errors as well:
>>
>> ERROR [MutationStage-9] 2020-02-18 14:21:47,267 Keyspace.java:593 - 
>> Attempting to mutate non-existant table 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1
>>
>> 4. We noticed messages like below:
>>
>> WARN  [BatchlogTasks:1] 2020-02-18 14:21:53,786 BatchlogManager.java:252 - 
>> Skipped batch replay of a19c6480-5294-11ea-9e09-3948c59ad0f5 due to {}
>>
>> 5. Hints file corrupted:
>>
>> WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,932 HintsReader.java:237 - 
>> Failed to read a hint for /10.X.X.X: f75e58e8-c255-4318-a553-06487b6bbe8c - 
>> table with id 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 is unknown in file 
>> f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints
>> ERROR [HintsDispatcher:6737] 2020-02-18 14:22:24,933 
>> HintsDispatchExecutor.java:243 - Failed to dispatch hints file 
>> f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints: file is 
>> corrupted ({})
>> org.apache.cassandra.io.FSReadError: java.io.IOException: Digest mismatch 
>> exception
>>
>>
>> 5. And then Cassandra shut down:
>>
>> ERROR [HintsDispatcher:6737] 2020-02-18 14:22:24,937 StorageService.java:430 
>> - Stopping gossiper
>> WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,937 StorageService.java:321 
>> - Stopping gossip by operator request
>> INFO  [HintsDispatcher:6737] 2020-02-18 14:22:24,937 Gossiper.java:1530 - 
>> Announcing shutdown
>>
>>
>> Any views? Below are the issues I
>>
>> https://support.datastax.com/hc/en-us/articles/36368126-Hints-file-with-unknown-CFID-can-cause-nodes-to-fail
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-13696
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-6822
>>
>> https://support.datastax.com/hc/en-us/articles/36368126-Hints-file-with-unknown-CFID-can-cause-nodes-to-fail
>>
>>
>>
>>
>>
>> On Wed, 12 Feb 2020 at 19:10, Surbhi Gupta 
>> wrote:
>>
>>> Thanks Eric ...
>>> This is helpful...
>>>
>>>
>>> On Wed, 12 Feb 2020 at 17:46, Erick Ramirez 
>>> wrote:
>>>
 There shouldn't be any negative impact from dropping MVs and there's
 certainly no risk to the base table if that is your concern. All it will do
 is remove all the data in the respective views plus drop any pending view
 mutations from the batch log. If anything, you should see some performance
 gain since updates to the base table will only trigger 4 view updates
 instead of the previous 11. Cheers!

 Erick Ramirez  |  Developer Relations

 erick.rami...@datastax.com | datastax.com 
 
  
  

 



 On Thu, 13 Feb 2020 at 04:26, Surbhi Gupta 
 wrote:

> Hi,
>
> So application team created 11 materialized views on a base table in
> production and we need to drop 7 Materialized views as they are not in 
> use.
> Wanted to understand the impact of dropping the materialized views.
> We are on Cassandra 3.11.1 , 

Re: Consequences of dropping Materialized views

[Jonathan Koppenhofer]

I believe we had something similar happen on 3.0.15 a while back. We had a
cluster that created mass havoc by creating MVs on a large existing
dataset. We thought we had stabilized the cluster, but saw similar issues
as you when we dropped the MVs.

We interpreted our errors to mean that we should not attempt to write to
base tables while also dropping downstream materialized views. We
essentially had the app stop their app, then drop the views 1 by 1 with
some pause between. That then seemed to work fine, but yes, be careful with
everything MVs.

We now disallow the use of MVs globally.

On Tue, Feb 18, 2020, 8:27 PM Surbhi Gupta  wrote:

> We are on cassandra 3.11 , we are using G1GC and using 16GB of heap.
>
> So we had to drop 7 MVs in production, as soon as we dropped the first
> Materialized View, our cluster became unstable and app started giving 100%
> error, what we noticed:
> 1. As soon as MV was dropped , cluster became unstable and nodes were
> showing down from each other.
> 2. We saw below warnings in system.log which is understood,
>  WARN [MessagingService-Incoming-/10.X.X.X] 2020-02-18 14:21:47,115
> IncomingTcpConnection.java:103 - UnknownColumnFamilyException reading from
> socket; closing
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for 
> cfId 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1. If a table was just created, this 
> is likely due to the schema not being fully propagated.  Please wait for 
> schema agreement on table creation.
>
> 3. We noticed below errors as well:
>
> ERROR [MutationStage-9] 2020-02-18 14:21:47,267 Keyspace.java:593 - 
> Attempting to mutate non-existant table 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1
>
> 4. We noticed messages like below:
>
> WARN  [BatchlogTasks:1] 2020-02-18 14:21:53,786 BatchlogManager.java:252 - 
> Skipped batch replay of a19c6480-5294-11ea-9e09-3948c59ad0f5 due to {}
>
> 5. Hints file corrupted:
>
> WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,932 HintsReader.java:237 - 
> Failed to read a hint for /10.X.X.X: f75e58e8-c255-4318-a553-06487b6bbe8c - 
> table with id 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 is unknown in file 
> f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints
> ERROR [HintsDispatcher:6737] 2020-02-18 14:22:24,933 
> HintsDispatchExecutor.java:243 - Failed to dispatch hints file 
> f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints: file is corrupted 
> ({})
> org.apache.cassandra.io.FSReadError: java.io.IOException: Digest mismatch 
> exception
>
>
> 5. And then Cassandra shut down:
>
> ERROR [HintsDispatcher:6737] 2020-02-18 14:22:24,937 StorageService.java:430 
> - Stopping gossiper
> WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,937 StorageService.java:321 
> - Stopping gossip by operator request
> INFO  [HintsDispatcher:6737] 2020-02-18 14:22:24,937 Gossiper.java:1530 - 
> Announcing shutdown
>
>
> Any views? Below are the issues I
>
> https://support.datastax.com/hc/en-us/articles/36368126-Hints-file-with-unknown-CFID-can-cause-nodes-to-fail
>
> https://issues.apache.org/jira/browse/CASSANDRA-13696
>
> https://issues.apache.org/jira/browse/CASSANDRA-6822
>
> https://support.datastax.com/hc/en-us/articles/36368126-Hints-file-with-unknown-CFID-can-cause-nodes-to-fail
>
>
>
>
>
> On Wed, 12 Feb 2020 at 19:10, Surbhi Gupta 
> wrote:
>
>> Thanks Eric ...
>> This is helpful...
>>
>>
>> On Wed, 12 Feb 2020 at 17:46, Erick Ramirez 
>> wrote:
>>
>>> There shouldn't be any negative impact from dropping MVs and there's
>>> certainly no risk to the base table if that is your concern. All it will do
>>> is remove all the data in the respective views plus drop any pending view
>>> mutations from the batch log. If anything, you should see some performance
>>> gain since updates to the base table will only trigger 4 view updates
>>> instead of the previous 11. Cheers!
>>>
>>> Erick Ramirez  |  Developer Relations
>>>
>>> erick.rami...@datastax.com | datastax.com 
>>> 
>>>  
>>>  
>>>
>>> 
>>>
>>>
>>>
>>> On Thu, 13 Feb 2020 at 04:26, Surbhi Gupta 
>>> wrote:
>>>
 Hi,

 So application team created 11 materialized views on a base table in
 production and we need to drop 7 Materialized views as they are not in use.
 Wanted to understand the impact of dropping the materialized views.
 We are on Cassandra 3.11.1 , multi datacenter with replication factor
 of 3 in each datacenter.
 We are using LOCAL_QUORUM for write consistency and LOCAL_ONE for read
 consistency.

 Any thoughts or suggestion to keep in mind before dropping the
 Materialized views.

 Thanks
 Surbhi

Re: Consequences of dropping Materialized views

[Surbhi Gupta]

We are on cassandra 3.11 , we are using G1GC and using 16GB of heap.

So we had to drop 7 MVs in production, as soon as we dropped the first
Materialized View, our cluster became unstable and app started giving 100%
error, what we noticed:
1. As soon as MV was dropped , cluster became unstable and nodes were
showing down from each other.
2. We saw below warnings in system.log which is understood,
 WARN [MessagingService-Incoming-/10.X.X.X] 2020-02-18 14:21:47,115
IncomingTcpConnection.java:103 - UnknownColumnFamilyException reading from
socket; closing

org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
table for cfId 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1. If a table was
just created, this is likely due to the schema not being fully
propagated.  Please wait for schema agreement on table creation.

3. We noticed below errors as well:

ERROR [MutationStage-9] 2020-02-18 14:21:47,267 Keyspace.java:593 -
Attempting to mutate non-existant table
7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1

4. We noticed messages like below:

WARN  [BatchlogTasks:1] 2020-02-18 14:21:53,786
BatchlogManager.java:252 - Skipped batch replay of
a19c6480-5294-11ea-9e09-3948c59ad0f5 due to {}

5. Hints file corrupted:

WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,932
HintsReader.java:237 - Failed to read a hint for /10.X.X.X:
f75e58e8-c255-4318-a553-06487b6bbe8c - table with id
7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 is unknown in file
f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints
ERROR [HintsDispatcher:6737] 2020-02-18 14:22:24,933
HintsDispatchExecutor.java:243 - Failed to dispatch hints file
f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints: file is
corrupted ({})
org.apache.cassandra.io.FSReadError: java.io.IOException: Digest
mismatch exception


5. And then Cassandra shut down:

ERROR [HintsDispatcher:6737] 2020-02-18 14:22:24,937
StorageService.java:430 - Stopping gossiper
WARN  [HintsDispatcher:6737] 2020-02-18 14:22:24,937
StorageService.java:321 - Stopping gossip by operator request
INFO  [HintsDispatcher:6737] 2020-02-18 14:22:24,937
Gossiper.java:1530 - Announcing shutdown


Any views? Below are the issues I

https://support.datastax.com/hc/en-us/articles/36368126-Hints-file-with-unknown-CFID-can-cause-nodes-to-fail

https://issues.apache.org/jira/browse/CASSANDRA-13696

https://issues.apache.org/jira/browse/CASSANDRA-6822

https://support.datastax.com/hc/en-us/articles/36368126-Hints-file-with-unknown-CFID-can-cause-nodes-to-fail





On Wed, 12 Feb 2020 at 19:10, Surbhi Gupta  wrote:

> Thanks Eric ...
> This is helpful...
>
>
> On Wed, 12 Feb 2020 at 17:46, Erick Ramirez 
> wrote:
>
>> There shouldn't be any negative impact from dropping MVs and there's
>> certainly no risk to the base table if that is your concern. All it will do
>> is remove all the data in the respective views plus drop any pending view
>> mutations from the batch log. If anything, you should see some performance
>> gain since updates to the base table will only trigger 4 view updates
>> instead of the previous 11. Cheers!
>>
>> Erick Ramirez  |  Developer Relations
>>
>> erick.rami...@datastax.com | datastax.com 
>> 
>>  
>>  
>>
>> 
>>
>>
>>
>> On Thu, 13 Feb 2020 at 04:26, Surbhi Gupta 
>> wrote:
>>
>>> Hi,
>>>
>>> So application team created 11 materialized views on a base table in
>>> production and we need to drop 7 Materialized views as they are not in use.
>>> Wanted to understand the impact of dropping the materialized views.
>>> We are on Cassandra 3.11.1 , multi datacenter with replication factor of
>>> 3 in each datacenter.
>>> We are using LOCAL_QUORUM for write consistency and LOCAL_ONE for read
>>> consistency.
>>>
>>> Any thoughts or suggestion to keep in mind before dropping the
>>> Materialized views.
>>>
>>> Thanks
>>> Surbhi
>>>
>>>
>>>
>>>

Re: Consequences of dropping Materialized views

[Surbhi Gupta]

Thanks Eric ...
This is helpful...


On Wed, 12 Feb 2020 at 17:46, Erick Ramirez 
wrote:

> There shouldn't be any negative impact from dropping MVs and there's
> certainly no risk to the base table if that is your concern. All it will do
> is remove all the data in the respective views plus drop any pending view
> mutations from the batch log. If anything, you should see some performance
> gain since updates to the base table will only trigger 4 view updates
> instead of the previous 11. Cheers!
>
> Erick Ramirez  |  Developer Relations
>
> erick.rami...@datastax.com | datastax.com 
> 
>  
>  
>
> 
>
>
>
> On Thu, 13 Feb 2020 at 04:26, Surbhi Gupta 
> wrote:
>
>> Hi,
>>
>> So application team created 11 materialized views on a base table in
>> production and we need to drop 7 Materialized views as they are not in use.
>> Wanted to understand the impact of dropping the materialized views.
>> We are on Cassandra 3.11.1 , multi datacenter with replication factor of
>> 3 in each datacenter.
>> We are using LOCAL_QUORUM for write consistency and LOCAL_ONE for read
>> consistency.
>>
>> Any thoughts or suggestion to keep in mind before dropping the
>> Materialized views.
>>
>> Thanks
>> Surbhi
>>
>>
>>
>>

Re: Consequences of dropping Materialized views

[Erick Ramirez]

There shouldn't be any negative impact from dropping MVs and there's
certainly no risk to the base table if that is your concern. All it will do
is remove all the data in the respective views plus drop any pending view
mutations from the batch log. If anything, you should see some performance
gain since updates to the base table will only trigger 4 view updates
instead of the previous 11. Cheers!

Erick Ramirez  |  Developer Relations

erick.rami...@datastax.com | datastax.com 

 
 





On Thu, 13 Feb 2020 at 04:26, Surbhi Gupta  wrote:

> Hi,
>
> So application team created 11 materialized views on a base table in
> production and we need to drop 7 Materialized views as they are not in use.
> Wanted to understand the impact of dropping the materialized views.
> We are on Cassandra 3.11.1 , multi datacenter with replication factor of 3
> in each datacenter.
> We are using LOCAL_QUORUM for write consistency and LOCAL_ONE for read
> consistency.
>
> Any thoughts or suggestion to keep in mind before dropping the
> Materialized views.
>
> Thanks
> Surbhi
>
>
>
>