Re: Corrupt SSTable Cassandra 3.11.2

[manish khandelwal]

Thanks all for your support.

I executed the discussed process (barring repair, as table was read for
reporting only) and it worked fine in production.

Regards
Manish

>

Re: Corrupt SSTable Cassandra 3.11.2

[Jeff Jirsa]

The risk is you violate consistency while you run repair

Assume you have three replicas for that range, a b c

At some point b misses a write, but it’s committed on a and c for quorum 
Now c has a corrupt sstable

You empty c and bring it back with no data and start repair

Then the app reads at quorum and selects b and c

You don’t see the data when you do a quorum read - this is technically incorrect

You could:

Stop the host with corrupt sstable
Run repair on the token range impacted using just the surviving hosts (this 
makes sure the two survivors have all of the data)
Clear all the data for that table on the host for the corrupt sstable (I think 
you can leave the commitlog in place but you probably want to flush and drain 
before you stop the host)
Then bring that host up and run repair


I think that’s strictly safe and you’re just rebuilding 5-6gb


> On Feb 13, 2020, at 11:23 PM, manish khandelwal 
>  wrote:
> 
> 
> Thanks Jeff for your response.
> 
> Do you see any risk in following approach
> 
> 1. Stop the node.
> 2. Remove all sstable files from  
> /var/lib/cassandra/data/keyspace/tablename-23dfadf32adf33d33s333s33s3s33 
> directory.
> 3. Start the node.
> 4. Run full repair on this particular table
> 
> I wanted to go this way because this table is small (5-6 GB).  I would like 
> to avoid 2-3 days of streaming in case of replacing the whole host.
> 
> Regards
> Manish
> 
>> On Fri, Feb 14, 2020 at 12:28 PM Jeff Jirsa  wrote:
>> Agree this is both strictly possible and more common with LCS. The only 
>> thing that's strictly correct to do is treat every corrupt sstable exception 
>> as a failed host, and replace it just like you would a failed host.
>> 
>> 
>>> On Thu, Feb 13, 2020 at 10:55 PM manish khandelwal 
>>>  wrote:
>>> Thanks Erick
>>> 
>>> I would like to explain how data resurrection can take place with single 
>>> SSTable deletion.
>>> 
>>> Consider this case of table with Levelled Compaction Strategy
>>> 
>>> 1. Data A written a long time back.
>>> 2. Data A is deleted and tombstone is created.
>>> 3. After GC grace tombstone is purgeable.
>>> 4. Now the SSTable containing purgeable tombstone in one node is corruputed.
>>> 4. The node with corrupt SSTable cannot compact the data and purgeable 
>>> tombstone
>>> 6. From other two nodes Data A is removed after compaction.
>>> 7. Remove the corrupt SSTable from impacted node.
>>> 8. When you run repair Data A is copied to all the nodes.
>>> 
>>> This table in quesiton is using Levelled Compaction Strategy.
>>> 
>>> Regards
>>> Manish
>>> 
 On Fri, Feb 14, 2020 at 12:00 PM Erick Ramirez 
  wrote:
 The log shows that the the problem occurs when decompressing the SSTable 
 but there's not much actionable info from it.
 
> I would like to know what will be "ordinary hammer" in this  case. Do you 
> want to suggest that  deleting only corrupt sstable file ( in this case 
> mc-1234-big-*.db) would be suffice ?
 
 Exactly. I mean if it's just a one-off, why go through the trouble of 
 blowing away all the files? :)
 
> I am afraid that this may cause data resurrection (I have prior 
> experience with same). 
 
 Whoa! That's a long bow to draw. Sounds like there's more history to it.
 
> Note that i am not willing to run the entire node rebuild as it will take 
> lots of time due to presence of multiple big tables (I am keeping it as 
> my last option)
 
 
 I wasn't going to suggest that at all. I didn't like the sledge hammer 
 approach. I certainly wouldn't recommend bringing in a wrecking ball. 😁
 
 Cheers!

Re: Corrupt SSTable Cassandra 3.11.2

[manish khandelwal]

Thanks Jeff for your response.

Do you see any risk in following approach

1. Stop the node.
2. Remove all sstable files from
*/var/lib/cassandra/data/keyspace/tablename-23dfadf32adf33d33s333s33s3s33 *
directory.
3. Start the node.
4. Run full repair on this particular table

I wanted to go this way because this table is small (5-6 GB).  I would like
to avoid 2-3 days of streaming in case of replacing the whole host.

Regards
Manish

On Fri, Feb 14, 2020 at 12:28 PM Jeff Jirsa  wrote:

> Agree this is both strictly possible and more common with LCS. The only
> thing that's strictly correct to do is treat every corrupt sstable
> exception as a failed host, and replace it just like you would a failed
> host.
>
>
> On Thu, Feb 13, 2020 at 10:55 PM manish khandelwal <
> manishkhandelwa...@gmail.com> wrote:
>
>> Thanks Erick
>>
>> I would like to explain how data resurrection can take place with single
>> SSTable deletion.
>>
>> Consider this case of table with Levelled Compaction Strategy
>>
>> 1. Data A written a long time back.
>> 2. Data A is deleted and tombstone is created.
>> 3. After GC grace tombstone is purgeable.
>> 4. Now the SSTable containing purgeable tombstone in one node is
>> corruputed.
>> 4. The node with corrupt SSTable cannot compact the data and purgeable
>> tombstone
>> 6. From other two nodes Data A is removed after compaction.
>> 7. Remove the corrupt SSTable from impacted node.
>> 8. When you run repair Data A is copied to all the nodes.
>>
>> This table in quesiton is using Levelled Compaction Strategy.
>>
>> Regards
>> Manish
>>
>> On Fri, Feb 14, 2020 at 12:00 PM Erick Ramirez <
>> erick.rami...@datastax.com> wrote:
>>
>>> The log shows that the the problem occurs when decompressing the SSTable
>>> but there's not much actionable info from it.
>>>
>>> I would like to know what will be "ordinary hammer" in this  case. Do
 you want to suggest that  deleting only corrupt sstable file ( in this case
 mc-1234-big-*.db) would be suffice ?
>>>
>>>
>>> Exactly. I mean if it's just a one-off, why go through the trouble of
>>> blowing away all the files? :)
>>>
>>> I am afraid that this may cause data resurrection (I have prior
 experience with same).
>>>
>>>
>>> Whoa! That's a long bow to draw. Sounds like there's more history to it.
>>>
>>> Note that i am not willing to run the entire node rebuild as it will
 take lots of time due to presence of multiple big tables (I am keeping it
 as my last option)
>>>
>>>
>>> I wasn't going to suggest that at all. I didn't like the sledge hammer
>>> approach. I certainly wouldn't recommend bringing in a wrecking ball. 😁
>>>
>>> Cheers!
>>>
>>

Re: Corrupt SSTable Cassandra 3.11.2

[Jeff Jirsa]

Agree this is both strictly possible and more common with LCS. The only
thing that's strictly correct to do is treat every corrupt sstable
exception as a failed host, and replace it just like you would a failed
host.


On Thu, Feb 13, 2020 at 10:55 PM manish khandelwal <
manishkhandelwa...@gmail.com> wrote:

> Thanks Erick
>
> I would like to explain how data resurrection can take place with single
> SSTable deletion.
>
> Consider this case of table with Levelled Compaction Strategy
>
> 1. Data A written a long time back.
> 2. Data A is deleted and tombstone is created.
> 3. After GC grace tombstone is purgeable.
> 4. Now the SSTable containing purgeable tombstone in one node is
> corruputed.
> 4. The node with corrupt SSTable cannot compact the data and purgeable
> tombstone
> 6. From other two nodes Data A is removed after compaction.
> 7. Remove the corrupt SSTable from impacted node.
> 8. When you run repair Data A is copied to all the nodes.
>
> This table in quesiton is using Levelled Compaction Strategy.
>
> Regards
> Manish
>
> On Fri, Feb 14, 2020 at 12:00 PM Erick Ramirez 
> wrote:
>
>> The log shows that the the problem occurs when decompressing the SSTable
>> but there's not much actionable info from it.
>>
>> I would like to know what will be "ordinary hammer" in this  case. Do you
>>> want to suggest that  deleting only corrupt sstable file ( in this case
>>> mc-1234-big-*.db) would be suffice ?
>>
>>
>> Exactly. I mean if it's just a one-off, why go through the trouble of
>> blowing away all the files? :)
>>
>> I am afraid that this may cause data resurrection (I have prior
>>> experience with same).
>>
>>
>> Whoa! That's a long bow to draw. Sounds like there's more history to it.
>>
>> Note that i am not willing to run the entire node rebuild as it will take
>>> lots of time due to presence of multiple big tables (I am keeping it as my
>>> last option)
>>
>>
>> I wasn't going to suggest that at all. I didn't like the sledge hammer
>> approach. I certainly wouldn't recommend bringing in a wrecking ball. 😁
>>
>> Cheers!
>>
>

Re: Corrupt SSTable Cassandra 3.11.2

[manish khandelwal]

Thanks Erick

I would like to explain how data resurrection can take place with single
SSTable deletion.

Consider this case of table with Levelled Compaction Strategy

1. Data A written a long time back.
2. Data A is deleted and tombstone is created.
3. After GC grace tombstone is purgeable.
4. Now the SSTable containing purgeable tombstone in one node is corruputed.
4. The node with corrupt SSTable cannot compact the data and purgeable
tombstone
6. From other two nodes Data A is removed after compaction.
7. Remove the corrupt SSTable from impacted node.
8. When you run repair Data A is copied to all the nodes.

This table in quesiton is using Levelled Compaction Strategy.

Regards
Manish

On Fri, Feb 14, 2020 at 12:00 PM Erick Ramirez 
wrote:

> The log shows that the the problem occurs when decompressing the SSTable
> but there's not much actionable info from it.
>
> I would like to know what will be "ordinary hammer" in this  case. Do you
>> want to suggest that  deleting only corrupt sstable file ( in this case
>> mc-1234-big-*.db) would be suffice ?
>
>
> Exactly. I mean if it's just a one-off, why go through the trouble of
> blowing away all the files? :)
>
> I am afraid that this may cause data resurrection (I have prior experience
>> with same).
>
>
> Whoa! That's a long bow to draw. Sounds like there's more history to it.
>
> Note that i am not willing to run the entire node rebuild as it will take
>> lots of time due to presence of multiple big tables (I am keeping it as my
>> last option)
>
>
> I wasn't going to suggest that at all. I didn't like the sledge hammer
> approach. I certainly wouldn't recommend bringing in a wrecking ball. 😁
>
> Cheers!
>

Re: Corrupt SSTable Cassandra 3.11.2

[Erick Ramirez]

The log shows that the the problem occurs when decompressing the SSTable
but there's not much actionable info from it.

I would like to know what will be "ordinary hammer" in this  case. Do you
> want to suggest that  deleting only corrupt sstable file ( in this case
> mc-1234-big-*.db) would be suffice ?


Exactly. I mean if it's just a one-off, why go through the trouble of
blowing away all the files? :)

I am afraid that this may cause data resurrection (I have prior experience
> with same).


Whoa! That's a long bow to draw. Sounds like there's more history to it.

Note that i am not willing to run the entire node rebuild as it will take
> lots of time due to presence of multiple big tables (I am keeping it as my
> last option)


I wasn't going to suggest that at all. I didn't like the sledge hammer
approach. I certainly wouldn't recommend bringing in a wrecking ball. 😁

Cheers!

Re: Corrupt SSTable Cassandra 3.11.2

[manish khandelwal]

Hi Erick

Thanks for your quick response. I have attached the full stacktrace which
show exception during validation phase of table repair.

I would like to know what will be "ordinary hammer" in this  case. Do you
want to suggest that  deleting only corrupt sstable file ( in this case
*mc-1234-big-*.db*) would be suffice ? I am afraid that this may cause data
resurrection (I have prior experience with same).
Or you are pointing towards running scrub ? Kindly explain.

Note that i am not willing to run the entire node rebuild as it will take
lots of time due to presence of multiple big tables (I am keeping it as my
last option)

Regards
Manish

On Fri, Feb 14, 2020 at 11:11 AM Erick Ramirez 
wrote:

> It will achieve the outcome you are after but I doubt anyone would
> recommend that approach. It's like using a sledgehammer when an ordinary
> hammer would suffice. And if you were hitting some bug then you'd run into
> the same problem anyway.
>
> Can you post the full stack trace? It might provide us some clues as to
> why you ran into the problem. Cheers!
>


error.log
Description: Binary data

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Corrupt SSTable Cassandra 3.11.2

[Erick Ramirez]

It will achieve the outcome you are after but I doubt anyone would
recommend that approach. It's like using a sledgehammer when an ordinary
hammer would suffice. And if you were hitting some bug then you'd run into
the same problem anyway.

Can you post the full stack trace? It might provide us some clues as to why
you ran into the problem. Cheers!

Re: Corrupt SSTable Cassandra 3.11.2

[manish khandelwal]

Hi Eric

Thanks for reply.

Reason for corruption is unknown to me. I just found the corrupt table when
scheduled repair failed with logs showing






*ERROR [ValidationExecutor:16] 2020-01-21 19:13:18,123
CassandraDaemon.java:228 - Exception in thread
Thread[ValidationExecutor:16,1,main]org.apache.cassandra.io.sstable.CorruptSSTableException:
Corrupted:
/var/lib/cassandra/data/keyspace/tablename-23dfadf32adf33d33s333s33s3s33/mc-1234-big-Data.db
 at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:134)
~[apache-cassandra-3.11.2.jar:3.11.2]   at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
~[apache-cassandra-3.11.2.jar:3.11.2]   at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
~[apache-cassandra-3.11.2.jar:3.11.2]   at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.11.2.jar:3.11.2]*


Regarding you question about removing all SSTable files of a table(column
family). I want quick recovery without any inconsistency. Since I have 3
node cluster with RF=3, my expectation is that repair would stream the data
from other two nodes. I just wanted to know is it correct to do this way

1. Stop the node.
2. Remove all sstable files from
*/var/lib/cassandra/data/keyspace/tablename-23dfadf32adf33d33s333s33s3s33
*directory.
3. Start the node.
4. Run full repair on this particular table

Regards
Manish







On Fri, Feb 14, 2020 at 4:44 AM Erick Ramirez 
wrote:

> You need to stop C* in order to run the offline sstable scrub utility.
> That's why it's referred to as "offline". :)
>
> Do you have any idea on what caused the corruption? It's highly unusual
> that you're thinking of removing all the files for just one table.
> Typically if the corruption was a result of a faulty disk or hardware
> failure, it wouldn't be isolated to just one table. If you provide a bit
> more background information, we would be able to give you a better
> response. Cheers!
>
> Erick Ramirez  |  Developer Relations
>
> erick.rami...@datastax.com | datastax.com 
> 
>  
>  
>
> 
>
>
>
> On Fri, 14 Feb 2020 at 04:39, manish khandelwal <
> manishkhandelwa...@gmail.com> wrote:
>
>> Hi
>>
>> I see a corrupt SSTable in one of my keyspace table on one node. Cluster
>> is 3 nodes with replication 3. Cassandra version is 3.11.2.
>> I am thinking on following lines to resolve the corrupt SSTable issue.
>> 1. Run nodetool scrub.
>> 2. If step 1 fails, run offline sstabablescrub.
>> 3. If step 2 fails, stop node. Remove all SSTables from problematic
>> table.Start the node and run full repair on table.I am removing all
>> SSTABLES of the particular table so as to avoid resurrection of data or any
>> data corruption.
>>
>> I would like to know are there any side effects of executing step 3 if
>> step 1 and step 2 fails.
>>
>> Regards
>> Manish
>>
>>
>>
>>
>>

Re: Corrupt SSTable Cassandra 3.11.2

[Erick Ramirez]

You need to stop C* in order to run the offline sstable scrub utility.
That's why it's referred to as "offline". :)

Do you have any idea on what caused the corruption? It's highly unusual
that you're thinking of removing all the files for just one table.
Typically if the corruption was a result of a faulty disk or hardware
failure, it wouldn't be isolated to just one table. If you provide a bit
more background information, we would be able to give you a better
response. Cheers!

Erick Ramirez  |  Developer Relations

erick.rami...@datastax.com | datastax.com 

 
 





On Fri, 14 Feb 2020 at 04:39, manish khandelwal <
manishkhandelwa...@gmail.com> wrote:

> Hi
>
> I see a corrupt SSTable in one of my keyspace table on one node. Cluster
> is 3 nodes with replication 3. Cassandra version is 3.11.2.
> I am thinking on following lines to resolve the corrupt SSTable issue.
> 1. Run nodetool scrub.
> 2. If step 1 fails, run offline sstabablescrub.
> 3. If step 2 fails, stop node. Remove all SSTables from problematic
> table.Start the node and run full repair on table.I am removing all
> SSTABLES of the particular table so as to avoid resurrection of data or any
> data corruption.
>
> I would like to know are there any side effects of executing step 3 if
> step 1 and step 2 fails.
>
> Regards
> Manish
>
>
>
>
>

Corrupt SSTable Cassandra 3.11.2

[manish khandelwal]

Hi

I see a corrupt SSTable in one of my keyspace table on one node. Cluster is
3 nodes with replication 3. Cassandra version is 3.11.2.
I am thinking on following lines to resolve the corrupt SSTable issue.
1. Run nodetool scrub.
2. If step 1 fails, run offline sstabablescrub.
3. If step 2 fails, stop node. Remove all SSTables from problematic
table.Start the node and run full repair on table.I am removing all
SSTABLES of the particular table so as to avoid resurrection of data or any
data corruption.

I would like to know are there any side effects of executing step 3 if step
1 and step 2 fails.

Regards
Manish