Jaydeepkumar Chovatia created CASSANDRA-17991:
-------------------------------------------------

             Summary: Possible data inconsistency during bootstrap/decommission
                 Key: CASSANDRA-17991
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-17991
             Project: Cassandra
          Issue Type: Bug
          Components: Consistency/Bootstrap and Decommission
            Reporter: Jaydeepkumar Chovatia


I am facing one corner case in which the deleted data resurrects. 

tl;dr: This could be because when we stream all the SSTables for a given token 
range to the new owner, then they are not sent atomically, so the new owner 
could do compaction on the partially received SSTables, which might remove the 
tombstones.
 
Here are the reproducible steps:

+*Prerequisite*+ # Three nodes Cassandra cluster n1, n2, and n3
 # 
{code:java}
CREATE KEYSPACE KS1 WITH replication = {'class': 'NetworkTopologyStrategy', 
'dc1': '3'};

CREATE TABLE KS1.T1 (
    key int,
    c1 int,
    c2 int,
    c3 int
    PRIMARY KEY (key)
) WITH CLUSTERING ORDER BY (key ASC)
 AND compaction = {'class': 
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 
'max_threshold': '32', 'min_threshold': '4'}
 AND gc_grace_seconds = 864000;
{code}

 

*Reproducible Steps*
 * Day1: Insert a new record
{code:java}
INSERT INTO KS1.T1 (key, c1, c2, c3) values (1, 10, 20, 30){code}

 * Day2: Other records are inserted into this table, and it goes through 
multiple compactions
 * Day3: Here is the data layout on SSTables on n1, n2, and n3 

{code:java}
SSTable1:
{
    "partition" : {
      "key" : [ "1" ],
      "position" : 900
    },
    "rows" : [
        "type" : "row",  
        "position" : 10,
        "liveness_info" : { "tstamp" : "2022-10-15T00:00:00.000001Z"},
        "cells" : [
            { "name" : "c1", "value" : 10 },
            { "name" : "c2", "value" : 20 },
            { "name" : "c3", "value" : 30 },
          ]
}
.....
SSTable2:
{
    "partition" : {
      "key" : [ "1" ],
      "position" : 900
    },
    "rows" : [
        "type" : "row",  
        "position" : 10,
        "liveness_info" : { "tstamp" : "2022-10-15T00:00:00.000001Z"},
        "cells" : [
            { "name" : "c1", "value" : 10 },
            { "name" : "c2", "value" : 20 },
            { "name" : "c3", "value" : 30 },
          ]
}
{code}
 * Day4: Delete the record
{code:java}
CONSISTENCY ALL; DELETE FROM KS1.T1 WHERE key = 1; {code}

 * Day5: Here is the data layout on SSTables on n1, n2, and n3 

{code:java}
SSTable1:
{
    "partition" : {
      "key" : [ "1" ],
      "position" : 900
    },
    "rows" : [
        "type" : "row",  
        "position" : 10,
        "liveness_info" : { "tstamp" : "2022-10-15T00:00:00.000001Z"},
        "cells" : [
            { "name" : "c1", "value" : 10 },
            { "name" : "c2", "value" : 20 },
            { "name" : "c3", "value" : 30 },
          ]
}
.....
SSTable2:
{
    "partition" : {
      "key" : [ "1" ],
      "position" : 900
    },
    "rows" : [
        "type" : "row",  
        "position" : 10,
        "liveness_info" : { "tstamp" : "2022-10-15T00:00:00.000001Z"},
        "cells" : [
            { "name" : "c1", "value" : 10 },
            { "name" : "c2", "value" : 20 },
            { "name" : "c3", "value" : 30 },
          ]
}
.....
SSTable3 (Tombstone):
{
    "partition" : {
      "key" : [ "1" ],
      "position" : 900
    },
    "rows" : [
        "type" : "row",  
        "position" : 10,
        "deletion_info" : { "marked_deleted" : "2022-10-19T00:00:00.000001Z", 
"local_delete_time" : "2022-10-19T00:00:00.000001Z" },
        "cells" : [ ]
}
{code}
 * Day20: Nothing happens for more than 10 days. Let's say the data layout on 
SSTables on n1, n2, and n3 is the same as Day5
 * Day20: A new node (n4) joins the ring, and it is going to be responsible for 
key "1". Let's say it streams the data from n3. The node _n3_ is supposed to 
stream out SSTable1, SSTable2, and SSTable3, but it does not happen atomically 
as per the streaming algorithm. Let's consider a scenario in that _n4_ receives 
SSTable1 and SSTable3, but not yet SSTable2, and _n4_ compacts SSTable1 and 
SSTable3. In this case, _n4_ would purge the key "1". So at this time, there 
are no traces of key "1" on {_}n4{_}. After some time, SSTable2 is streamed, 
and at this time it will stream the key "1" as well.
 * Day20: _n4_ becomes normal 
{code:java}
Query on n4:
$> CONSISTENCY LOCAL_ONE; SELECT * FROM KS1.T1 WHERE key = 1;
// 1 | 10 | 20 | 30 <-- A record is returned

Query on n1:
$> CONSISTENCY LOCAL_ONE; SELECT * FROM KS1.T1 WHERE key = 1;
// <empty> //no output{code}

 

Does this make sense?{*}{*}

*Possible Solution*
 * One of the solutions is to maybe not purge tombstones while there are token 
range movements in the ring



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

Reply via email to