[
https://issues.apache.org/jira/browse/CASSANDRA-13125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15831816#comment-15831816
]
Sylvain Lebresne commented on CASSANDRA-13125:
----------------------------------------------
Thanks a lot for the very detailed reproduction case and analysis. You are
absolutely right that it's abnormal for 3.0 to transform a range like
{{\[c-c!\]\[d-d!\]}} into {{\[c-c!\]\[c-d!\]}}. And having duplicate row
entries is only one of the problem this triggers, as with more collections you
could this ending up to delete data that it shouldn't. In any case, the problem
is definitively in 3.0 generating this in the first place, so that's what we
should fix, so fixing this in LegacyRangeTombstone constructor (Plan-A above)
is too late.
So the problem is when building the {{LegacyRangeTombstonList}} on 3.0, though
it's not in the {{insertFrom}} method and the Plan-B patch would likely create
other problems. The problem is in the {{LegacyBoundComparator}}: it delegates
to {{clusteringComparator.compare()}} before it even checks if the bounds it
compares have a {{collectionName}}, but that's incorrect.
Anyway, attaching below a fix for this as well as a simple unit test showing
where the problem lies:
| [13125-3.0|https://github.com/pcmanus/cassandra/commits/13125-3.0] |
[utests|http://cassci.datastax.com/job/pcmanus-13125-3.0-testall] |
[dtests|http://cassci.datastax.com/job/pcmanus-13125-3.0-dtest] |
| [13125-3.11|https://github.com/pcmanus/cassandra/commits/13125-3.11] |
[utests|http://cassci.datastax.com/job/pcmanus-13125-3.11-testall] |
[dtests|http://cassci.datastax.com/job/pcmanus-13125-3.11-dtest] |
[~Yasuharu], if you could check if this patch does properly work in your case,
that would be much appreciated. [~thobbs], since you're familiar with 3.0
compatibility code, would you also mind having a quick look at the patch?
> Duplicate rows after upgrading from 2.1.16 to 3.0.10/3.9
> --------------------------------------------------------
>
> Key: CASSANDRA-13125
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13125
> Project: Cassandra
> Issue Type: Bug
> Reporter: Zhongxiang Zheng
> Assignee: Yasuharu Goto
> Priority: Critical
> Attachments: diff-a.patch, diff-b.patch
>
>
> I found that rows are splitting and duplicated after upgrading the cluster
> from 2.1.x to 3.0.x.
> I found the way to reproduce the problem as below.
> {code}
> $ ccm create test -v 2.1.16 -n 3 -s
>
> Current cluster is now: test
> $ ccm node1 cqlsh -e "CREATE KEYSPACE test WITH replication =
> {'class':'SimpleStrategy', 'replication_factor':3}"
> $ ccm node1 cqlsh -e "CREATE TABLE test.test (id text PRIMARY KEY, value1
> set<text>, value2 set<text>);"
> # Upgrade node1
> $ for i in 1; do ccm node${i} stop; ccm node${i} setdir -v3.0.10; ccm
> node${i} start;ccm node${i} nodetool upgradesstables; done
> # Insert a row through node1(3.0.10)
> $ ccm node1 cqlsh -e "INSERT INTO test.test (id, value1, value2) values
> ('aaa', {'aaa', 'bbb'}, {'ccc', 'ddd'});"
> # Insert a row through node2(2.1.16)
> $ ccm node2 cqlsh -e "INSERT INTO test.test (id, value1, value2) values
> ('bbb', {'aaa', 'bbb'}, {'ccc', 'ddd'});"
> # The row inserted from node1 is splitting
> $ ccm node1 cqlsh -e "SELECT * FROM test.test ;"
> id | value1 | value2
> -----+----------------+----------------
> aaa | null | null
> aaa | {'aaa', 'bbb'} | {'ccc', 'ddd'}
> bbb | {'aaa', 'bbb'} | {'ccc', 'ddd'}
> $ for i in 1 2; do ccm node${i} nodetool flush; done
> # Results of sstable2json of node2. The row inserted from node1(3.0.10) is
> different from the row inserted from node2(2.1.16).
> $ ccm node2 json -k test -c test
> running
> ['/home/zzheng/.ccm/test/node2/data0/test/test-5406ee80dbdb11e6a175f57c4c7c85f3/test-test-ka-1-Data.db']
> -- test-test-ka-1-Data.db -----
> [
> {"key": "aaa",
> "cells": [["","",1484564624769577],
> ["value1","value2:!",1484564624769576,"t",1484564624],
> ["value1:616161","",1484564624769577],
> ["value1:626262","",1484564624769577],
> ["value2:636363","",1484564624769577],
> ["value2:646464","",1484564624769577]]},
> {"key": "bbb",
> "cells": [["","",1484564634508029],
> ["value1:_","value1:!",1484564634508028,"t",1484564634],
> ["value1:616161","",1484564634508029],
> ["value1:626262","",1484564634508029],
> ["value2:_","value2:!",1484564634508028,"t",1484564634],
> ["value2:636363","",1484564634508029],
> ["value2:646464","",1484564634508029]]}
> ]
> # Upgrade node2,3
> $ for i in `seq 2 3`; do ccm node${i} stop; ccm node${i} setdir -v3.0.10; ccm
> node${i} start;ccm node${i} nodetool upgradesstables; done
> # After upgrade node2,3, the row inserted from node1 is splitting in node2,3
> $ ccm node2 cqlsh -e "SELECT * FROM test.test ;"
>
> id | value1 | value2
> -----+----------------+----------------
> aaa | null | null
> aaa | {'aaa', 'bbb'} | {'ccc', 'ddd'}
> bbb | {'aaa', 'bbb'} | {'ccc', 'ddd'}
> (3 rows)
> # Results of sstabledump
> # node1
> [
> {
> "partition" : {
> "key" : [ "aaa" ],
> "position" : 0
> },
> "rows" : [
> {
> "type" : "row",
> "position" : 17,
> "liveness_info" : { "tstamp" : "2017-01-16T11:03:44.769577Z" },
> "cells" : [
> { "name" : "value1", "deletion_info" : { "marked_deleted" :
> "2017-01-16T11:03:44.769576Z", "local_delete_time" : "2017-01-16T11:03:44Z" }
> },
> { "name" : "value1", "path" : [ "aaa" ], "value" : "" },
> { "name" : "value1", "path" : [ "bbb" ], "value" : "" },
> { "name" : "value2", "deletion_info" : { "marked_deleted" :
> "2017-01-16T11:03:44.769576Z", "local_delete_time" : "2017-01-16T11:03:44Z" }
> },
> { "name" : "value2", "path" : [ "ccc" ], "value" : "" },
> { "name" : "value2", "path" : [ "ddd" ], "value" : "" }
> ]
> }
> ]
> },
> {
> "partition" : {
> "key" : [ "bbb" ],
> "position" : 48
> },
> "rows" : [
> {
> "type" : "row",
> "position" : 65,
> "liveness_info" : { "tstamp" : "2017-01-16T11:03:54.508029Z" },
> "cells" : [
> { "name" : "value1", "deletion_info" : { "marked_deleted" :
> "2017-01-16T11:03:54.508028Z", "local_delete_time" : "2017-01-16T11:03:54Z" }
> },
> { "name" : "value1", "path" : [ "aaa" ], "value" : "" },
> { "name" : "value1", "path" : [ "bbb" ], "value" : "" },
> { "name" : "value2", "deletion_info" : { "marked_deleted" :
> "2017-01-16T11:03:54.508028Z", "local_delete_time" : "2017-01-16T11:03:54Z" }
> },
> { "name" : "value2", "path" : [ "ccc" ], "value" : "" },
> { "name" : "value2", "path" : [ "ddd" ], "value" : "" }
> ]
> }
> ]
> }
> ]
>
> # node2
> [
> {
> "partition" : {
> "key" : [ "aaa" ],
> "position" : 0
> },
> "rows" : [
> {
> "type" : "row",
> "position" : 17,
> "liveness_info" : { "tstamp" : "2017-01-16T11:03:44.769577Z" },
> "cells" : [ ]
> },
> {
> "type" : "row",
> "position" : 22,
> "deletion_info" : { "marked_deleted" : "2017-01-16T11:03:44.769576Z",
> "local_delete_time" : "2017-01-16T11:03:44Z" },
> "cells" : [
> { "name" : "value1", "path" : [ "aaa" ], "value" : "", "tstamp" :
> "2017-01-16T11:03:44.769577Z" },
> { "name" : "value1", "path" : [ "bbb" ], "value" : "", "tstamp" :
> "2017-01-16T11:03:44.769577Z" },
> { "name" : "value2", "path" : [ "ccc" ], "value" : "", "tstamp" :
> "2017-01-16T11:03:44.769577Z" },
> { "name" : "value2", "path" : [ "ddd" ], "value" : "", "tstamp" :
> "2017-01-16T11:03:44.769577Z" }
> ]
> }
> ]
> },
> {
> "partition" : {
> "key" : [ "bbb" ],
> "position" : 57
> },
> "rows" : [
> {
> "type" : "row",
> "position" : 74,
> "liveness_info" : { "tstamp" : "2017-01-16T11:03:54.508029Z" },
> "cells" : [
> { "name" : "value1", "deletion_info" : { "marked_deleted" :
> "2017-01-16T11:03:54.508028Z", "local_delete_time" : "2017-01-16T11:03:54Z" }
> },
> { "name" : "value1", "path" : [ "aaa" ], "value" : "" },
> { "name" : "value1", "path" : [ "bbb" ], "value" : "" },
> { "name" : "value2", "deletion_info" : { "marked_deleted" :
> "2017-01-16T11:03:54.508028Z", "local_delete_time" : "2017-01-16T11:03:54Z" }
> },
> { "name" : "value2", "path" : [ "ccc" ], "value" : "" },
> { "name" : "value2", "path" : [ "ddd" ], "value" : "" }
> ]
> }
> ]
> }
> ]
> {code}
> Another example of row splitting is as follows.
> {code}
> $ ccm create test2 -v 2.1.16 -n 3 -s
>
> Current cluster is now: test2
> $ ccm node1 cqlsh -e "CREATE KEYSPACE test WITH replication =
> {'class':'SimpleStrategy', 'replication_factor':3}"
> $ ccm node1 cqlsh -e "CREATE TABLE test.text_set_set (id text PRIMARY KEY,
> value1 text, value2 set<text>, value3 set<text>);"
> $ for i in `seq 1`; do ccm node${i} stop; ccm node${i} setdir -v3.0.10; ccm
> node${i} start;ccm node${i} nodetool upgradesstables; done
> $ ccm node1 cqlsh -e "INSERT INTO test.text_set_set (id, value1, value2,
> value3) values ('aaa', 'aaa', {'aaa', 'bbb'}, {'ccc', 'ddd'});"
> $ ccm node1 cqlsh -e "SELECT * FROM test.text_set_set;"
>
> id | value1 | value2 | value3
> -----+--------+----------------+----------------
> aaa | aaa | null | null
> aaa | null | {'aaa', 'bbb'} | {'ccc', 'ddd'}
> (2 rows)
> {code}
> As far as I investigated, the occurrence conditions are as follows.
> * Table schema contains multiple collections.
> * Insert a row, which values of the collection column are not null through
> 3.x node while both 2.1 and 3.x nodes exist in a cluster.
> * Rows in sstables of node which version was 2.1 at the time the row was
> inserted is splitting after upgrading to 3.x.
> Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
