[
https://issues.apache.org/jira/browse/CASSANDRA-2918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067896#comment-13067896
]
Cathy Daw commented on CASSANDRA-2918:
--------------------------------------
This looks more to me like a WRITE with CL.Q bug as the source of the problem.
But shouldn't have repair detected that node1 and node2 were out of sync?
* 3 node cluster / KS with RF=3 / 2 nodes up
* Insert 1000 rows on node2 with CL=QUORUM
** node1:691 rows
** node2:1000 rows
** node3: down
* Repair node3
** node1:691 rows
** node2:1000 rows
** node3: 691 rows
> After repair, one row missing from query when two rows expected
> ---------------------------------------------------------------
>
> Key: CASSANDRA-2918
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2918
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.2
> Environment: Cassandra-0.8 branch @ 07/18 around 1pm PST.
> Reporter: Cathy Daw
> Assignee: Sylvain Lebresne
>
> *Cluster Config*
> {code}
> cathy1 - 50.57.114.45 - Token: 0
> cathy2 - 50.57.107.176 - Token: 56713727820156410577229101238628035242
> cathy3 - 50.57.114.39 - Token: 113427455640312821154458202477256070484
> {code}
> *+1) Create Seed Data+*
> {code}
> create keyspace testKS with placement_strategy = 'SimpleStrategy' and
> strategy_options = [{replication_factor : 3}];
> {code}
> *+2) Kill cathy3: 50.57.114.39+*
> {code}
> root@cathy2:~/cass-0.8/bin# ./nodetool -h localhost ring
> Address DC Rack Status State Load Owns
> Token
>
> 113427455640312821154458202477256070484
> 50.57.114.45 datacenter1 rack1 Up Normal 59.84 KB 33.33%
> 0
> 50.57.107.176 datacenter1 rack1 Up Normal 59.85 KB 33.33%
> 56713727820156410577229101238628035242
> 50.57.114.39 datacenter1 rack1 Down Normal 59.85 KB 33.33%
> 113427455640312821154458202477256070484
> {code}
> *+3) Run cassandra-cli+*
> {code}
> use testKS;
> create column family metadataCF
> with key_validation_class = 'AsciiType'
> and comparator = 'AsciiType'
> and column_metadata = [
> {column_name: ascii_col, validation_class:AsciiType, index_type: KEYS},
> {column_name: byte_col, validation_class: BytesType, index_type: KEYS},
> {column_name: uuid_col, validation_class: LexicalUUIDType, index_type: KEYS},
> {column_name: int_col, validation_class: IntegerType, index_type: KEYS},
> {column_name: long_col, validation_class: LongType, index_type: KEYS},
> {column_name: utf8_col, validation_class: UTF8Type, index_type: KEYS}];
> set metadataCF['key1']['ascii_col']=ascii('this is data inserted into ascii
> column');
> set metadataCF['key1']['byte_col']=bytes('10101010');
> set metadataCF['key1']['uuid_col']=timeuuid();
> set metadataCF['key1']['int_col']=integer(1000);
> set metadataCF['key1']['long_col']=long(44444444444);
> set metadataCF['key1']['utf8_col']=utf8('this is data inserted into UTF8
> column');
> //Please note: forget to change CL before inserting 'key1', so that was
> inserted with CL=ONE by default.
> consistencylevel as TWO;
> set metadataCF['key2']['ascii_col']=ascii('key2: this is data inserted into
> ascii column');
> set metadataCF['key2']['byte_col']=bytes('201010102');
> set metadataCF['key2']['uuid_col']=timeuuid();
> set metadataCF['key2']['int_col']=integer(2000);
> set metadataCF['key2']['long_col']=long(222222222);
> set metadataCF['key2']['utf8_col']=utf8('key2-this is data inserted into UTF8
> column');
> //Assumed that the following read would be done on CL=TWO and will the second
> replica would be guaranteed to be fix for 'key1'.
> list metadataCF;
> {code}
> {code}
> [default@testKS] list metadataCF;
> Using default limit of 100
> -------------------
> RowKey: key1
> => (column=ascii_col, value=this is data inserted into ascii column,
> timestamp=1311035260656000)
> => (column=byte_col, value=10101010, timestamp=1311035260662000)
> => (column=int_col, value=1000, timestamp=1311035260669000)
> => (column=long_col, value=44444444444, timestamp=1311035260674000)
> => (column=utf8_col, value=this is data inserted into UTF8 column,
> timestamp=1311035260678000)
> => (column=uuid_col, value=e9811e90-b19d-11e0-0000-2069cd105fbf,
> timestamp=1311035260666000)
> -------------------
> RowKey: key2
> => (column=ascii_col, value=key2: this is data inserted into ascii column,
> timestamp=1311035260682000)
> => (column=byte_col, value=0201010102, timestamp=1311035260685000)
> => (column=int_col, value=2000, timestamp=1311035260692000)
> => (column=long_col, value=222222222, timestamp=1311035260695000)
> => (column=utf8_col, value=key2-this is data inserted into UTF8 column,
> timestamp=1311035260699000)
> => (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf,
> timestamp=1311035260689000)
> 2 Rows Returned.
> {code}
> *+4) Start Cassandra on cathy3+*
> *+5) Run repair on cathy3+*
> {code}
> nodetool -h cathy3 repair testKS metadataCF
> {code}
> {code}
> INFO 00:38:32,696 Waiting for repair requests: [#<TreeRequest
> manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.114.45,
> (testKS,metadataCF), (0,56713727820156410577229101238628035242]>,
> #<TreeRequest manual-repair-9b989c5e-66a1-459d-93ed-793acc401794,
> /50.57.107.176, (testKS,metadataCF),
> (0,56713727820156410577229101238628035242]>]
> INFO 00:38:32,696 Waiting for repair requests: [#<TreeRequest
> manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45,
> (testKS,metadataCF), (113427455640312821154458202477256070484,0]>,
> #<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e,
> /50.57.107.176, (testKS,metadataCF),
> (113427455640312821154458202477256070484,0]>]
> INFO 00:38:32,697 Waiting for repair requests: [#<TreeRequest
> manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.107.176,
> (testKS,metadataCF),
> (56713727820156410577229101238628035242,113427455640312821154458202477256070484]>,
> #<TreeRequest manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc,
> /50.57.114.45, (testKS,metadataCF),
> (56713727820156410577229101238628035242,113427455640312821154458202477256070484]>]
> INFO 00:38:33,052 Sending AEService tree for #<TreeRequest
> manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.39,
> (testKS,metadataCF), (113427455640312821154458202477256070484,0]>
> INFO 00:38:33,100 Sending AEService tree for #<TreeRequest
> manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.114.39,
> (testKS,metadataCF),
> (56713727820156410577229101238628035242,113427455640312821154458202477256070484]>
> INFO 00:38:33,203 Sending AEService tree for #<TreeRequest
> manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.114.39,
> (testKS,metadataCF), (0,56713727820156410577229101238628035242]>
> INFO 00:38:33,242 Queueing comparison #<Differencer #<TreeRequest
> manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.107.176,
> (testKS,metadataCF),
> (56713727820156410577229101238628035242,113427455640312821154458202477256070484]>>
> INFO 00:38:33,270 Queueing comparison #<Differencer #<TreeRequest
> manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.114.45,
> (testKS,metadataCF), (0,56713727820156410577229101238628035242]>>
> INFO 00:38:33,363 Endpoints /50.57.114.39 and /50.57.107.176 are consistent
> for (testKS,metadataCF) on
> (56713727820156410577229101238628035242,113427455640312821154458202477256070484]
> INFO 00:38:33,364 #<TreeRequest
> manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.107.176,
> (testKS,metadataCF),
> (56713727820156410577229101238628035242,113427455640312821154458202477256070484]>
> completed successfully: 1 outstanding.
> INFO 00:38:33,428 Endpoints /50.57.114.39 and /50.57.114.45 are consistent
> for (testKS,metadataCF) on (0,56713727820156410577229101238628035242]
> INFO 00:38:33,429 #<TreeRequest
> manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.114.45,
> (testKS,metadataCF), (0,56713727820156410577229101238628035242]> completed
> successfully: 1 outstanding.
> INFO 00:38:33,441 Queueing comparison #<Differencer #<TreeRequest
> manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.107.176,
> (testKS,metadataCF), (113427455640312821154458202477256070484,0]>>
> INFO 00:38:33,453 Queueing comparison #<Differencer #<TreeRequest
> manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.114.45,
> (testKS,metadataCF),
> (56713727820156410577229101238628035242,113427455640312821154458202477256070484]>>
> INFO 00:38:33,519 Endpoints /50.57.114.39 and /50.57.107.176 are consistent
> for (testKS,metadataCF) on (113427455640312821154458202477256070484,0]
> INFO 00:38:33,519 #<TreeRequest
> manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.107.176,
> (testKS,metadataCF), (113427455640312821154458202477256070484,0]> completed
> successfully: 1 outstanding.
> INFO 00:38:33,554 Endpoints /50.57.114.39 and /50.57.114.45 are consistent
> for (testKS,metadataCF) on
> (56713727820156410577229101238628035242,113427455640312821154458202477256070484]
> INFO 00:38:33,555 #<TreeRequest
> manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.114.45,
> (testKS,metadataCF),
> (56713727820156410577229101238628035242,113427455640312821154458202477256070484]>
> completed successfully: 0 outstanding.
> INFO 00:38:33,555 Repair session
> manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc completed successfully.
> INFO 00:38:33,567 Queueing comparison #<Differencer #<TreeRequest
> manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45,
> (testKS,metadataCF), (113427455640312821154458202477256070484,0]>>
> INFO 00:38:33,598 Endpoints /50.57.114.39 and /50.57.114.45 have 1 range(s)
> out of sync for (testKS,metadataCF) on
> (113427455640312821154458202477256070484,0]
> INFO 00:38:33,599 Performing streaming repair of 1 ranges for #<TreeRequest
> manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45,
> (testKS,metadataCF), (113427455640312821154458202477256070484,0]>
> INFO 00:38:33,626 Stream context metadata [], 0 sstables.
> INFO 00:38:33,643 Streaming to /50.57.114.45
> INFO 00:38:33,732 Opening /var/lib/cassandra/data/testKS/metadataCF-g-1
> INFO 00:38:33,736 Submitting index build of
> ascii_col,byte_col,int_col,long_col,utf8_col,uuid_col, for data in
> SSTableReader(path='/var/lib/cassandra/data/testKS/metadataCF-g-1-Data.db')
> INFO 00:38:33,741 Enqueuing flush of
> Memtable-metadataCF.ascii_col_idx@1146840108(19/23 serialized/live bytes, 1
> ops)
> INFO 00:38:33,742 Writing Memtable-metadataCF.ascii_col_idx@1146840108(19/23
> serialized/live bytes, 1 ops)
> INFO 00:38:33,748 Completed flushing
> /var/lib/cassandra/data/testKS/metadataCF.ascii_col_idx-g-1-Data.db (114
> bytes)
> INFO 00:38:33,750 Enqueuing flush of
> Memtable-metadataCF.byte_col_idx@1562533825(19/23 serialized/live bytes, 1
> ops)
> INFO 00:38:33,750 Writing Memtable-metadataCF.byte_col_idx@1562533825(19/23
> serialized/live bytes, 1 ops)
> INFO 00:38:33,756 Completed flushing
> /var/lib/cassandra/data/testKS/metadataCF.byte_col_idx-g-1-Data.db (74 bytes)
> INFO 00:38:33,758 Enqueuing flush of
> Memtable-metadataCF.int_col_idx@1515686296(19/23 serialized/live bytes, 1 ops)
> INFO 00:38:33,758 Writing Memtable-metadataCF.int_col_idx@1515686296(19/23
> serialized/live bytes, 1 ops)
> INFO 00:38:33,820 Completed flushing
> /var/lib/cassandra/data/testKS/metadataCF.int_col_idx-g-1-Data.db (71 bytes)
> INFO 00:38:33,821 Enqueuing flush of
> Memtable-metadataCF.long_col_idx@507274563(19/23 serialized/live bytes, 1 ops)
> INFO 00:38:33,821 Writing Memtable-metadataCF.long_col_idx@507274563(19/23
> serialized/live bytes, 1 ops)
> INFO 00:38:33,828 Completed flushing
> /var/lib/cassandra/data/testKS/metadataCF.long_col_idx-g-1-Data.db (77 bytes)
> INFO 00:38:33,829 Enqueuing flush of
> Memtable-metadataCF.utf8_col_idx@1201340161(19/23 serialized/live bytes, 1
> ops)
> INFO 00:38:33,830 Writing Memtable-metadataCF.utf8_col_idx@1201340161(19/23
> serialized/live bytes, 1 ops)
> INFO 00:38:33,836 Completed flushing
> /var/lib/cassandra/data/testKS/metadataCF.utf8_col_idx-g-1-Data.db (112 bytes)
> INFO 00:38:33,837 Enqueuing flush of
> Memtable-metadataCF.uuid_col_idx@703489457(19/23 serialized/live bytes, 1 ops)
> INFO 00:38:33,838 Writing Memtable-metadataCF.uuid_col_idx@703489457(19/23
> serialized/live bytes, 1 ops)
> INFO 00:38:33,857 Completed flushing
> /var/lib/cassandra/data/testKS/metadataCF.uuid_col_idx-g-1-Data.db (85 bytes)
> INFO 00:38:33,858 Index build of
> ascii_col,byte_col,int_col,long_col,utf8_col,uuid_col, complete
> INFO 00:38:33,858 Finished streaming session 4937677751163 from /50.57.114.45
> INFO 00:38:33,859 Finished streaming repair for #<TreeRequest
> manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45,
> (testKS,metadataCF), (113427455640312821154458202477256070484,0]>
> INFO 00:38:33,859 #<TreeRequest
> manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45,
> (testKS,metadataCF), (113427455640312821154458202477256070484,0]> completed
> successfully: 0 outstanding.
> INFO 00:38:33,859 Repair session
> manual-repair-9a957823-9969-4b91-900a-96a00659a85e completed successfully.
> INFO 00:38:36,026 Queueing comparison #<Differencer #<TreeRequest
> manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.107.176,
> (testKS,metadataCF), (0,56713727820156410577229101238628035242]>>
> INFO 00:38:36,057 Endpoints /50.57.114.39 and /50.57.107.176 are consistent
> for (testKS,metadataCF) on (0,56713727820156410577229101238628035242]
> INFO 00:38:36,058 #<TreeRequest
> manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.107.176,
> (testKS,metadataCF), (0,56713727820156410577229101238628035242]> completed
> successfully: 0 outstanding.
> INFO 00:38:36,058 Repair session
> manual-repair-9b989c5e-66a1-459d-93ed-793acc401794 completed successfully.
> {code}
> *+6) Kill cathy1 and cathy2+*
> {code}
> root@cathy3:~/cass-0.8/bin# ./nodetool -h cathy3 ring
> Address DC Rack Status State Load Owns
> Token
>
> 113427455640312821154458202477256070484
> 50.57.114.45 datacenter1 rack1 Down Normal 105.46 KB 33.33%
> 0
> 50.57.107.176 datacenter1 rack1 Down Normal 106 KB 33.33%
> 56713727820156410577229101238628035242
> 50.57.114.39 datacenter1 rack1 Up Normal 331.33 KB 33.33%
> 113427455640312821154458202477256070484
> {code}
> *+7) Log into cassandra-cli on node3+*
> {code}
> ./cassandra-cli -h cathy3
> Connected to: "Test Cluster" on cathy3/9160
> Welcome to the Cassandra CLI.
> Type 'help;' or '?' for help.
> Type 'quit;' or 'exit;' to quit.
> [default@unknown] use testKS;
> Authenticated to keyspace: testKS
> [default@testKS] consistencylevel as ONE;
> Consistency level is set to 'ONE'.
> [default@testKS] list metadataCF;
> Using default limit of 100
> -------------------
> RowKey: key2
> => (column=ascii_col, value=key2: this is data inserted into ascii column,
> timestamp=1311035260682000)
> => (column=byte_col, value=0201010102, timestamp=1311035260685000)
> => (column=int_col, value=2000, timestamp=1311035260692000)
> => (column=long_col, value=222222222, timestamp=1311035260695000)
> => (column=utf8_col, value=key2-this is data inserted into UTF8 column,
> timestamp=1311035260699000)
> => (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf,
> timestamp=1311035260689000)
> 1 Row Returned.
> {code}
> *+8) Bring up nodes 1 and 2 and query+*
> *NODE1 - cassandra-cli*
> {code}
> ./cassandra-cli -h cathy1
> Connected to: "Test Cluster" on cathy1/9160
> Welcome to the Cassandra CLI.
> Type 'help;' or '?' for help.
> Type 'quit;' or 'exit;' to quit.
> [default@unknown] use testKS;
> Authenticated to keyspace: testKS
> [default@testKS] list metadataCF;
> Using default limit of 100
> -------------------
> RowKey: key2
> => (column=ascii_col, value=key2: this is data inserted into ascii column,
> timestamp=1311035260682000)
> => (column=byte_col, value=0201010102, timestamp=1311035260685000)
> => (column=int_col, value=2000, timestamp=1311035260692000)
> => (column=long_col, value=222222222, timestamp=1311035260695000)
> => (column=utf8_col, value=key2-this is data inserted into UTF8 column,
> timestamp=1311035260699000)
> => (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf,
> timestamp=1311035260689000)
> 1 Row Returned.
> {code}
> *NODE2 - cassandra-cli*
> {code}
> ./cassandra-cli -h cathy2
> Connected to: "Test Cluster" on cathy2/9160
> Welcome to the Cassandra CLI.
> Type 'help;' or '?' for help.
> Type 'quit;' or 'exit;' to quit.
> [default@unknown] use testKS;
> Authenticated to keyspace: testKS
> [default@testKS] list metadataCF;
> Using default limit of 100
> -------------------
> RowKey: key1
> => (column=ascii_col, value=this is data inserted into ascii column,
> timestamp=1311035260656000)
> => (column=byte_col, value=10101010, timestamp=1311035260662000)
> => (column=int_col, value=1000, timestamp=1311035260669000)
> => (column=long_col, value=44444444444, timestamp=1311035260674000)
> => (column=utf8_col, value=this is data inserted into UTF8 column,
> timestamp=1311035260678000)
> => (column=uuid_col, value=e9811e90-b19d-11e0-0000-2069cd105fbf,
> timestamp=1311035260666000)
> -------------------
> RowKey: key2
> => (column=ascii_col, value=key2: this is data inserted into ascii column,
> timestamp=1311035260682000)
> => (column=byte_col, value=0201010102, timestamp=1311035260685000)
> => (column=int_col, value=2000, timestamp=1311035260692000)
> => (column=long_col, value=222222222, timestamp=1311035260695000)
> => (column=utf8_col, value=key2-this is data inserted into UTF8 column,
> timestamp=1311035260699000)
> => (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf,
> timestamp=1311035260689000)
> 2 Rows Returned.
> {code}
> *NODE3 - cassandra-cli*
> {code}
> ./cassandra-cli -h cathy3
> Connected to: "Test Cluster" on cathy3/9160
> Welcome to the Cassandra CLI.
> Type 'help;' or '?' for help.
> Type 'quit;' or 'exit;' to quit.
> [default@unknown] use testKS;
> Authenticated to keyspace: testKS
> [default@testKS] list metadataCF;
> Using default limit of 100
> -------------------
> RowKey: key2
> => (column=ascii_col, value=key2: this is data inserted into ascii column,
> timestamp=1311035260682000)
> => (column=byte_col, value=0201010102, timestamp=1311035260685000)
> => (column=int_col, value=2000, timestamp=1311035260692000)
> => (column=long_col, value=222222222, timestamp=1311035260695000)
> => (column=utf8_col, value=key2-this is data inserted into UTF8 column,
> timestamp=1311035260699000)
> => (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf,
> timestamp=1311035260689000)
> 1 Row Returned.
> {code}
> *+9) Force Read Repair on node1+*
> {code}
> consistencylevel as THREE;
> Consistency level is set to 'THREE'.
> [default@testKS] list metadataCF;
> Using default limit of 100
> -------------------
> RowKey: key1
> => (column=ascii_col, value=this is data inserted into ascii column,
> timestamp=1311035260656000)
> => (column=byte_col, value=10101010, timestamp=1311035260662000)
> => (column=int_col, value=1000, timestamp=1311035260669000)
> => (column=long_col, value=44444444444, timestamp=1311035260674000)
> => (column=utf8_col, value=this is data inserted into UTF8 column,
> timestamp=1311035260678000)
> => (column=uuid_col, value=e9811e90-b19d-11e0-0000-2069cd105fbf,
> timestamp=1311035260666000)
> -------------------
> RowKey: key2
> => (column=ascii_col, value=key2: this is data inserted into ascii column,
> timestamp=1311035260682000)
> => (column=byte_col, value=0201010102, timestamp=1311035260685000)
> => (column=int_col, value=2000, timestamp=1311035260692000)
> => (column=long_col, value=222222222, timestamp=1311035260695000)
> => (column=utf8_col, value=key2-this is data inserted into UTF8 column,
> timestamp=1311035260699000)
> => (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf,
> timestamp=1311035260689000)
> 2 Rows Returned.
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
