[ https://issues.apache.org/jira/browse/CASSANDRA-2918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067882#comment-13067882 ]
Cathy Daw commented on CASSANDRA-2918: -------------------------------------- Interestingly, I did the same test case using the hector stress tool which writes @ CL=QUORUM. *Insert 1000 rows. 691 Rows Returned.* * Bring up 3 nodes. Create a KS with RF=3. {code} [default@StressKeyspace] describe keyspace StressKeyspace; Keyspace: StressKeyspace: Replication Strategy: org.apache.cassandra.locator.SimpleStrategy Durable Writes: true Options: [replication_factor:3] Column Families: ColumnFamily: StressStandard Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.UTF8Type Row cache size / save period in seconds: 0.0/0 Key cache size / save period in seconds: 10000.0/14400 Memtable thresholds: 0.2859375/1440/32 (millions of ops/MB/minutes) GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 1.0 Replicate on write: true Built indexes: [] {code} * Kill one node * Insert 1000 rows with 10 columns each in batches of 100 into standard table. * Bring up dead node. Run Repair. * Kill two other nodes. * Set: consistencylevel as ONE. * Run: list StressStandard limit 2000; * 691 Rows Returned. > After repair, one row missing from query when two rows expected > --------------------------------------------------------------- > > Key: CASSANDRA-2918 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2918 > Project: Cassandra > Issue Type: Bug > Components: Core > Affects Versions: 0.8.2 > Environment: Cassandra-0.8 branch @ 07/18 around 1pm PST. > Reporter: Cathy Daw > Assignee: Sylvain Lebresne > > *Cluster Config* > {code} > cathy1 - 50.57.114.45 - Token: 0 > cathy2 - 50.57.107.176 - Token: 56713727820156410577229101238628035242 > cathy3 - 50.57.114.39 - Token: 113427455640312821154458202477256070484 > {code} > *+1) Create Seed Data+* > {code} > create keyspace testKS with placement_strategy = 'SimpleStrategy' and > strategy_options = [{replication_factor : 3}]; > {code} > *+2) Kill cathy3: 50.57.114.39+* > {code} > root@cathy2:~/cass-0.8/bin# ./nodetool -h localhost ring > Address DC Rack Status State Load Owns > Token > > 113427455640312821154458202477256070484 > 50.57.114.45 datacenter1 rack1 Up Normal 59.84 KB 33.33% > 0 > 50.57.107.176 datacenter1 rack1 Up Normal 59.85 KB 33.33% > 56713727820156410577229101238628035242 > 50.57.114.39 datacenter1 rack1 Down Normal 59.85 KB 33.33% > 113427455640312821154458202477256070484 > {code} > *+3) Run cassandra-cli+* > {code} > use testKS; > create column family metadataCF > with key_validation_class = 'AsciiType' > and comparator = 'AsciiType' > and column_metadata = [ > {column_name: ascii_col, validation_class:AsciiType, index_type: KEYS}, > {column_name: byte_col, validation_class: BytesType, index_type: KEYS}, > {column_name: uuid_col, validation_class: LexicalUUIDType, index_type: KEYS}, > {column_name: int_col, validation_class: IntegerType, index_type: KEYS}, > {column_name: long_col, validation_class: LongType, index_type: KEYS}, > {column_name: utf8_col, validation_class: UTF8Type, index_type: KEYS}]; > set metadataCF['key1']['ascii_col']=ascii('this is data inserted into ascii > column'); > set metadataCF['key1']['byte_col']=bytes('10101010'); > set metadataCF['key1']['uuid_col']=timeuuid(); > set metadataCF['key1']['int_col']=integer(1000); > set metadataCF['key1']['long_col']=long(44444444444); > set metadataCF['key1']['utf8_col']=utf8('this is data inserted into UTF8 > column'); > //Please note: forget to change CL before inserting 'key1', so that was > inserted with CL=ONE by default. > consistencylevel as TWO; > set metadataCF['key2']['ascii_col']=ascii('key2: this is data inserted into > ascii column'); > set metadataCF['key2']['byte_col']=bytes('201010102'); > set metadataCF['key2']['uuid_col']=timeuuid(); > set metadataCF['key2']['int_col']=integer(2000); > set metadataCF['key2']['long_col']=long(222222222); > set metadataCF['key2']['utf8_col']=utf8('key2-this is data inserted into UTF8 > column'); > //Assumed that the following read would be done on CL=TWO and will the second > replica would be guaranteed to be fix for 'key1'. > list metadataCF; > {code} > {code} > [default@testKS] list metadataCF; > Using default limit of 100 > ------------------- > RowKey: key1 > => (column=ascii_col, value=this is data inserted into ascii column, > timestamp=1311035260656000) > => (column=byte_col, value=10101010, timestamp=1311035260662000) > => (column=int_col, value=1000, timestamp=1311035260669000) > => (column=long_col, value=44444444444, timestamp=1311035260674000) > => (column=utf8_col, value=this is data inserted into UTF8 column, > timestamp=1311035260678000) > => (column=uuid_col, value=e9811e90-b19d-11e0-0000-2069cd105fbf, > timestamp=1311035260666000) > ------------------- > RowKey: key2 > => (column=ascii_col, value=key2: this is data inserted into ascii column, > timestamp=1311035260682000) > => (column=byte_col, value=0201010102, timestamp=1311035260685000) > => (column=int_col, value=2000, timestamp=1311035260692000) > => (column=long_col, value=222222222, timestamp=1311035260695000) > => (column=utf8_col, value=key2-this is data inserted into UTF8 column, > timestamp=1311035260699000) > => (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf, > timestamp=1311035260689000) > 2 Rows Returned. > {code} > *+4) Start Cassandra on cathy3+* > *+5) Run repair on cathy3+* > {code} > nodetool -h cathy3 repair testKS metadataCF > {code} > {code} > INFO 00:38:32,696 Waiting for repair requests: [#<TreeRequest > manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.114.45, > (testKS,metadataCF), (0,56713727820156410577229101238628035242]>, > #<TreeRequest manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, > /50.57.107.176, (testKS,metadataCF), > (0,56713727820156410577229101238628035242]>] > INFO 00:38:32,696 Waiting for repair requests: [#<TreeRequest > manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45, > (testKS,metadataCF), (113427455640312821154458202477256070484,0]>, > #<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e, > /50.57.107.176, (testKS,metadataCF), > (113427455640312821154458202477256070484,0]>] > INFO 00:38:32,697 Waiting for repair requests: [#<TreeRequest > manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.107.176, > (testKS,metadataCF), > (56713727820156410577229101238628035242,113427455640312821154458202477256070484]>, > #<TreeRequest manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, > /50.57.114.45, (testKS,metadataCF), > (56713727820156410577229101238628035242,113427455640312821154458202477256070484]>] > INFO 00:38:33,052 Sending AEService tree for #<TreeRequest > manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.39, > (testKS,metadataCF), (113427455640312821154458202477256070484,0]> > INFO 00:38:33,100 Sending AEService tree for #<TreeRequest > manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.114.39, > (testKS,metadataCF), > (56713727820156410577229101238628035242,113427455640312821154458202477256070484]> > INFO 00:38:33,203 Sending AEService tree for #<TreeRequest > manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.114.39, > (testKS,metadataCF), (0,56713727820156410577229101238628035242]> > INFO 00:38:33,242 Queueing comparison #<Differencer #<TreeRequest > manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.107.176, > (testKS,metadataCF), > (56713727820156410577229101238628035242,113427455640312821154458202477256070484]>> > INFO 00:38:33,270 Queueing comparison #<Differencer #<TreeRequest > manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.114.45, > (testKS,metadataCF), (0,56713727820156410577229101238628035242]>> > INFO 00:38:33,363 Endpoints /50.57.114.39 and /50.57.107.176 are consistent > for (testKS,metadataCF) on > (56713727820156410577229101238628035242,113427455640312821154458202477256070484] > INFO 00:38:33,364 #<TreeRequest > manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.107.176, > (testKS,metadataCF), > (56713727820156410577229101238628035242,113427455640312821154458202477256070484]> > completed successfully: 1 outstanding. > INFO 00:38:33,428 Endpoints /50.57.114.39 and /50.57.114.45 are consistent > for (testKS,metadataCF) on (0,56713727820156410577229101238628035242] > INFO 00:38:33,429 #<TreeRequest > manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.114.45, > (testKS,metadataCF), (0,56713727820156410577229101238628035242]> completed > successfully: 1 outstanding. > INFO 00:38:33,441 Queueing comparison #<Differencer #<TreeRequest > manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.107.176, > (testKS,metadataCF), (113427455640312821154458202477256070484,0]>> > INFO 00:38:33,453 Queueing comparison #<Differencer #<TreeRequest > manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.114.45, > (testKS,metadataCF), > (56713727820156410577229101238628035242,113427455640312821154458202477256070484]>> > INFO 00:38:33,519 Endpoints /50.57.114.39 and /50.57.107.176 are consistent > for (testKS,metadataCF) on (113427455640312821154458202477256070484,0] > INFO 00:38:33,519 #<TreeRequest > manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.107.176, > (testKS,metadataCF), (113427455640312821154458202477256070484,0]> completed > successfully: 1 outstanding. > INFO 00:38:33,554 Endpoints /50.57.114.39 and /50.57.114.45 are consistent > for (testKS,metadataCF) on > (56713727820156410577229101238628035242,113427455640312821154458202477256070484] > INFO 00:38:33,555 #<TreeRequest > manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.114.45, > (testKS,metadataCF), > (56713727820156410577229101238628035242,113427455640312821154458202477256070484]> > completed successfully: 0 outstanding. > INFO 00:38:33,555 Repair session > manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc completed successfully. > INFO 00:38:33,567 Queueing comparison #<Differencer #<TreeRequest > manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45, > (testKS,metadataCF), (113427455640312821154458202477256070484,0]>> > INFO 00:38:33,598 Endpoints /50.57.114.39 and /50.57.114.45 have 1 range(s) > out of sync for (testKS,metadataCF) on > (113427455640312821154458202477256070484,0] > INFO 00:38:33,599 Performing streaming repair of 1 ranges for #<TreeRequest > manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45, > (testKS,metadataCF), (113427455640312821154458202477256070484,0]> > INFO 00:38:33,626 Stream context metadata [], 0 sstables. > INFO 00:38:33,643 Streaming to /50.57.114.45 > INFO 00:38:33,732 Opening /var/lib/cassandra/data/testKS/metadataCF-g-1 > INFO 00:38:33,736 Submitting index build of > ascii_col,byte_col,int_col,long_col,utf8_col,uuid_col, for data in > SSTableReader(path='/var/lib/cassandra/data/testKS/metadataCF-g-1-Data.db') > INFO 00:38:33,741 Enqueuing flush of > Memtable-metadataCF.ascii_col_idx@1146840108(19/23 serialized/live bytes, 1 > ops) > INFO 00:38:33,742 Writing Memtable-metadataCF.ascii_col_idx@1146840108(19/23 > serialized/live bytes, 1 ops) > INFO 00:38:33,748 Completed flushing > /var/lib/cassandra/data/testKS/metadataCF.ascii_col_idx-g-1-Data.db (114 > bytes) > INFO 00:38:33,750 Enqueuing flush of > Memtable-metadataCF.byte_col_idx@1562533825(19/23 serialized/live bytes, 1 > ops) > INFO 00:38:33,750 Writing Memtable-metadataCF.byte_col_idx@1562533825(19/23 > serialized/live bytes, 1 ops) > INFO 00:38:33,756 Completed flushing > /var/lib/cassandra/data/testKS/metadataCF.byte_col_idx-g-1-Data.db (74 bytes) > INFO 00:38:33,758 Enqueuing flush of > Memtable-metadataCF.int_col_idx@1515686296(19/23 serialized/live bytes, 1 ops) > INFO 00:38:33,758 Writing Memtable-metadataCF.int_col_idx@1515686296(19/23 > serialized/live bytes, 1 ops) > INFO 00:38:33,820 Completed flushing > /var/lib/cassandra/data/testKS/metadataCF.int_col_idx-g-1-Data.db (71 bytes) > INFO 00:38:33,821 Enqueuing flush of > Memtable-metadataCF.long_col_idx@507274563(19/23 serialized/live bytes, 1 ops) > INFO 00:38:33,821 Writing Memtable-metadataCF.long_col_idx@507274563(19/23 > serialized/live bytes, 1 ops) > INFO 00:38:33,828 Completed flushing > /var/lib/cassandra/data/testKS/metadataCF.long_col_idx-g-1-Data.db (77 bytes) > INFO 00:38:33,829 Enqueuing flush of > Memtable-metadataCF.utf8_col_idx@1201340161(19/23 serialized/live bytes, 1 > ops) > INFO 00:38:33,830 Writing Memtable-metadataCF.utf8_col_idx@1201340161(19/23 > serialized/live bytes, 1 ops) > INFO 00:38:33,836 Completed flushing > /var/lib/cassandra/data/testKS/metadataCF.utf8_col_idx-g-1-Data.db (112 bytes) > INFO 00:38:33,837 Enqueuing flush of > Memtable-metadataCF.uuid_col_idx@703489457(19/23 serialized/live bytes, 1 ops) > INFO 00:38:33,838 Writing Memtable-metadataCF.uuid_col_idx@703489457(19/23 > serialized/live bytes, 1 ops) > INFO 00:38:33,857 Completed flushing > /var/lib/cassandra/data/testKS/metadataCF.uuid_col_idx-g-1-Data.db (85 bytes) > INFO 00:38:33,858 Index build of > ascii_col,byte_col,int_col,long_col,utf8_col,uuid_col, complete > INFO 00:38:33,858 Finished streaming session 4937677751163 from /50.57.114.45 > INFO 00:38:33,859 Finished streaming repair for #<TreeRequest > manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45, > (testKS,metadataCF), (113427455640312821154458202477256070484,0]> > INFO 00:38:33,859 #<TreeRequest > manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45, > (testKS,metadataCF), (113427455640312821154458202477256070484,0]> completed > successfully: 0 outstanding. > INFO 00:38:33,859 Repair session > manual-repair-9a957823-9969-4b91-900a-96a00659a85e completed successfully. > INFO 00:38:36,026 Queueing comparison #<Differencer #<TreeRequest > manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.107.176, > (testKS,metadataCF), (0,56713727820156410577229101238628035242]>> > INFO 00:38:36,057 Endpoints /50.57.114.39 and /50.57.107.176 are consistent > for (testKS,metadataCF) on (0,56713727820156410577229101238628035242] > INFO 00:38:36,058 #<TreeRequest > manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.107.176, > (testKS,metadataCF), (0,56713727820156410577229101238628035242]> completed > successfully: 0 outstanding. > INFO 00:38:36,058 Repair session > manual-repair-9b989c5e-66a1-459d-93ed-793acc401794 completed successfully. > {code} > *+6) Kill cathy1 and cathy2+* > {code} > root@cathy3:~/cass-0.8/bin# ./nodetool -h cathy3 ring > Address DC Rack Status State Load Owns > Token > > 113427455640312821154458202477256070484 > 50.57.114.45 datacenter1 rack1 Down Normal 105.46 KB 33.33% > 0 > 50.57.107.176 datacenter1 rack1 Down Normal 106 KB 33.33% > 56713727820156410577229101238628035242 > 50.57.114.39 datacenter1 rack1 Up Normal 331.33 KB 33.33% > 113427455640312821154458202477256070484 > {code} > *+7) Log into cassandra-cli on node3+* > {code} > ./cassandra-cli -h cathy3 > Connected to: "Test Cluster" on cathy3/9160 > Welcome to the Cassandra CLI. > Type 'help;' or '?' for help. > Type 'quit;' or 'exit;' to quit. > [default@unknown] use testKS; > Authenticated to keyspace: testKS > [default@testKS] consistencylevel as ONE; > Consistency level is set to 'ONE'. > [default@testKS] list metadataCF; > Using default limit of 100 > ------------------- > RowKey: key2 > => (column=ascii_col, value=key2: this is data inserted into ascii column, > timestamp=1311035260682000) > => (column=byte_col, value=0201010102, timestamp=1311035260685000) > => (column=int_col, value=2000, timestamp=1311035260692000) > => (column=long_col, value=222222222, timestamp=1311035260695000) > => (column=utf8_col, value=key2-this is data inserted into UTF8 column, > timestamp=1311035260699000) > => (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf, > timestamp=1311035260689000) > 1 Row Returned. > {code} > *+8) Bring up nodes 1 and 2 and query+* > *NODE1 - cassandra-cli* > {code} > ./cassandra-cli -h cathy1 > Connected to: "Test Cluster" on cathy1/9160 > Welcome to the Cassandra CLI. > Type 'help;' or '?' for help. > Type 'quit;' or 'exit;' to quit. > [default@unknown] use testKS; > Authenticated to keyspace: testKS > [default@testKS] list metadataCF; > Using default limit of 100 > ------------------- > RowKey: key2 > => (column=ascii_col, value=key2: this is data inserted into ascii column, > timestamp=1311035260682000) > => (column=byte_col, value=0201010102, timestamp=1311035260685000) > => (column=int_col, value=2000, timestamp=1311035260692000) > => (column=long_col, value=222222222, timestamp=1311035260695000) > => (column=utf8_col, value=key2-this is data inserted into UTF8 column, > timestamp=1311035260699000) > => (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf, > timestamp=1311035260689000) > 1 Row Returned. > {code} > *NODE2 - cassandra-cli* > {code} > ./cassandra-cli -h cathy2 > Connected to: "Test Cluster" on cathy2/9160 > Welcome to the Cassandra CLI. > Type 'help;' or '?' for help. > Type 'quit;' or 'exit;' to quit. > [default@unknown] use testKS; > Authenticated to keyspace: testKS > [default@testKS] list metadataCF; > Using default limit of 100 > ------------------- > RowKey: key1 > => (column=ascii_col, value=this is data inserted into ascii column, > timestamp=1311035260656000) > => (column=byte_col, value=10101010, timestamp=1311035260662000) > => (column=int_col, value=1000, timestamp=1311035260669000) > => (column=long_col, value=44444444444, timestamp=1311035260674000) > => (column=utf8_col, value=this is data inserted into UTF8 column, > timestamp=1311035260678000) > => (column=uuid_col, value=e9811e90-b19d-11e0-0000-2069cd105fbf, > timestamp=1311035260666000) > ------------------- > RowKey: key2 > => (column=ascii_col, value=key2: this is data inserted into ascii column, > timestamp=1311035260682000) > => (column=byte_col, value=0201010102, timestamp=1311035260685000) > => (column=int_col, value=2000, timestamp=1311035260692000) > => (column=long_col, value=222222222, timestamp=1311035260695000) > => (column=utf8_col, value=key2-this is data inserted into UTF8 column, > timestamp=1311035260699000) > => (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf, > timestamp=1311035260689000) > 2 Rows Returned. > {code} > *NODE3 - cassandra-cli* > {code} > ./cassandra-cli -h cathy3 > Connected to: "Test Cluster" on cathy3/9160 > Welcome to the Cassandra CLI. > Type 'help;' or '?' for help. > Type 'quit;' or 'exit;' to quit. > [default@unknown] use testKS; > Authenticated to keyspace: testKS > [default@testKS] list metadataCF; > Using default limit of 100 > ------------------- > RowKey: key2 > => (column=ascii_col, value=key2: this is data inserted into ascii column, > timestamp=1311035260682000) > => (column=byte_col, value=0201010102, timestamp=1311035260685000) > => (column=int_col, value=2000, timestamp=1311035260692000) > => (column=long_col, value=222222222, timestamp=1311035260695000) > => (column=utf8_col, value=key2-this is data inserted into UTF8 column, > timestamp=1311035260699000) > => (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf, > timestamp=1311035260689000) > 1 Row Returned. > {code} > *+9) Force Read Repair on node1+* > {code} > consistencylevel as THREE; > Consistency level is set to 'THREE'. > [default@testKS] list metadataCF; > Using default limit of 100 > ------------------- > RowKey: key1 > => (column=ascii_col, value=this is data inserted into ascii column, > timestamp=1311035260656000) > => (column=byte_col, value=10101010, timestamp=1311035260662000) > => (column=int_col, value=1000, timestamp=1311035260669000) > => (column=long_col, value=44444444444, timestamp=1311035260674000) > => (column=utf8_col, value=this is data inserted into UTF8 column, > timestamp=1311035260678000) > => (column=uuid_col, value=e9811e90-b19d-11e0-0000-2069cd105fbf, > timestamp=1311035260666000) > ------------------- > RowKey: key2 > => (column=ascii_col, value=key2: this is data inserted into ascii column, > timestamp=1311035260682000) > => (column=byte_col, value=0201010102, timestamp=1311035260685000) > => (column=int_col, value=2000, timestamp=1311035260692000) > => (column=long_col, value=222222222, timestamp=1311035260695000) > => (column=utf8_col, value=key2-this is data inserted into UTF8 column, > timestamp=1311035260699000) > => (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf, > timestamp=1311035260689000) > 2 Rows Returned. > {code} -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira