After repair, one row missing from query when two rows expected
---------------------------------------------------------------
Key: CASSANDRA-2918
URL: https://issues.apache.org/jira/browse/CASSANDRA-2918
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.8.2
Environment: Cassandra-0.8 branch @ 07/18 around 1pm PST.
Reporter: Cathy Daw
*Cluster Config*
{code}
cathy1 - 50.57.114.45 - Token: 0
cathy2 - 50.57.107.176 - Token: 56713727820156410577229101238628035242
cathy3 - 50.57.114.39 - Token: 113427455640312821154458202477256070484
{code}
*1) Create Seed Data*
{code}
create keyspace testKS with placement_strategy = 'SimpleStrategy' and
strategy_options = [{replication_factor : 3}];
{code}
*2) Kill cathy3: 50.57.114.39*
{code}
root@cathy2:~/cass-0.8/bin# ./nodetool -h localhost ring
Address DC Rack Status State Load Owns
Token
113427455640312821154458202477256070484
50.57.114.45 datacenter1 rack1 Up Normal 59.84 KB 33.33%
0
50.57.107.176 datacenter1 rack1 Up Normal 59.85 KB 33.33%
56713727820156410577229101238628035242
50.57.114.39 datacenter1 rack1 Down Normal 59.85 KB 33.33%
113427455640312821154458202477256070484
{code}
*3) Run cassandra-cli*
{code}
use testKS;
create column family metadataCF
with key_validation_class = 'AsciiType'
and comparator = 'AsciiType'
and column_metadata = [
{column_name: ascii_col, validation_class:AsciiType, index_type: KEYS},
{column_name: byte_col, validation_class: BytesType, index_type: KEYS},
{column_name: uuid_col, validation_class: LexicalUUIDType, index_type: KEYS},
{column_name: int_col, validation_class: IntegerType, index_type: KEYS},
{column_name: long_col, validation_class: LongType, index_type: KEYS},
{column_name: utf8_col, validation_class: UTF8Type, index_type: KEYS}];
set metadataCF['key1']['ascii_col']=ascii('this is data inserted into ascii
column');
set metadataCF['key1']['byte_col']=bytes('10101010');
set metadataCF['key1']['uuid_col']=timeuuid();
set metadataCF['key1']['int_col']=integer(1000);
set metadataCF['key1']['long_col']=long(44444444444);
set metadataCF['key1']['utf8_col']=utf8('this is data inserted into UTF8
column');
consistencylevel as TWO;
set metadataCF['key2']['ascii_col']=ascii('key2: this is data inserted into
ascii column');
set metadataCF['key2']['byte_col']=bytes('201010102');
set metadataCF['key2']['uuid_col']=timeuuid();
set metadataCF['key2']['int_col']=integer(2000);
set metadataCF['key2']['long_col']=long(222222222);
set metadataCF['key2']['utf8_col']=utf8('key2-this is data inserted into UTF8
column');
{code}
{code}
[default@testKS] list metadataCF;
Using default limit of 100
-------------------
RowKey: key1
=> (column=ascii_col, value=this is data inserted into ascii column,
timestamp=1311035260656000)
=> (column=byte_col, value=10101010, timestamp=1311035260662000)
=> (column=int_col, value=1000, timestamp=1311035260669000)
=> (column=long_col, value=44444444444, timestamp=1311035260674000)
=> (column=utf8_col, value=this is data inserted into UTF8 column,
timestamp=1311035260678000)
=> (column=uuid_col, value=e9811e90-b19d-11e0-0000-2069cd105fbf,
timestamp=1311035260666000)
-------------------
RowKey: key2
=> (column=ascii_col, value=key2: this is data inserted into ascii column,
timestamp=1311035260682000)
=> (column=byte_col, value=0201010102, timestamp=1311035260685000)
=> (column=int_col, value=2000, timestamp=1311035260692000)
=> (column=long_col, value=222222222, timestamp=1311035260695000)
=> (column=utf8_col, value=key2-this is data inserted into UTF8 column,
timestamp=1311035260699000)
=> (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf,
timestamp=1311035260689000)
2 Rows Returned.
{code}
*4) Start Cassandra on cathy3*
*5) Run repair on cathy3*
{code}
nodetool -h cathy3 repair testKS metadataCF
{code}
{code}
INFO 00:38:32,696 Waiting for repair requests: [#<TreeRequest
manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.114.45,
(testKS,metadataCF), (0,56713727820156410577229101238628035242]>, #<TreeRequest
manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.107.176,
(testKS,metadataCF), (0,56713727820156410577229101238628035242]>]
INFO 00:38:32,696 Waiting for repair requests: [#<TreeRequest
manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45,
(testKS,metadataCF), (113427455640312821154458202477256070484,0]>,
#<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e,
/50.57.107.176, (testKS,metadataCF),
(113427455640312821154458202477256070484,0]>]
INFO 00:38:32,697 Waiting for repair requests: [#<TreeRequest
manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.107.176,
(testKS,metadataCF),
(56713727820156410577229101238628035242,113427455640312821154458202477256070484]>,
#<TreeRequest manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc,
/50.57.114.45, (testKS,metadataCF),
(56713727820156410577229101238628035242,113427455640312821154458202477256070484]>]
INFO 00:38:33,052 Sending AEService tree for #<TreeRequest
manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.39,
(testKS,metadataCF), (113427455640312821154458202477256070484,0]>
INFO 00:38:33,100 Sending AEService tree for #<TreeRequest
manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.114.39,
(testKS,metadataCF),
(56713727820156410577229101238628035242,113427455640312821154458202477256070484]>
INFO 00:38:33,203 Sending AEService tree for #<TreeRequest
manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.114.39,
(testKS,metadataCF), (0,56713727820156410577229101238628035242]>
INFO 00:38:33,242 Queueing comparison #<Differencer #<TreeRequest
manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.107.176,
(testKS,metadataCF),
(56713727820156410577229101238628035242,113427455640312821154458202477256070484]>>
INFO 00:38:33,270 Queueing comparison #<Differencer #<TreeRequest
manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.114.45,
(testKS,metadataCF), (0,56713727820156410577229101238628035242]>>
INFO 00:38:33,363 Endpoints /50.57.114.39 and /50.57.107.176 are consistent
for (testKS,metadataCF) on
(56713727820156410577229101238628035242,113427455640312821154458202477256070484]
INFO 00:38:33,364 #<TreeRequest
manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.107.176,
(testKS,metadataCF),
(56713727820156410577229101238628035242,113427455640312821154458202477256070484]>
completed successfully: 1 outstanding.
INFO 00:38:33,428 Endpoints /50.57.114.39 and /50.57.114.45 are consistent for
(testKS,metadataCF) on (0,56713727820156410577229101238628035242]
INFO 00:38:33,429 #<TreeRequest
manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.114.45,
(testKS,metadataCF), (0,56713727820156410577229101238628035242]> completed
successfully: 1 outstanding.
INFO 00:38:33,441 Queueing comparison #<Differencer #<TreeRequest
manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.107.176,
(testKS,metadataCF), (113427455640312821154458202477256070484,0]>>
INFO 00:38:33,453 Queueing comparison #<Differencer #<TreeRequest
manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.114.45,
(testKS,metadataCF),
(56713727820156410577229101238628035242,113427455640312821154458202477256070484]>>
INFO 00:38:33,519 Endpoints /50.57.114.39 and /50.57.107.176 are consistent
for (testKS,metadataCF) on (113427455640312821154458202477256070484,0]
INFO 00:38:33,519 #<TreeRequest
manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.107.176,
(testKS,metadataCF), (113427455640312821154458202477256070484,0]> completed
successfully: 1 outstanding.
INFO 00:38:33,554 Endpoints /50.57.114.39 and /50.57.114.45 are consistent for
(testKS,metadataCF) on
(56713727820156410577229101238628035242,113427455640312821154458202477256070484]
INFO 00:38:33,555 #<TreeRequest
manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.114.45,
(testKS,metadataCF),
(56713727820156410577229101238628035242,113427455640312821154458202477256070484]>
completed successfully: 0 outstanding.
INFO 00:38:33,555 Repair session
manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc completed successfully.
INFO 00:38:33,567 Queueing comparison #<Differencer #<TreeRequest
manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45,
(testKS,metadataCF), (113427455640312821154458202477256070484,0]>>
INFO 00:38:33,598 Endpoints /50.57.114.39 and /50.57.114.45 have 1 range(s)
out of sync for (testKS,metadataCF) on
(113427455640312821154458202477256070484,0]
INFO 00:38:33,599 Performing streaming repair of 1 ranges for #<TreeRequest
manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45,
(testKS,metadataCF), (113427455640312821154458202477256070484,0]>
INFO 00:38:33,626 Stream context metadata [], 0 sstables.
INFO 00:38:33,643 Streaming to /50.57.114.45
INFO 00:38:33,732 Opening /var/lib/cassandra/data/testKS/metadataCF-g-1
INFO 00:38:33,736 Submitting index build of
ascii_col,byte_col,int_col,long_col,utf8_col,uuid_col, for data in
SSTableReader(path='/var/lib/cassandra/data/testKS/metadataCF-g-1-Data.db')
INFO 00:38:33,741 Enqueuing flush of
Memtable-metadataCF.ascii_col_idx@1146840108(19/23 serialized/live bytes, 1 ops)
INFO 00:38:33,742 Writing Memtable-metadataCF.ascii_col_idx@1146840108(19/23
serialized/live bytes, 1 ops)
INFO 00:38:33,748 Completed flushing
/var/lib/cassandra/data/testKS/metadataCF.ascii_col_idx-g-1-Data.db (114 bytes)
INFO 00:38:33,750 Enqueuing flush of
Memtable-metadataCF.byte_col_idx@1562533825(19/23 serialized/live bytes, 1 ops)
INFO 00:38:33,750 Writing Memtable-metadataCF.byte_col_idx@1562533825(19/23
serialized/live bytes, 1 ops)
INFO 00:38:33,756 Completed flushing
/var/lib/cassandra/data/testKS/metadataCF.byte_col_idx-g-1-Data.db (74 bytes)
INFO 00:38:33,758 Enqueuing flush of
Memtable-metadataCF.int_col_idx@1515686296(19/23 serialized/live bytes, 1 ops)
INFO 00:38:33,758 Writing Memtable-metadataCF.int_col_idx@1515686296(19/23
serialized/live bytes, 1 ops)
INFO 00:38:33,820 Completed flushing
/var/lib/cassandra/data/testKS/metadataCF.int_col_idx-g-1-Data.db (71 bytes)
INFO 00:38:33,821 Enqueuing flush of
Memtable-metadataCF.long_col_idx@507274563(19/23 serialized/live bytes, 1 ops)
INFO 00:38:33,821 Writing Memtable-metadataCF.long_col_idx@507274563(19/23
serialized/live bytes, 1 ops)
INFO 00:38:33,828 Completed flushing
/var/lib/cassandra/data/testKS/metadataCF.long_col_idx-g-1-Data.db (77 bytes)
INFO 00:38:33,829 Enqueuing flush of
Memtable-metadataCF.utf8_col_idx@1201340161(19/23 serialized/live bytes, 1 ops)
INFO 00:38:33,830 Writing Memtable-metadataCF.utf8_col_idx@1201340161(19/23
serialized/live bytes, 1 ops)
INFO 00:38:33,836 Completed flushing
/var/lib/cassandra/data/testKS/metadataCF.utf8_col_idx-g-1-Data.db (112 bytes)
INFO 00:38:33,837 Enqueuing flush of
Memtable-metadataCF.uuid_col_idx@703489457(19/23 serialized/live bytes, 1 ops)
INFO 00:38:33,838 Writing Memtable-metadataCF.uuid_col_idx@703489457(19/23
serialized/live bytes, 1 ops)
INFO 00:38:33,857 Completed flushing
/var/lib/cassandra/data/testKS/metadataCF.uuid_col_idx-g-1-Data.db (85 bytes)
INFO 00:38:33,858 Index build of
ascii_col,byte_col,int_col,long_col,utf8_col,uuid_col, complete
INFO 00:38:33,858 Finished streaming session 4937677751163 from /50.57.114.45
INFO 00:38:33,859 Finished streaming repair for #<TreeRequest
manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45,
(testKS,metadataCF), (113427455640312821154458202477256070484,0]>
INFO 00:38:33,859 #<TreeRequest
manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45,
(testKS,metadataCF), (113427455640312821154458202477256070484,0]> completed
successfully: 0 outstanding.
INFO 00:38:33,859 Repair session
manual-repair-9a957823-9969-4b91-900a-96a00659a85e completed successfully.
INFO 00:38:36,026 Queueing comparison #<Differencer #<TreeRequest
manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.107.176,
(testKS,metadataCF), (0,56713727820156410577229101238628035242]>>
INFO 00:38:36,057 Endpoints /50.57.114.39 and /50.57.107.176 are consistent
for (testKS,metadataCF) on (0,56713727820156410577229101238628035242]
INFO 00:38:36,058 #<TreeRequest
manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.107.176,
(testKS,metadataCF), (0,56713727820156410577229101238628035242]> completed
successfully: 0 outstanding.
INFO 00:38:36,058 Repair session
manual-repair-9b989c5e-66a1-459d-93ed-793acc401794 completed successfully.
{code}
*6) Kill cathy1 and cathy2*
{code}
root@cathy3:~/cass-0.8/bin# ./nodetool -h cathy3 ring
Address DC Rack Status State Load Owns
Token
113427455640312821154458202477256070484
50.57.114.45 datacenter1 rack1 Down Normal 105.46 KB 33.33%
0
50.57.107.176 datacenter1 rack1 Down Normal 106 KB 33.33%
56713727820156410577229101238628035242
50.57.114.39 datacenter1 rack1 Up Normal 331.33 KB 33.33%
113427455640312821154458202477256070484
{code}
*7) Log into cassandra-cli on node3*
{code}
./cassandra-cli -h cathy3
Connected to: "Test Cluster" on cathy3/9160
Welcome to the Cassandra CLI.
Type 'help;' or '?' for help.
Type 'quit;' or 'exit;' to quit.
[default@unknown] use testKS;
Authenticated to keyspace: testKS
[default@testKS] consistencylevel as ONE;
Consistency level is set to 'ONE'.
[default@testKS] list metadataCF;
Using default limit of 100
-------------------
RowKey: key2
=> (column=ascii_col, value=key2: this is data inserted into ascii column,
timestamp=1311035260682000)
=> (column=byte_col, value=0201010102, timestamp=1311035260685000)
=> (column=int_col, value=2000, timestamp=1311035260692000)
=> (column=long_col, value=222222222, timestamp=1311035260695000)
=> (column=utf8_col, value=key2-this is data inserted into UTF8 column,
timestamp=1311035260699000)
=> (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf,
timestamp=1311035260689000)
1 Row Returned.
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
