[
https://issues.apache.org/jira/browse/CASSANDRA-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16061717#comment-16061717
]
Jay Zhuang edited comment on CASSANDRA-13632 at 6/24/17 1:16 AM:
-----------------------------------------------------------------
Hi [~whangsf], could you please help me to reproduce the problem locally, here
is what I did:
{noformat}
CREATE KEYSPACE foo WITH replication = {'class': 'NetworkTopologyStrategy',
'dc1': '3' };
CREATE TABLE foo.foo ( key int, foo int, col int, PRIMARY KEY (key, foo) ) with
dclocal_read_repair_chance=0;
CONSISTENCY QUORUM;
INSERT INTO foo.foo (key, foo) VALUES ( 1,1);
TRACING ON;
SELECT * FROM foo.foo WHERE key = 1 and foo =2;
{noformat}
But no Repair event:
{noformat}
cqlsh> SELECT * FROM foo.foo WHERE key = 1 and foo =2;
key | foo | col
-----+-----+-----
(0 rows)
Tracing session: e9751120-5879-11e7-be43-dfd5ff1ad595
activity
| timestamp | source | source_elapsed
------------------------------------------------------------------------------------------+----------------------------+-----------+----------------
Execute
CQL3 query | 2017-06-24 01:10:28.914000 | 127.0.0.1 | 0
Parsing SELECT * FROM foo.foo WHERE key = 1 and foo =2;
[SharedPool-Worker-1] | 2017-06-24 01:10:28.914000 | 127.0.0.1 | 220
Preparing statement
[SharedPool-Worker-1] | 2017-06-24 01:10:28.914000 | 127.0.0.1 | 423
READ message received from /127.0.0.1
[MessagingService-Incoming-/127.0.0.1] | 2017-06-24 01:10:28.915000 | 127.0.0.2
| 40
reading digest from /127.0.0.2
[SharedPool-Worker-1] | 2017-06-24 01:10:28.915000 | 127.0.0.1 | 1115
Executing single-partition query on foo
[SharedPool-Worker-2] | 2017-06-24 01:10:28.915000 | 127.0.0.1 | 1123
Acquiring sstable references
[SharedPool-Worker-2] | 2017-06-24 01:10:28.915000 | 127.0.0.1 | 1189
Merging memtable contents
[SharedPool-Worker-2] | 2017-06-24 01:10:28.915000 | 127.0.0.1 | 1228
Sending READ message to /127.0.0.2
[MessagingService-Outgoing-/127.0.0.2] | 2017-06-24 01:10:28.915000 | 127.0.0.1
| 1326
Read 0 live and 0 tombstone cells
[SharedPool-Worker-2] | 2017-06-24 01:10:28.915000 | 127.0.0.1 | 1424
Executing single-partition query on foo
[SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 | 221
REQUEST_RESPONSE message received from /127.0.0.2
[MessagingService-Incoming-/127.0.0.2] | 2017-06-24 01:10:28.916000 | 127.0.0.1
| 2817
Acquiring sstable references
[SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 | 342
Merging memtable contents
[SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 | 402
Read 0 live and 0 tombstone cells
[SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 | 640
Read 0 live and 0 tombstone cells
[SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 | 714
Enqueuing response to /127.0.0.1
[SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 | 770
Sending REQUEST_RESPONSE message to /127.0.0.1
[MessagingService-Outgoing-/127.0.0.1] | 2017-06-24 01:10:28.916001 | 127.0.0.2
| 917
Processing response from /127.0.0.2
[SharedPool-Worker-2] | 2017-06-24 01:10:28.917000 | 127.0.0.1 | 2916
Request complete | 2017-06-24 01:10:28.917133 | 127.0.0.1 | 3133
{noformat}
was (Author: jay.zhuang):
Hi [~whangsf], could you please help me to reproduce the problem locally, here
is what I did:
{noformat}
CREATE KEYSPACE foo WITH replication = {'class': 'NetworkTopologyStrategy',
'dc1': '3' };
CREATE TABLE foo.foo ( key int, foo int, col int, PRIMARY KEY (key, foo) ) with
dclocal_read_repair_chance=0;
CONSISTENCY QUORUM;
INSERT INTO foo.foo (key, foo) VALUES ( 1,1);
TRACING ON;
SELECT * FROM foo.foo WHERE key = 1 and foo =2;
{noformat}
But Ino Repair event:
{noformat}
cqlsh> SELECT * FROM foo.foo WHERE key = 1 and foo =2;
key | foo | col
-----+-----+-----
(0 rows)
Tracing session: e9751120-5879-11e7-be43-dfd5ff1ad595
activity
| timestamp | source | source_elapsed
------------------------------------------------------------------------------------------+----------------------------+-----------+----------------
Execute
CQL3 query | 2017-06-24 01:10:28.914000 | 127.0.0.1 | 0
Parsing SELECT * FROM foo.foo WHERE key = 1 and foo =2;
[SharedPool-Worker-1] | 2017-06-24 01:10:28.914000 | 127.0.0.1 | 220
Preparing statement
[SharedPool-Worker-1] | 2017-06-24 01:10:28.914000 | 127.0.0.1 | 423
READ message received from /127.0.0.1
[MessagingService-Incoming-/127.0.0.1] | 2017-06-24 01:10:28.915000 | 127.0.0.2
| 40
reading digest from /127.0.0.2
[SharedPool-Worker-1] | 2017-06-24 01:10:28.915000 | 127.0.0.1 | 1115
Executing single-partition query on foo
[SharedPool-Worker-2] | 2017-06-24 01:10:28.915000 | 127.0.0.1 | 1123
Acquiring sstable references
[SharedPool-Worker-2] | 2017-06-24 01:10:28.915000 | 127.0.0.1 | 1189
Merging memtable contents
[SharedPool-Worker-2] | 2017-06-24 01:10:28.915000 | 127.0.0.1 | 1228
Sending READ message to /127.0.0.2
[MessagingService-Outgoing-/127.0.0.2] | 2017-06-24 01:10:28.915000 | 127.0.0.1
| 1326
Read 0 live and 0 tombstone cells
[SharedPool-Worker-2] | 2017-06-24 01:10:28.915000 | 127.0.0.1 | 1424
Executing single-partition query on foo
[SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 | 221
REQUEST_RESPONSE message received from /127.0.0.2
[MessagingService-Incoming-/127.0.0.2] | 2017-06-24 01:10:28.916000 | 127.0.0.1
| 2817
Acquiring sstable references
[SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 | 342
Merging memtable contents
[SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 | 402
Read 0 live and 0 tombstone cells
[SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 | 640
Read 0 live and 0 tombstone cells
[SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 | 714
Enqueuing response to /127.0.0.1
[SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 | 770
Sending REQUEST_RESPONSE message to /127.0.0.1
[MessagingService-Outgoing-/127.0.0.1] | 2017-06-24 01:10:28.916001 | 127.0.0.2
| 917
Processing response from /127.0.0.2
[SharedPool-Worker-2] | 2017-06-24 01:10:28.917000 | 127.0.0.1 | 2916
Request complete | 2017-06-24 01:10:28.917133 | 127.0.0.1 | 3133
{noformat}
> Digest mismatch if row is empty
> -------------------------------
>
> Key: CASSANDRA-13632
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13632
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Reporter: Andrew Whang
> Assignee: Andrew Whang
> Fix For: 3.0.x
>
>
> This issue is similar to CASSANDRA-12090. Quorum read queries that include a
> column selector (non-wildcard) result in digest mismatch when the row is
> empty (key does not exist). It seems the data serialization path checks if
> rowIterator.isEmpty() and if so ignores column names (by setting IS_EMPTY
> flag). However, the digest serialization path does not perform this check and
> includes column names. The digest comparison results in a mismatch. The
> mismatch does not end up issuing a read repair mutation since the underlying
> data is the same.
> The mismatch on the read path ends up doubling our p99 read latency. We
> discovered this issue while testing a 2.2.5 to 3.0.13 upgrade.
> One thing to note is that we're using thrift, which ends up handling the
> ColumnFilter differently than the CQL path.
> As with CASSANDRA-12090, fixing the digest seems sensible.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
