[jira] [Comment Edited] (CASSANDRA-13632) Digest mismatch if row is empty
[
https://issues.apache.org/jira/browse/CASSANDRA-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166318#comment-16166318
]
Jason Brown edited comment on CASSANDRA-13632 at 9/14/17 1:39 PM:
--
bq. the data serialization path checks if rowIterator.isEmpty() and if so
ignores column names (by setting IS_EMPTY flag). However, the digest
serialization path does not perform this check and includes column names.
This is correct.
bq. The digest comparison results in a mismatch.
This is not correct. In
[{{DigestResolver}}|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/DigestResolver.java#L87],
the coordinator gets the digest from each response, even for the data that
sent back the full {{DataResponse}}. So, while there is the difference in what
the data nodes send back to the coordindator, it resolves to the same digest
value.
[~whangsf] what indications do you have that a DigestMismatch is happening?
was (Author: jasobrown):
bq. the data serialization path checks if rowIterator.isEmpty() and if so
ignores column names (by setting IS_EMPTY flag). However, the digest
serialization path does not perform this check and includes column names.
This is correct.
bq. The digest comparison results in a mismatch.
This is not correct. In
[{{DigestResolver}}|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/DigestResolver.java#L87],
the coordinator gets the digest from each response, even for the data that
sent back the full {{DataResponse}}. So, while there is the difference in what
the data nodes send back to the coordindator, is resolves to the same digest
value.
[~whangsf] what indications do you have that a DigestMismatch is happening?
> Digest mismatch if row is empty
> ---
>
> Key: CASSANDRA-13632
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13632
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
>Reporter: Andrew Whang
>Assignee: Andrew Whang
> Fix For: 3.0.x
>
>
> This issue is similar to CASSANDRA-12090. Quorum read queries that include a
> column selector (non-wildcard) result in digest mismatch when the row is
> empty (key does not exist). It seems the data serialization path checks if
> rowIterator.isEmpty() and if so ignores column names (by setting IS_EMPTY
> flag). However, the digest serialization path does not perform this check and
> includes column names. The digest comparison results in a mismatch. The
> mismatch does not end up issuing a read repair mutation since the underlying
> data is the same.
> The mismatch on the read path ends up doubling our p99 read latency. We
> discovered this issue while testing a 2.2.5 to 3.0.13 upgrade.
> One thing to note is that we're using thrift, which ends up handling the
> ColumnFilter differently than the CQL path.
> As with CASSANDRA-12090, fixing the digest seems sensible.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13632) Digest mismatch if row is empty
[
https://issues.apache.org/jira/browse/CASSANDRA-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16061717#comment-16061717
]
Jay Zhuang edited comment on CASSANDRA-13632 at 6/24/17 2:24 AM:
-
Hi [~whangsf], could you please help me to reproduce the problem locally, here
is what I did:
{noformat}
CREATE KEYSPACE foo WITH replication = {'class': 'NetworkTopologyStrategy',
'dc1': '3' };
CREATE TABLE foo.foo ( key int, foo int, col int, PRIMARY KEY (key, foo) ) with
dclocal_read_repair_chance=0;
CONSISTENCY QUORUM;
INSERT INTO foo.foo (key, foo) VALUES ( 1,1);
TRACING ON;
SELECT key FROM foo.foo WHERE key = 1 and foo =2;
{noformat}
But no Repair event:
{noformat}
cqlsh> SELECT key FROM foo.foo WHERE key = 1 and foo =2;
key
-
(0 rows)
Tracing session: 2cb5aa80-5884-11e7-be43-dfd5ff1ad595
activity
| timestamp | source| source_elapsed
--++---+
Execute
CQL3 query | 2017-06-24 02:23:56.712000 | 127.0.0.1 | 0
Parsing SELECT key FROM foo.foo WHERE key = 1 and foo =2;
[SharedPool-Worker-1] | 2017-06-24 02:23:56.712000 | 127.0.0.1 |187
Preparing statement
[SharedPool-Worker-1] | 2017-06-24 02:23:56.713000 | 127.0.0.1 |411
reading digest from /127.0.0.3
[SharedPool-Worker-1] | 2017-06-24 02:23:56.713000 | 127.0.0.1 |976
Executing single-partition query on foo
[SharedPool-Worker-2] | 2017-06-24 02:23:56.713000 | 127.0.0.1 | 1004
Acquiring sstable references
[SharedPool-Worker-2] | 2017-06-24 02:23:56.713000 | 127.0.0.1 | 1057
Merging memtable contents
[SharedPool-Worker-2] | 2017-06-24 02:23:56.713000 | 127.0.0.1 | 1083
Sending READ message to /127.0.0.3
[MessagingService-Outgoing-/127.0.0.3] | 2017-06-24 02:23:56.713000 | 127.0.0.1
| 1124
READ message received from /127.0.0.1
[MessagingService-Incoming-/127.0.0.1] | 2017-06-24 02:23:56.714000 | 127.0.0.3
| 21
Read 0 live and 0 tombstone cells
[SharedPool-Worker-2] | 2017-06-24 02:23:56.714000 | 127.0.0.1 | 1249
Executing single-partition query on foo
[SharedPool-Worker-1] | 2017-06-24 02:23:56.714000 | 127.0.0.3 |145
REQUEST_RESPONSE message received from /127.0.0.3
[MessagingService-Incoming-/127.0.0.3] | 2017-06-24 02:23:56.714000 | 127.0.0.1
| 2198
Acquiring sstable references
[SharedPool-Worker-1] | 2017-06-24 02:23:56.714000 | 127.0.0.3 |214
Merging memtable contents
[SharedPool-Worker-1] | 2017-06-24 02:23:56.714000 | 127.0.0.3 |265
Read 0 live and 0 tombstone cells
[SharedPool-Worker-1] | 2017-06-24 02:23:56.714000 | 127.0.0.3 |466
Read 0 live and 0 tombstone cells
[SharedPool-Worker-1] | 2017-06-24 02:23:56.714000 | 127.0.0.3 |538
Enqueuing response to /127.0.0.1
[SharedPool-Worker-1] | 2017-06-24 02:23:56.714001 | 127.0.0.3 |596
Sending REQUEST_RESPONSE message to /127.0.0.1
[MessagingService-Outgoing-/127.0.0.1] | 2017-06-24 02:23:56.714001 | 127.0.0.3
|671
Processing response from /127.0.0.3
[SharedPool-Worker-2] | 2017-06-24 02:23:56.715000 | 127.0.0.1 | 2324
Request complete | 2017-06-24 02:23:56.714453 | 127.0.0.1 | 2453
{noformat}
was (Author: jay.zhuang):
Hi [~whangsf], could you please help me to reproduce the problem locally, here
is what I did:
{noformat}
CREATE KEYSPACE foo WITH replication = {'class': 'NetworkTopologyStrategy',
'dc1': '3' };
CREATE TABLE foo.foo ( key int, foo int, col int, PRIMARY KEY (key, foo) ) with
dclocal_read_repair_chance=0;
CONSISTENCY QUORUM;
INSERT INTO foo.foo (key, foo) VALUES ( 1,1);
TRACING ON;
SELECT * FROM foo.foo WHERE key = 1 and foo =2;
{noformat}
But no Repair event:
{noformat}
cqlsh> SELECT * FROM foo.foo WHERE key = 1 and foo =2;
key | foo | col
-+-+-
(0 rows)
Tracing session: e9751120-5879-11e7-be43-dfd5ff1ad595
activity
| timestamp | source
[jira] [Comment Edited] (CASSANDRA-13632) Digest mismatch if row is empty
[
https://issues.apache.org/jira/browse/CASSANDRA-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16061717#comment-16061717
]
Jay Zhuang edited comment on CASSANDRA-13632 at 6/24/17 1:16 AM:
-
Hi [~whangsf], could you please help me to reproduce the problem locally, here
is what I did:
{noformat}
CREATE KEYSPACE foo WITH replication = {'class': 'NetworkTopologyStrategy',
'dc1': '3' };
CREATE TABLE foo.foo ( key int, foo int, col int, PRIMARY KEY (key, foo) ) with
dclocal_read_repair_chance=0;
CONSISTENCY QUORUM;
INSERT INTO foo.foo (key, foo) VALUES ( 1,1);
TRACING ON;
SELECT * FROM foo.foo WHERE key = 1 and foo =2;
{noformat}
But no Repair event:
{noformat}
cqlsh> SELECT * FROM foo.foo WHERE key = 1 and foo =2;
key | foo | col
-+-+-
(0 rows)
Tracing session: e9751120-5879-11e7-be43-dfd5ff1ad595
activity
| timestamp | source| source_elapsed
--++---+
Execute
CQL3 query | 2017-06-24 01:10:28.914000 | 127.0.0.1 | 0
Parsing SELECT * FROM foo.foo WHERE key = 1 and foo =2;
[SharedPool-Worker-1] | 2017-06-24 01:10:28.914000 | 127.0.0.1 |220
Preparing statement
[SharedPool-Worker-1] | 2017-06-24 01:10:28.914000 | 127.0.0.1 |423
READ message received from /127.0.0.1
[MessagingService-Incoming-/127.0.0.1] | 2017-06-24 01:10:28.915000 | 127.0.0.2
| 40
reading digest from /127.0.0.2
[SharedPool-Worker-1] | 2017-06-24 01:10:28.915000 | 127.0.0.1 | 1115
Executing single-partition query on foo
[SharedPool-Worker-2] | 2017-06-24 01:10:28.915000 | 127.0.0.1 | 1123
Acquiring sstable references
[SharedPool-Worker-2] | 2017-06-24 01:10:28.915000 | 127.0.0.1 | 1189
Merging memtable contents
[SharedPool-Worker-2] | 2017-06-24 01:10:28.915000 | 127.0.0.1 | 1228
Sending READ message to /127.0.0.2
[MessagingService-Outgoing-/127.0.0.2] | 2017-06-24 01:10:28.915000 | 127.0.0.1
| 1326
Read 0 live and 0 tombstone cells
[SharedPool-Worker-2] | 2017-06-24 01:10:28.915000 | 127.0.0.1 | 1424
Executing single-partition query on foo
[SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 |221
REQUEST_RESPONSE message received from /127.0.0.2
[MessagingService-Incoming-/127.0.0.2] | 2017-06-24 01:10:28.916000 | 127.0.0.1
| 2817
Acquiring sstable references
[SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 |342
Merging memtable contents
[SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 |402
Read 0 live and 0 tombstone cells
[SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 |640
Read 0 live and 0 tombstone cells
[SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 |714
Enqueuing response to /127.0.0.1
[SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 |770
Sending REQUEST_RESPONSE message to /127.0.0.1
[MessagingService-Outgoing-/127.0.0.1] | 2017-06-24 01:10:28.916001 | 127.0.0.2
|917
Processing response from /127.0.0.2
[SharedPool-Worker-2] | 2017-06-24 01:10:28.917000 | 127.0.0.1 | 2916
Request complete | 2017-06-24 01:10:28.917133 | 127.0.0.1 | 3133
{noformat}
was (Author: jay.zhuang):
Hi [~whangsf], could you please help me to reproduce the problem locally, here
is what I did:
{noformat}
CREATE KEYSPACE foo WITH replication = {'class': 'NetworkTopologyStrategy',
'dc1': '3' };
CREATE TABLE foo.foo ( key int, foo int, col int, PRIMARY KEY (key, foo) ) with
dclocal_read_repair_chance=0;
CONSISTENCY QUORUM;
INSERT INTO foo.foo (key, foo) VALUES ( 1,1);
TRACING ON;
SELECT * FROM foo.foo WHERE key = 1 and foo =2;
{noformat}
But Ino Repair event:
{noformat}
cqlsh> SELECT * FROM foo.foo WHERE key = 1 and foo =2;
key | foo | col
-+-+-
(0 rows)
Tracing session: e9751120-5879-11e7-be43-dfd5ff1ad595
activity
| timestamp
[jira] [Comment Edited] (CASSANDRA-13632) Digest mismatch if row is empty
[
https://issues.apache.org/jira/browse/CASSANDRA-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16061717#comment-16061717
]
Jay Zhuang edited comment on CASSANDRA-13632 at 6/24/17 1:15 AM:
-
Hi [~whangsf], could you please help me to reproduce the problem locally, here
is what I did:
{noformat}
CREATE KEYSPACE foo WITH replication = {'class': 'NetworkTopologyStrategy',
'dc1': '3' };
CREATE TABLE foo.foo ( key int, foo int, col int, PRIMARY KEY (key, foo) ) with
dclocal_read_repair_chance=0;
CONSISTENCY QUORUM;
INSERT INTO foo.foo (key, foo) VALUES ( 1,1);
TRACING ON;
SELECT * FROM foo.foo WHERE key = 1 and foo =2;
{noformat}
But Ino Repair event:
{noformat}
cqlsh> SELECT * FROM foo.foo WHERE key = 1 and foo =2;
key | foo | col
-+-+-
(0 rows)
Tracing session: e9751120-5879-11e7-be43-dfd5ff1ad595
activity
| timestamp | source| source_elapsed
--++---+
Execute
CQL3 query | 2017-06-24 01:10:28.914000 | 127.0.0.1 | 0
Parsing SELECT * FROM foo.foo WHERE key = 1 and foo =2;
[SharedPool-Worker-1] | 2017-06-24 01:10:28.914000 | 127.0.0.1 |220
Preparing statement
[SharedPool-Worker-1] | 2017-06-24 01:10:28.914000 | 127.0.0.1 |423
READ message received from /127.0.0.1
[MessagingService-Incoming-/127.0.0.1] | 2017-06-24 01:10:28.915000 | 127.0.0.2
| 40
reading digest from /127.0.0.2
[SharedPool-Worker-1] | 2017-06-24 01:10:28.915000 | 127.0.0.1 | 1115
Executing single-partition query on foo
[SharedPool-Worker-2] | 2017-06-24 01:10:28.915000 | 127.0.0.1 | 1123
Acquiring sstable references
[SharedPool-Worker-2] | 2017-06-24 01:10:28.915000 | 127.0.0.1 | 1189
Merging memtable contents
[SharedPool-Worker-2] | 2017-06-24 01:10:28.915000 | 127.0.0.1 | 1228
Sending READ message to /127.0.0.2
[MessagingService-Outgoing-/127.0.0.2] | 2017-06-24 01:10:28.915000 | 127.0.0.1
| 1326
Read 0 live and 0 tombstone cells
[SharedPool-Worker-2] | 2017-06-24 01:10:28.915000 | 127.0.0.1 | 1424
Executing single-partition query on foo
[SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 |221
REQUEST_RESPONSE message received from /127.0.0.2
[MessagingService-Incoming-/127.0.0.2] | 2017-06-24 01:10:28.916000 | 127.0.0.1
| 2817
Acquiring sstable references
[SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 |342
Merging memtable contents
[SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 |402
Read 0 live and 0 tombstone cells
[SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 |640
Read 0 live and 0 tombstone cells
[SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 |714
Enqueuing response to /127.0.0.1
[SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 |770
Sending REQUEST_RESPONSE message to /127.0.0.1
[MessagingService-Outgoing-/127.0.0.1] | 2017-06-24 01:10:28.916001 | 127.0.0.2
|917
Processing response from /127.0.0.2
[SharedPool-Worker-2] | 2017-06-24 01:10:28.917000 | 127.0.0.1 | 2916
Request complete | 2017-06-24 01:10:28.917133 | 127.0.0.1 | 3133
{noformat}
was (Author: jay.zhuang):
Hi [~whangsf], could you please help me to reproduce the problem locally, here
is what I did:
{noformat}
CREATE KEYSPACE foo WITH replication = {'class': 'NetworkTopologyStrategy',
'dc1': '3' };
CREATE TABLE foo.foo ( key int, foo int, col int, PRIMARY KEY (key, foo) ) with
dclocal_read_repair_chance=0;
CONSISTENCY QUORUM;
INSERT INTO foo.foo (key, foo) VALUES ( 1,1);
TRACING ON;
SELECT * FROM foo.foo WHERE key = 1 and foo =2;
{noformat}
But I don't see Repair event:
{noformat}
cqlsh> SELECT * FROM foo.foo WHERE key = 1 and foo =2;
key | foo | col
-+-+-
(0 rows)
Tracing session: e9751120-5879-11e7-be43-dfd5ff1ad595
activity
|
