[ https://issues.apache.org/jira/browse/CASSANDRA-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16061717#comment-16061717 ]
Jay Zhuang edited comment on CASSANDRA-13632 at 6/24/17 2:24 AM: ----------------------------------------------------------------- Hi [~whangsf], could you please help me to reproduce the problem locally, here is what I did: {noformat} CREATE KEYSPACE foo WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': '3' }; CREATE TABLE foo.foo ( key int, foo int, col int, PRIMARY KEY (key, foo) ) with dclocal_read_repair_chance=0; CONSISTENCY QUORUM; INSERT INTO foo.foo (key, foo) VALUES ( 1,1); TRACING ON; SELECT key FROM foo.foo WHERE key = 1 and foo =2; {noformat} But no Repair event: {noformat} cqlsh> SELECT key FROM foo.foo WHERE key = 1 and foo =2; key ----- (0 rows) Tracing session: 2cb5aa80-5884-11e7-be43-dfd5ff1ad595 activity | timestamp | source | source_elapsed ------------------------------------------------------------------------------------------+----------------------------+-----------+---------------- Execute CQL3 query | 2017-06-24 02:23:56.712000 | 127.0.0.1 | 0 Parsing SELECT key FROM foo.foo WHERE key = 1 and foo =2; [SharedPool-Worker-1] | 2017-06-24 02:23:56.712000 | 127.0.0.1 | 187 Preparing statement [SharedPool-Worker-1] | 2017-06-24 02:23:56.713000 | 127.0.0.1 | 411 reading digest from /127.0.0.3 [SharedPool-Worker-1] | 2017-06-24 02:23:56.713000 | 127.0.0.1 | 976 Executing single-partition query on foo [SharedPool-Worker-2] | 2017-06-24 02:23:56.713000 | 127.0.0.1 | 1004 Acquiring sstable references [SharedPool-Worker-2] | 2017-06-24 02:23:56.713000 | 127.0.0.1 | 1057 Merging memtable contents [SharedPool-Worker-2] | 2017-06-24 02:23:56.713000 | 127.0.0.1 | 1083 Sending READ message to /127.0.0.3 [MessagingService-Outgoing-/127.0.0.3] | 2017-06-24 02:23:56.713000 | 127.0.0.1 | 1124 READ message received from /127.0.0.1 [MessagingService-Incoming-/127.0.0.1] | 2017-06-24 02:23:56.714000 | 127.0.0.3 | 21 Read 0 live and 0 tombstone cells [SharedPool-Worker-2] | 2017-06-24 02:23:56.714000 | 127.0.0.1 | 1249 Executing single-partition query on foo [SharedPool-Worker-1] | 2017-06-24 02:23:56.714000 | 127.0.0.3 | 145 REQUEST_RESPONSE message received from /127.0.0.3 [MessagingService-Incoming-/127.0.0.3] | 2017-06-24 02:23:56.714000 | 127.0.0.1 | 2198 Acquiring sstable references [SharedPool-Worker-1] | 2017-06-24 02:23:56.714000 | 127.0.0.3 | 214 Merging memtable contents [SharedPool-Worker-1] | 2017-06-24 02:23:56.714000 | 127.0.0.3 | 265 Read 0 live and 0 tombstone cells [SharedPool-Worker-1] | 2017-06-24 02:23:56.714000 | 127.0.0.3 | 466 Read 0 live and 0 tombstone cells [SharedPool-Worker-1] | 2017-06-24 02:23:56.714000 | 127.0.0.3 | 538 Enqueuing response to /127.0.0.1 [SharedPool-Worker-1] | 2017-06-24 02:23:56.714001 | 127.0.0.3 | 596 Sending REQUEST_RESPONSE message to /127.0.0.1 [MessagingService-Outgoing-/127.0.0.1] | 2017-06-24 02:23:56.714001 | 127.0.0.3 | 671 Processing response from /127.0.0.3 [SharedPool-Worker-2] | 2017-06-24 02:23:56.715000 | 127.0.0.1 | 2324 Request complete | 2017-06-24 02:23:56.714453 | 127.0.0.1 | 2453 {noformat} was (Author: jay.zhuang): Hi [~whangsf], could you please help me to reproduce the problem locally, here is what I did: {noformat} CREATE KEYSPACE foo WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': '3' }; CREATE TABLE foo.foo ( key int, foo int, col int, PRIMARY KEY (key, foo) ) with dclocal_read_repair_chance=0; CONSISTENCY QUORUM; INSERT INTO foo.foo (key, foo) VALUES ( 1,1); TRACING ON; SELECT * FROM foo.foo WHERE key = 1 and foo =2; {noformat} But no Repair event: {noformat} cqlsh> SELECT * FROM foo.foo WHERE key = 1 and foo =2; key | foo | col -----+-----+----- (0 rows) Tracing session: e9751120-5879-11e7-be43-dfd5ff1ad595 activity | timestamp | source | source_elapsed ------------------------------------------------------------------------------------------+----------------------------+-----------+---------------- Execute CQL3 query | 2017-06-24 01:10:28.914000 | 127.0.0.1 | 0 Parsing SELECT * FROM foo.foo WHERE key = 1 and foo =2; [SharedPool-Worker-1] | 2017-06-24 01:10:28.914000 | 127.0.0.1 | 220 Preparing statement [SharedPool-Worker-1] | 2017-06-24 01:10:28.914000 | 127.0.0.1 | 423 READ message received from /127.0.0.1 [MessagingService-Incoming-/127.0.0.1] | 2017-06-24 01:10:28.915000 | 127.0.0.2 | 40 reading digest from /127.0.0.2 [SharedPool-Worker-1] | 2017-06-24 01:10:28.915000 | 127.0.0.1 | 1115 Executing single-partition query on foo [SharedPool-Worker-2] | 2017-06-24 01:10:28.915000 | 127.0.0.1 | 1123 Acquiring sstable references [SharedPool-Worker-2] | 2017-06-24 01:10:28.915000 | 127.0.0.1 | 1189 Merging memtable contents [SharedPool-Worker-2] | 2017-06-24 01:10:28.915000 | 127.0.0.1 | 1228 Sending READ message to /127.0.0.2 [MessagingService-Outgoing-/127.0.0.2] | 2017-06-24 01:10:28.915000 | 127.0.0.1 | 1326 Read 0 live and 0 tombstone cells [SharedPool-Worker-2] | 2017-06-24 01:10:28.915000 | 127.0.0.1 | 1424 Executing single-partition query on foo [SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 | 221 REQUEST_RESPONSE message received from /127.0.0.2 [MessagingService-Incoming-/127.0.0.2] | 2017-06-24 01:10:28.916000 | 127.0.0.1 | 2817 Acquiring sstable references [SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 | 342 Merging memtable contents [SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 | 402 Read 0 live and 0 tombstone cells [SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 | 640 Read 0 live and 0 tombstone cells [SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 | 714 Enqueuing response to /127.0.0.1 [SharedPool-Worker-1] | 2017-06-24 01:10:28.916000 | 127.0.0.2 | 770 Sending REQUEST_RESPONSE message to /127.0.0.1 [MessagingService-Outgoing-/127.0.0.1] | 2017-06-24 01:10:28.916001 | 127.0.0.2 | 917 Processing response from /127.0.0.2 [SharedPool-Worker-2] | 2017-06-24 01:10:28.917000 | 127.0.0.1 | 2916 Request complete | 2017-06-24 01:10:28.917133 | 127.0.0.1 | 3133 {noformat} > Digest mismatch if row is empty > ------------------------------- > > Key: CASSANDRA-13632 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13632 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths > Reporter: Andrew Whang > Assignee: Andrew Whang > Fix For: 3.0.x > > > This issue is similar to CASSANDRA-12090. Quorum read queries that include a > column selector (non-wildcard) result in digest mismatch when the row is > empty (key does not exist). It seems the data serialization path checks if > rowIterator.isEmpty() and if so ignores column names (by setting IS_EMPTY > flag). However, the digest serialization path does not perform this check and > includes column names. The digest comparison results in a mismatch. The > mismatch does not end up issuing a read repair mutation since the underlying > data is the same. > The mismatch on the read path ends up doubling our p99 read latency. We > discovered this issue while testing a 2.2.5 to 3.0.13 upgrade. > One thing to note is that we're using thrift, which ends up handling the > ColumnFilter differently than the CQL path. > As with CASSANDRA-12090, fixing the digest seems sensible. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org