This is a known issue, and we should perhaps open a JIRA on it. The original Dynamo approach was to have 3 mechanisms -- HintedHandoff, read-repair, and Merk trees to guarantee convergence (eventual consistency). Cassandra only has the first two. There are some corner cases where hinted-handoff alone can't be relied on to guarantee convergence which is why there's read-repair on every read.
Turning off read repair is a relatively simple (but risky) change to the code. However, minimizing unnecessary read repair is a lot trickier :) On Fri, Sep 25, 2009 at 5:39 PM, Edmond Lau <[email protected]> wrote: > I have a 3 node cluster with a replication factor of 2, running on 0.4 > RC1. I've set both my read and write consistency levels to use a > quorum. > > I'm observing that quorum reads keep invoking read repair and log > DigestMismatchExceptions from the StorageProxy. Obviously, this > significantly reduces my read throughput. In the absence of any > additional inserts, I'd expect that read repair would happen at most > once before the 2 nodes responsible for the data both have fresh views > of the data. > > Here's what I see in my debug log for one machine on two consecutive > quorum reads for the data. I get similar messages when querying any > of the 3 nodes. Similar messages are logged on subsequent queries for > the exact same row/column. The issue happens when reading both > supercolumns or columns. Restarting the cluster has no effect. > > DEBUG [pool-1-thread-1] 2009-09-26 00:26:20,317 CassandraServer.java > (line 305) multiget > DEBUG [pool-1-thread-1] 2009-09-26 00:26:20,360 StorageProxy.java > (line 375) strongread reading data for > SliceByNamesReadCommand(table='Analytics', key='test', > columnParent='QueryPath(columnFamilyName='Domain', > superColumnName='null', co\ > lumnName='null')', columns=[www.ooyala.com,]) from [email protected]:7000 > DEBUG [pool-1-thread-1] 2009-09-26 00:26:20,365 StorageProxy.java > (line 382) strongread reading digest for > SliceByNamesReadCommand(table='Analytics', key='test', > columnParent='QueryPath(columnFamilyName='Domain', > superColumnName='null', \ > columnName='null')', columns=[www.ooyala.com,]) from [email protected]:7000 > DEBUG [ROW-READ-STAGE:1] 2009-09-26 00:26:20,380 ReadVerbHandler.java > (line 100) Read key test; sending response to > [email protected]:7000 > DEBUG [RESPONSE-STAGE:1] 2009-09-26 00:26:20,387 > ResponseVerbHandler.java (line 34) Processing response on a callback > from [email protected]:7000 > DEBUG [RESPONSE-STAGE:2] 2009-09-26 00:26:20,449 > ResponseVerbHandler.java (line 34) Processing response on a callback > from [email protected]:7000 > DEBUG [pool-1-thread-1] 2009-09-26 00:26:20,474 > ReadResponseResolver.java (line 84) Response deserialization time : 0 > ms. > DEBUG [pool-1-thread-1] 2009-09-26 00:26:20,474 > ReadResponseResolver.java (line 84) Response deserialization time : 0 > ms. > INFO [pool-1-thread-1] 2009-09-26 00:26:20,475 StorageProxy.java > (line 411) DigestMismatchException: test > DEBUG [ROW-READ-STAGE:2] 2009-09-26 00:26:20,477 ReadVerbHandler.java > (line 100) Read key test; sending response to [email protected]:7000 > DEBUG [RESPONSE-STAGE:3] 2009-09-26 00:26:20,478 > ResponseVerbHandler.java (line 34) Processing response on a callback > from [email protected]:7000 > DEBUG [RESPONSE-STAGE:4] 2009-09-26 00:26:20,480 > ResponseVerbHandler.java (line 34) Processing response on a callback > from [email protected]:7000 > DEBUG [pool-1-thread-1] 2009-09-26 00:26:20,481 > ReadResponseResolver.java (line 84) Response deserialization time : 0 > ms. > DEBUG [pool-1-thread-1] 2009-09-26 00:26:20,481 > ReadResponseResolver.java (line 84) Response deserialization time : 0 > ms. > INFO [pool-1-thread-1] 2009-09-26 00:26:20,482 > ReadResponseResolver.java (line 148) resolve: 1 ms. > DEBUG [pool-1-thread-2] 2009-09-26 00:27:22,099 CassandraServer.java > (line 305) multiget > DEBUG [pool-1-thread-2] 2009-09-26 00:27:22,100 StorageProxy.java > (line 375) strongread reading data for > SliceByNamesReadCommand(table='Analytics', key='test', > columnParent='QueryPath(columnFamilyName='Domain', > superColumnName='null', co\ > lumnName='null')', columns=[www.ooyala.com,]) from [email protected]:7000 > DEBUG [pool-1-thread-2] 2009-09-26 00:27:22,100 StorageProxy.java > (line 382) strongread reading digest for > SliceByNamesReadCommand(table='Analytics', key='test', > columnParent='QueryPath(columnFamilyName='Domain', > superColumnName='null', \ > columnName='null')', columns=[www.ooyala.com,]) from [email protected]:7000 > DEBUG [ROW-READ-STAGE:1] 2009-09-26 00:27:22,103 ReadVerbHandler.java > (line 100) Read key test; sending response to > [email protected]:7000 > DEBUG [RESPONSE-STAGE:1] 2009-09-26 00:27:22,103 > ResponseVerbHandler.java (line 34) Processing response on a callback > from [email protected]:7000 > DEBUG [RESPONSE-STAGE:2] 2009-09-26 00:27:22,107 > ResponseVerbHandler.java (line 34) Processing response on a callback > from [email protected]:7000 > DEBUG [pool-1-thread-2] 2009-09-26 00:27:22,108 > ReadResponseResolver.java (line 84) Response deserialization time : 1 > ms. > DEBUG [pool-1-thread-2] 2009-09-26 00:27:22,108 > ReadResponseResolver.java (line 84) Response deserialization time : 0 > ms. > INFO [pool-1-thread-2] 2009-09-26 00:27:22,109 StorageProxy.java > (line 411) DigestMismatchException: test > DEBUG [ROW-READ-STAGE:2] 2009-09-26 00:27:22,114 ReadVerbHandler.java > (line 100) Read key test; sending response to [email protected]:7000 > DEBUG [RESPONSE-STAGE:3] 2009-09-26 00:27:22,114 > ResponseVerbHandler.java (line 34) Processing response on a callback > from [email protected]:7000 > DEBUG [RESPONSE-STAGE:4] 2009-09-26 00:27:22,205 > ResponseVerbHandler.java (line 34) Processing response on a callback > from [email protected]:7000 > DEBUG [pool-1-thread-2] 2009-09-26 00:27:22,206 > ReadResponseResolver.java (line 84) Response deserialization time : 0 > ms. > DEBUG [pool-1-thread-2] 2009-09-26 00:27:22,206 > ReadResponseResolver.java (line 84) Response deserialization time : 0 > ms. > INFO [pool-1-thread-2] 2009-09-26 00:27:22,207 > ReadResponseResolver.java (line 148) resolve: 1 ms. > > Thoughts? > > Edmond >
