Hi We are running a Cassandra 1.0.7, 2 node setup on EC2 using ephemeral disks for storage. Debugging inconsistent dara between 2 nodes we saw logging below. Following the counter CF design, I assume this means the READ step reconciliating the memtable value with the SSTables on the replica receiving the increment is failing. To my understanding client will not be notified because this read step is asynchronously. Also the other replica cannot be updated properly (replicate-on-write stage is backing up). Question is: - Will the counter value eventually become correct? - Read repair is turned off for the moment, but would this (not reconciliated) memtable value be merged correctly if read repair would be enabled? - What with the final value after running nodetool repairs or compaction process...
Load shouldn't be that high on this trial setup but we need to investigate more before concluding. Txs, David INFO [ScheduledTasks:1] 2012-11-28 16:58:28,924 MessagingService.java (line 607) 1175 READ messages dropped in last 5000ms INFO [ScheduledTasks:1] 2012-11-28 16:58:28,924 StatusLogger.java (line 50) Pool Name Active Pending Blocked INFO [ScheduledTasks:1] 2012-11-28 16:58:28,925 StatusLogger.java (line 65) ReadStage 0 0 0 INFO [ScheduledTasks:1] 2012-11-28 16:58:28,925 StatusLogger.java (line 65) RequestResponseStage 0 0 0 INFO [ScheduledTasks:1] 2012-11-28 16:58:28,925 StatusLogger.java (line 65) ReadRepairStage 0 0 0 INFO [ScheduledTasks:1] 2012-11-28 16:58:28,926 StatusLogger.java (line 65) MutationStage 0 0 0 INFO [ScheduledTasks:1] 2012-11-28 16:58:28,926 StatusLogger.java (line 65) ReplicateOnWriteStage 32 106 0 INFO [ScheduledTasks:1] 2012-11-28 16:58:28,926 StatusLogger.java (line 65) GossipStage 0 0 0 INFO [ScheduledTasks:1] 2012-11-28 16:58:28,926 StatusLogger.java (line 65) AntiEntropyStage 0 0 0 INFO [ScheduledTasks:1] 2012-11-28 16:58:28,927 StatusLogger.java (line 65) MigrationStage 0 0 0 INFO [ScheduledTasks:1] 2012-11-28 16:58:28,927 StatusLogger.java (line 65) StreamStage 0 0 0 INFO [ScheduledTasks:1] 2012-11-28 16:58:28,927 StatusLogger.java (line 65) MemtablePostFlusher 0 0 0 INFO [ScheduledTasks:1] 2012-11-28 16:58:28,928 StatusLogger.java (line 65) FlushWriter 0 0 0 INFO [ScheduledTasks:1] 2012-11-28 16:58:28,928 StatusLogger.java (line 65) MiscStage 0 0 0 INFO [ScheduledTasks:1] 2012-11-28 16:58:28,928 StatusLogger.java (line 65) InternalResponseStage 0 0 0 INFO [ScheduledTasks:1] 2012-11-28 16:58:28,928 StatusLogger.java (line 65) AntiEntropySessions 0 0 0 INFO [ScheduledTasks:1] 2012-11-28 16:58:28,929 StatusLogger.java (line 65) HintedHandoff 0 0 0 INFO [ScheduledTasks:1] 2012-11-28 16:58:28,929 StatusLogger.java (line 69) CompactionManager n/a 2 INFO [ScheduledTasks:1] 2012-11-28 16:58:28,929 StatusLogger.java (line 81) MessagingService n/a 0,0
