Hello, While under load, we have occasionally been seeing "messages dropped" errors in our cassandra log. Doing some research, I understand this is part of Cassandra's design to shed load, and we should look at the tpstats-like output to determine what should be done to resolve the situation. Typically, you will see lots of messages blocked or pending, and that might be an indicator that a specific part of hardware needs to be improved/tuned/upgraded.
However, looking at the output we are getting, I'm finding it difficult to see what needs to be tuned, as it looks to me cassandra is handling the load within the mutation stage: NFO [ScheduledTasks:1] 2012-09-17 06:28:03,266 MessagingService.java (line 658) 119 MUTATION messages dropped in last 5000ms INFO [ScheduledTasks:1] 2012-09-17 06:28:03,645 StatusLogger.java (line 57) Pool Name Active Pending Blocked INFO [ScheduledTasks:1] 2012-09-17 06:28:03,836 StatusLogger.java (line 72) ReadStage 3 3 0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,837 StatusLogger.java (line 72) RequestResponseStage 0 0 0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,837 StatusLogger.java (line 72) ReadRepairStage 0 0 0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,837 StatusLogger.java (line 72) MutationStage 0 0 0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,838 StatusLogger.java (line 72) ReplicateOnWriteStage 0 0 0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,838 StatusLogger.java (line 72) GossipStage 0 0 0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,839 StatusLogger.java (line 72) AntiEntropyStage 0 0 0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,839 StatusLogger.java (line 72) MigrationStage 0 0 0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,839 StatusLogger.java (line 72) StreamStage 0 0 0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,839 StatusLogger.java (line 72) MemtablePostFlusher 1 5 0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,840 StatusLogger.java (line 72) FlushWriter 1 5 0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,840 StatusLogger.java (line 72) MiscStage 0 0 0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,840 StatusLogger.java (line 72) commitlog_archiver 0 0 0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,841 StatusLogger.java (line 72) InternalResponseStage 0 0 0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,841 StatusLogger.java (line 72) AntiEntropySessions 0 0 0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,851 StatusLogger.java (line 72) HintedHandoff 0 0 0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,851 StatusLogger.java (line 77) CompactionManager 0 0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,852 StatusLogger.java (line 89) MessagingService n/a 0,0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,852 StatusLogger.java (line 99) Cache Type Size Capacity KeysToSave Provider INFO [ScheduledTasks:1] 2012-09-17 06:28:03,853 StatusLogger.java (line 100) KeyCache 2184533 2184533 all INFO [ScheduledTasks:1] 2012-09-17 06:28:03,853 StatusLogger.java (line 106) RowCache 0 0 all org.apache.cassandra.cache.SerializingCacheProvider INFO [ScheduledTasks:1] 2012-09-17 06:28:03,853 StatusLogger.java (line 113) ColumnFamily Memtable ops,data INFO [ScheduledTasks:1] 2012-09-17 06:28:03,853 StatusLogger.java (line 116) system.NodeIdInfo 0,0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,854 StatusLogger.java (line 116) system.IndexInfo 0,0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,854 StatusLogger.java (line 116) system.LocationInfo 0,0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,854 StatusLogger.java (line 116) system.Versions 0,0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,855 StatusLogger.java (line 116) system.schema_keyspaces 0,0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,855 StatusLogger.java (line 116) system.Migrations 0,0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,855 StatusLogger.java (line 116) system.schema_columnfamilies 0,0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,856 StatusLogger.java (line 116) system.schema_columns 0,0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,856 StatusLogger.java (line 116) system.HintsColumnFamily 0,0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,856 StatusLogger.java (line 116) system.Schema 0,0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,857 StatusLogger.java (line 116) prod.comp 0,0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,857 StatusLogger.java (line 116) prod.byp 224,61863 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,857 StatusLogger.java (line 116) prod.byn 30,6106 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,857 StatusLogger.java (line 116) prod.p 3354,3792954 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,858 StatusLogger.java (line 116) prod.rel 788,153294 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,858 StatusLogger.java (line 116) prod.i 46,271517 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,859 StatusLogger.java (line 116) prod.users 18489,5650405 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,859 StatusLogger.java (line 116) prod.caches 40,7512998 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,859 StatusLogger.java (line 116) prod.sessions 8085,13063121 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,859 StatusLogger.java (line 116) prod.content 0,0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,860 StatusLogger.java (line 116) prod.enr 436,341190 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,860 StatusLogger.java (line 116) prod.byt 23800,9880339 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,861 StatusLogger.java (line 116) prod.aliases 63603,10475769 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,861 StatusLogger.java (line 116) prod.ran 1500,329387 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,861 StatusLogger.java (line 116) prod.cmap 0,0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,861 StatusLogger.java (line 116) prod.pmap 73032,61237450 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,862 StatusLogger.java (line 116) prod.pic 5421,4438066 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,862 StatusLogger.java (line 116) prod.watchdog 0,0 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,863 StatusLogger.java (line 116) prod.s 31,40924 INFO [ScheduledTasks:1] 2012-09-17 06:28:03,863 StatusLogger.java (line 116) prod.m 21690,64262284 We are using a write consistency level of LOCAL_QUROUM over a six node cluster. Any observations are welcome, Thanks, -Mike
