> I increased the size of the cluster also the concurrent_writes parameter. > Still there is a node which keeps on dropping the mutation messages. Ensure all the nodes have the same spec, and the nodes have the same config. In a virtual environment consider moving the node.
> Is this due to some improper load balancing? What does nodetool ring say and what sort of queries (and RF and CL) are you sending. Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 6/03/2012, at 3:58 AM, Tiwari, Dushyant wrote: > Hey Aaron, > > I increased the size of the cluster also the concurrent_writes parameter. > Still there is a node which keeps on dropping the mutation messages. The > other nodes are not dropping mutation messages. I am using Hector API and had > done nothing for load balancing so far. Just provided the host:port of the > nodes in the Cassandrahostconfig. Is this due to some improper load > balancing? Also the physical host where the node is hosted is relatively > heavier than other nodes’ host. What can I do to improve? > PS: The node is seed of the cluster. > > Thanks, > Dushyant > > From: aaron morton [mailto:aa...@thelastpickle.com] > Sent: Monday, March 05, 2012 4:15 PM > To: user@cassandra.apache.org > Subject: Re: Mutation Dropped Messages > > 1. Which parameters to tune in the config files? – Especially looking > for heavy writes > The node is overloaded. It may be because there are no enough nodes, or the > node is under temporary stress such as GC or repair. > If you have spare IO / CPU capacity you could increase the current_writes to > increase throughput on the write stage. You then need to ensure the commit > log and, to a lesser degree, the data volumes can keep up. > > 2. What is the difference between TimedOutException and silently > dropping mutation messages while operating on a CL of QUORUM. > TimedOutExceptions means CL nodes did not respond to the coordinator before > rpc_timeout. Dropping messages happens when a message is removed from the > queue in the a thread pool after rpc_timeout has occurred. it is a feature of > the architecture, and correct behaviour under stress. > Inconsistencies created by dropped messages are repaired via reads as high > CL, HH (in 1.+), Read Repair or Anti Entropy. > > Cheers > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 5/03/2012, at 11:32 PM, Tiwari, Dushyant wrote: > > > Hi All, > > While benchmarking Cassandra I found “Mutation Dropped” messages in the logs. > Now I know this is a good old question. It will be really great if someone > can provide a check list to recover when such a thing happens. I am looking > for answers of the following questions - > > 1. Which parameters to tune in the config files? – Especially looking > for heavy writes > 2. What is the difference between TimedOutException and silently > dropping mutation messages while operating on a CL of QUORUM. > > > Regards, > Dushyant > NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions > or views contained herein are not intended to be, and do not constitute, > advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform > and Consumer Protection Act. If you have received this communication in > error, please destroy all electronic and paper copies and notify the sender > immediately. Mistransmission is not intended to waive confidentiality or > privilege. Morgan Stanley reserves the right, to the extent permitted under > applicable law, to monitor electronic communications. This message is subject > to terms available at the following link: > http://www.morganstanley.com/disclaimers. If you cannot access these links, > please notify us by reply message and we will send the contents to you. By > messaging with Morgan Stanley you consent to the foregoing. > > NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions > or views contained herein are not intended to be, and do not constitute, > advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform > and Consumer Protection Act. If you have received this communication in > error, please destroy all electronic and paper copies and notify the sender > immediately. Mistransmission is not intended to waive confidentiality or > privilege. Morgan Stanley reserves the right, to the extent permitted under > applicable law, to monitor electronic communications. This message is subject > to terms available at the following > link:http://www.morganstanley.com/disclaimers. If you cannot access these > links, please notify us by reply message and we will send the contents to > you. By messaging with Morgan Stanley you consent to the foregoing.
