Mutation dropped and Read-Repair performance issue

2020-12-19 Thread sunil pawar
Hi All, We are facing problems of failure of Read-Repair stages with error Digest Mismatch and count is 300+ per day per node. At the same time, we are experiencing node is getting overloaded for a quick couple of seconds due to long GC pauses (of around 7-8 seconds). We are not running a repair

Re: Mutation dropped

2013-02-23 Thread Víctor Hugo Oliveira Molinar
: Mutation dropped What does rpc_timeout control? Only the reads/writes? Yes. like data stream, streaming_socket_timeout_in_ms in the yaml merkle tree request? Either no time out or a number of days, cannot remember which right now. What is the side effect if it's set to a really small number

Re: Mutation dropped

2013-02-22 Thread aaron morton
under 3ms. Any suggestion is very much appreciated. Thanks. -Wei - Original Message - From: aaron morton aa...@thelastpickle.com To: Cassandra User user@cassandra.apache.org Sent: Thursday, February 21, 2013 9:20:49 AM Subject: Re: Mutation dropped What does rpc_timeout

Re: Mutation dropped

2013-02-21 Thread aaron morton
To: user@cassandra.apache.org Sent: Tuesday, February 19, 2013 7:32 PM Subject: Re: Mutation dropped Does the rpc_timeout not control the client timeout ? No it is how long a node will wait for a response from other nodes before raising a TimedOutException if less than CL nodes have

Re: Mutation dropped

2013-02-21 Thread Wei Zhu
is a bit high. Our 50th percentile was great under 3ms. Any suggestion is very much appreciated. Thanks. -Wei - Original Message - From: aaron morton aa...@thelastpickle.com To: Cassandra User user@cassandra.apache.org Sent: Thursday, February 21, 2013 9:20:49 AM Subject: Re: Mutation dropped

Re: Mutation dropped

2013-02-20 Thread Wei Zhu
, say 20ms? Thanks. -Wei From: aaron morton aa...@thelastpickle.com To: user@cassandra.apache.org Sent: Tuesday, February 19, 2013 7:32 PM Subject: Re: Mutation dropped Does the rpc_timeout not control the client timeout ?No it is how long a node will wait

Re: Mutation dropped

2013-02-19 Thread aaron morton
UN 10.x.x.x 880.77 GB 256 100.0% 95d59054-be99-455f-90d1-f43981d3d778 rack1 Once we hit a very high TPS (around 50k/sec of inserts), the nodes start falling behind and we see the mutation dropped messages. But there are no failures on the client. Does that mean

RE: Mutation dropped

2013-02-18 Thread Kanwar Sangha
...@thelastpickle.com] Sent: 17 February 2013 11:26 To: user@cassandra.apache.org Subject: Re: Mutation dropped You are hitting the maximum throughput on the cluster. The messages are dropped because the node fails to start processing them before rpc_timeout. However the request is still a success because

Re: Mutation dropped

2013-02-17 Thread aaron morton
100.0% 95d59054-be99-455f-90d1-f43981d3d778 rack1 Once we hit a very high TPS (around 50k/sec of inserts), the nodes start falling behind and we see the mutation dropped messages. But there are no failures on the client. Does that mean other node is not able to persist

Mutation dropped

2013-02-14 Thread Kanwar Sangha
Hi - I am doing a load test using YCSB across 2 nodes in a cluster and seeing a lot of mutation dropped messages. I understand that this is due to the replica not being written to the other node ? RF = 2, CL =1. From the wiki - For MUTATION messages this means that the mutation was not applied

RE: Mutation dropped

2013-02-14 Thread Kanwar Sangha
:08 To: user@cassandra.apache.org Subject: Mutation dropped Hi - I am doing a load test using YCSB across 2 nodes in a cluster and seeing a lot of mutation dropped messages. I understand that this is due to the replica not being written to the other node ? RF = 2, CL =1. From the wiki

RE: Mutation Dropped Messages

2012-03-06 Thread Tiwari, Dushyant
Subject: Re: Mutation Dropped Messages I increased the size of the cluster also the concurrent_writes parameter. Still there is a node which keeps on dropping the mutation messages. Ensure all the nodes have the same spec, and the nodes have the same config. In a virtual environment consider moving

Re: Mutation Dropped Messages

2012-03-06 Thread aaron morton
@cassandra.apache.org Subject: Re: Mutation Dropped Messages I increased the size of the cluster also the concurrent_writes parameter. Still there is a node which keeps on dropping the mutation messages. Ensure all the nodes have the same spec, and the nodes have the same config. In a virtual environment

Mutation Dropped Messages

2012-03-05 Thread Tiwari, Dushyant
Hi All, While benchmarking Cassandra I found Mutation Dropped messages in the logs. Now I know this is a good old question. It will be really great if someone can provide a check list to recover when such a thing happens. I am looking for answers of the following questions - 1. Which

Re: Mutation Dropped Messages

2012-03-05 Thread aaron morton
“Mutation Dropped” messages in the logs. Now I know this is a good old question. It will be really great if someone can provide a check list to recover when such a thing happens. I am looking for answers of the following questions - 1. Which parameters to tune in the config files

Re: Mutation Dropped Messages

2012-03-05 Thread aaron morton
...@thelastpickle.com] Sent: Monday, March 05, 2012 4:15 PM To: user@cassandra.apache.org Subject: Re: Mutation Dropped Messages 1. Which parameters to tune in the config files? – Especially looking for heavy writes The node is overloaded. It may be because there are no enough nodes