Re: best practices for time-series data with massive amounts of records
Hi all, Thanks for the responses, this was very helpful. I don't know yet what the distribution of clicks and users will be, but I expect to see a few users with an enormous amount of interactions and most users having very few. The idea of doing some additional manual partitioning, and then maintaining another table that contains the head partition for each user makes sense, although it would add additional latency when we want to get say the most recent 1000 interactions for a given user (which is something that we have to do sometimes for applications with tight SLAs). FWIW I doubt that any users will have so many interactions that they exceed what we could reasonably put in a row, but I wanted to have a strategy to deal with this. Having a nice design pattern in Cassandra for maintaining a row with the N-most-recent interactions would also solve this reasonably well, but I don't know of any way to implement that without running batch jobs that periodically clean out data (which might be okay). Best regards, Clint On Tue, Mar 3, 2015 at 8:10 AM, mck m...@apache.org wrote: Here partition is a random digit from 0 to (N*M) where N=nodes in cluster, and M=arbitrary number. Hopefully it was obvious, but here (unless you've got hot partitions), you don't need N. ~mck
timeout when using secondary index
Hi,
Ran into RPC timeout exception when execution a query that involve
secondary index of a Boolean column when for example the company has more
than 1k person.
select * from company where company_id= and isMale = true;
such extreme low cardinality of secondary index like the other docs
stated, will result in basically 2 large row those values. However, I
thought since I also bounded the query with my primary partition key, won't
that be first consulted and then further narrow down the result and be
efficient?
Also, if I simply do
select * from company where company_id= ;
(without the AND clause on secondary index, it return right away)
Or mayb Cassandra server internal always parsing the secondary index result
first?
thanks
I have a simple table
create table company {
company_id uuid,
person_id uuid,
isMale Boolean,
PRIMARY KEY (company_id, person_id)
)
Re: Upgrade from 2.0.9 to 2.1.3
Note for anyone who accidentally or otherwise ends up with 2.1.3 in a situation they cannot downgrade, feel free to look at https://github.com/vast-engineering/cassandra/tree/vast-cassandra-2.1.3 https://github.com/vast-engineering/cassandra/tree/vast-cassandra-2.1.3 We sometimes make custom versions incorporating as many important patches as we reasonably can that we need to run a newer C* environment successfully. Obviously use at your own risk blah blah… basically install procedure would be to replace the main cassandra jar on a 2.1.3 node while it is down. On Mar 6, 2015, at 3:15 PM, Robert Coli rc...@eventbrite.com wrote: On Fri, Mar 6, 2015 at 6:25 AM, graham sanderson gra...@vast.com mailto:gra...@vast.com wrote: I would definitely wait for at least 2.1.4 +1 https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ =Rob smime.p7s Description: S/MIME cryptographic signature
Re: best practices for time-series data with massive amounts of records
Note that using static column(s) for the “head” value, and trailing TTLed values behind is something we’re considering. Note this is especially nice if your head state includes say a map which is updated by small deltas (individual keys) We have not yet studied the effect of static columns on say DTCS On Mar 6, 2015, at 4:42 PM, Clint Kelly clint.ke...@gmail.com wrote: Hi all, Thanks for the responses, this was very helpful. I don't know yet what the distribution of clicks and users will be, but I expect to see a few users with an enormous amount of interactions and most users having very few. The idea of doing some additional manual partitioning, and then maintaining another table that contains the head partition for each user makes sense, although it would add additional latency when we want to get say the most recent 1000 interactions for a given user (which is something that we have to do sometimes for applications with tight SLAs). FWIW I doubt that any users will have so many interactions that they exceed what we could reasonably put in a row, but I wanted to have a strategy to deal with this. Having a nice design pattern in Cassandra for maintaining a row with the N-most-recent interactions would also solve this reasonably well, but I don't know of any way to implement that without running batch jobs that periodically clean out data (which might be okay). Best regards, Clint On Tue, Mar 3, 2015 at 8:10 AM, mck m...@apache.org mailto:m...@apache.org wrote: Here partition is a random digit from 0 to (N*M) where N=nodes in cluster, and M=arbitrary number. Hopefully it was obvious, but here (unless you've got hot partitions), you don't need N. ~mck smime.p7s Description: S/MIME cryptographic signature
Re: Upgrade from 2.0.9 to 2.1.3
I would definitely wait for at least 2.1.4 On Mar 6, 2015, at 8:13 AM, Fredrik Larsson Stigbäck fredrik.l.stigb...@sitevision.se wrote: So no upgradeSSTables are required? /Fredrik 6 mar 2015 kl. 15:11 skrev Carlos Rolo r...@pythian.com mailto:r...@pythian.com: I would not recommend an upgrade to 2.1.x for now. Do you have any specific reason to upgrade? For upgrading from 2.0.9 you can just do a direct upgrade. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo Tel: 1649 www.pythian.com http://www.pythian.com/ On Fri, Mar 6, 2015 at 3:03 PM, Fredrik Larsson Stigbäck fredrik.l.stigb...@sitevision.se mailto:fredrik.l.stigb...@sitevision.se wrote: What’s the recommended way of upgrading from 2.0.9 to 2.1.3? Is upgradeSSTables required? According to http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeC_c.html http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeC_c.html it should be possible to just start the on 2.1.3 directly after 2.0.9. Regards Fredrik -- smime.p7s Description: S/MIME cryptographic signature
Upgrade from 2.0.9 to 2.1.3
What’s the recommended way of upgrading from 2.0.9 to 2.1.3? Is upgradeSSTables required? According to http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeC_c.html http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeC_c.html it should be possible to just start the on 2.1.3 directly after 2.0.9. Regards Fredrik
Re: Upgrade from 2.0.9 to 2.1.3
I would not recommend an upgrade to 2.1.x for now. Do you have any specific reason to upgrade? For upgrading from 2.0.9 you can just do a direct upgrade. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo* Tel: 1649 www.pythian.com On Fri, Mar 6, 2015 at 3:03 PM, Fredrik Larsson Stigbäck fredrik.l.stigb...@sitevision.se wrote: What’s the recommended way of upgrading from 2.0.9 to 2.1.3? Is upgradeSSTables required? According to http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeC_c.html it should be possible to just start the on 2.1.3 directly after 2.0.9. Regards Fredrik -- --
Re: Upgrade from 2.0.9 to 2.1.3
So no upgradeSSTables are required? /Fredrik 6 mar 2015 kl. 15:11 skrev Carlos Rolo r...@pythian.com: I would not recommend an upgrade to 2.1.x for now. Do you have any specific reason to upgrade? For upgrading from 2.0.9 you can just do a direct upgrade. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo Tel: 1649 www.pythian.com http://www.pythian.com/ On Fri, Mar 6, 2015 at 3:03 PM, Fredrik Larsson Stigbäck fredrik.l.stigb...@sitevision.se mailto:fredrik.l.stigb...@sitevision.se wrote: What’s the recommended way of upgrading from 2.0.9 to 2.1.3? Is upgradeSSTables required? According to http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeC_c.html http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeC_c.html it should be possible to just start the on 2.1.3 directly after 2.0.9. Regards Fredrik --
Downgrade Cassandra from 2.1.x to 2.0.x
Hi there, What is the best way to downgrade a C* 2.1.3 cluster to the stable 2.0.12? I know it's not supported, but we are getting too many issues with the 2.1.x... It is leading us to think that the best solution is to use the stable version. Is there a safe way to do that? Cheers, Roni
Re: Downgrade Cassandra from 2.1.x to 2.0.x
AFAIK downgrading is not officially supported. how much data do you have? If all possible I would dump all my data out and bulk load them into the 2.0.x cluster. This is the only way I feel safe. On Mar 6, 2015 5:55 AM, Roni Balthazar ronibaltha...@gmail.com wrote: Hi there, What is the best way to downgrade a C* 2.1.3 cluster to the stable 2.0.12? I know it's not supported, but we are getting too many issues with the 2.1.x... It is leading us to think that the best solution is to use the stable version. Is there a safe way to do that? Cheers, Roni
Re: cassandra node jvm stall intermittently
well, StatusLogger.java started shown in cassandra system.log, MessagingService.java also shown some stage (e.g. read, mutation) dropped. It's strange it only happen in this node but this type of message does not shown in other node log file at the same time... Jason On Thu, Mar 5, 2015 at 4:26 AM, Jan cne...@yahoo.com wrote: HI Jason; Whats in the log files at the moment jstat shows 100%. What is the activity on the cluster the node at the specific point in time (reads/ writes/ joins etc) Jan/ On Wednesday, March 4, 2015 5:59 AM, Jason Wee peich...@gmail.com wrote: Hi, our cassandra node using java 7 update 72 and we ran jstat on one of the node, and notice some strange behaviour as indicated by output below. any idea why when eden space stay the same for few seconds like 100% and 18.02% for few seconds? we suspect such stalling cause timeout to our cluster. any idea what happened, what went wrong and what could cause this? $ jstat -gcutil 32276 1s 0.00 5.78 91.21 70.94 60.07 2657 73.437 40.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 40.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 40.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 40.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 40.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 40.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 40.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 40.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 40.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 40.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 40.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 40.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 40.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 40.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 40.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 40.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 40.056 73.493 0.00 4.65 29.66 71.00 60.07 2659 73.488 40.056 73.544 0.00 4.65 70.88 71.00 60.07 2659 73.488 40.056 73.544 0.00 4.65 71.58 71.00 60.07 2659 73.488 40.056 73.544 0.00 4.65 72.15 71.00 60.07 2659 73.488 40.056 73.544 0.00 4.65 72.33 71.00 60.07 2659 73.488 40.056 73.544 0.00 4.65 72.73 71.00 60.07 2659 73.488 40.056 73.544 0.00 4.65 73.20 71.00 60.07 2659 73.488 40.056 73.544 0.00 4.65 73.71 71.00 60.07 2659 73.488 40.056 73.544 0.00 4.65 73.84 71.00 60.07 2659 73.488 40.056 73.544 0.00 4.65 73.91 71.00 60.07 2659 73.488 40.056 73.544 0.00 4.65 74.18 71.00 60.07 2659 73.488 40.056 73.544 0.00 4.65 74.29 71.00 60.07 2659 73.488 40.056 73.544 0.00 4.65 74.29 71.00 60.07 2659 73.488 40.056 73.544 0.00 4.65 74.29 71.00 60.07 2659 73.488 40.056 73.544 0.00 4.65 74.29 71.00 60.07 2659 73.488 40.056 73.544 0.00 4.65 74.29 71.00 60.07 2659 73.488 40.056 73.544 0.00 4.65 74.29 71.00 60.07 2659 73.488 40.056 73.544 0.00 4.65 74.29 71.00 60.07 2659 73.488 40.056 73.544 0.00 5.43 12.64 71.09 60.07 2661 73.534 40.056 73.590 0.00 5.43 18.02 71.09 60.07 2661 73.534 40.056 73.590 0.00 5.43 18.02 71.09 60.07 2661 73.534 40.056 73.590 0.00 5.43 18.02 71.09 60.07 2661 73.534 40.056 73.590 0.00 5.43 18.02 71.09 60.07 2661 73.534 40.056 73.590 0.00 5.43 18.02 71.09 60.07 2661 73.534 40.056 73.590 0.00 5.43 18.02 71.09 60.07 2661 73.534 40.056 73.590 0.00 5.43 18.02 71.09 60.07 2661 73.534 40.056 73.590 0.00 5.43 18.02 71.09 60.07 2661 73.534 40.056 73.590 0.00 5.43 18.02 71.09 60.07 2661 73.534 40.056 73.590 0.00 5.43 18.02 71.09 60.07 2661 73.534 40.056 73.590 0.00 5.43 18.02 71.09 60.07 2661 73.534 40.056 73.590 0.00 5.43 18.02 71.09 60.07 2661 73.534 40.056 73.590 0.00 5.43 18.02 71.09 60.07 2661 73.534 40.056 73.590 0.00 5.43 18.02 71.09 60.07 2661 73.534 40.056 73.590 0.00 5.43 18.02 71.09 60.07 2661 73.534 40.056 73.590 0.00 5.43 18.02 71.09 60.07 2661 73.534 40.056 73.590 0.00 5.43 18.02 71.09 60.07 2661 73.534
Re: Write timeout under load but Read is fine
I am using QUORUM CQL No SSDs (Anyway my IOPs is quite low so I dont think so it matters) No compaction is running when I receive timeout On Fri, Mar 6, 2015 at 12:35 AM, Carlos Rolo r...@pythian.com wrote: What is the consistency level you are using? Are you using Thrift or CQL? Are you using SSDs? Check if compactions are running when you get the timeouts. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo* Tel: 1649 www.pythian.com On Thu, Mar 5, 2015 at 9:51 PM, Jan cne...@yahoo.com wrote: Hello Jaydeep; Run *cassandra-stress* with R/W options enabled for about the same time and check if you have dropped packets. It would eliminate the client as the source of the error also give you a replicable tool to base subsequent tests/ findings. Jan/ On Thursday, March 5, 2015 12:19 PM, Jaydeep Chovatia chovatia.jayd...@gmail.com wrote: I have tried increasing timeout to 1 but no help. Also verified that there is no network lost packets. Jaydeep On Wed, Mar 4, 2015 at 12:19 PM, Jan cne...@yahoo.com wrote: HI Jaydeep; - look at the i/o on all three nodes - Increase the write_request_timeout_in_ms: 1 - check the time-outs if any on the client inserting the Writes - check the Network for dropped/lost packets hope this helps Jan/ On Wednesday, March 4, 2015 12:26 PM, Jaydeep Chovatia chovatia.jayd...@gmail.com wrote: Hi, In my test program when I increase load then I keep getting few write timeout from Cassandra say every 10~15 mins. My read:write ratio is 50:50. My reads are fine but only writes time out. Here is my Cassandra details: Version: 2.0.11 Ring of 3 nodes with RF=3 Node configuration: 24 core + 64GB RAM + 2TB write_request_timeout_in_ms: 5000, rest of Cassandra.yaml configuration is default I've also checked IO on Cassandra nodes and looks very low (around 5%). I've also checked Cassandra log file and do not see any GC happening. Also CPU on Cassandra is low (around 20%). I have 20GB data on each node. My test program creates connection to all three Cassandra nodes and sends read+write request randomly. Any idea what should I look for? Jaydeep -- Jaydeep -- -- Jaydeep
Re: Downgrade Cassandra from 2.1.x to 2.0.x
Hi Roni, We had the same problem and had to build a new, separate cluster. We read from both clusters in the beginning and copied the data in the background by paging over our data in the old 2.1.3 cluster. This allowed us to make the downgrade seemless. Just watch out for 2.0.12's bug with nodetool cleanup and make sure you build the cluster with enough nodes so you don't need to add another node before version 2.0.13 gets released. Best, Tobias On Fri, Mar 6, 2015 at 7:59 AM, Kai Wang dep...@gmail.com wrote: AFAIK downgrading is not officially supported. how much data do you have? If all possible I would dump all my data out and bulk load them into the 2.0.x cluster. This is the only way I feel safe. On Mar 6, 2015 5:55 AM, Roni Balthazar ronibaltha...@gmail.com wrote: Hi there, What is the best way to downgrade a C* 2.1.3 cluster to the stable 2.0.12? I know it's not supported, but we are getting too many issues with the 2.1.x... It is leading us to think that the best solution is to use the stable version. Is there a safe way to do that? Cheers, Roni
Run witch repair cmd when increase replication factor
I want fo increase replication factor in my C* 2.1.3 cluster(rf chang from 2 to 3 for some keyspaces). I read the doc of Updating the replication factor http://www.datastax.com/documentation/cql/3.1/cql/cql_using/update_ks_rf_t.html . The step two is run the nodetool repair.But as I know nodetool repair default is full repair,seems againt to the step three.So run witch repair cmd when increase replication factor? Thanks all. -- Ranger Tsao
Re: Upgrade from 2.0.9 to 2.1.3
On Fri, Mar 6, 2015 at 6:25 AM, graham sanderson gra...@vast.com wrote: I would definitely wait for at least 2.1.4 +1 https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ =Rob
Re: cassandra node jvm stall intermittently
HI Jason; The single node showing the anomaly is a hint that the problem is probably local to a node (as you suspected). - How many nodes do you have on the ring ? - What is the activity when this occurs - reads / writes/ compactions ? - Is there anything that is unique about this node that makes it different from the other nodes ? - Is this a periodic occurance OR a single occurence - I am trying to determine a pattern about when this shows up. - What is the load distribution the ring (ie: is this node carrying more load than the others). The system.log should have more info., about it. hope this helpsJan/ On Friday, March 6, 2015 4:50 AM, Jason Wee peich...@gmail.com wrote: well, StatusLogger.java started shown in cassandra system.log, MessagingService.java also shown some stage (e.g. read, mutation) dropped. It's strange it only happen in this node but this type of message does not shown in other node log file at the same time... Jason On Thu, Mar 5, 2015 at 4:26 AM, Jan cne...@yahoo.com wrote: HI Jason; Whats in the log files at the moment jstat shows 100%. What is the activity on the cluster the node at the specific point in time (reads/ writes/ joins etc) Jan/ On Wednesday, March 4, 2015 5:59 AM, Jason Wee peich...@gmail.com wrote: Hi, our cassandra node using java 7 update 72 and we ran jstat on one of the node, and notice some strange behaviour as indicated by output below. any idea why when eden space stay the same for few seconds like 100% and 18.02% for few seconds? we suspect such stalling cause timeout to our cluster. any idea what happened, what went wrong and what could cause this? $ jstat -gcutil 32276 1s 0.00 5.78 91.21 70.94 60.07 2657 73.437 4 0.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 0.00 5.78 100.00 70.94 60.07 2657 73.437 4 0.056 73.493 0.00 4.65 29.66 71.00 60.07 2659 73.488 4 0.056 73.544 0.00 4.65 70.88 71.00 60.07 2659 73.488 4 0.056 73.544 0.00 4.65 71.58 71.00 60.07 2659 73.488 4 0.056 73.544 0.00 4.65 72.15 71.00 60.07 2659 73.488 4 0.056 73.544 0.00 4.65 72.33 71.00 60.07 2659 73.488 4 0.056 73.544 0.00 4.65 72.73 71.00 60.07 2659 73.488 4 0.056 73.544 0.00 4.65 73.20 71.00 60.07 2659 73.488 4 0.056 73.544 0.00 4.65 73.71 71.00 60.07 2659 73.488 4 0.056 73.544 0.00 4.65 73.84 71.00 60.07 2659 73.488 4 0.056 73.544 0.00 4.65 73.91 71.00 60.07 2659 73.488 4 0.056 73.544 0.00 4.65 74.18 71.00 60.07 2659 73.488 4 0.056 73.544 0.00 4.65 74.29 71.00 60.07 2659 73.488 4 0.056 73.544 0.00 4.65 74.29 71.00 60.07 2659 73.488 4 0.056 73.544 0.00 4.65 74.29 71.00 60.07 2659 73.488 4 0.056 73.544 0.00 4.65 74.29 71.00 60.07 2659 73.488 4 0.056 73.544 0.00 4.65 74.29 71.00 60.07 2659 73.488 4 0.056 73.544 0.00 4.65 74.29 71.00 60.07 2659 73.488 4 0.056 73.544 0.00 4.65 74.29 71.00 60.07 2659 73.488 4 0.056 73.544 0.00 5.43 12.64 71.09 60.07 2661 73.534 4 0.056 73.590 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 0.00 5.43 18.02 71.09 60.07 2661 73.534 4 0.056 73.590 0.00 5.43
Re: Write timeout under load but Read is fine
What is the consistency level you are using? Are you using Thrift or CQL? Are you using SSDs? Check if compactions are running when you get the timeouts. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo* Tel: 1649 www.pythian.com On Thu, Mar 5, 2015 at 9:51 PM, Jan cne...@yahoo.com wrote: Hello Jaydeep; Run *cassandra-stress* with R/W options enabled for about the same time and check if you have dropped packets. It would eliminate the client as the source of the error also give you a replicable tool to base subsequent tests/ findings. Jan/ On Thursday, March 5, 2015 12:19 PM, Jaydeep Chovatia chovatia.jayd...@gmail.com wrote: I have tried increasing timeout to 1 but no help. Also verified that there is no network lost packets. Jaydeep On Wed, Mar 4, 2015 at 12:19 PM, Jan cne...@yahoo.com wrote: HI Jaydeep; - look at the i/o on all three nodes - Increase the write_request_timeout_in_ms: 1 - check the time-outs if any on the client inserting the Writes - check the Network for dropped/lost packets hope this helps Jan/ On Wednesday, March 4, 2015 12:26 PM, Jaydeep Chovatia chovatia.jayd...@gmail.com wrote: Hi, In my test program when I increase load then I keep getting few write timeout from Cassandra say every 10~15 mins. My read:write ratio is 50:50. My reads are fine but only writes time out. Here is my Cassandra details: Version: 2.0.11 Ring of 3 nodes with RF=3 Node configuration: 24 core + 64GB RAM + 2TB write_request_timeout_in_ms: 5000, rest of Cassandra.yaml configuration is default I've also checked IO on Cassandra nodes and looks very low (around 5%). I've also checked Cassandra log file and do not see any GC happening. Also CPU on Cassandra is low (around 20%). I have 20GB data on each node. My test program creates connection to all three Cassandra nodes and sends read+write request randomly. Any idea what should I look for? Jaydeep -- Jaydeep -- --
