Re: Write timeout under load but Read is fine
I am using QUORUM CQL No SSDs (Anyway my IOPs is quite low so I dont think so it matters) No compaction is running when I receive timeout On Fri, Mar 6, 2015 at 12:35 AM, Carlos Rolo r...@pythian.com wrote: What is the consistency level you are using? Are you using Thrift or CQL? Are you using SSDs? Check if compactions are running when you get the timeouts. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo* Tel: 1649 www.pythian.com On Thu, Mar 5, 2015 at 9:51 PM, Jan cne...@yahoo.com wrote: Hello Jaydeep; Run *cassandra-stress* with R/W options enabled for about the same time and check if you have dropped packets. It would eliminate the client as the source of the error also give you a replicable tool to base subsequent tests/ findings. Jan/ On Thursday, March 5, 2015 12:19 PM, Jaydeep Chovatia chovatia.jayd...@gmail.com wrote: I have tried increasing timeout to 1 but no help. Also verified that there is no network lost packets. Jaydeep On Wed, Mar 4, 2015 at 12:19 PM, Jan cne...@yahoo.com wrote: HI Jaydeep; - look at the i/o on all three nodes - Increase the write_request_timeout_in_ms: 1 - check the time-outs if any on the client inserting the Writes - check the Network for dropped/lost packets hope this helps Jan/ On Wednesday, March 4, 2015 12:26 PM, Jaydeep Chovatia chovatia.jayd...@gmail.com wrote: Hi, In my test program when I increase load then I keep getting few write timeout from Cassandra say every 10~15 mins. My read:write ratio is 50:50. My reads are fine but only writes time out. Here is my Cassandra details: Version: 2.0.11 Ring of 3 nodes with RF=3 Node configuration: 24 core + 64GB RAM + 2TB write_request_timeout_in_ms: 5000, rest of Cassandra.yaml configuration is default I've also checked IO on Cassandra nodes and looks very low (around 5%). I've also checked Cassandra log file and do not see any GC happening. Also CPU on Cassandra is low (around 20%). I have 20GB data on each node. My test program creates connection to all three Cassandra nodes and sends read+write request randomly. Any idea what should I look for? Jaydeep -- Jaydeep -- -- Jaydeep
Re: Write timeout under load but Read is fine
What is the consistency level you are using? Are you using Thrift or CQL? Are you using SSDs? Check if compactions are running when you get the timeouts. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo* Tel: 1649 www.pythian.com On Thu, Mar 5, 2015 at 9:51 PM, Jan cne...@yahoo.com wrote: Hello Jaydeep; Run *cassandra-stress* with R/W options enabled for about the same time and check if you have dropped packets. It would eliminate the client as the source of the error also give you a replicable tool to base subsequent tests/ findings. Jan/ On Thursday, March 5, 2015 12:19 PM, Jaydeep Chovatia chovatia.jayd...@gmail.com wrote: I have tried increasing timeout to 1 but no help. Also verified that there is no network lost packets. Jaydeep On Wed, Mar 4, 2015 at 12:19 PM, Jan cne...@yahoo.com wrote: HI Jaydeep; - look at the i/o on all three nodes - Increase the write_request_timeout_in_ms: 1 - check the time-outs if any on the client inserting the Writes - check the Network for dropped/lost packets hope this helps Jan/ On Wednesday, March 4, 2015 12:26 PM, Jaydeep Chovatia chovatia.jayd...@gmail.com wrote: Hi, In my test program when I increase load then I keep getting few write timeout from Cassandra say every 10~15 mins. My read:write ratio is 50:50. My reads are fine but only writes time out. Here is my Cassandra details: Version: 2.0.11 Ring of 3 nodes with RF=3 Node configuration: 24 core + 64GB RAM + 2TB write_request_timeout_in_ms: 5000, rest of Cassandra.yaml configuration is default I've also checked IO on Cassandra nodes and looks very low (around 5%). I've also checked Cassandra log file and do not see any GC happening. Also CPU on Cassandra is low (around 20%). I have 20GB data on each node. My test program creates connection to all three Cassandra nodes and sends read+write request randomly. Any idea what should I look for? Jaydeep -- Jaydeep -- --
Re: Write timeout under load but Read is fine
I have tried increasing timeout to 1 but no help. Also verified that there is no network lost packets. Jaydeep On Wed, Mar 4, 2015 at 12:19 PM, Jan cne...@yahoo.com wrote: HI Jaydeep; - look at the i/o on all three nodes - Increase the write_request_timeout_in_ms: 1 - check the time-outs if any on the client inserting the Writes - check the Network for dropped/lost packets hope this helps Jan/ On Wednesday, March 4, 2015 12:26 PM, Jaydeep Chovatia chovatia.jayd...@gmail.com wrote: Hi, In my test program when I increase load then I keep getting few write timeout from Cassandra say every 10~15 mins. My read:write ratio is 50:50. My reads are fine but only writes time out. Here is my Cassandra details: Version: 2.0.11 Ring of 3 nodes with RF=3 Node configuration: 24 core + 64GB RAM + 2TB write_request_timeout_in_ms: 5000, rest of Cassandra.yaml configuration is default I've also checked IO on Cassandra nodes and looks very low (around 5%). I've also checked Cassandra log file and do not see any GC happening. Also CPU on Cassandra is low (around 20%). I have 20GB data on each node. My test program creates connection to all three Cassandra nodes and sends read+write request randomly. Any idea what should I look for? Jaydeep -- Jaydeep
Re: Write timeout under load but Read is fine
Hello Jaydeep; Run cassandra-stress with R/W options enabled for about the same time and check if you have dropped packets. It would eliminate the client as the source of the error also give you a replicable tool to base subsequent tests/ findings. Jan/ On Thursday, March 5, 2015 12:19 PM, Jaydeep Chovatia chovatia.jayd...@gmail.com wrote: I have tried increasing timeout to 1 but no help. Also verified that there is no network lost packets. Jaydeep On Wed, Mar 4, 2015 at 12:19 PM, Jan cne...@yahoo.com wrote: HI Jaydeep; - look at the i/o on all three nodes - Increase the write_request_timeout_in_ms: 1 - check the time-outs if any on the client inserting the Writes - check the Network for dropped/lost packets hope this helpsJan/ On Wednesday, March 4, 2015 12:26 PM, Jaydeep Chovatia chovatia.jayd...@gmail.com wrote: Hi, In my test program when I increase load then I keep getting few write timeout from Cassandra say every 10~15 mins. My read:write ratio is 50:50. My reads are fine but only writes time out. Here is my Cassandra details:Version: 2.0.11 Ring of 3 nodes with RF=3Node configuration: 24 core + 64GB RAM + 2TB write_request_timeout_in_ms: 5000, rest of Cassandra.yaml configuration is default I've also checked IO on Cassandra nodes and looks very low (around 5%). I've also checked Cassandra log file and do not see any GC happening. Also CPU on Cassandra is low (around 20%). I have 20GB data on each node. My test program creates connection to all three Cassandra nodes and sends read+write request randomly. Any idea what should I look for? Jaydeep -- Jaydeep
Write timeout under load but Read is fine
Hi, In my test program when I increase load then I keep getting few write timeout from Cassandra say every 10~15 mins. My read:write ratio is 50:50. My reads are fine but only writes time out. Here is my Cassandra details: Version: 2.0.11 Ring of 3 nodes with RF=3 Node configuration: 24 core + 64GB RAM + 2TB write_request_timeout_in_ms: 5000, rest of Cassandra.yaml configuration is default I've also checked IO on Cassandra nodes and looks very low (around 5%). I've also checked Cassandra log file and do not see any GC happening. Also CPU on Cassandra is low (around 20%). I have 20GB data on each node. My test program creates connection to all three Cassandra nodes and sends read+write request randomly. Any idea what should I look for? Jaydeep
Re: Write timeout under load but Read is fine
HI Jaydeep; - look at the i/o on all three nodes - Increase the write_request_timeout_in_ms: 1 - check the time-outs if any on the client inserting the Writes - check the Network for dropped/lost packets hope this helpsJan/ On Wednesday, March 4, 2015 12:26 PM, Jaydeep Chovatia chovatia.jayd...@gmail.com wrote: Hi, In my test program when I increase load then I keep getting few write timeout from Cassandra say every 10~15 mins. My read:write ratio is 50:50. My reads are fine but only writes time out. Here is my Cassandra details:Version: 2.0.11 Ring of 3 nodes with RF=3Node configuration: 24 core + 64GB RAM + 2TB write_request_timeout_in_ms: 5000, rest of Cassandra.yaml configuration is default I've also checked IO on Cassandra nodes and looks very low (around 5%). I've also checked Cassandra log file and do not see any GC happening. Also CPU on Cassandra is low (around 20%). I have 20GB data on each node. My test program creates connection to all three Cassandra nodes and sends read+write request randomly. Any idea what should I look for? Jaydeep
