Re: Write timeout on other nodes when joing a new node (in new DC)
Hi all,
so after deep investigation, we found out that this is this problem
https://issues.apache.org/jira/browse/CASSANDRA-8058
Jiri Horky
On 10/20/2015 12:00 PM, Jiri Horky wrote:
> Hi all,
>
> we are experiencing a strange behavior when we are trying to bootstrap a
> new node. The problem is that the Recent Write Latency goes to 2s on all
> the other Cassandra nodes (which are receiving user traffic), which
> corresponds to our setting of "write_request_timeout_in_ms: 2000".
>
> We use Cassandra 2.0.10 and trying to convert to vnodes and increase a
> replication factor. So we are adding a new node in new DC (marked as
> DCXA) as the only node in new DC with replication factor 3. The reason
> for higher RF is that we will be converting another 2 existing servers
> to new DC (vnodes) and we want them to get all the data.
>
> The replication settings look like this:
> ALTER KEYSPACE slw WITH replication = {
> 'class': 'NetworkTopologyStrategy',
> 'DC4': '1',
> 'DC5': '1',
> 'DC2': '1',
> 'DC3': '1',
> 'DC0': '1',
> 'DC1': '1',
> 'DC0A': '3',
> 'DC1A': '3',
> 'DC2A': '3',
> 'DC3A': '3',
> 'DC4A': '3',
> 'DC5A': '3'
> };
>
> We were adding the nodes to DC0A->DC4A without any effects on existing
> nodes (DCX without A). When we are trying to add DC5A, the abovemention
> problem happens, 100% reproducibly.
>
> I tried to increase number of concurrent_writers from 32 to 128 on the
> old nodes, also tried to increase number of flush writers, both with no
> effect. The strange thing is that the load, CPU usage, GC, network
> throughput - everything is fine on the old nodes which are reporting 2s
> of write latency. Nodetool tpstats does not show any blocked/pending
> operations.
>
> I think I must be hitting some limit (because of overall of replicas?)
> somewhere.
>
> Any input would be greatly appreciated.
>
> Thanks
> Jirka H.
>
RE: Write timeout on other nodes when joing a new node (in new DC)
UNSUBSCRIBE
Re: write timeout
Forgot to mention I am using Cassandra 2.0.13 On Mon, Mar 23, 2015 at 5:59 PM, Anishek Agarwal anis...@gmail.com wrote: Hello, I am using a single node server class machine with 16 CPUs with 32GB RAM with a single drive attached to it. my table structure is as below CREATE TABLE t1(id bigint, ts timestamp, cat1 settext, cat2 settext, lat float, lon float, a bigint, primary key (id, ts)); I am trying to insert 300 entries per partition key with 4000 partition keys using 25 threads. Configurations write_request_timeout_in_ms: 5000 concurrent_writes: 32 heap space : 8GB Client side timeout is 12 sec using datastax java driver. Consistency level: ONE With the above configuration i try to run it 10 times to eventually generate around 300 * 4000 * 10 = 1200 entries, When i run this after the first few runs i get a WriteTimeout exception at client with 1 replica were required but only 0 acknowledged the write message. There are no errors in server log. Why does this error come how do i know what is the limit I should limit concurrent writes to a single node to. Looking at iostat disk utilization seems to be at 1-3% when running this. Please let me know if anything else is required. Regards, Anishek
Re: write timeout
My group is seeing the same thing and also can not figure out why its happening. On Mon, Mar 23, 2015 at 8:36 AM, Anishek Agarwal anis...@gmail.com wrote: Forgot to mention I am using Cassandra 2.0.13 On Mon, Mar 23, 2015 at 5:59 PM, Anishek Agarwal anis...@gmail.com wrote: Hello, I am using a single node server class machine with 16 CPUs with 32GB RAM with a single drive attached to it. my table structure is as below CREATE TABLE t1(id bigint, ts timestamp, cat1 settext, cat2 settext, lat float, lon float, a bigint, primary key (id, ts)); I am trying to insert 300 entries per partition key with 4000 partition keys using 25 threads. Configurations write_request_timeout_in_ms: 5000 concurrent_writes: 32 heap space : 8GB Client side timeout is 12 sec using datastax java driver. Consistency level: ONE With the above configuration i try to run it 10 times to eventually generate around 300 * 4000 * 10 = 1200 entries, When i run this after the first few runs i get a WriteTimeout exception at client with 1 replica were required but only 0 acknowledged the write message. There are no errors in server log. Why does this error come how do i know what is the limit I should limit concurrent writes to a single node to. Looking at iostat disk utilization seems to be at 1-3% when running this. Please let me know if anything else is required. Regards, Anishek -- http://about.me/BrianTarbox
Re: write timeout
On Mon, Mar 23, 2015 at 7:27 AM, Brian Tarbox briantar...@gmail.com wrote: My group is seeing the same thing and also can not figure out why its happening. On Mon, Mar 23, 2015 at 8:36 AM, Anishek Agarwal anis...@gmail.com wrote: Forgot to mention I am using Cassandra 2.0.13 This seems like a rather significant bug in the most recent stable version. In this case, I would tend to file a JIRA first and then ask the mailing list second. Could one or both of you file steps-to-reproduce with a JIRA at http://issues.apache.org? =Rob
Re: Write timeout under load but Read is fine
I am using QUORUM CQL No SSDs (Anyway my IOPs is quite low so I dont think so it matters) No compaction is running when I receive timeout On Fri, Mar 6, 2015 at 12:35 AM, Carlos Rolo r...@pythian.com wrote: What is the consistency level you are using? Are you using Thrift or CQL? Are you using SSDs? Check if compactions are running when you get the timeouts. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo* Tel: 1649 www.pythian.com On Thu, Mar 5, 2015 at 9:51 PM, Jan cne...@yahoo.com wrote: Hello Jaydeep; Run *cassandra-stress* with R/W options enabled for about the same time and check if you have dropped packets. It would eliminate the client as the source of the error also give you a replicable tool to base subsequent tests/ findings. Jan/ On Thursday, March 5, 2015 12:19 PM, Jaydeep Chovatia chovatia.jayd...@gmail.com wrote: I have tried increasing timeout to 1 but no help. Also verified that there is no network lost packets. Jaydeep On Wed, Mar 4, 2015 at 12:19 PM, Jan cne...@yahoo.com wrote: HI Jaydeep; - look at the i/o on all three nodes - Increase the write_request_timeout_in_ms: 1 - check the time-outs if any on the client inserting the Writes - check the Network for dropped/lost packets hope this helps Jan/ On Wednesday, March 4, 2015 12:26 PM, Jaydeep Chovatia chovatia.jayd...@gmail.com wrote: Hi, In my test program when I increase load then I keep getting few write timeout from Cassandra say every 10~15 mins. My read:write ratio is 50:50. My reads are fine but only writes time out. Here is my Cassandra details: Version: 2.0.11 Ring of 3 nodes with RF=3 Node configuration: 24 core + 64GB RAM + 2TB write_request_timeout_in_ms: 5000, rest of Cassandra.yaml configuration is default I've also checked IO on Cassandra nodes and looks very low (around 5%). I've also checked Cassandra log file and do not see any GC happening. Also CPU on Cassandra is low (around 20%). I have 20GB data on each node. My test program creates connection to all three Cassandra nodes and sends read+write request randomly. Any idea what should I look for? Jaydeep -- Jaydeep -- -- Jaydeep
Re: Write timeout under load but Read is fine
What is the consistency level you are using? Are you using Thrift or CQL? Are you using SSDs? Check if compactions are running when you get the timeouts. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo* Tel: 1649 www.pythian.com On Thu, Mar 5, 2015 at 9:51 PM, Jan cne...@yahoo.com wrote: Hello Jaydeep; Run *cassandra-stress* with R/W options enabled for about the same time and check if you have dropped packets. It would eliminate the client as the source of the error also give you a replicable tool to base subsequent tests/ findings. Jan/ On Thursday, March 5, 2015 12:19 PM, Jaydeep Chovatia chovatia.jayd...@gmail.com wrote: I have tried increasing timeout to 1 but no help. Also verified that there is no network lost packets. Jaydeep On Wed, Mar 4, 2015 at 12:19 PM, Jan cne...@yahoo.com wrote: HI Jaydeep; - look at the i/o on all three nodes - Increase the write_request_timeout_in_ms: 1 - check the time-outs if any on the client inserting the Writes - check the Network for dropped/lost packets hope this helps Jan/ On Wednesday, March 4, 2015 12:26 PM, Jaydeep Chovatia chovatia.jayd...@gmail.com wrote: Hi, In my test program when I increase load then I keep getting few write timeout from Cassandra say every 10~15 mins. My read:write ratio is 50:50. My reads are fine but only writes time out. Here is my Cassandra details: Version: 2.0.11 Ring of 3 nodes with RF=3 Node configuration: 24 core + 64GB RAM + 2TB write_request_timeout_in_ms: 5000, rest of Cassandra.yaml configuration is default I've also checked IO on Cassandra nodes and looks very low (around 5%). I've also checked Cassandra log file and do not see any GC happening. Also CPU on Cassandra is low (around 20%). I have 20GB data on each node. My test program creates connection to all three Cassandra nodes and sends read+write request randomly. Any idea what should I look for? Jaydeep -- Jaydeep -- --
Re: Write timeout under load but Read is fine
I have tried increasing timeout to 1 but no help. Also verified that there is no network lost packets. Jaydeep On Wed, Mar 4, 2015 at 12:19 PM, Jan cne...@yahoo.com wrote: HI Jaydeep; - look at the i/o on all three nodes - Increase the write_request_timeout_in_ms: 1 - check the time-outs if any on the client inserting the Writes - check the Network for dropped/lost packets hope this helps Jan/ On Wednesday, March 4, 2015 12:26 PM, Jaydeep Chovatia chovatia.jayd...@gmail.com wrote: Hi, In my test program when I increase load then I keep getting few write timeout from Cassandra say every 10~15 mins. My read:write ratio is 50:50. My reads are fine but only writes time out. Here is my Cassandra details: Version: 2.0.11 Ring of 3 nodes with RF=3 Node configuration: 24 core + 64GB RAM + 2TB write_request_timeout_in_ms: 5000, rest of Cassandra.yaml configuration is default I've also checked IO on Cassandra nodes and looks very low (around 5%). I've also checked Cassandra log file and do not see any GC happening. Also CPU on Cassandra is low (around 20%). I have 20GB data on each node. My test program creates connection to all three Cassandra nodes and sends read+write request randomly. Any idea what should I look for? Jaydeep -- Jaydeep
Re: Write timeout under load but Read is fine
Hello Jaydeep; Run cassandra-stress with R/W options enabled for about the same time and check if you have dropped packets. It would eliminate the client as the source of the error also give you a replicable tool to base subsequent tests/ findings. Jan/ On Thursday, March 5, 2015 12:19 PM, Jaydeep Chovatia chovatia.jayd...@gmail.com wrote: I have tried increasing timeout to 1 but no help. Also verified that there is no network lost packets. Jaydeep On Wed, Mar 4, 2015 at 12:19 PM, Jan cne...@yahoo.com wrote: HI Jaydeep; - look at the i/o on all three nodes - Increase the write_request_timeout_in_ms: 1 - check the time-outs if any on the client inserting the Writes - check the Network for dropped/lost packets hope this helpsJan/ On Wednesday, March 4, 2015 12:26 PM, Jaydeep Chovatia chovatia.jayd...@gmail.com wrote: Hi, In my test program when I increase load then I keep getting few write timeout from Cassandra say every 10~15 mins. My read:write ratio is 50:50. My reads are fine but only writes time out. Here is my Cassandra details:Version: 2.0.11 Ring of 3 nodes with RF=3Node configuration: 24 core + 64GB RAM + 2TB write_request_timeout_in_ms: 5000, rest of Cassandra.yaml configuration is default I've also checked IO on Cassandra nodes and looks very low (around 5%). I've also checked Cassandra log file and do not see any GC happening. Also CPU on Cassandra is low (around 20%). I have 20GB data on each node. My test program creates connection to all three Cassandra nodes and sends read+write request randomly. Any idea what should I look for? Jaydeep -- Jaydeep
Re: Write timeout under load but Read is fine
HI Jaydeep; - look at the i/o on all three nodes - Increase the write_request_timeout_in_ms: 1 - check the time-outs if any on the client inserting the Writes - check the Network for dropped/lost packets hope this helpsJan/ On Wednesday, March 4, 2015 12:26 PM, Jaydeep Chovatia chovatia.jayd...@gmail.com wrote: Hi, In my test program when I increase load then I keep getting few write timeout from Cassandra say every 10~15 mins. My read:write ratio is 50:50. My reads are fine but only writes time out. Here is my Cassandra details:Version: 2.0.11 Ring of 3 nodes with RF=3Node configuration: 24 core + 64GB RAM + 2TB write_request_timeout_in_ms: 5000, rest of Cassandra.yaml configuration is default I've also checked IO on Cassandra nodes and looks very low (around 5%). I've also checked Cassandra log file and do not see any GC happening. Also CPU on Cassandra is low (around 20%). I have 20GB data on each node. My test program creates connection to all three Cassandra nodes and sends read+write request randomly. Any idea what should I look for? Jaydeep
