Cassandra DC2 nodes down after increasing write requests on DC1 nodes

2014-11-16 Thread Gabriel Menegatti
Hello, We are using Cassandra 2.1.2 in a multi dc cluster (30 servers on DC1 and 10 on DC2) with a key space replication factor of 1 on DC1 and 2 on DC2. For some reason when we increase the volume of write requests on DC1 (using ONE or LOCAL_ONE), the Cassandra java process on DC2 nodes goes

Re: Two writers appending to a set to see which one wins?

2014-11-16 Thread Eric Stevens
You may be able to do something with conditional updates, however trying to use Cassandra for this kind of coordination smells to me a lot like typical antipatterns (eg write then read or read then write). You probably would do better if you need one writer to consistently win a race condition

Re: Reading the write time of each value in a set?

2014-11-16 Thread Eric Stevens
I'm not aware of a way to query TTL or writetime on collections from CQL yet. You can access this information from Thrift though. On Sat Nov 15 2014 at 12:51:55 AM DuyHai Doan doanduy...@gmail.com wrote: Why don't you use map to store write time as value and data as key? Le 15 nov. 2014

Re: Cassandra DC2 nodes down after increasing write requests on DC1 nodes

2014-11-16 Thread Eric Stevens
load average on DC1 nodes are around 3-5 and on DC2 around 7-10 Anecdotally I can say that loads in the 7-10 range have been dangerously high. When we had a cluster running in this range, the cluster was falling behind on important tasks such as compaction, and we really struggled to

Re: Cassandra DC2 nodes down after increasing write requests on DC1 nodes

2014-11-16 Thread Gabriel Menegatti
Hi Eric, Thanks for your reply. I said that load was not a big deal, because ops center shows this loads as green, not as yellow or red at all. Also, our servers have many processors/threads, so I *think* this load is not problematic. My assumption is that for some reason the DC2 10 nodes

Problem with performance, memory consumption, and RLIMIT_MEMLOCK

2014-11-16 Thread Dmitri Dmitrienko
Hi, I have a very simple table in cassandra that contains only three columns: id, time and blob with data. I added 1M rows of data and now the database is about 12GB on disk. 1M is only part of data I want to store in the database, it's necessary to synchronize this table with external source. In

What time range does nodetool cfhistograms use?

2014-11-16 Thread Clint Kelly
Hi all, Over what time range does nodetool cfhistograms operate? I am using Cassandra 2.0.8.39. I am trying to debug some very high 95th and 99th percentile read latencies in an application that I'm working on. I tried running nodetool cfhistograms to get a flavor for the distribution of read

Re: What time range does nodetool cfhistograms use?

2014-11-16 Thread Mark Reddy
Hi Clint, The values of SSTables, Write Latency and Read Latency will be reset on node start/restart and after running the cfhistograms command itself. The values of Row Size and Column Count are calculated at startup and then re-evaluated during compaction. Mark On 16 November 2014 17:12,

Re: Cassandra DC2 nodes down after increasing write requests on DC1 nodes

2014-11-16 Thread Tim Heckman
Hello Gabriel, On Sun, Nov 16, 2014 at 7:25 AM, Gabriel Menegatti gabr...@s1mbi0se.com.br wrote: I said that load was not a big deal, because ops center shows this loads as green, not as yellow or red at all. Also, our servers have many processors/threads, so I *think* this load is not

Re: Reading the write time of each value in a set?

2014-11-16 Thread Kevin Burton
Thanks. I’ll probably file a bug for this, assuming one doesn’t already exist. On Sun, Nov 16, 2014 at 6:20 AM, Eric Stevens migh...@gmail.com wrote: I'm not aware of a way to query TTL or writetime on collections from CQL yet. You can access this information from Thrift though. On Sat Nov

Cassandra 2.1.1 Out of Memory Errors

2014-11-16 Thread Pardeep
I'm running a 4 node cluster with RF=3, CL of QUORUM for writes and ONE for reads. Each node has 3.7GB RAM with 32GB SSD HD, commitlog is on another HD. Currently each node has about 12GB of data. Cluster is always normal unless repair happens, that's when some nodes go to medium health in

Re: Cassandra 2.1.1 Out of Memory Errors

2014-11-16 Thread DuyHai Doan
I do use rows that can span few thousands to a few million, I'm not sure if range slices happen using the memory. What are your query patterns ? For a given partition, take a slice of xxx columns ? Or give me a range of partitions ? For the 1st scenario, depending on how many columns you want

Best practices for route tracing

2014-11-16 Thread Clint Kelly
Hi all, I am trying to debug some high-latency outliers (99th percentile) in an application I'm working on. I thought that I could turn on route tracing, print the route traces to logs, and then examine my logs after a load test to find the highest-latency paths and figure out what is going on.

Re: What time range does nodetool cfhistograms use?

2014-11-16 Thread DuyHai Doan
Does it show statistics for just the current node on which I am running, or for the entire cluster? -- only current node. Are the read latencies shown the latencies within a single host, or are they the end-to-end latencies from the coordinator node? -- cfhistograms shows metrics at table/node

Re: What time range does nodetool cfhistograms use?

2014-11-16 Thread Clint Kelly
Thanks for the quick reply! I completely forgot about proxyhistograms! On Sun, Nov 16, 2014 at 1:56 PM, DuyHai Doan doanduy...@gmail.com wrote: Does it show statistics for just the current node on which I am running, or for the entire cluster? -- only current node. Are the read latencies

Re: Cassandra 2.1.1 Out of Memory Errors

2014-11-16 Thread Pardeep
DuyHai Doan doanduyhai at gmail.com writes: I do use rows that can span few thousands to a few million, I'm not sure if rangeslices happen using the memory. What are your query patterns ? For a given partition, take a slice of xxx columns ? Or give me a range of partitions ?  For the

read repair across DC and latency

2014-11-16 Thread Jimmy Lin
I have a CF that use the default, read_repair_chance (0.1) and dc_read_repair_chance(0). Our read and write is all local_quorum, on one of the 2 DC, replication of 3. so a read will have 10% chance trigger a read repair to other DC. # I have read that read repair suppose to be running as

Re: Cassandra 2.1.1 Out of Memory Errors

2014-11-16 Thread DuyHai Doan
If the table is fragmented on many sstables on disk, you may run into trouble. Let me explain the reason. Your query is perfectly fine, but if you're querying a partition of, let's say 1 millions of rows spread across 10 SSTables, Cassandra may need to read the partition splits in all those

conditional batches across two tables?

2014-11-16 Thread Kevin Burton
I’m trying to have some code acquire a lock by first at performing a table mutation, and then if it wins, performing a second table insert. I don’t think this is possible with batches though. I don’t think I can say “update this table, and if you are able to set the value, and the value doesn’t

Re: Cassandra 2.1.1 Out of Memory Errors

2014-11-16 Thread Pardeep
DuyHai Doan, For wide rows, would it be better to switch to LeveledCompactionStrategy. The number of SSTables will decrease and it's also optimized for reading data. I have read in quite a few places that LeveledCompactionStrategy is better for wide rows. Is it true and would you recommend it?

Re: conditional batches across two tables?

2014-11-16 Thread Daniel Compton
Hi Kevin Have you looked at Cassandra's lightweight transactions? It sounds like they might work for you http://www.datastax.com/documentation/cassandra/2.0/cassandra/dml/dml_ltwt_transaction_c.html. -- Daniel On 17/11/2014, at 1:51 pm, Kevin Burton bur...@spinn3r.com wrote: I’m trying