Re: Cassandra freezes

2013-04-04 Thread Hiller, Dean
y-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" mailto:user@cassandra.apache.org>> Date: Thursday, April 4, 2013 5:49 AM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" mailto:user@cassandra.apache.org>> Subject: Re: Cassand

Re: Cassandra freezes

2013-04-03 Thread Joel Samuelsson
to > handle both the data and GC. Increasing the number of CPU cores made > everything run smoothly at the same load. > > > 2013/3/21 Andras Szerdahelyi > >> Neat! >> Thanks. >> >> From: Sylvain Lebresne >> Reply-To: "user@cassandra.apache.org&

Re: Cassandra freezes

2013-03-22 Thread Joel Samuelsson
> Reply-To: "user@cassandra.apache.org" > Date: Thursday 21 March 2013 10:10 > To: "user@cassandra.apache.org" > Subject: Re: Cassandra freezes > > Prior to 1.2 the index summaries were not saved on disk, and were thus > computed on startup while the sstable was l

Re: Cassandra freezes

2013-03-21 Thread Andras Szerdahelyi
@cassandra.apache.org>" mailto:user@cassandra.apache.org>> Subject: Re: Cassandra freezes Prior to 1.2 the index summaries were not saved on disk, and were thus computed on startup while the sstable was loaded. In 1.2 they now are saved on disk to make startup faster (https://issues.apache

Re: Cassandra freezes

2013-03-21 Thread Sylvain Lebresne
Prior to 1.2 the index summaries were not saved on disk, and were thus computed on startup while the sstable was loaded. In 1.2 they now are saved on disk to make startup faster ( https://issues.apache.org/jira/browse/CASSANDRA-2392). That being said, if the index_interval value used by a summary s

Re: Cassandra freezes

2013-03-21 Thread Michal Michalski
OK, I took a look at the source code and for now it seems to me that we both are partially right ( ;-) ), but changing index_interval does NOT require rebuilding SSTables: Yes, index sample file can be persisted (see io/sstable/IndexSummary.java, serialize/deserialize methods + io/sstable/SST

Re: Cassandra freezes

2013-03-21 Thread Andras Szerdahelyi
I can not find the reference that notes having to upgradesstables when you change this. I really hope such complex assumptions are not formulating in my head just on their own and there actually exists some kind of reliable reference that clears this up :-) but, # index_interval controls the samp

Re: Cassandra freezes

2013-03-21 Thread Michal Michalski
About index_interval: 1) you have to rebuild stables ( not an issue if you are evaluating, doing test writes.. Etc, not so much in production ) Are you sure of this? As I understand indexes, it's not required because this parameter defines an interval of in-memory index sample, which is crea

Re: Cassandra freezes

2013-03-20 Thread Hiller, Dean
e settings in cassandra.yaml >>> >>>memtable_total_space_in_mb: >>>memtable_flush_writers: >>>memtable_flush_queue_size: >>>compaction_throughput_mb_per_sec: >>> >>>concurrent_writes: >>> >>> >>> >>>Which v

Re: Cassandra freezes

2013-03-20 Thread Andras Szerdahelyi
? I assume ONE or ANY if >>you have a single node. >> >>What are the values for these settings in cassandra.yaml >> >>memtable_total_space_in_mb: >>memtable_flush_writers: >>memtable_flush_queue_size: >>compaction_throughput_mb_per_sec: >> >>concurrent_writes: >>

Re: Cassandra freezes

2013-03-20 Thread Hiller, Dean
gt;Regards, >Andras > >From: Joel Samuelsson >Reply-To: "user@cassandra.apache.org" >Date: Wednesday 20 March 2013 13:06 >To: "user@cassandra.apache.org" >Subject: Cassandra freezes > > >Hello, > >I've been trying to load test a one n

Re: Cassandra freezes

2013-03-20 Thread Tupshin Harper
What is in your Cassandra log right before and after that freeze? -Tupshin On Mar 20, 2013 8:06 AM, "Joel Samuelsson" wrote: > Hello, > > I've been trying to load test a one node cassandra cluster. When I add > lots of data, the Cassandra node freezes for 4-5 minutes during which > neither reads

Re: Cassandra freezes

2013-03-20 Thread Andras Szerdahelyi
ion_throughput_mb_per_sec: concurrent_writes: Which version of Cassandra? Regards, Andras From: Joel Samuelsson Reply-To: "user@cassandra.apache.org" Date: Wednesday 20 March 2013 13:06 To: "user@cassandra.apache.org" Subject: Cassandra freezes Hello, I've bee

Re: Cassandra freezes under load when using libc6 2.11.1-0ubuntu7.5

2011-01-17 Thread Erik Onnen
Unfortunately, the previous AMI we used to provision the 7.5 version is no longer available. More unfortunately, the two test nodes we spun up in each AZ did not get Nehalem architectures so the only things I can say for certain after running Mike's test 10x on each test node are: 1) I could not r

Re: Cassandra freezes under load when using libc6 2.11.1-0ubuntu7.5

2011-01-14 Thread Mike Malone
That's interesting. For us, the 7.5 version of libc was causing problems. Either way, I'm looking forward to hearing about anything you find. Mike On Thu, Jan 13, 2011 at 11:47 PM, Erik Onnen wrote: > Too similar to be a coincidence I'd say: > > Good node (old AZ): 2.11.1-0ubuntu7.5 > Bad node

Re: Cassandra freezes under load when using libc6 2.11.1-0ubuntu7.5

2011-01-13 Thread Erik Onnen
Too similar to be a coincidence I'd say: Good node (old AZ): 2.11.1-0ubuntu7.5 Bad node (new AZ): 2.11.1-0ubuntu7.6 You beat me to the punch with the test program. I was working on something similar to test it out and got side tracked. I'll try the test app tomorrow and verify the versions of t

Re: Cassandra freezes under load when using libc6 2.11.1-0ubuntu7.5

2011-01-13 Thread Mike Malone
Erik, the scenario you're describing is almost identical to what we've been experiencing. Sounds like you've been pulling your hair out too! You're also running the same distro and kernel as us. And we also run without swap. Which begs the question... what version of libc6 are you running!? Here's

Re: Cassandra freezes under load when using libc6 2.11.1-0ubuntu7.5

2011-01-13 Thread Erik Onnen
Forgot one critical point, we use zero swap on any of these hosts.

Re: Cassandra freezes under load when using libc6 2.11.1-0ubuntu7.5

2011-01-13 Thread Erik Onnen
May or may not be related but I thought I'd recount a similar experience we had in EC2 in hopes it helps someone else. As background, we had been running several servers in a 0.6.8 ring with no Cassandra issues (some EC2 issues, but none related to Cassandra) on multiple EC2 XL instances in a sing

Cassandra freezes under load when using libc6 2.11.1-0ubuntu7.5

2011-01-13 Thread Mike Malone
Hey folks, We've discovered an issue on Ubuntu/Lenny with libc6 2.11.1-0ubuntu7.5 (it may also affect versions between 2.11.1-0ubuntu7.1 and 2.11.1-0ubuntu7.4). The bug affects systems when a large number of threads (or processes) are created rapidly. Once triggered, the system will become complet