Re: Unbalanced disk load

2015-07-18 Thread J. Ryan Earl
Even with https://issues.apache.org/jira/browse/CASSANDRA-7386 data balancing across JBOD setups is pretty horrible. Having used JBOD for about 2 years from 1.2.x and up, it is my opinion JBOD on Cassandra is nascent at best and far from mature. For a variety of reasons, JBOD should perform

Re: error='Cannot allocate memory' (errno=12)

2015-05-12 Thread J. Ryan Earl
What's your ulimit -a output? Did you adjust nproc and nofile ulimits up? Do you have JNA installed? What about memlock ulimit and in sysctl.conf = kernel.shmmax? What's in cassandra.log? On Mon, May 11, 2015 at 7:24 AM, Rahul Bhardwaj rahul.bhard...@indiamart.com wrote: Hi Robert, I saw

Re: error='Cannot allocate memory' (errno=12)

2015-05-12 Thread J. Ryan Earl
, similar to how pipes, sockets, etc can count against file descriptor limits. On Tue, May 12, 2015 at 7:55 PM, J. Ryan Earl o...@jryanearl.us wrote: What's your ulimit -a output? Did you adjust nproc and nofile ulimits up? Do you have JNA installed? What about memlock ulimit and in sysctl.conf

Re: OOM and high SSTables count

2015-03-04 Thread J. Ryan Earl
We think it is this bug: https://issues.apache.org/jira/browse/CASSANDRA-8860 We're rolling a patch to beta before rolling it into production. On Wed, Mar 4, 2015 at 4:12 PM, graham sanderson gra...@vast.com wrote: We can confirm a problem on 2.1.3 (sadly our beta sstable state obviously did

Re: Setting up JNA on CentOS 6.6. with cassandra20-2.0.12 and Oracle Java 1.7.0_75

2015-02-25 Thread J. Ryan Earl
We've been using jna-3.2.4-2.el6.x86_64 with the Sun/Oracle JDK for probably 2-years now, and it works just fine. Where are you seeing 3.2.7 required at? I searched the pages you link and that string isn't even in there. Regardless, I assure you the newest jna that ships in the EL6 repo works

Re: hs_err_pid3013.log, out of memory?

2014-09-18 Thread J. Ryan Earl
On Wed, Sep 17, 2014 at 8:35 PM, Yatong Zhang bluefl...@gmail.com wrote: @Chris Lohfink I have 16G memory per node, all the other settings are default @J. Ryan Earl I am not sure. I am using the default settings. But I've found out it might be because some settings in '/etc/sysctl.conf'. I

Re: ava.lang.OutOfMemoryError: unable to create new native thread

2014-09-18 Thread J. Ryan Earl
What's the 'ulimit -a' output of the user cassandra runs as? From this and your previous OOM thread, is sounds like you skipped the requisite OS configuration. On Wed, Sep 17, 2014 at 9:43 AM, Yatong Zhang bluefl...@gmail.com wrote: Hi there, I am using leveled compaction strategy and have

Re: hs_err_pid3013.log, out of memory?

2014-09-16 Thread J. Ryan Earl
Are you using JNA? Did you adjust your memlock limit? On Tue, Sep 16, 2014 at 9:46 AM, Chris Lohfink clohf...@blackbirdit.com wrote: How much memory does your system have? How much memory is system utilizing before starting Cassandra (use command free)? What are the heap setting it tries to

Re: too many open files

2014-08-08 Thread J. Ryan Earl
Yes, definitely look how many open files are actual file handles versus networks sockets. We found a file handle leak in 2.0 but it was patched in 2.0.3 or .5 I think. A million open files is way too high. On Fri, Aug 8, 2014 at 5:19 PM, Andrey Ilinykh ailin...@gmail.com wrote: You may have

Re: Write performance with 1.2.12

2013-12-12 Thread J. Ryan Earl
Why did you switch to RandomPartitioner away from Murmur3Partitioner? Have you tried with Murmur3? 1. # partitioner: org.apache.cassandra.dht.Murmur3Partitioner 2. partitioner: org.apache.cassandra.dht.RandomPartitioner On Fri, Dec 6, 2013 at 10:36 AM, srmore comom...@gmail.com wrote:

Re: cassandra performance problems

2013-12-06 Thread J. Ryan Earl
On Thu, Dec 5, 2013 at 6:33 AM, Alexander Shutyaev shuty...@gmail.comwrote: We've plugged it into our production environment as a cache in front of postgres. Everything worked fine, we even stressed it by explicitly propagating about 30G (10G/node) data from postgres to cassandra. If you

Re: Getting into Too many open files issues

2013-11-20 Thread J. Ryan Earl
There was a bug introduced in 2.0.0-beta1 related to TTL, a patch just came available in: https://issues.apache.org/jira/browse/CASSANDRA-6275 On Thu, Nov 7, 2013 at 5:15 AM, Murthy Chelankuri kmurt...@gmail.comwrote: I have experimenting cassandra latest version for storing the huge the in

Re: Cass 2.0.0: Extensive memory allocation when row_cache enabled

2013-11-14 Thread J. Ryan Earl
First off, I'm curious what hardware (system specs) you're running this on? Secondly, here are some observations: * You're not running the newest JDK7, I can tell by your stack-size. Consider getting the newest. * Cassandra 2.0.2 has a lot of improvements, consider upgrading. We noticed

Re: Multi-dc restart impact

2013-10-10 Thread J. Ryan Earl
Are you doing QUORUM reads instead of LOCAL_QUORUM reads? On Wed, Oct 9, 2013 at 7:41 PM, Chris Burroughs chris.burrou...@gmail.comwrote: I have not been able to do the test with the 2nd cluster, but have been given a disturbing data point. We had a disk slowly fail causing a significant