Re: OOM while reading key cache

2013-11-14 Thread olek.stas...@gmail.com
Yes, as I wrote in first e-mail. When I removed key cache file cassandra started without further problems. regards Olek 2013/11/13 Robert Coli rc...@eventbrite.com: On Wed, Nov 13, 2013 at 12:35 AM, Tom van den Berge t...@drillster.com wrote: I'm having the same problem, after upgrading

Cassand is holding too many deleted file descriptors

2013-11-14 Thread Murthy Chelankuri
I See lots of these deleted file descriptors cassandra is holding in my case out of 90K file descriptors 80.5K is having these descriptors Because of this cassandra is not performing well. Can some one please tell what i am doing wrong. lr-x-- 1 root root 64 Nov 14 08:25 10875 -

Re: OOM while reading key cache

2013-11-14 Thread Fabien Rousseau
A few month ago, we've got a similar issue on 1.2.6 : https://issues.apache.org/jira/browse/CASSANDRA-5706 But it has been fixed and did not encountered this issue anymore (we're also on 1.2.10) 2013/11/14 olek.stas...@gmail.com olek.stas...@gmail.com Yes, as I wrote in first e-mail. When I

Re: Cassand is holding too many deleted file descriptors

2013-11-14 Thread Marcus Eriksson
yeah this is known, and we are looking for a fix https://issues.apache.org/jira/browse/CASSANDRA-6275 if you have a simple way of reproducing, please add a comment On Thu, Nov 14, 2013 at 10:53 AM, Murthy Chelankuri kmurt...@gmail.comwrote: I See lots of these deleted file descriptors

Re: Modeling multi-tenanted Cassandra schema

2013-11-14 Thread Ben Hood
OK, so in the end I elected to go for option (c), which makes my table definition look like this: create table tenanted_foo_table ( tenant ascii, application_key bigint, timestamp timestamp, other non-key columns PRIMARY KEY ((tenant, application_key), timestamp) ) such

Hints still exist for a removed node

2013-11-14 Thread Cyril Scetbon
Hi, I saw on http://www.datastax.com/dev/blog/modern-hinted-handoff (wrote on december 2012) that hints targeting a removed node (our case) are automatically removed. However, a compaction has been done on our cf and hints for the removed node are still stored. We're using version 1.2.2

Risk of not doing repair

2013-11-14 Thread olek.stas...@gmail.com
Hello, I'm facing bug https://issues.apache.org/jira/browse/CASSANDRA-6277. After migration to 2.0.2 I can't perform repair on my cluster (six nodes). Repair on the biggest CF breaks with error described in Jira. I know, that probably there is a solution in repository, but it's not included in any

Re: Cass 2.0.0: Extensive memory allocation when row_cache enabled

2013-11-14 Thread J. Ryan Earl
First off, I'm curious what hardware (system specs) you're running this on? Secondly, here are some observations: * You're not running the newest JDK7, I can tell by your stack-size. Consider getting the newest. * Cassandra 2.0.2 has a lot of improvements, consider upgrading. We noticed

db file missing error

2013-11-14 Thread Langston, Jim
Hi all, When I run nodetool repair, I'm getting an error that indicates that several of the Data.db files are missing. Is there a way to correct this error ? The files that the error message is referencing are indeed missing, I'm not sure why it is looking for them to begin with. AFAIK nothing

Re: db file missing error

2013-11-14 Thread Langston, Jim
Found it, had a second repair running which was generating the error. Jim From: Langston, Jim jim.langs...@compuware.commailto:jim.langs...@compuware.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Thu, 14 Nov 2013 18:34:19 + To:

Re: Cass 2.0.0: Extensive memory allocation when row_cache enabled

2013-11-14 Thread Robert Coli
On Thu, Nov 14, 2013 at 10:05 AM, J. Ryan Earl o...@jryanearl.us wrote: * Cassandra 2.0.2 has a lot of improvements, consider upgrading. We noticed improved heap usage compared to 2.0.2 https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ And especially if you're using

Re: Risk of not doing repair

2013-11-14 Thread Robert Coli
On Thu, Nov 14, 2013 at 6:25 AM, olek.stas...@gmail.com olek.stas...@gmail.com wrote: After migration to 2.0.2 I can't perform repair on my cluster (six nodes). ... If it's not really neccessary, i would avoid building unstable version of cass from sources and install it in prod environ

Re: Hints still exist for a removed node

2013-11-14 Thread Robert Coli
On Thu, Nov 14, 2013 at 6:08 AM, Cyril Scetbon cyril.scet...@free.frwrote: I saw on http://www.datastax.com/dev/blog/modern-hinted-handoff (wrote on december 2012) that hints targeting a removed node (our case) are automatically removed. However, a compaction has been done on our cf and hints

Read inconsistency after backup and restore to different cluster

2013-11-14 Thread David Laube
Hi All, After running through our backup and restore process FROM our test production TO our staging environment, we are seeing inconsistent reads from the cluster we restored to. We have the same number of nodes in both clusters. For example, we will select data from a column family on the

Re: Read inconsistency after backup and restore to different cluster

2013-11-14 Thread Robert Coli
On Thu, Nov 14, 2013 at 12:37 PM, David Laube d...@stormpath.com wrote: It is almost as if the data only exists on some of the nodes, or perhaps the token ranges are dramatically different --again, we are using vnodes so I am not exactly sure how this plays into the equation. The token

Re: Read inconsistency after backup and restore to different cluster

2013-11-14 Thread David Laube
Thank you for the detailed reply Rob! I have replied to your comments in-line below; On Nov 14, 2013, at 1:15 PM, Robert Coli rc...@eventbrite.com wrote: On Thu, Nov 14, 2013 at 12:37 PM, David Laube d...@stormpath.com wrote: It is almost as if the data only exists on some of the nodes, or

making sense of output from Eclipse Memory Analyzer tool taken from .hprof file

2013-11-14 Thread Mike Koh
I am investigating Java Out of memory heap errors. So I created an .hprof file and loaded it into Eclipse Memory Analyzer Tool which gave some Problem Suspects. First one looks like: One instance of org.apache.cassandra.db.ColumnFamilyStore loaded by sun.misc.Launcher$AppClassLoader @