Re: Size Tiered - Leveled Compaction

2013-03-08 Thread Alain RODRIGUEZ
I'm still wondering about how to chose the size of the sstable under LCS. Defaul is 5MB, people use to configure it to 10MB and now you configure it at 128MB. What are the benefits or inconveniants of a very small size (let's say 5 MB) vs big size (like 128MB) ? Alain 2013/3/8 Al Tobey

Row cache off-heap ?

2013-03-08 Thread Alain RODRIGUEZ
Hi, We have some issue having a high read throughput. I wanted to alleviate things by turning the row cache ON. I set the row cache to 200 on one node and enable caching 'ALL' on the 3 most read CF. There is the effect this operation had on my JVM:

Using WHERE IN with Wide Rows

2013-03-08 Thread Adam Venturella
TL;DR: Is it possible to use WHERE IN on wide rows but only have it return the 1st column of each of the rows in the IN()? First, I am aware that WHERE IN (id1, id2, id3...N) is not the most performant, and should not be used on large sets. Assuming there is also little difference from just

Kundera 2.4 Released

2013-03-08 Thread Amresh Kumar Singh
Hi All, We are happy to announce the release of Kundera 2.4. Kundera is a JPA 2.0 compliant, object-datastore mapping library for NoSQL datastores. The idea behind Kundera is to make working with NoSQL databases drop-dead simple and fun. It currently supports Cassandra, HBase, MongoDB, Redis,

Re: Bloom filters and LCS

2013-03-08 Thread Michael Theroux
I think my impression that Bloom Filters were off in 1.1.9 was a misinterpretation of this thread: http://www.mail-archive.com/user@cassandra.apache.org/msg27787.html and this bug: https://issues.apache.org/jira/browse/CASSANDRA-5029 I read it that Bloom filters were added to 1.2.2 for

Re: Need help

2013-03-08 Thread Jason Wee
Shouldn't be difficult to google what you want for starter... but here are some below, http://wiki.apache.org/cassandra/GettingStarted http://wiki.apache.org/cassandra/ClientOptions http://wiki.apache.org/cassandra/HadoopSupport http://www.slideshare.net/jeromatron/cassandrahadoop-integration

Re: Cassandra OOM, many deletedColumn

2013-03-08 Thread aaron morton
You need to provide some details of the machine and the JVM configuration. But lets say you need to have 4Gb to 8GB for the JVM heap. If you have many deleted columns I would say you have a *lot* of garbage in each row. Consider reducing the gc_grace seconds so the columns are purged more

Re: data model to store large volume syslog

2013-03-08 Thread aaron morton
1). create a column family 'cfrawlog' which stores raw log as received. row key could be 'ddmmhh'(new row is added for each hour or less), each 'column name' is uuid with 'value' is raw log data. Since we are also going to use this log for forensics purpose, so it will help us to have

Re: Hinted handoff

2013-03-08 Thread aaron morton
For example in my case running 'upgradesstables' on ~300GB node takes about 30+ hours. +1 And repair. 1 TB at 1Gb/s will take roughly 2.2hrs assume we stream from say 100 nodes… All of these suggestions on node size boil down to: make sure compaction, repair, node replacement and node moves

Re: Can't replace dead node

2013-03-08 Thread aaron morton
If it does not have the schema check the logs for errors and ensure it is actually part of the cluster. You may have better luck with Priam specific questions on https://github.com/Netflix/Priam Cheers - Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton

Re: Write latency spikes

2013-03-08 Thread aaron morton
Looking at the histograms, it looks like the latency is in the hot write path not the request wait time. Because at first glance the request latency and the cf latency match up. Consider graphing the request and write latencies to see how much time is spent waiting. So it could be waiting for

Time series data and deletion

2013-03-08 Thread Flavio Baronti
Hello, we are using Cassandra for storing time series data. We never update, only append; we plan to store 1 year worth of data, occupying something around 200Gb. I'm trying to understand what will happen when we start deleting the old data. With size tiered compaction, suppose we have one

Re: Size Tiered - Leveled Compaction

2013-03-08 Thread Wei Zhu
I have the same wonder.  We started with the default 5M and the compaction after repair takes too long on 200G node, so we increase the size to 10M sort of arbitrarily since there is not much documentation around it. Our tech op team still thinks there are too many files in one directory. To

Re: Size Tiered - Leveled Compaction

2013-03-08 Thread Hiller, Dean
+1 (I would love to know this info). Dean From: Wei Zhu wz1...@yahoo.commailto:wz1...@yahoo.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org, Wei Zhu wz1...@yahoo.commailto:wz1...@yahoo.com Date: Friday, March 8,

Re: Size Tiered - Leveled Compaction

2013-03-08 Thread Michael Theroux
I've asked this myself in the past... fairly arbitrarily chose 10MB based on Wei's experience, -Mike On Mar 8, 2013, at 1:50 PM, Hiller, Dean wrote: +1 (I would love to know this info). Dean From: Wei Zhu wz1...@yahoo.commailto:wz1...@yahoo.com Reply-To:

has anyone used dynamic snitch at all

2013-03-08 Thread Hiller, Dean
Our test setup 4 nodes, RF=3, reads at CL=QUOROM and we tried CL=TWO Tell the network card to slow down every packet on node 2 After fixing astyanax to not go to node 2 anymore, we are still seeing cassandra have issues as it seems to be involving node 2 somehow. If we take node 2 down, it all

Re: has anyone used dynamic snitch at all

2013-03-08 Thread Edward Capriolo
dynamic_snitch=true is the default. So it is usually on wrapping other snitches. I have found several scenarios where it does not work exactly as your would expect. On Fri, Mar 8, 2013 at 2:26 PM, Hiller, Dean dean.hil...@nrel.gov wrote: Our test setup 4 nodes, RF=3, reads at CL=QUOROM and we

leveled compaction

2013-03-08 Thread Kanwar Sangha
Hi - Can someone explain the meaning for the levelled compaction in cfstats - SSTables in each level: [40/4, 442/10, 97, 967, 7691, 0, 0, 0] SSTables in each level: [61/4, 9, 92, 945, 8146, 0, 0, 0] SSTables in each level: [34/4, 1000/10, 100, 953, 8184, 0, 0, 0 Thanks, Kanwar

Re: leveled compaction

2013-03-08 Thread Yuki Morishita
It is SSTable counts in each level. SSTables in each level: [40/4, 442/10, 97, 967, 7691, 0, 0, 0] So you have 40 SSTables in L0, 442 in L1, 97 in L2 and so forth. '40/4' and '442/10' have numbers after slash, those are expected maximum number of SSTables in that level and only displayed when

RE: leveled compaction

2013-03-08 Thread Kanwar Sangha
Cool ! So of we exceed the threshold, is that an issue… ? From: Yuki Morishita [mailto:mor.y...@gmail.com] Sent: 08 March 2013 15:57 To: user@cassandra.apache.org Subject: Re: leveled compaction It is SSTable counts in each level. SSTables in each level: [40/4, 442/10, 97, 967, 7691, 0, 0, 0]

Re: leveled compaction

2013-03-08 Thread Yuki Morishita
no. sstables are eventually compacted and moved to next level. On Friday, March 8, 2013, Kanwar Sangha wrote: Cool ! So of we exceed the threshold, is that an issue… ? ** ** *From:* Yuki Morishita [mailto:mor.y...@gmail.com javascript:_e({}, 'cvml', 'mor.y...@gmail.com');] *Sent:* 08

Re: Nodetool drain automatically shutting down node?

2013-03-08 Thread Andrew Bialecki
If it's helps, here's the log with debug log statements. Possibly issue with that exception? INFO [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:32,402 StorageService.java (line 774) DRAINING: starting drain process INFO [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:32,403

Startup Exception During Upgrade 1.1.6 to 1.2.2 during LCS Migration and Corrupt Tables

2013-03-08 Thread Arya Goudarzi
Hi, I am exercising the rolling upgrade from 1.1.6 to 1.2.2. When I upgraded to 1.2.2 on the first node, during startup I got this exception: ERROR [main] 2013-03-09 04:24:30,771 CassandraDaemon.java (line 213) Could not migrate old leveled manifest. Move away the .json file in the data

Re: Startup Exception During Upgrade 1.1.6 to 1.2.2 during LCS Migration and Corrupt Tables

2013-03-08 Thread Yuki Morishita
Are you sure you are using 1.2.2? Because LegacyLeveledManifest is from unreleased development version. On Friday, March 8, 2013 at 11:02 PM, Arya Goudarzi wrote: Hi, I am exercising the rolling upgrade from 1.1.6 to 1.2.2. When I upgraded to 1.2.2 on the first node, during startup I got

Re: Incompatible Gossip 1.1.6 to 1.2.1 Upgrade?

2013-03-08 Thread Arya Goudarzi
OK. I upgraded one node from 1.1.6 to 1.2.2 today. Despite some new problems that I had and I posted them in a separate email, this issue still exists but now it is only on 1.2.2 node. This means that the nodes running 1.1.6 see all other nodes including 1.2.2 as Up. Here is the ring and gossip

Re: Startup Exception During Upgrade 1.1.6 to 1.2.2 during LCS Migration and Corrupt Tables

2013-03-08 Thread Arya Goudarzi
*face palm* You are totally right. I built from the wrong branch. I am so sorry. But at least you got yourself a development bug to figure out. :) The specific commit I built from was this: f3e0aa683f3f310678d62ba8345fe33633b709e0. On Fri, Mar 8, 2013 at 9:14 PM, Yuki Morishita mor.y...@gmail.com

Re: Cassandra cluster setup issue

2013-03-08 Thread amulya rattan
I just saw that the failed to connect to 'x.x.x.x:7199' : connection refused error for nodetool that I am facing used to exist earlier also, case in point https://issues.apache.org/jira/browse/CASSANDRA-4494 I can see this problem resurfacing in Cassandra 1.1.9 on my system. I am using RHEL 6.0