Re: [RELEASE CANDIDATE] Apache Cassandra 1.1.0-rc1 released

2012-04-17 Thread Sylvain Lebresne
On Mon, Apr 16, 2012 at 10:45 PM, Bryce Godfrey bryce.godf...@azaleos.com wrote: I keep running into this with my testing (on a windows box), Is this just a OOM for RAM? How much RAM do you have? Do you use completely standard settings? Do you also OOM if you try the same test with Cassandra

Re: Poor write performance with seconrady index

2012-04-17 Thread aaron morton
Secondary indexes require a read and a write (potentially two) for every update. Regular mutations are no look writes and are much faster. Just like in a RDBMS, it's more efficient to insert data and then create the index than to insert data with the index present. An alternative is to

Re: Cassandra Network Protocols

2012-04-17 Thread aaron morton
Hand rolled headers and binary serialisation over TCP. See o.a.c.net.IncomingTcpConnection for the starting point for an incoming connection. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 17/04/2012, at 9:49 AM, Tom Duffield (Mailing

Re: java.nio.BufferOverflowException from cassandra server

2012-04-17 Thread aaron morton
What version are you on ? it's on odd error for sure. Try eliminates changes until you get to a stable base: use the default install out of the box and the tools/stress app in the source distro. If that works try it with your C++ client. Hope that helps. - Aaron Morton

Re: Is the secondary index re-built under compaction?

2012-04-17 Thread aaron morton
Yes secondary index builds are done via the compaction manager. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 17/04/2012, at 1:06 PM, Maxim Potekhin wrote: I noticed that nodetool compactionstats shows the building of the secondary

Re: Poor write performance with seconrady index

2012-04-17 Thread Patrik Modesto
Hi Aaron, thanks for the reply. I suspected it might be the read-and-write that causes the slower updates. Regards, P. On Tue, Apr 17, 2012 at 11:52, aaron morton aa...@thelastpickle.com wrote: Secondary indexes require a read and a write (potentially two) for every update. Regular mutations

User authorized for modify-keyspace cannot create CFs

2012-04-17 Thread Michal Michalski
Hi, I'm suffering a problem, which maybe is a feature ( ;) ), but for me it's rather an annoying problem. I use SimpleAuthenticator and I have user who should be a kind of Cassandra's keyspace root - he should be allowed to do everything. So I set: modify-keyspaces=master Unluckily, when I

Re: Is the secondary index re-built under compaction?

2012-04-17 Thread Maxim Potekhin
Thanks Aaaron. Just to be clear, every time I do a compaction, I rebuild all indexes from scratch. Right? Maxim On 4/17/2012 6:16 AM, aaron morton wrote: Yes secondary index builds are done via the compaction manager. Cheers - Aaron Morton Freelance Developer @aaronmorton

Re: Is the secondary index re-built under compaction?

2012-04-17 Thread Jake Luciani
No, the indexes are not rebuilt every compaction. Only if you manually rebuild or bootstrap a new node does it use compaction manager to rebuild. On Tue, Apr 17, 2012 at 9:47 AM, Maxim Potekhin potek...@bnl.gov wrote: Thanks Aaaron. Just to be clear, every time I do a compaction, I rebuild

Re: exceptions after upgrading from 1.0.7 to 1.0.9

2012-04-17 Thread Watanabe Maki
You need to install cql driver for python as it says. % easy_install cql If you don't have easy_install, you need to install it first. You will be able to find easy_install by querying easy_install python on google. maki On 2012/04/17, at 20:18, Tamar Fraenkel ta...@tok-media.com wrote:

Re: Is the secondary index re-built under compaction?

2012-04-17 Thread Maxim Potekhin
Thanks Jake. Then I am definitely seeing weirdness, as there are tons of pending tasks in compaction stats, and tons of index files created in the data directory. Plus it does tell me that it is building the secondary index, and that seems to be happening at an amazingly glacial pace. I have 2

Re: Is the secondary index re-built under compaction?

2012-04-17 Thread Jake Luciani
Well, the since the secondary indexes are themselves column families they too are compacted along with everything else. On Tue, Apr 17, 2012 at 10:02 AM, Maxim Potekhin potek...@bnl.gov wrote: Thanks Jake. Then I am definitely seeing weirdness, as there are tons of pending tasks in compaction

Re: Is the secondary index re-built under compaction?

2012-04-17 Thread Maxim Potekhin
I understand that indexes are CFs. But the compaction stats says it's building the index, not compacting the corresponding CF. Either that's an ambiguous diagnostic, or indeed something is not right with my rig as of late. Maxim On 4/17/2012 10:05 AM, Jake Luciani wrote: Well, the since the

Re: Is the secondary index re-built under compaction?

2012-04-17 Thread Jake Luciani
Hmm that does sound fishy. When you run show keyspaces from cassandra-cli it shows which indexes are built. Are they marked built in your column family? -Jake On Tue, Apr 17, 2012 at 10:09 AM, Maxim Potekhin potek...@bnl.gov wrote: I understand that indexes are CFs. But the compaction stats

Re: Is the secondary index re-built under compaction?

2012-04-17 Thread Maxim Potekhin
Yes. Sorry I didn't mention this, but of course I'm checking on indexes once in a while. So yes, they are marked as built. All of this started happening after a few days of continuous loading process. Since the nodes have good hardware (24 cores + SSD), the apparent load on each node was

Re: Is the secondary index re-built under compaction?

2012-04-17 Thread Jake Luciani
How many indexes are there? On Tue, Apr 17, 2012 at 10:16 AM, Maxim Potekhin potek...@bnl.gov wrote: Yes. Sorry I didn't mention this, but of course I'm checking on indexes once in a while. So yes, they are marked as built. All of this started happening after a few days of continuous

Re: Is the secondary index re-built under compaction?

2012-04-17 Thread Maxim Potekhin
The offending CF only has one. The other one, that seems to behave well, has nine. Maxim On 4/17/2012 10:20 AM, Jake Luciani wrote: How many indexes are there? On Tue, Apr 17, 2012 at 10:16 AM, Maxim Potekhin potek...@bnl.gov mailto:potek...@bnl.gov wrote: Yes. Sorry I didn't mention

Re: exceptions after upgrading from 1.0.7 to 1.0.9

2012-04-17 Thread Tamar Fraenkel
Thanks!!! Two simple actions 1. sudo apt-get install python-setuptools 2. sudo easy_install cql And it did the trick! But just to be on the safe side, before I move to upgrade our staging environment, does anyone know a detailed description of how to upgrade cassandra installed from

Multi Master replication : rejoining a node after split network

2012-04-17 Thread Samba
Hi all, We are evaluating Cassandra for a geographically distributed deployment that requires multi master replication. We have a few questions regarding how replication is handled in Cassandra, like: 1. Which mechanism is used to replicate the changes from one system to another:

Counter column family

2012-04-17 Thread Tamar Fraenkel
Hi! I want to understand how incrementing of counter works. - I have a 3 node ring, - I use FailoverPolicy.FAIL_FAST, - RF is 2, I have the following counter column family ColumnFamily: tk_counters Key Validation Class: org.apache.cassandra.db.marshal.CompositeType(

RE: [RELEASE CANDIDATE] Apache Cassandra 1.1.0-rc1 released

2012-04-17 Thread Bryce Godfrey
Sorry, I found the issue. The server I was using had 32bit java installed. -Original Message- From: Sylvain Lebresne [mailto:sylv...@datastax.com] Sent: Monday, April 16, 2012 11:39 PM To: user@cassandra.apache.org Subject: Re: [RELEASE CANDIDATE] Apache Cassandra 1.1.0-rc1 released On

Re: [RELEASE CANDIDATE] Apache Cassandra 1.1.0-rc1 released

2012-04-17 Thread Jonathan Ellis
64bit is recommended where that's available. If you actually did have a 32bit machine or VM, then you should dramatically reduce the commitlog space cap to the minimum of 128MB so it doesn't need to mmap so much. On Tue, Apr 17, 2012 at 1:45 PM, Bryce Godfrey bryce.godf...@azaleos.com wrote:

Re: size tiered compaction - improvement

2012-04-17 Thread Jonathan Ellis
On Sat, Apr 14, 2012 at 3:27 AM, Radim Kolar h...@filez.com wrote: forceUserDefinedCompaction would be more usefull if you could do compaction on 2 tables. You absolutely can. That's what the user defined part is: you give it the exact list of sstables you want compacted. -- Jonathan Ellis

Re: size tiered compaction - improvement

2012-04-17 Thread Jonathan Ellis
On Sat, Apr 14, 2012 at 4:08 AM, Igor i...@4friends.od.ua wrote: Assume I insert all my data with TTL=2weeks and let we have sstable A which was created week ago at the time T, so I know that right now it contain: 1) some data that were inserted not later than T and may-be not expired yet 2)

Re: Off-heap row cache and mmapped sstables

2012-04-17 Thread Jonathan Ellis
Absolutely. Best practice is still to disable swap entirely on server machines; mlockall is just our best attempt to at least keep your JVM from swapping if you've forgotten this. On Thu, Apr 12, 2012 at 11:15 AM, Omid Aladini omidalad...@gmail.com wrote: Hi, Cassandra issues an mlockall [1]

Re: Long start-up times

2012-04-17 Thread Jonathan Ellis
On Sun, Apr 15, 2012 at 2:47 PM, sj.climber sj.clim...@gmail.com wrote: Also, I see in 1.0.9 there's a fix for a potentially related issue (see https://issues.apache.org/jira/browse/CASSANDRA-4023).  Any thoughts on this? My thought is, upgrading is a no-brainer if that's a pain point for you.

Re: swap grows

2012-04-17 Thread Jonathan Ellis
Swappiness is actually a fairly weak hint to linux: http://www.linuxvox.com/2009/10/what-is-the-linux-kernel-parameter-vm-swappiness On Sat, Apr 14, 2012 at 1:39 PM, aaron morton aa...@thelastpickle.com wrote: From https://help.ubuntu.com/community/SwapFaq swappiness=0 tells the kernel to

Re: java.nio.BufferOverflowException from cassandra server

2012-04-17 Thread Jonathan Ellis
If I were to take a wild guess, it would be that you're using a single Thrift connection in multiple threads, which isn't supported. On Mon, Apr 16, 2012 at 6:43 PM, Aniket Chakrabarti chakr...@cse.ohio-state.edu wrote: Hi, I have set up a 4 node cassandra cluster. I am using the Thrift C++

Re: size tiered compaction - improvement

2012-04-17 Thread Igor
Thank you Jonatathan, I missed this point about converting TTL data to tombstones first. When you say: You absolutely can. That's what the user defined part is: you give it the exact list of sstables you want compacted. does it mean that I can use list (not just one) of sstables as

Re: size tiered compaction - improvement

2012-04-17 Thread Jonathan Ellis
On Tue, Apr 17, 2012 at 11:26 PM, Igor i...@4friends.od.ua wrote: You absolutely can. That's what the user defined part is: you give it the exact list of sstables you want compacted. does it mean that I can use list (not just one) of sstables as second parameter for userDefinedCompaction?