Thanks for the detailed response. It is really helpful to understand what is going on behind the covers.
We are using "RandomPartitioner." However, I have noticed that some of the boxes have significantly more data (in /var/cassandra/data and /var/cassandra/commitlog) than others (like 30 X more). I have 50 client processes doing the read/write workload (and had 50 doing the load) but had them round-robined between servers. E.g. the cassandra server to connect to was clientid % 6. I have been reading and writing a single column family, so perhaps that's part of the issue. Incidentally, the system is quite fun to play with, and the startup is very easy (just start the nodes and they all find each other.) Writing the client (e.g. dealing with thrift) was much harder. I wonder whether a lot of users had tried to write C++ clients; this was a little non-trivial as the documentation I could find favors the java and PHP cases. (I didn't realize the FB folks had disconnected from the OSS project.) brian ________________________________________ From: Jonathan Ellis [[email protected]] Sent: Wednesday, August 19, 2009 5:32 PM To: [email protected] Subject: Re: Anybody experience one Cassandra server locking up? On Wed, Aug 19, 2009 at 5:19 PM, Brian Frank Cooper<[email protected]> wrote: > We are trying to learn what we can about the performance of Cassandra. I hope > to have some results to share publicly in the next couple of weeks. > > The 0.4 version seems to have handled the insert load better, but is having > trouble with a 50/50 read/write workload. One server again has a busy core > with the other 7 cores (and the other servers) idle or near idle. Any ideas? Writes are serialized per columnfamily. There are some ways we can improve that but right now you may need multiple CFs to max write throughput. (Reads are not serialized like that though so I am a little surprised that the idleness difference is so complete.) If only one server is getting all the load something is wrong. Is that the server all your clients are connecting to? It's designed to have the clients spread around the cluster. Or, are you using OrderPreservingPartitioner? Load balancing won't be in until 0.5 so unless you manually pick your tokens carefully and/or do writes in a non-sequential manner one server will get all the keys. Or just use RandomPartitioner (which of course means giving up range queries). > (Incidentally, we are reading and writing 10 KB records; does the large data > size have any impact?) Since unlike a traditional K/V store you can update and retrieve individual columns separately, most column sizes are for < 1KB but 10KB isn't totally unreasonable. > I was hoping to chat with some of you Cassandra folks when we visited FB last > week...perhaps we can grab coffee sometime and chat about these issues... The FB guys haven't been involved with the OSS project for some time, unfortunately. -Jonathan
