Hi Paul, Concerning large rows which are not compacting, I've probably managed to reproduce your problem. I suppose you're using collections, but also TTLs ?
Anyway, I opened an issue here : https://issues.apache.org/jira/browse/CASSANDRA-5799 Hope this helps 2013/7/24 Christopher Wirt <chris.w...@struq.com> > Hi Paul,**** > > ** ** > > Sorry to hear you’re having a low point.**** > > ** ** > > We ended up not using the collection features of 1.2. **** > > Instead storing a compressed string containing the map and handling client > side.**** > > ** ** > > We only have fixed schema short rows so no experience with large row > compaction.**** > > ** ** > > File descriptors have never got that high for us. But, if you only have a > couple physical nodes with loads of data and small ss-tables maybe they > could get that high?**** > > ** ** > > Only time I’ve had file descriptors get out of hand was then compaction > got slightly confused with a new schema when I dropped and recreated > instead of truncating. > https://issues.apache.org/jira/browse/CASSANDRA-4857 restarting the node > fixed the issue.**** > > ** ** > > ** ** > > From my limited experience I think Cassandra is a dangerous choice for an > young limited funding/experience start-up expecting to scale fast. We are a > fairly mature start-up with funding. We’ve just spent 3-5 months moving > from Mongo to Cassandra. It’s been expensive and painful getting Cassandra > to read like Mongo, but we’ve made it J**** > > ** ** > > ** ** > > ** ** > > ** ** > > *From:* Paul Ingalls [mailto:paulinga...@gmail.com] > *Sent:* 24 July 2013 06:01 > *To:* user@cassandra.apache.org > *Subject:* disappointed**** > > ** ** > > I want to check in. I'm sad, mad and afraid. I've been trying to get a > 1.2 cluster up and working with my data set for three weeks with no > success. I've been running a 1.1 cluster for 8 months now with no hiccups, > but for me at least 1.2 has been a disaster. I had high hopes for > leveraging the new features of 1.2, specifically vnodes and collections. > But at this point I can't release my system into production, and will > probably need to find a new back end. As a small startup, this could be > catastrophic. I'm mostly mad at myself. I took a risk moving to the new > tech. I forgot sometimes when you gamble, you lose.**** > > ** ** > > First, the performance of 1.2.6 was horrible when using collections. I > wasn't able to push through 500k rows before the cluster became unusable. > With a lot of digging, and way too much time, I discovered I was hitting a > bug that had just been fixed, but was unreleased. This scared me, because > the release was already at 1.2.6 and I would have expected something as > https://issues.apache.org/jira/browse/CASSANDRA-5677 would have been > addressed long before. But gamely I grabbed the latest code from the 1.2 > branch, built it and I was finally able to get past half a million rows. > **** > > ** ** > > But, then I hit ~4 million rows, and a multitude of problems. Even with > the fix above, I was still seeing a ton of compactions failing, > specifically the ones for large rows. Not a single large row will compact, > they all assert with the wrong size. Worse, and this is what kills the > whole thing, I keep hitting a wall with open files, even after dumping the > whole DB, dropping vnodes and trying again. Seriously, 650k open file > descriptors? When it hits this limit, the whole DB craps out and is > basically unusable. This isn't that many rows. I have close to a half a > billion in 1.1…**** > > ** ** > > I'm now at a standstill. I figure I have two options unless someone here > can help me. Neither of them involve 1.2. I can either go back to 1.1 and > remove the features that collections added to my service, or I find another > data backend that has similar performance characteristics to cassandra but > allows collections type behavior in a scalable manner. Cause as far as I > can tell, 1.2 doesn't scale. Which makes me sad, I was proud of what I > accomplished with 1.1….**** > > ** ** > > Does anyone know why there are so many open file descriptors? Any ideas > on why a large row won't compact?**** > > ** ** > > Paul**** > -- Fabien Rousseau * * <aur...@yakaz.com>www.yakaz.com
