i do not have any pending tasks in the compaction pool but i have 1164 files in my data directory. one thing to note about my situation is that i did run out of disk space during my test. cassandra _seemed_ to recover nicely.
tim, is your's recovering? i plan to rerun the test tonight with a slightly smaller data set, however, the 'get' performance was dwindling before the node ran out of disk space. does compaction only happen when idle? and yes, i'm running with 0.5-beta1, but not trunk. On Thu, 2009-12-03 at 11:03 -0800, B. Todd Burruss wrote: > i am seeing this as well. i did a test with just 1 cassandra node, > ReplicationFactor=1, 'get' ConsistencyLevel.ONE, 'put' > ConsistencyLevel.QUORUM. The first test was writing and reading random > values starting from a fresh database. The put performance is staying > reasonabe, but the read performance falls off dramatically as the data > grows. The get performance fell from approx 6500 get/sec to 150 get/sec > (as reported by my client stats.) The database has grown to approx > 500gig. i have the stats recorded on 5 second intervals and i see a > very linear drop off as the data grows. > > i stopped the server and restarted it, let it do its thing during > restart and then rerun a read-only test using the exact same data. i am > still at about 150 get/sec. via JMX i can see the read latency at about > 60, but this varies as the app runs. > > my keyspace is simple: > > <Keyspaces> > <Keyspace Name="uds"> > <KeysCachedFraction>0.01</KeysCachedFraction> > <ColumnFamily CompareWith="BytesType" Name="bucket" /> > </Keyspace> > </Keyspaces> > > all values are exactly the same and are 2k in length. > > i've tried to do some tuning to make things faster but don't necessarily > understand the options. here are some of the params i've changed in the > config file: > > <CommitLogRotationThresholdInMB>256</CommitLogRotationThresholdInMB> > <MemtableSizeInMB>1024</MemtableSizeInMB> > <MemtableObjectCountInMillions>0.6</MemtableObjectCountInMillions> > <CommitLogSyncPeriodInMS>1000</CommitLogSyncPeriodInMS> > <MemtableFlushAfterMinutes>1440</MemtableFlushAfterMinutes> > > hope this data helps, and any help you can provide is much appreciated. > > > On Tue, 2009-12-01 at 20:18 -0600, Jonathan Ellis wrote: > > On Tue, Dec 1, 2009 at 7:31 PM, Freeman, Tim <[email protected]> wrote: > > > Looking at the Cassandra mbean's, the attributes of ROW-MUTATION-STAGE > > > and ROW-READ-STAGE and RESPONSE-STAGE are all less than 10. > > > MINOR-COMPACTION-POOL reports 1218 pending tasks. > > > > That's probably the culprit right there. Something is wrong if you > > have 1200 pending compactions. > > > > This is something that upgrading to trunk will help with right away > > since we parallelize compactions there. > > > > Another thing you can do is increase the memtable limits so you are > > not flushing + compacting so often with your insert traffic. > > > > -Jonathan > >
