It looks like what you're seeing is, stress far outpaced the ability of compaction to keep up (which is normal for our default settings, which prioritize maintaining request throughput over compaction), so LCS will grab a bunch of L0 sstables, compact them together with L1 resulting in a spike of L1 sstables, then compact those upwards into higher levels, gradually lowering the sstable count.
It's unclear how to improve the "LCS can't keep up" case [1]. But it's worth noting that a single large stress insert run, consisting as it does of a large volume of unique rows, is the worst case for LCS. This is the primary reason LCS is not the default: if you have an append-mostly write load with few overwrites or deletes, LCS will do a lot of extra i/o for no real benefit. [1] https://issues.apache.org/jira/browse/CASSANDRA-3854 On Sun, Jan 22, 2012 at 10:26 PM, Chris Burroughs <[email protected]> wrote: > I inserted a large number of keys to a single node using stress.java [1] > and let things sit for a while (several hours with no more inserts). > After a bit I decided something might be up and started sampling the > number of files in the data directory for 250 minutes while I played The > Legend of Zelda. At the start there were 78291 files, and the end > 78599. All I see in the log is a lot of "Compacting to" and "Compacted" > messages. The output of compactionstatus also seemed odd: > > $ ./bin/nodetool -h localhost -p 10101 compactionstats > pending tasks: 3177 > compaction type keyspace column family bytes > compacted bytes total progress > Compaction Keyspace1 Standard1 > 250298718 0 n/a > > > Below is a graph showing an oscillation in the number of files. > > Is this how leveled compaction strategy is expected to behave? If so, > is it ever 'done'? > > http://img836.imageshack.us/img836/7294/levelcompactionfiles.png > > [1] (ran three times) ./bin/stress -d HOST --random -l 1 -o insert -c 25 > -e ONE --average-size-values -C 100 -t 75 -n 75000000 > > with this config (dupliate options in original, but I don't think that > should matter) > > update column family Standard1 with rows_cached=1000000 and > keys_cached=0 and compaction_strategy = 'LeveledCompactionStrategy' and > compaction_strategy_options = {sstable_size_in_mb:10} and > compaction_strategy_options = {sstable_size_in_mb:10} and > compression_options={sstable_compression:SnappyCompressor, > chunk_length_kb:64} and row_cache_provider = > 'ConcurrentLinkedHashCacheProvider' and row_cache_keys_to_save = 20000 > and row_cache_save_period = 120; -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
