1.2 to 2.0 was a long time ago for many of us, but I don’t recall anything that should have doubled size other than perhaps temporarily during the sstable rewrite or snapshots (which may? Be automatic on upgrade).
The bloom filters, sstable count, compression ratio in cfstats all look similar, only the size is double, so that sorta hints st maybe a snapshot You have few sstables, looks like STCS, so it’d be possible that if the upgrade is still running, maybe one sstable of the old version still (temporarily) exists on disk causing it to be double counted. -- Jeff Jirsa > On Dec 29, 2017, at 4:33 PM, Dan Washusen <d...@reactive.org> wrote: > > Hi All, > We're taking advantage of the lull in traffic to go through a production > cluster upgrade from 1.2.x (latest) to 2.0.x (latest). We have three nodes > with a replication factor of three. I've noticed that the 'space used' has > almost doubled as a result of running 'nodetool upgradesstables'. > > Anyone have any ideas? Is that to be expected? > > For comparison, on a node (pre-upgrade): >> nodetool cfstats dp.s_evt >> Keyspace: dp >> Read Count: 190570567 >> Read Latency: 2.6280611004164145 ms. >> Write Count: 46213651 >> Write Latency: 0.08166790944519835 ms. >> Pending Tasks: 0 >> Column Family: s_evt >> SSTable count: 8 >> Space used (live): 36269415929 >> Space used (total): 36274282945 >> SSTable Compression Ratio: 0.23450301499990572 >> Number of Keys (estimate): 3213696 >> Memtable Columns Count: 2934 >> Memtable Data Size: 9561951 >> Memtable Switch Count: 1974 >> Read Count: 190570567 >> Read Latency: 2.628 ms. >> Write Count: 46213651 >> Write Latency: 0.082 ms. >> Pending Tasks: 0 >> Bloom Filter False Positives: 1162636 >> Bloom Filter False Ratio: 0.73869 >> Bloom Filter Space Used: 4492256 >> Compacted row minimum size: 373 >> Compacted row maximum size: 1996099046 >> Compacted row mean size: 63595 >> Average live cells per slice (last five minutes): 11.0 >> Average tombstones per slice (last five minutes): 0.0 > > And after upgrading and running 'upgradesstables' (different node): >> nodetool cfstats dp.s_evt >> Keyspace: dp >> Read Count: 1461617 >> Read Latency: 4.9734411921864625 ms. >> Write Count: 359250 >> Write Latency: 0.11328054279749478 ms. >> Pending Tasks: 0 >> Table: s_evt >> SSTable count: 6 >> Space used (live), bytes: 71266932602 >> Space used (total), bytes: 71266932602 >> Off heap memory used (total), bytes: 44853104 >> SSTable Compression Ratio: 0.2387480210082192 >> Number of keys (estimate): 3307776 >> Memtable cell count: 603223 >> Memtable data size, bytes: 121913569 >> Memtable switch count: 9 >> Local read count: 1461617 >> Local read latency: 7.248 ms >> Local write count: 359250 >> Local write latency: 0.110 ms >> Pending tasks: 0 >> Bloom filter false positives: 2501 >> Bloom filter false ratio: 0.01118 >> Bloom filter space used, bytes: 4135248 >> Bloom filter off heap memory used, bytes: 4135200 >> Index summary off heap memory used, bytes: 723576 >> Compression metadata off heap memory used, bytes: 39994328 >> Compacted partition minimum bytes: 536 >> Compacted partition maximum bytes: 2874382626 >> Compacted partition mean bytes: 108773 >> Average live cells per slice (last five minutes): 11.0 >> Average tombstones per slice (last five minutes): 17.0 > > Column familiy definition: >> create column family s_evt with column_type = 'Super' and comparator = >> 'TimeUUIDType' and subcomparator = 'UTF8Type'; > > > Also curious why the 'Average tombstones per slice' value has gone from 0 to > 17. Note sure if its relevant but way back when we used to write values to > that (super) column family with a TTL, but for a long time now its been > append only (with no TTL)... > > Thanks, > Dan
