Dan,
What partitioner are you using and did you just swap out the binary?
Going from 70GB to 200GB+ is extremely odd in any scenario.
Maybe Carlos Rolo has an idea about this issue. He did a ton of 1.2 cluster
upgrades.
As for the tombstones, its the stat for the last five minutes. You could
have had it on the old cluster but didn't catch it with cfstats.
Side note:
Looking at the outputs looks like you have something wrong with the data
model too. 2GB partition size is kinda big.
Best,
Matija
On Sun, Dec 31, 2017 at 3:47 AM, Dan Washusen <d...@reactive.org> wrote:
> Nope, just ran out of disk space... So on 1.2.x I had 70GB used on a 200GB
> disk and everything was great, with 2.0.x I'm now at 99% used and getting
> exception while compacting about insufficient disk space. FML...
>
> Dan Washusen
>
>
> On Sun, Dec 31, 2017 at 6:47 AM, Dan Washusen <d...@reactive.org> wrote:
>
>> Thanks for the response Jeff. It wasn't snapshots but after running
>> upgradesstables on all nodes I started a repair and it seems like the file
>> sizes are reducing:
>>
>> INFO [CompactionExecutor:1626] 2017-12-30 19:42:36,065
>> CompactionTask.java (line 299) Compacted 2 sstables to
>> [/appdata/lib/cassandra/data/dp/s_evt/dp-s_evt-jb-302,]. 9,663,696,752
>> bytes to 4,834,895,601 (~50% of original) in 3,899,888ms = 1.182320MB/s.
>> 90,533 total partitions merged to 45,278. Partition merge counts were
>> {1:23, 2:45255, }
>>
>> Dan Washusen
>>
>>
>> On Sun, Dec 31, 2017 at 1:51 AM, Jeff Jirsa <jji...@gmail.com> wrote:
>>
>>> 1.2 to 2.0 was a long time ago for many of us, but I don’t recall
>>> anything that should have doubled size other than perhaps temporarily
>>> during the sstable rewrite or snapshots (which may? Be automatic on
>>> upgrade).
>>>
>>> The bloom filters, sstable count, compression ratio in cfstats all look
>>> similar, only the size is double, so that sorta hints st maybe a snapshot
>>>
>>> You have few sstables, looks like STCS, so it’d be possible that if the
>>> upgrade is still running, maybe one sstable of the old version still
>>> (temporarily) exists on disk causing it to be double counted.
>>>
>>>
>>>
>>> --
>>> Jeff Jirsa
>>>
>>>
>>> On Dec 29, 2017, at 4:33 PM, Dan Washusen <d...@reactive.org> wrote:
>>>
>>> Hi All,
>>> We're taking advantage of the lull in traffic to go through a production
>>> cluster upgrade from 1.2.x (latest) to 2.0.x (latest). We have three nodes
>>> with a replication factor of three. I've noticed that the 'space used' has
>>> almost doubled as a result of running 'nodetool upgradesstables'.
>>>
>>> Anyone have any ideas? Is that to be expected?
>>>
>>> For comparison, on a node (pre-upgrade):
>>>
>>>> nodetool cfstats dp.s_evt
>>>> Keyspace: dp
>>>> Read Count: 190570567
>>>> Read Latency: 2.6280611004164145 ms.
>>>> Write Count: 46213651
>>>> Write Latency: 0.08166790944519835 ms.
>>>> Pending Tasks: 0
>>>> Column Family: s_evt
>>>> SSTable count: 8
>>>> Space used (live): 36269415929
>>>> Space used (total): 36274282945
>>>> SSTable Compression Ratio: 0.23450301499990572
>>>> Number of Keys (estimate): 3213696
>>>> Memtable Columns Count: 2934
>>>> Memtable Data Size: 9561951
>>>> Memtable Switch Count: 1974
>>>> Read Count: 190570567
>>>> Read Latency: 2.628 ms.
>>>> Write Count: 46213651
>>>> Write Latency: 0.082 ms.
>>>> Pending Tasks: 0
>>>> Bloom Filter False Positives: 1162636
>>>> Bloom Filter False Ratio: 0.73869
>>>> Bloom Filter Space Used: 4492256
>>>> Compacted row minimum size: 373
>>>> Compacted row maximum size: 1996099046
>>>> Compacted row mean size: 63595
>>>> Average live cells per slice (last five minutes): 11.0
>>>> Average tombstones per slice (last five minutes): 0.0
>>>
>>>
>>> And after upgrading and running 'upgradesstables' (different node):
>>>
>>>> nodetool cfstats dp.s_evt
>>>> Keyspace: dp
>>>> Read Count: 1461617
>>>> Read Latency: 4.9734411921864625 ms.
>>>> Write Count: 359250
>>>> Write Latency: 0.11328054279749478 ms.
>>>> Pending Tasks: 0
>>>> Table: s_evt
>>>> SSTable count: 6
>>>> Space used (live), bytes: 71266932602
>>>> Space used (total), bytes: 71266932602
>>>> Off heap memory used (total), bytes: 44853104
>>>> SSTable Compression Ratio: 0.2387480210082192
>>>> Number of keys (estimate): 3307776
>>>> Memtable cell count: 603223
>>>> Memtable data size, bytes: 121913569
>>>> Memtable switch count: 9
>>>> Local read count: 1461617
>>>> Local read latency: 7.248 ms
>>>> Local write count: 359250
>>>> Local write latency: 0.110 ms
>>>> Pending tasks: 0
>>>> Bloom filter false positives: 2501
>>>> Bloom filter false ratio: 0.01118
>>>> Bloom filter space used, bytes: 4135248
>>>> Bloom filter off heap memory used, bytes: 4135200
>>>> Index summary off heap memory used, bytes: 723576
>>>> Compression metadata off heap memory used, bytes:
>>>> 39994328
>>>> Compacted partition minimum bytes: 536
>>>> Compacted partition maximum bytes: 2874382626
>>>> Compacted partition mean bytes: 108773
>>>> Average live cells per slice (last five minutes): 11.0
>>>> Average tombstones per slice (last five minutes): 17.0
>>>
>>>
>>> Column familiy definition:
>>>
>>>> create column family s_evt with column_type = 'Super' and comparator =
>>>> 'TimeUUIDType' and subcomparator = 'UTF8Type';
>>>
>>>
>>>
>>> Also curious why the 'Average tombstones per slice' value has gone from
>>> 0 to 17. Note sure if its relevant but way back when we used to write
>>> values to that (super) column family with a TTL, but for a long time now
>>> its been append only (with no TTL)...
>>>
>>> Thanks,
>>> Dan
>>>
>>>
>>
>
