Agreed that you tend to add capacity to nodes or add nodes once you know you 
have no unneeded data in the cluster.


From: Alain RODRIGUEZ [] 
Sent: Wednesday, April 04, 2018 9:10 AM
To: user
Subject: Re: Urgent Problem - Disk full




When the disks are full, here are the options I can think of depending on the 
situation and how 'full' the disk really is:

- Add capacity - Add a disk, use JBOD adding a second location folder for the 
sstables and move some of them around then restart Cassandra. Or add a new node.
- Reduce disk space used. Some options come to my mind to reduce space used:

1 - Clean tombstones if any (use sstablemetadata for example to check the 
number of tombstones). If you have some not being purged, my first guess would 
be to set 'unchecked_tombstone_compaction' to 'true' at the node level. Yet be 
aware that this will trigger some compactions, that before freeing space, start 
by taking some more temporary!


If remaining space is really low on one node, you can control to compact only 
on the sstables having the higher tombstone ratio after you made the change 
above and that fit in the disk space you have left. It can even be scripted. It 
worked for me in the past with disk 100% full. If you do so, you might have to 
disable/reenable automatic compactions at key moments as well


2 -  If you added nodes recently to the data center you can consider running a 
'nodetool cleanup', but here again, it will start by using more space for 
temporary sstables, and might have no positive impacts if the node only own 
data for its token ranges.


3 - Another common way to easily claim space is to clear snapshots that are not 
needed and might have been forgotten or taken by Cassandra: 'nodetool 
clearsnapshot'. This has no other risk than removing a useful backup.

4 - Delete data from this table or another table (effectively), directly 
removing the sstables indeed - as you use TWCS. If you don't need the data 


5 - Truncate one of those other tables we tend to have that are written 'just 
in case' and actually never used and never read for months. It has been a 
powerful way out of this situation for me in the past too :). I would say: be 
sure that the disk space is used properly.



There is zero reason to believe a full repair would make this better and a lot 
of reason to believe it’ll make it worse


I second that too, just in case. Really, do not run a repair. The only thing it 
could do is bring more data to a node that really don't need it for now.


Finally, when this is behind you, the disk size is something you could consider 
monitoring as it is way easier to fix it when the disk is not completely full 
and it can be fixed preemptively. Usually, 50 to 20% of free disk is 
recommended depending on your use case.




Alain Rodriguez - @arodream -

France / Spain


The Last Pickle - Apache Cassandra Consulting <> 


2018-04-04 15:34 GMT+01:00 Kenneth Brotman <>:

There's also the old snapshots to remove that could be a significant amount of 

-----Original Message-----
From: Kenneth Brotman []
Sent: Wednesday, April 04, 2018 7:28 AM
Subject: RE: Urgent Problem - Disk full


Just wondering: why wouldn't the answer be to:
        1. move anything you want to archive to colder storage off the cluster,
        2. nodetool cleanup
        3. snapshot
        4. use delete command to remove archived data.

Kenneth Brotman

-----Original Message-----
From: Jeff Jirsa []
Sent: Wednesday, April 04, 2018 7:10 AM
Subject: Re: Urgent Problem - Disk full

Yes, this works in TWCS.

Note though that if you have tombstone compaction subproperties set, there may 
be sstables with newer filesystem timestamps that actually hold older Cassandra 
data, in which case sstablemetadata can help finding the sstables with truly 
old timestamps

Also if you’ve expanded the cluster over time and you see an imbalance of disk 
usage on the oldest hosts, “nodetool cleanup” will likely free up some of that 

Jeff Jirsa

> On Apr 4, 2018, at 4:32 AM, Jürgen Albersdorfer 
> <> wrote:
> Hi,
> I have an urgent Problem. - I will run out of disk space in near future.
> Largest Table is a Time-Series Table with TimeWindowCompactionStrategy (TWCS) 
> and default_time_to_live = 0
> Keyspace Replication Factor RF=3. I run C* Version 3.11.2
> We have grown the Cluster over time, so SSTable files have different Dates on 
> different Nodes.
> From Application Standpoint it would be safe to loose some of the oldest Data.
> Is it safe to delete some of the oldest SSTable Files, which will no longer 
> get touched by TWCS Compaction any more, while Node is clean Shutdown? - And 
> doing so for one Node after another?
> Or maybe there is a different way to free some disk space? - Any suggestions?
> best regards
> Jürgen Albersdorfer
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail: 
> <> 

To unsubscribe, e-mail: 
For additional commands, e-mail:

To unsubscribe, e-mail: 
For additional commands, e-mail:

To unsubscribe, e-mail: 
For additional commands, e-mail:


Reply via email to