Agreed that you tend to add capacity to nodes or add nodes once you know you have no unneeded data in the cluster.
From: Alain RODRIGUEZ [mailto:arodr...@gmail.com] Sent: Wednesday, April 04, 2018 9:10 AM To: user cassandra.apache.org Subject: Re: Urgent Problem - Disk full Hi, When the disks are full, here are the options I can think of depending on the situation and how 'full' the disk really is: - Add capacity - Add a disk, use JBOD adding a second location folder for the sstables and move some of them around then restart Cassandra. Or add a new node. - Reduce disk space used. Some options come to my mind to reduce space used: 1 - Clean tombstones if any (use sstablemetadata for example to check the number of tombstones). If you have some not being purged, my first guess would be to set 'unchecked_tombstone_compaction' to 'true' at the node level. Yet be aware that this will trigger some compactions, that before freeing space, start by taking some more temporary! If remaining space is really low on one node, you can control to compact only on the sstables having the higher tombstone ratio after you made the change above and that fit in the disk space you have left. It can even be scripted. It worked for me in the past with disk 100% full. If you do so, you might have to disable/reenable automatic compactions at key moments as well 2 - If you added nodes recently to the data center you can consider running a 'nodetool cleanup', but here again, it will start by using more space for temporary sstables, and might have no positive impacts if the node only own data for its token ranges. 3 - Another common way to easily claim space is to clear snapshots that are not needed and might have been forgotten or taken by Cassandra: 'nodetool clearsnapshot'. This has no other risk than removing a useful backup. 4 - Delete data from this table or another table (effectively), directly removing the sstables indeed - as you use TWCS. If you don't need the data anyway. 5 - Truncate one of those other tables we tend to have that are written 'just in case' and actually never used and never read for months. It has been a powerful way out of this situation for me in the past too :). I would say: be sure that the disk space is used properly. There is zero reason to believe a full repair would make this better and a lot of reason to believe it’ll make it worse I second that too, just in case. Really, do not run a repair. The only thing it could do is bring more data to a node that really don't need it for now. Finally, when this is behind you, the disk size is something you could consider monitoring as it is way easier to fix it when the disk is not completely full and it can be fixed preemptively. Usually, 50 to 20% of free disk is recommended depending on your use case. C*heers, ----------------------- Alain Rodriguez - @arodream - al...@thelastpickle.com France / Spain The Last Pickle - Apache Cassandra Consulting http://www.thelastpickle.com <http://wwwthelastpickle.com> 2018-04-04 15:34 GMT+01:00 Kenneth Brotman <kenbrot...@yahoo.com.invalid>: There's also the old snapshots to remove that could be a significant amount of memory. -----Original Message----- From: Kenneth Brotman [mailto:kenbrot...@yahoo.com.INVALID] Sent: Wednesday, April 04, 2018 7:28 AM To: user@cassandra.apache.org Subject: RE: Urgent Problem - Disk full Jeff, Just wondering: why wouldn't the answer be to: 1. move anything you want to archive to colder storage off the cluster, 2. nodetool cleanup 3. snapshot 4. use delete command to remove archived data. Kenneth Brotman -----Original Message----- From: Jeff Jirsa [mailto:jji...@gmail.com] Sent: Wednesday, April 04, 2018 7:10 AM To: user@cassandra.apache.org Subject: Re: Urgent Problem - Disk full Yes, this works in TWCS. Note though that if you have tombstone compaction subproperties set, there may be sstables with newer filesystem timestamps that actually hold older Cassandra data, in which case sstablemetadata can help finding the sstables with truly old timestamps Also if you’ve expanded the cluster over time and you see an imbalance of disk usage on the oldest hosts, “nodetool cleanup” will likely free up some of that data -- Jeff Jirsa > On Apr 4, 2018, at 4:32 AM, Jürgen Albersdorfer > <juergen.albersdor...@zweiradteile.net> wrote: > > Hi, > > I have an urgent Problem. - I will run out of disk space in near future. > Largest Table is a Time-Series Table with TimeWindowCompactionStrategy (TWCS) > and default_time_to_live = 0 > Keyspace Replication Factor RF=3. I run C* Version 3.11.2 > We have grown the Cluster over time, so SSTable files have different Dates on > different Nodes. > > From Application Standpoint it would be safe to loose some of the oldest Data. > > Is it safe to delete some of the oldest SSTable Files, which will no longer > get touched by TWCS Compaction any more, while Node is clean Shutdown? - And > doing so for one Node after another? > > Or maybe there is a different way to free some disk space? - Any suggestions? > > best regards > Jürgen Albersdorfer > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > <mailto:user-h...@cassandraapache.org> --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org <mailto:user-unsubscribe@cassandra.apacheorg> For additional commands, e-mail: user-h...@cassandra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org <mailto:user-unsubscribe@cassandra.apacheorg> For additional commands, e-mail: user-h...@cassandra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org <mailto:user-unsubscribe@cassandra.apacheorg> For additional commands, e-mail: user-h...@cassandra.apache.org
