Re: Cassandra is consuming a lot of disk space

2016-01-14 Thread Rahul Ramesh
Hi Jan, I checked it. There are no old Key Spaces or tables. Thanks for your pointer, I started looking inside the directories. I see lot of snapshots directory inside the table directory. These directories are consuming space. However these snapshots are not shown when I issue listsnapshots

Re: Cassandra is consuming a lot of disk space

2016-01-14 Thread Rahul Ramesh
One update. I cleared the snapshot using nodetool clearsnapshot command. Disk space is recovered now. Because of this issue, I have mounted one more drive to the server and there are some data files there. How can I migrate the data so that I can decommission the drive? Will it work if I just

Re: Cassandra is consuming a lot of disk space

2016-01-14 Thread Jan Kesten
Hi Rahul, it should work as you would expect - simply copy over the sstables from your extra disk to the original one. To minimize downtime of the node you can do something like this: - rsync the files while the node is still running (sstables are immutable) to copy most of the data - edit

Re: Cassandra is consuming a lot of disk space

2016-01-13 Thread Carlos Rolo
You can check if the snapshot exists in the snapshot folder. Repairs stream sstables over, than can temporary increase disk space. But I think Carlos Alonso might be correct. Running compactions might be the issue. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data

Re: Cassandra is consuming a lot of disk space

2016-01-13 Thread Carlos Alonso
I'd have a look also at possible running compactions. If you have big column families with STCS then large compactions may be happening. Check it with nodetool compactionstats Carlos Alonso | Software Engineer | @calonso On 13 January 2016 at 05:22, Kevin O'Connor

Re: Cassandra is consuming a lot of disk space

2016-01-13 Thread Rahul Ramesh
Thanks for your suggestion. Compaction was happening on one of the large tables. The disk space did not decrease much after the compaction. So I ran an external compaction. The disk space decreased by around 10%. However it is still consuming close to 750Gb for load of 250Gb. I even restarted

Re: Cassandra is consuming a lot of disk space

2016-01-13 Thread Jan Kesten
Hi Rahul, just an idea, did you have a look at the data directorys on disk (/var/lib/cassandra/data)? It could be that there are some from old keyspaces that have been deleted and snapshoted before. Try something like "du -sh /var/lib/cassandra/data/*" to verify which keyspace is consuming

Re: Cassandra is consuming a lot of disk space

2016-01-12 Thread Kevin O'Connor
Have you tried restarting? It's possible there's open file handles to sstables that have been compacted away. You can verify by doing lsof and grepping for DEL or deleted. If it's not that, you can run nodetool cleanup on each node to scan all of the sstables on disk and remove anything that it's