File descriptors to sstables not closed (even for sstables that have been
deleted due to compaction)
----------------------------------------------------------------------------------------------------
Key: CASSANDRA-1858
URL: https://issues.apache.org/jira/browse/CASSANDRA-1858
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.7.0 rc 1
Environment: Ubuntu 8.04 with Java 1.6.0_22-b04
Reporter: Josep M. Blanquer
The Cassandra process doesn't let go of filedescriptors to sstables.
It was initially spotted due to the fact that the disk utilization was really
high, while the data dirs of sstables held much less (nicely compacted) data.
Performing an "lsof" lists the Cassandra process having basically all open FD's
pointing to lots and lots of deleted files. In fact, I saw the "-1-Data.db" in
the list...which tells me that not even the first sstable was let go.
Restarting the cassandra process obviously gets rid of the references. I've
also manually triggered a GC to try to make sure compaction is completed...to
no avail. Holding FD's for a high-throughput write servers is a big problem
since it accumulates a huge amount of disk space due to compactions..etc...so
it's very easy to run out of space.
There's nothing special in this setup, so I think it'd be easily reproducible.
I get this in a:
* 4-node cluster, 1 Keyspace, 1 CF, RF=2
* the cluster has basically been stood up, and blasted with inserts (now it has
200GB, 50 each)...but it doesn't get reads (I'm performing just loading data
tests).
* While the loading is happening, one can observe that the process keeps
compacting and creating new sstables...but never lets the FD's go.
* even if I trigger compactions and cleanups...the disk util only continues to
go up...i.e., all FD's are still open, plus the compated sstables are added.
* at a particular node I see about 11 fd's for current sstables, and 1250 fd's
pointing to deleted sstables.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
