[
https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105596#comment-14105596
]
Jeremiah Jordan commented on CASSANDRA-7042:
--------------------------------------------
This is possibly CASSANDRA-7511. Are you truncating ever? Is your commit log
or your sstable data what is taking up the space? The fact that it gets fixed
on restart makes me think commit log.
> Disk space growth until restart
> -------------------------------
>
> Key: CASSANDRA-7042
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7042
> Project: Cassandra
> Issue Type: Bug
> Environment: Ubuntu 12.04
> Sun Java 7
> Cassandra 2.0.6
> Reporter: Zach Aller
> Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot
> 2014-04-18 at 11.47.30 AM.png, Screen Shot 2014-04-22 at 1.40.41 PM.png,
> after.log, before.log, tabledump_after_restart.txt,
> tabledump_before_restart.txt
>
>
> Cassandra will constantly eat disk space not sure whats causing it the only
> thing that seems to fix it is a restart of cassandra this happens about every
> 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we
> restart cassandra it usually all clears itself up and disks return to normal
> for a while then something triggers its and starts climbing again. Sometimes
> when we restart compactions pending skyrocket and if we restart a second time
> the compactions pending drop off back to a normal level. One other thing to
> note is the space is not free'd until cassandra starts back up and not when
> shutdown.
> I will get a clean log of before and after restarting next time it happens
> and post it.
> Here is a common ERROR in our logs that might be related
> {noformat}
> ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java
> (line 196) Exception in thread Thread[CompactionExecutor:46,1,main]
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
> at
> org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355)
> at
> org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:67)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(Unknown Source)
> at
> org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:58)
> at
> org.apache.cassandra.io.util.ThrottledReader.<init>(ThrottledReader.java:35)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49)
> ... 17 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
