Failed compactation doesn't delete temporary files causing hd to fill
---------------------------------------------------------------------
Key: CASSANDRA-2424
URL: https://issues.apache.org/jira/browse/CASSANDRA-2424
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.7.4
Reporter: Thibaut
Fix For: 0.7.5
Hi,
We seem to have a few incorrect keys in one of our columns (Seems to be a
memory bit flip). Cassandra will try to compact and compact those tables again
but won't delete any temporary files.
INFO [CompactionExecutor:1] 2011-04-05 17:52:45,494 SSTableWriter.java (line
108) Last written key :
DecoratedKey(dsearch_1300960988716456<398199492444161_1301855754714_4564397053271578441,
647365617263685f313330303936303938383731363435363c3339383139393439323434343136315f313330313835353735343731345f34353634333937303533323731353738343431)
INFO [CompactionExecutor:1] 2011-04-05 17:52:45,521 SSTableWriter.java (line
109) Current key :
DecoratedKey(dsearch_13009609887164564398199492444161_1301855754760_4564395937046043595,
647365617263685f31333030393630393838373136343536343339383139393439323434343136315f313330313835353735343736305f34353634333935393337303436303433353935)
INFO [CompactionExecutor:1] 2011-04-05 17:52:45,521 SSTableWriter.java (line
110) Writing into file
/cassandra/data/table_userentries/table_userentries-tmp-f-3550-Data.db
ERROR [CompactionExecutor:1] 2011-04-05 17:52:45,522
AbstractCassandraDaemon.java (line 112) Fatal exception in thread
Thread[CompactionExecutor:1,1,main]
java.io.IOException: Keys must be written in ascending order.
at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
at
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:452)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
ls -l /cassandra/data/table_userentries/table_userentries-tmp*
-rw-r--r-- 1 root root 13G 2011-04-05 16:09
/cassandra/data/table_userentries/table_userentries-tmp-f-3507-Data.db
-rw-r--r-- 1 root root 512M 2011-04-05 16:09
/cassandra/data/table_userentries/table_userentries-tmp-f-3507-Index.db
-rw-r--r-- 1 root root 13G 2011-04-05 16:21
/cassandra/data/table_userentries/table_userentries-tmp-f-3512-Data.db
-rw-r--r-- 1 root root 512M 2011-04-05 16:21
/cassandra/data/table_userentries/table_userentries-tmp-f-3512-Index.db
-rw-r--r-- 1 root root 13G 2011-04-05 16:33
/cassandra/data/table_userentries/table_userentries-tmp-f-3517-Data.db
-rw-r--r-- 1 root root 512M 2011-04-05 16:33
/cassandra/data/table_userentries/table_userentries-tmp-f-3517-Index.db
-rw-r--r-- 1 root root 13G 2011-04-05 16:44
/cassandra/data/table_userentries/table_userentries-tmp-f-3521-Data.db
-rw-r--r-- 1 root root 512M 2011-04-05 16:44
/cassandra/data/table_userentries/table_userentries-tmp-f-3521-Index.db
-rw-r--r-- 1 root root 13G 2011-04-05 16:55
/cassandra/data/table_userentries/table_userentries-tmp-f-3527-Data.db
-rw-r--r-- 1 root root 512M 2011-04-05 16:55
/cassandra/data/table_userentries/table_userentries-tmp-f-3527-Index.db
-rw-r--r-- 1 root root 13G 2011-04-05 17:05
/cassandra/data/table_userentries/table_userentries-tmp-f-3531-Data.db
-rw-r--r-- 1 root root 512M 2011-04-05 17:06
/cassandra/data/table_userentries/table_userentries-tmp-f-3531-Index.db
-rw-r--r-- 1 root root 13G 2011-04-05 17:16
/cassandra/data/table_userentries/table_userentries-tmp-f-3535-Data.db
-rw-r--r-- 1 root root 512M 2011-04-05 17:16
/cassandra/data/table_userentries/table_userentries-tmp-f-3535-Index.db
-rw-r--r-- 1 root root 13G 2011-04-05 17:28
/cassandra/data/table_userentries/table_userentries-tmp-f-3540-Data.db
-rw-r--r-- 1 root root 512M 2011-04-05 17:28
/cassandra/data/table_userentries/table_userentries-tmp-f-3540-Index.db
-rw-r--r-- 1 root root 13G 2011-04-05 17:39
/cassandra/data/table_userentries/table_userentries-tmp-f-3545-Data.db
-rw-r--r-- 1 root root 512M 2011-04-05 17:40
/cassandra/data/table_userentries/table_userentries-tmp-f-3545-Index.db
-rw-r--r-- 1 root root 13G 2011-04-05 17:52
/cassandra/data/table_userentries/table_userentries-tmp-f-3550-Data.db
-rw-r--r-- 1 root root 512M 2011-04-05 17:52
/cassandra/data/table_userentries/table_userentries-tmp-f-3550-Index.db
-rw-r--r-- 1 root root 13G 2011-04-05 18:04
/cassandra/data/table_userentries/table_userentries-tmp-f-3556-Data.db
-rw-r--r-- 1 root root 512M 2011-04-05 18:04
/cassandra/data/table_userentries/table_userentries-tmp-f-3556-Index.db
-rw-r--r-- 1 root root 13G 2011-04-05 18:15
/cassandra/data/table_userentries/table_userentries-tmp-f-3562-Data.db
-rw-r--r-- 1 root root 512M 2011-04-05 18:15
/cassandra/data/table_userentries/table_userentries-tmp-f-3562-Index.db
-rw-r--r-- 1 root root 13G 2011-04-05 18:28
/cassandra/data/table_userentries/table_userentries-tmp-f-3566-Data.db
-rw-r--r-- 1 root root 512M 2011-04-05 18:28
/cassandra/data/table_userentries/table_userentries-tmp-f-3566-Index.db
-rw-r--r-- 1 root root 13G 2011-04-05 18:40
/cassandra/data/table_userentries/table_userentries-tmp-f-3572-Data.db
-rw-r--r-- 1 root root 512M 2011-04-05 18:40
/cassandra/data/table_userentries/table_userentries-tmp-f-3572-Index.db
-rw-r--r-- 1 root root 13G 2011-04-05 18:53
/cassandra/data/table_userentries/table_userentries-tmp-f-3577-Data.db
-rw-r--r-- 1 root root 512M 2011-04-05 18:53
/cassandra/data/table_userentries/table_userentries-tmp-f-3577-Index.db
-rw-r--r-- 1 root root 13G 2011-04-05 19:06
/cassandra/data/table_userentries/table_userentries-tmp-f-3582-Data.db
-rw-r--r-- 1 root root 512M 2011-04-05 19:07
/cassandra/data/table_userentries/table_userentries-tmp-f-3582-Index.db
-rw-r--r-- 1 root root 19G 2011-04-05 19:29
/cassandra/data/table_userentries/table_userentries-tmp-f-3592-Data.db
-rw-r--r-- 1 root root 736M 2011-04-05 19:29
/cassandra/data/table_userentries/table_userentries-tmp-f-3592-Index.db
-rw-r--r-- 1 root root 19G 2011-04-05 19:42
/cassandra/data/table_userentries/table_userentries-tmp-f-3598-Data.db
-rw-r--r-- 1 root root 736M 2011-04-05 19:42
/cassandra/data/table_userentries/table_userentries-tmp-f-3598-Index.db
-rw-r--r-- 1 root root 19G 2011-04-05 19:56
/cassandra/data/table_userentries/table_userentries-tmp-f-3602-Data.db
-rw-r--r-- 1 root root 736M 2011-04-05 19:56
/cassandra/data/table_userentries/table_userentries-tmp-f-3602-Index.db
-rw-r--r-- 1 root root 19G 2011-04-05 20:12
/cassandra/data/table_userentries/table_userentries-tmp-f-3606-Data.db
-rw-r--r-- 1 root root 736M 2011-04-05 20:12
/cassandra/data/table_userentries/table_userentries-tmp-f-3606-Index.db
-rw-r--r-- 1 root root 19G 2011-04-05 20:28
/cassandra/data/table_userentries/table_userentries-tmp-f-3612-Data.db
-rw-r--r-- 1 root root 736M 2011-04-05 20:28
/cassandra/data/table_userentries/table_userentries-tmp-f-3612-Index.db
-rw-r--r-- 1 root root 19G 2011-04-05 20:44
/cassandra/data/table_userentries/table_userentries-tmp-f-3617-Data.db
-rw-r--r-- 1 root root 736M 2011-04-05 20:44
/cassandra/data/table_userentries/table_userentries-tmp-f-3617-Index.db
-rw-r--r-- 1 root root 19G 2011-04-05 21:01
/cassandra/data/table_userentries/table_userentries-tmp-f-3622-Data.db
-rw-r--r-- 1 root root 736M 2011-04-05 21:01
/cassandra/data/table_userentries/table_userentries-tmp-f-3622-Index.db
-rw-r--r-- 1 root root 19G 2011-04-05 21:23
/cassandra/data/table_userentries/table_userentries-tmp-f-3627-Data.db
-rw-r--r-- 1 root root 736M 2011-04-05 21:23
/cassandra/data/table_userentries/table_userentries-tmp-f-3627-Index.db
-rw-r--r-- 1 root root 19G 2011-04-05 21:48
/cassandra/data/table_userentries/table_userentries-tmp-f-3635-Data.db
-rw-r--r-- 1 root root 736M 2011-04-05 21:48
/cassandra/data/table_userentries/table_userentries-tmp-f-3635-Index.db
-rw-r--r-- 1 root root 19G 2011-04-05 22:16
/cassandra/data/table_userentries/table_userentries-tmp-f-3646-Data.db
-rw-r--r-- 1 root root 736M 2011-04-05 22:16
/cassandra/data/table_userentries/table_userentries-tmp-f-3646-Index.db
-rw-r--r-- 1 root root 19G 2011-04-05 22:48
/cassandra/data/table_userentries/table_userentries-tmp-f-3658-Data.db
-rw-r--r-- 1 root root 736M 2011-04-05 22:48
/cassandra/data/table_userentries/table_userentries-tmp-f-3658-Index.db
-rw-r--r-- 1 root root 19G 2011-04-05 23:22
/cassandra/data/table_userentries/table_userentries-tmp-f-3672-Data.db
-rw-r--r-- 1 root root 736M 2011-04-05 23:21
/cassandra/data/table_userentries/table_userentries-tmp-f-3672-Index.db
-rw-r--r-- 1 root root 19G 2011-04-05 23:53
/cassandra/data/table_userentries/table_userentries-tmp-f-3685-Data.db
-rw-r--r-- 1 root root 736M 2011-04-05 23:52
/cassandra/data/table_userentries/table_userentries-tmp-f-3685-Index.db
-rw-r--r-- 1 root root 19G 2011-04-06 00:29
/cassandra/data/table_userentries/table_userentries-tmp-f-3698-Data.db
-rw-r--r-- 1 root root 736M 2011-04-06 00:28
/cassandra/data/table_userentries/table_userentries-tmp-f-3698-Index.db
-rw-r--r-- 1 root root 19G 2011-04-06 01:01
/cassandra/data/table_userentries/table_userentries-tmp-f-3712-Data.db
-rw-r--r-- 1 root root 736M 2011-04-06 01:01
/cassandra/data/table_userentries/table_userentries-tmp-f-3712-Index.db
-rw-r--r-- 1 root root 19G 2011-04-06 01:36
/cassandra/data/table_userentries/table_userentries-tmp-f-3723-Data.db
-rw-r--r-- 1 root root 736M 2011-04-06 01:36
/cassandra/data/table_userentries/table_userentries-tmp-f-3723-Index.db
-rw-r--r-- 1 root root 19G 2011-04-06 02:08
/cassandra/data/table_userentries/table_userentries-tmp-f-3735-Data.db
-rw-r--r-- 1 root root 736M 2011-04-06 02:08
/cassandra/data/table_userentries/table_userentries-tmp-f-3735-Index.db
-rw-r--r-- 1 root root 13G 2011-04-06 02:32
/cassandra/data/table_userentries/table_userentries-tmp-f-3742-Data.db
-rw-r--r-- 1 root root 512M 2011-04-06 02:32
/cassandra/data/table_userentries/table_userentries-tmp-f-3742-Index.db
-rw-r--r-- 1 root root 4.5G 2011-04-06 02:42
/cassandra/data/table_userentries/table_userentries-tmp-f-3748-Data.db
-rw-r--r-- 1 root root 168M 2011-04-06 02:42
/cassandra/data/table_userentries/table_userentries-tmp-f-3748-Index.db
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
