[
https://issues.apache.org/jira/browse/CASSANDRA-4021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594321#comment-13594321
]
Boris Yen edited comment on CASSANDRA-4021 at 3/6/13 3:57 AM:
--------------------------------------------------------------
We saw this issue again on 1.0.10. It seems this happens when the
file-name-change after compaction is not completed. The following are the
sstables on our file system and the error we saw (the -ea option is removed):
-rw-r--r-- 1 byan admin 66305 Mar 5 10:07 indexBytes-hd-111-Data.db
-rw-r--r-- 1 byan admin 67 Mar 5 10:07 indexBytes-hd-111-Digest.sha1
-rw-r--r-- 1 byan admin 1936 Mar 5 10:07 indexBytes-hd-111-Filter.db
-rw-r--r-- 1 byan admin 333 Mar 5 10:07 indexBytes-hd-111-Index.db
-rw-r--r-- 1 byan admin 4336 Mar 5 10:07 indexBytes-hd-111-Statistics.db
-rw-r--r-- 1 byan admin 240 Mar 5 12:50 indexBytes-hd-112-Data.db
-rw-r--r-- 1 byan admin 67 Mar 5 12:50 indexBytes-hd-112-Digest.sha1
-rw-r--r-- 1 byan admin 16 Mar 5 12:50 indexBytes-hd-112-Filter.db
-rw-r--r-- 1 byan admin 32 Mar 5 12:50 indexBytes-hd-112-Index.db
-rw-r--r-- 1 byan admin 4336 Mar 5 12:50 indexBytes-hd-112-Statistics.db
-rw-r--r-- 1 byan admin 240 Mar 5 15:18 indexBytes-hd-113-Data.db
-rw-r--r-- 1 byan admin 67 Mar 5 15:18 indexBytes-hd-113-Digest.sha1
-rw-r--r-- 1 byan admin 16 Mar 5 15:18 indexBytes-hd-113-Filter.db
-rw-r--r-- 1 byan admin 32 Mar 5 15:18 indexBytes-hd-113-Index.db
-rw-r--r-- 1 byan admin 4336 Mar 5 15:18 indexBytes-hd-113-Statistics.db
-rw-r--r-- 1 byan admin 120 Mar 5 15:26 indexBytes-hd-114-Data.db
-rw-r--r-- 1 byan admin 67 Mar 5 15:26 indexBytes-hd-114-Digest.sha1
-rw-r--r-- 1 byan admin 16 Mar 5 15:26 indexBytes-hd-114-Filter.db
-rw-r--r-- 1 byan admin 16 Mar 5 15:26 indexBytes-hd-114-Index.db
-rw-r--r-- 1 byan admin 4336 Mar 5 15:26 indexBytes-hd-114-Statistics.db
-rw-r--r-- 1 byan admin 333 Mar 5 15:26 indexBytes-hd-115-Index.db
-rw-r--r-- 1 byan admin 66305 Mar 5 15:26 indexBytes-tmp-hd-115-Data.db
-rw-r--r-- 1 byan admin 67 Mar 5 15:26 indexBytes-tmp-hd-115-Digest.sha1
-rw-r--r-- 1 byan admin 1936 Mar 5 15:26 indexBytes-tmp-hd-115-Filter.db
WARN [main] 2013-03-05 10:43:22,494 ColumnFamilyStore.java (line 383) Removing
orphans for /opt/test/tt/indexBytes-hd-115: [Digest.sha1, Index.db, Filter.db,
Data.db]
ERROR [main] 2013-03-05 10:43:22,494 AbstractCassandraDaemon.java (line 373)
Exception encountered during startup
java.io.IOError: java.io.IOException: Failed to delete
/opt/test/tt/indexBytes-hd-115-Digest.sha1
at
org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:392)
at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:193)
at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:356)
at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:107)
Caused by: java.io.IOException: Failed to delete
/opt/ruckuswireless/wsg/db/data/wsg/indexBytes-hd-115-Digest.sha1
at
org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:54)
at
org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:44)
at
org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:388)
... 3 more
It seems the "Descriptor" does not take the "temporary" flag into account in
the "hash" and "equals" function. I was wondering if anyone could confirm this.
was (Author: yulinyen):
We saw this issue again. It seems this happens when the file-name-change
after compaction is not completed. The following are the sstables on our file
system and the error we saw (the -ea option is removed):
-rw-r--r-- 1 byan admin 66305 Mar 5 10:07 indexBytes-hd-111-Data.db
-rw-r--r-- 1 byan admin 67 Mar 5 10:07 indexBytes-hd-111-Digest.sha1
-rw-r--r-- 1 byan admin 1936 Mar 5 10:07 indexBytes-hd-111-Filter.db
-rw-r--r-- 1 byan admin 333 Mar 5 10:07 indexBytes-hd-111-Index.db
-rw-r--r-- 1 byan admin 4336 Mar 5 10:07 indexBytes-hd-111-Statistics.db
-rw-r--r-- 1 byan admin 240 Mar 5 12:50 indexBytes-hd-112-Data.db
-rw-r--r-- 1 byan admin 67 Mar 5 12:50 indexBytes-hd-112-Digest.sha1
-rw-r--r-- 1 byan admin 16 Mar 5 12:50 indexBytes-hd-112-Filter.db
-rw-r--r-- 1 byan admin 32 Mar 5 12:50 indexBytes-hd-112-Index.db
-rw-r--r-- 1 byan admin 4336 Mar 5 12:50 indexBytes-hd-112-Statistics.db
-rw-r--r-- 1 byan admin 240 Mar 5 15:18 indexBytes-hd-113-Data.db
-rw-r--r-- 1 byan admin 67 Mar 5 15:18 indexBytes-hd-113-Digest.sha1
-rw-r--r-- 1 byan admin 16 Mar 5 15:18 indexBytes-hd-113-Filter.db
-rw-r--r-- 1 byan admin 32 Mar 5 15:18 indexBytes-hd-113-Index.db
-rw-r--r-- 1 byan admin 4336 Mar 5 15:18 indexBytes-hd-113-Statistics.db
-rw-r--r-- 1 byan admin 120 Mar 5 15:26 indexBytes-hd-114-Data.db
-rw-r--r-- 1 byan admin 67 Mar 5 15:26 indexBytes-hd-114-Digest.sha1
-rw-r--r-- 1 byan admin 16 Mar 5 15:26 indexBytes-hd-114-Filter.db
-rw-r--r-- 1 byan admin 16 Mar 5 15:26 indexBytes-hd-114-Index.db
-rw-r--r-- 1 byan admin 4336 Mar 5 15:26 indexBytes-hd-114-Statistics.db
-rw-r--r-- 1 byan admin 333 Mar 5 15:26 indexBytes-hd-115-Index.db
-rw-r--r-- 1 byan admin 66305 Mar 5 15:26 indexBytes-tmp-hd-115-Data.db
-rw-r--r-- 1 byan admin 67 Mar 5 15:26 indexBytes-tmp-hd-115-Digest.sha1
-rw-r--r-- 1 byan admin 1936 Mar 5 15:26 indexBytes-tmp-hd-115-Filter.db
WARN [main] 2013-03-05 10:43:22,494 ColumnFamilyStore.java (line 383) Removing
orphans for /opt/ruckuswireless/wsg/db/data/wsg/indexBytes-hd-115:
[Digest.sha1, Index.db, Filter.db, Data.db]
ERROR [main] 2013-03-05 10:43:22,494 AbstractCassandraDaemon.java (line 373)
Exception encountered during startup
java.io.IOError: java.io.IOException: Failed to delete
/opt/ruckuswireless/wsg/db/data/wsg/indexBytes-hd-115-Digest.sha1
at
org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:392)
at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:193)
at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:356)
at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:107)
Caused by: java.io.IOException: Failed to delete
/opt/ruckuswireless/wsg/db/data/wsg/indexBytes-hd-115-Digest.sha1
at
org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:54)
at
org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:44)
at
org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:388)
It seems the "Descriptor" does not take the "temporary" flag into account in
the "hash" and "equals" function. I was wondering if anyone could confirm this.
> CFS.scrubDataDirectories tries to delete nonexistent orphans
> ------------------------------------------------------------
>
> Key: CASSANDRA-4021
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4021
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 2
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Priority: Minor
> Labels: datastax_qa
> Attachments: 4021.txt, node1.log
>
>
> The check only looks for a missing data file, then deletes all other
> components, however it's possible for the data file and another component to
> be missing, causing an error:
> {noformat}
> WARN 17:19:28,765 Removing orphans for
> /var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-24492:
> [Index.db, Filter.db, Digest.sha1, Statistics.db, Data.db]
> ERROR 17:19:28,766 Exception encountered during startup
> java.lang.AssertionError: attempted to delete non-existing file
> system-HintsColumnFamily-hd-24492-Index.db
> at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:49)
> at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:44)
> at
> org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:357)
> at
> org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:167)
> at
> org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:352)
> at
> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:105)
> java.lang.AssertionError: attempted to delete non-existing file
> system-HintsColumnFamily-hd-24492-Index.db
> at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:49)
> at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:44)
> at
> org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:357)
> at
> org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:167)
> at
> org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:352)
> at
> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:105)
> Exception encountered during startup: attempted to delete non-existing file
> system-HintsColumnFamily-hd-24492-Index.db
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
