[
https://issues.apache.org/jira/browse/CASSANDRA-2598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270362#comment-13270362
]
André Cruz commented on CASSANDRA-2598:
---------------------------------------
I think I have a similar issue but I don't have incremental_backups or
snapshot_before_compaction enabled, and I'm using 1.1.
Since I upgraded to Cassandra 1.1, I get the following error when trying to
delete a CF. After this happens the CF is not accessible anymore, but I cannot
create another one with the same name until I restart the server.
INFO [MigrationStage:1] 2012-05-07 18:10:12,682 ColumnFamilyStore.java (line
634) Enqueuing flush of Memtable-schema_columnfamilies@1128094887(978/1222
serialized/live bytes, 21 ops)
INFO [FlushWriter:2] 2012-05-07 18:10:12,682 Memtable.java (line 266) Writing
Memtable-schema_columnfamilies@1128094887(978/1222 serialized/live bytes, 21
ops)
INFO [FlushWriter:2] 2012-05-07 18:10:12,720 Memtable.java (line 307) Completed
flushing
/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-28-Data.db
(1041 bytes)
INFO [MigrationStage:1] 2012-05-07 18:10:12,721 ColumnFamilyStore.java (line
634) Enqueuing flush of Memtable-schema_columns@1599271050(392/490
serialized/live bytes, 8 ops)
INFO [FlushWriter:2] 2012-05-07 18:10:12,722 Memtable.java (line 266) Writing
Memtable-schema_columns@1599271050(392/490 serialized/live bytes, 8 ops)
INFO [CompactionExecutor:8] 2012-05-07 18:10:12,722 CompactionTask.java (line
114) Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-26-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-28-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfam
ilies-hc-27-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-25-Data.db')]
INFO [FlushWriter:2] 2012-05-07 18:10:12,806 Memtable.java (line 307) Completed
flushing
/var/lib/cassandra/data/system/schema_columns/system-schema_columns-hc-23-Data.db
(447 bytes)
INFO [CompactionExecutor:8] 2012-05-07 18:10:12,811 CompactionTask.java (line
225) Compacted to
[/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-29-Data.db,].
24,797 to 21,431
(~86% of original) bytes for 2 keys at 0.232252MB/s. Time: 88ms.
ERROR [MigrationStage:1] 2012-05-07 18:10:12,895 CLibrary.java (line 158)
Unable to create hard link
com.sun.jna.LastErrorException: errno was 17
at org.apache.cassandra.utils.CLibrary.link(Native Method)
at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:150)
at
org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483)
at org.apache.cassandra.db.DefsTable.dropColumnFamily(DefsTable.java:512)
at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:403)
at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
at
org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR [Thrift:17] 2012-05-07 18:10:12,898 CustomTThreadPoolServer.java (line
204) Error occurred during processing of message.
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.io.IOError: java.io.IOException: Unable to create hard link from
/var/lib/cassandra/data/Disco/Client/Client.json to /var/lib/cassandra/data/
Disco/Client/snapshots/1336410612893-Client/Client.json (errno 17)
at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:372)
at
org.apache.cassandra.service.MigrationManager.announce(MigrationManager.java:191)
at
org.apache.cassandra.service.MigrationManager.announceColumnFamilyDrop(MigrationManager.java:182)
at
org.apache.cassandra.thrift.CassandraServer.system_drop_column_family(CassandraServer.java:948)
at
org.apache.cassandra.thrift.Cassandra$Processor$system_drop_column_family.getResult(Cassandra.java:3348)
at
org.apache.cassandra.thrift.Cassandra$Processor$system_drop_column_family.getResult(Cassandra.java:3336)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.util.concurrent.ExecutionException: java.io.IOError:
java.io.IOException: Unable to create hard link from
/var/lib/cassandra/data/Disco/Client/Client.json to
/var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json
(errno 17)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:368)
... 11 more
Caused by: java.io.IOError: java.io.IOException: Unable to create hard link
from /var/lib/cassandra/data/Disco/Client/Client.json to
/var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json
(errno 17)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1454)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483)
at org.apache.cassandra.db.DefsTable.dropColumnFamily(DefsTable.java:512)
at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:403)
at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
at
org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
... 3 more
Caused by: java.io.IOException: Unable to create hard link from
/var/lib/cassandra/data/Disco/Client/Client.json to
/var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json
(errno 17)
at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:163)
at
org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450)
... 10 more
ERROR [MigrationStage:1] 2012-05-07 18:10:12,899 AbstractCassandraDaemon.java
(line 134) Exception in thread Thread[MigrationStage:1,5,main]
java.io.IOError: java.io.IOException: Unable to create hard link from
/var/lib/cassandra/data/Disco/Client/Client.json to
/var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json
(errno 17)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1454)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483)
at org.apache.cassandra.db.DefsTable.dropColumnFamily(DefsTable.java:512)
at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:403)
at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
at
org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Unable to create hard link from
/var/lib/cassandra/data/Disco/Client/Client.json to
/var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json
(errno 17)
at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:163)
at
org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450)
... 10 more
I've tried recreating the data dirs, to see if this was some kind of
permissions problem, but the error happens every time and with any CF. I'm
using the Cassandra Debian package on Debian squeeze and the Sun JVM (build
1.6.0_26-b03).
> incremental_backups and snapshot_before_compaction duplicate hard links
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-2598
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2598
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8 beta 1
> Environment: linux & jna
> Reporter: Mck SembWever
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: 2598.txt
>
>
> See discussion @ http://thread.gmane.org/gmane.comp.db.cassandra.user/15933/
> Enabling both incremental_backups and snapshot_before_compaction leads to the
> same hard links trying to be created.
> This gives stacktraces like
> java.io.IOError: java.io.IOException: Unable to create hard link from
> /cassandra-data/<keyspace>/<cf>-f-3875-Data.db
> to
> /cassandra-data/<keyspace>/snapshots/compact-<cf>/<cf>-f-3875-Data.db
> (errno 17)
> at
> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1629)
> at
> org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1654)
> at org.apache.cassandra.db.Table.snapshot(Table.java:198)
> at
> org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:504)
> at
> org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:146)
> at
> org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:112)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: Unable to create hard link from
> /cassandra-data/<keyspace>/<cf>-f-3875-Data.db
> to
> /cassandra-data/<keyspace>/snapshots/compact-<cf>/<cf>-f-3875-Data.db
> (errno 17)
> at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:155)
> at
> org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:713)
> at
> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1622)
> ... 10 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
