[
https://issues.apache.org/jira/browse/CASSANDRA-730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12803417#action_12803417
]
Anthony Molinaro commented on CASSANDRA-730:
--------------------------------------------
Not a problem, just glad it's on the radar, and hopefully it'll make it into
0.5.1.
By the way, a workaround when you get into this situation is to switch to a
single directory in the config for the bootstrapping node and restart it.
Just stopping the bootstrapping node, actually puts the system into a broken
state, where you constantly get these exceptions
ERROR [pool-1-thread-64] 2010-01-21 07:37:42,610 Cassandra.java (line 1064)
Internal error processing insert
java.lang.AssertionError
at
org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:212)
at
org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:129)
at
org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
at
org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1186)
at
org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:169)
at
org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:466)
at
org.apache.cassandra.service.CassandraServer.insert(CassandraServer.java:417)
at
org.apache.cassandra.service.Cassandra$Processor$insert.process(Cassandra.java:1056)
at
org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
So I had to restart the bootstrapping node. This seems like it could be an
issue if for whatever reason you start bootstrapping a node, then have to undo
that action. Maybe there's the need for some sort of 'recover' command which
helps you get out of a failed bootstrap?
> When multiple DataFileDirectory entries exist in config files for the same
> sstable can end up in different directories.
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-730
> URL: https://issues.apache.org/jira/browse/CASSANDRA-730
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.5
> Reporter: Anthony Molinaro
>
> I have a cluster of 8 cassandra nodes, each node has 3 disks and I have the
> following in my storage-conf.xml
> <DataFileDirectories>
> <DataFileDirectory>/var/lib/cassandra/data1</DataFileDirectory>
> <DataFileDirectory>/var/lib/cassandra/data2</DataFileDirectory>
> <DataFileDirectory>/var/lib/cassandra/data3</DataFileDirectory>
> </DataFileDirectories>
> Upon adding a new node, setting AutoBootstrap to true and starting it, it
> seemed to start bootstrapping, but then
> failed with the following exception.
> java.io.IOException: rename failed of
> /var/lib/cassandra/data3/retarget/user_to_acctids-1-Index.db
> java.io.IOError: java.io.IOException: rename failed of
> /var/lib/cassandra/data3/retarget/user_to_acctids-1-Index.db
> at
> org.apache.cassandra.io.SSTableWriter.rename(SSTableWriter.java:154)
> at
> org.apache.cassandra.io.SSTableWriter.renameAndOpen(SSTableWriter.java:161)
> at
> org.apache.cassandra.io.Streaming$StreamCompletionHandler.onStreamCompletion(Streaming.java:301)
> at
> org.apache.cassandra.net.io.ContentStreamState.handleStreamCompletion(ContentStreamState.java:108)
> at
> org.apache.cassandra.net.io.ContentStreamState.read(ContentStreamState.java:90)
> at org.apache.cassandra.net.io.TcpReader.read(TcpReader.java:95)
> at
> org.apache.cassandra.net.TcpConnection$ReadWorkItem.run(TcpConnection.java:445)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.IOException: rename failed of
> /var/lib/cassandra/data3/retarget/user_to_acctids-1-Index.db
> at
> org.apache.cassandra.utils.FBUtilities.renameWithConfirm(FBUtilities.java:306)
> at
> org.apache.cassandra.io.SSTableWriter.rename(SSTableWriter.java:150)
> ... 9 more
> Upon inspecting the directories I noticed the following
> find /var/lib/cassandra -follow
> /var/lib/cassandra
> /var/lib/cassandra/data2
> /var/lib/cassandra/data2/system
> /var/lib/cassandra/data2/retarget
> /var/lib/cassandra/data2/retarget/user_to_acctids-tmp-1-Filter.db
> /var/lib/cassandra/.cassandra-placeholder
> /var/lib/cassandra/data3
> /var/lib/cassandra/data3/system
> /var/lib/cassandra/data3/retarget
> /var/lib/cassandra/data3/retarget/user_to_acctids-tmp-1-Data.db
> /var/lib/cassandra/logs
> /var/lib/cassandra/logs/system.log
> /var/lib/cassandra/data1
> /var/lib/cassandra/data1/system
> /var/lib/cassandra/data1/retarget
> /var/lib/cassandra/data1/retarget/user_to_acctids-tmp-1-Index.db
> /var/lib/cassandra/commitlog
> /var/lib/cassandra/commitlog/CommitLog-1264050238700.log
> showing the sstable files were in different directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
