[
https://issues.apache.org/jira/browse/CASSANDRA-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435597#comment-13435597
]
Florent Clairambault edited comment on CASSANDRA-4481 at 8/16/12 10:18 AM:
---------------------------------------------------------------------------
So, I didn't find (or tried to find) a solution to reproduce this bug. But I
found an (incomplete) fix. I'm on Debian/6.0.5, still with Cassandra/1.1.3:
For a keyspace named "dom2dom" (so that I don't have to replace any name).
In my case I removed all the commitlog files that were created prior to 1.1.3
{code:title=Shell commands|borderStyle=solid}
# 1. Flush cassandra
nodetool flush
# 2. Stop cassandra
service cassandra stop
# 3. Move the sstable files to an other directory
mkdir /var/lib/cassandra/toload
mv /var/lib/cassandra/data/dom2dom /var/lib/cassandra/toload/m2mp
# In my case, I had to create a 127.0.0.2 loopback interface
# and update the cassandra.yaml file to change rpc_address and listen_address
settings
# to 127.0.0.2 so that sstableloader could work.
# 4. Start cassandra
service cassandra stop
# At that point the commitlogs should work again and you should have some new
sstable created
du -sh /var/lib/cassandra/dom2dom
# Returns: 236K
# You now have the new data and not the old one, so you need to load the old
data using sstableloader:
find /var/lib/cassandra/toload/dom2dom/ -type d -exec sstableloader -d
127.0.0.2 {} \;
# In my case, I had to put back localhost in the cassandra.yaml for the
rpc_address and listen_address settings
# You can delete the /var/lib/cassandra/toload folder
{code}
*IMPORTANT NOTE:*
I'm don't think back the old (prior to 1.1.3) commitlog files will work. From
what I've quickly tested it doesn't.
Still, it would be very good to have some kind of error/warning logs around
this commit logs and sstables incompatibility issue. Because you currently only
discover it when you restart your cassandra server.
was (Author: superfc):
So, I didn't find (or tried to find) a solution to reproduce this bug. But
I found an (incomplete) fix. I'm on Debian/6.0.5, still with Cassandra/1.1.3:
For a keyspace named "dom2dom" (so that I don't have to replace any name).
# In my case I removed all the commitlog files that were created prior to 1.1.3
# 1. Flush cassandra
nodetool flush
# 2. Stop cassandra
service cassandra stop
# 3. Move the sstable files to an other directory
mkdir /var/lib/cassandra/toload
mv /var/lib/cassandra/data/dom2dom /var/lib/cassandra/toload/m2mp
# In my case, I had to create a 127.0.0.2 loopback interface
# and update the cassandra.yaml file to change rpc_address and listen_address
settings
# to 127.0.0.2 so that sstableloader could work.
# 4. Start cassandra
service cassandra stop
# At that point the commitlogs should work again and you should have some new
sstable created
du -sh /var/lib/cassandra/dom2dom
# Returns: 236K
# You now have the new data and not the old one, so you need to load the old
data using sstableloader:
find /var/lib/cassandra/toload/dom2dom/ -type d -exec sstableloader -d
127.0.0.2 {} \;
# In my case, I had to put back localhost in the cassandra.yaml for the
rpc_address and listen_address settings
# You can delete the /var/lib/cassandra/toload folder
IMPORTANT NOTE:
I'm not sure that putting back the old (prior to 1.1.3) commitlog files will
work. From what I've quickly tested it doesn't.
Still, I think there's should be more information in the logs to get what is
happening. It seems very strange that the system could get stuck like this
without any a single error message.
> Commitlog not replayed after restart - data lost
> ------------------------------------------------
>
> Key: CASSANDRA-4481
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4481
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.2
> Environment: Single node cluster on 64Bit CentOS
> Reporter: Ivo Meißner
> Priority: Critical
>
> When data is written to the commitlog and I restart the machine, all commited
> data is lost that has not been flushed to disk.
> In the startup logs it says that it replays the commitlog successfully, but
> the data is not available then.
> When I open the commitlog file in an editor I can see the added data, but
> after the restart it cannot be fetched from cassandra.
> {code}
> INFO 09:59:45,362 Replaying
> /var/myproject/cassandra/commitlog/CommitLog-83203377067.log
> INFO 09:59:45,476 Finished reading
> /var/myproject/cassandra/commitlog/CommitLog-83203377067.log
> INFO 09:59:45,476 Log replay complete, 0 replayed mutations
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
