[
https://issues.apache.org/jira/browse/CASSANDRA-8290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14208487#comment-14208487
]
Sam Tunnicliffe commented on CASSANDRA-8290:
--------------------------------------------
The specific problem here is that we pre-emptively archive segments once we
know they're not going to receive any further mutations (i.e. they're full).
However, the segment may not actually be discarded or recycled immediately as
it can still contain mutations for unflushed column families. If the server is
stopped while we're in this state, the original segment file will remain in the
commitlog directory. When the server is restarted, it tries to archive any
pre-existing segment files (CASSANDRA-6904), but in this case the target
already exists, which leads to the reported error.
We could help this somewhat by moving the archival to the point where the
segment is recycled or discarded, shortening the window in which we have
archived, but not active files present (of course, it wouldn't completely close
that window).
However, there's a bigger issue here which is that due to recycling, CL
segments are not really immutable and so hardlinking probably isn't the right
way to archive them. I would modify the command you're using to copy the files
to the archive location instead.
I noticed we use {{ln}} in the example command, so I've attached a patch that
changes that to {{cp}}
> archiving commitlogs after restart fails
> -----------------------------------------
>
> Key: CASSANDRA-8290
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8290
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra 2.0.11
> Debian wheezy
> Reporter: Manuel Lausch
> Assignee: Sam Tunnicliffe
> Priority: Minor
>
> After update to Cassandra 2.0.11 Cassandra mostly fails during startup while
> archiving commitlogs
> see logfile:
> {noformat}
> RROR [main] 2014-11-03 13:08:59,388 CassandraDaemon.java (line 513) Exception
> encountered during startup
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: java.io.IOException: Exception while executing
> the command: /bin/ln
> /var/lib/cassandra/commitlog/CommitLog-3-1413451666161.log
> /var/lib/cassandra/archive/CommitLog-3-1413451666161.log, command error Code:
> 1, command output: /bin/ln: failed to create hard link
> `/var/lib/cassandra/archive/CommitLog-3-1413451666161.log': File exists
> at
> org.apache.cassandra.db.commitlog.CommitLogArchiver.maybeWaitForArchiving(CommitLogArchiver.java:158)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:124)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:336)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: java.io.IOException: Exception while executing
> the command: /bin/ln
> /var/lib/cassandra/commitlog/CommitLog-3-1413451666161.log
> /var/lib/cassandra/archive/CommitLog-3-1413451666161.log, command error Code:
> 1, command output: /bin/ln: failed to create hard link
> `/var/lib/cassandra/archive/CommitLog-3-1413451666161.log': File exists
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:188)
> at
> org.apache.cassandra.db.commitlog.CommitLogArchiver.maybeWaitForArchiving(CommitLogArchiver.java:145)
> ... 4 more
> Caused by: java.lang.RuntimeException: java.io.IOException: Exception while
> executing the command: /bin/ln
> /var/lib/cassandra/commitlog/CommitLog-3-1413451666161.log
> /var/lib/cassandra/archive/CommitLog-3-1413451666161.log, command error Code:
> 1, command output: /bin/ln: failed to create hard link
> `/var/lib/cassandra/archive/CommitLog-3-1413451666161.log': File exists
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Exception while executing the command:
> /bin/ln /var/lib/cassandra/commitlog/CommitLog-3-1413451666161.log
> /var/lib/cassandra/archive/CommitLog-3-1413451666161.log, command error Code:
> 1, command output: /bin/ln: failed to create hard link
> `/var/lib/cassandra/archive/CommitLog-3-1413451666161.log': File exists
> at org.apache.cassandra.utils.FBUtilities.exec(FBUtilities.java:604)
> at
> org.apache.cassandra.db.commitlog.CommitLogArchiver.exec(CommitLogArchiver.java:197)
> at
> org.apache.cassandra.db.commitlog.CommitLogArchiver.access$100(CommitLogArchiver.java:44)
> at
> org.apache.cassandra.db.commitlog.CommitLogArchiver$1.runMayThrow(CommitLogArchiver.java:132)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ... 5 more
> ERROR [commitlog_archiver:1] 2014-11-03 13:08:59,388 CassandraDaemon.java
> (line 199) Exception in thread Thread[commitlog_archiver:1,5,main]
> java.lang.RuntimeException: java.io.IOException: Exception while executing
> the command: /bin/ln
> /var/lib/cassandra/commitlog/CommitLog-3-1413451666161.log
> /var/lib/cassandra/archive/CommitLog-3-1413451666161.log, command error Code:
> 1, command output: /bin/ln: failed to create hard link
> `/var/lib/cassandra/archive/CommitLog-3-1413451666161.log': File exists
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Exception while executing the command:
> /bin/ln /var/lib/cassandra/commitlog/CommitLog-3-1413451666161.log
> /var/lib/cassandra/archive/CommitLog-3-1413451666161.log, command error Code:
> 1, command output: /bin/ln: failed to create hard link
> `/var/lib/cassandra/archive/CommitLog-3-1413451666161.log': File exists
> at org.apache.cassandra.utils.FBUtilities.exec(FBUtilities.java:604)
> at
> org.apache.cassandra.db.commitlog.CommitLogArchiver.exec(CommitLogArchiver.java:197)
> at
> org.apache.cassandra.db.commitlog.CommitLogArchiver.access$100(CommitLogArchiver.java:44)
> at
> org.apache.cassandra.db.commitlog.CommitLogArchiver$1.runMayThrow(CommitLogArchiver.java:132)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ... 5 more
> {noformat}
> The commitlogs are archived already. In this case it makes no sense to crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
