[jira] [Commented] (CASSANDRA-12539) Empty CommitLog prevents restart
[
https://issues.apache.org/jira/browse/CASSANDRA-12539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860269#comment-15860269
]
Joshua McKenzie commented on CASSANDRA-12539:
-
+1
> Empty CommitLog prevents restart
>
>
> Key: CASSANDRA-12539
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12539
> Project: Cassandra
> Issue Type: Bug
>Reporter: Stefano Ortolani
>Assignee: Benjamin Lerer
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
>
> A node just crashed (known cause: CASSANDRA-11594) but to my surprise (unlike
> other time) restarting simply fails.
> Checking the logs showed:
> {noformat}
> ERROR [main] 2016-08-25 17:05:22,611 JVMStabilityInspector.java:82 - Exiting
> due to error while processing commit log during initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Could not read commit log descriptor in file
> /data/cassandra/commitlog/CommitLog-6-1468235564433.log
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:650)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:327)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:148)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:181)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:161)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:289)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:685)
> [apache-cassandra-3.0.8.jar:3.0.8]
> INFO [main] 2016-08-25 17:08:56,944 YamlConfigurationLoader.java:85 -
> Configuration location: file:/etc/cassandra/cassandra.yaml
> {noformat}
> Deleting the empty file fixes the problem.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CASSANDRA-12539) Empty CommitLog prevents restart
[
https://issues.apache.org/jira/browse/CASSANDRA-12539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15858384#comment-15858384
]
Joshua McKenzie commented on CASSANDRA-12539:
-
Sure - should be able to get to this tomorrow.
> Empty CommitLog prevents restart
>
>
> Key: CASSANDRA-12539
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12539
> Project: Cassandra
> Issue Type: Bug
>Reporter: Stefano Ortolani
>Assignee: Benjamin Lerer
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
>
> A node just crashed (known cause: CASSANDRA-11594) but to my surprise (unlike
> other time) restarting simply fails.
> Checking the logs showed:
> {noformat}
> ERROR [main] 2016-08-25 17:05:22,611 JVMStabilityInspector.java:82 - Exiting
> due to error while processing commit log during initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Could not read commit log descriptor in file
> /data/cassandra/commitlog/CommitLog-6-1468235564433.log
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:650)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:327)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:148)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:181)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:161)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:289)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:685)
> [apache-cassandra-3.0.8.jar:3.0.8]
> INFO [main] 2016-08-25 17:08:56,944 YamlConfigurationLoader.java:85 -
> Configuration location: file:/etc/cassandra/cassandra.yaml
> {noformat}
> Deleting the empty file fixes the problem.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CASSANDRA-12539) Empty CommitLog prevents restart
[
https://issues.apache.org/jira/browse/CASSANDRA-12539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15858248#comment-15858248
]
Benjamin Lerer commented on CASSANDRA-12539:
||[2.2|https://github.com/apache/cassandra/compare/trunk...blerer:12539-2.2]|[utests|http://cassci.datastax.com/view/Dev/view/blerer/job/blerer-12539-2.2-testall/]|[dtests|http://cassci.datastax.com/view/Dev/view/blerer/job/blerer-12539-2.2-dtest/]|
||[3.0|https://github.com/apache/cassandra/compare/trunk...blerer:12539-3.0]|[utests|http://cassci.datastax.com/view/Dev/view/blerer/job/blerer-12539-3.0-testall/]|[dtests|http://cassci.datastax.com/view/Dev/view/blerer/job/blerer-12539-3.0-dtest/]|
||[3.11|https://github.com/apache/cassandra/compare/trunk...blerer:12539-3.11]|[utests|http://cassci.datastax.com/view/Dev/view/blerer/job/blerer-12539-3.11-testall/]|[dtests|http://cassci.datastax.com/view/Dev/view/blerer/job/blerer-12539-3.11-dtest/]|
||[trunk|https://github.com/apache/cassandra/compare/trunk...blerer:12539-trunk]|[utests|http://cassci.datastax.com/view/Dev/view/blerer/job/blerer-12539-trunk-testall/]|[dtests|http://cassci.datastax.com/view/Dev/view/blerer/job/blerer-12539-trunk-dtest/]|
The patches for 2.2 and 3.0 are the same and the patches for 3.11 and trunk
arethe same.
The patch removes the use of {{RandomAccessFile}} to resize the commitlog
segment file.
As we do not recycle the CommitLog files in 2.2, it is not needed anymore. The
creation of the {{MappedByteBuffer}} will automatically set the size of the
file to {{DatabaseDescriptor.getCommitLogSegmentSize()}}.
By not using an extra file pointer with the {{RandomAccessFile}}, we avoid the
problem that had cause the empty commit log (which was running out of file
pointers when resizing the file segement).
[~JoshuaMcKenzie] do you want to review it?
> Empty CommitLog prevents restart
>
>
> Key: CASSANDRA-12539
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12539
> Project: Cassandra
> Issue Type: Bug
>Reporter: Stefano Ortolani
>Assignee: Benjamin Lerer
>
> A node just crashed (known cause: CASSANDRA-11594) but to my surprise (unlike
> other time) restarting simply fails.
> Checking the logs showed:
> {noformat}
> ERROR [main] 2016-08-25 17:05:22,611 JVMStabilityInspector.java:82 - Exiting
> due to error while processing commit log during initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Could not read commit log descriptor in file
> /data/cassandra/commitlog/CommitLog-6-1468235564433.log
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:650)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:327)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:148)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:181)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:161)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:289)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:685)
> [apache-cassandra-3.0.8.jar:3.0.8]
> INFO [main] 2016-08-25 17:08:56,944 YamlConfigurationLoader.java:85 -
> Configuration location: file:/etc/cassandra/cassandra.yaml
> {noformat}
> Deleting the empty file fixes the problem.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CASSANDRA-12539) Empty CommitLog prevents restart
[
https://issues.apache.org/jira/browse/CASSANDRA-12539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15621732#comment-15621732
]
Benjamin Lerer commented on CASSANDRA-12539:
bq. There seems to be a very easy way to reproduce this condition
Which condition? Having an empty commit log segment?
An empty commit log segment can occurs for different reasons. Running out of
file pointers is only one of them. Cassandra has no way to know that it has an
empty commit log segment because it run out of file pointers before crashing.
An empty commit log segment might have been caused by another problem were you
actually lost some data. In this case, you will actually want to be notify of
the problem.
Now, the real problem here is not that Cassandra does not ignore empty commit
log in startup. It is that it should not create an empty commit log in this
corner case. I will look at what I can do about it.
> Empty CommitLog prevents restart
>
>
> Key: CASSANDRA-12539
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12539
> Project: Cassandra
> Issue Type: Bug
>Reporter: Stefano Ortolani
>
> A node just crashed (known cause: CASSANDRA-11594) but to my surprise (unlike
> other time) restarting simply fails.
> Checking the logs showed:
> {noformat}
> ERROR [main] 2016-08-25 17:05:22,611 JVMStabilityInspector.java:82 - Exiting
> due to error while processing commit log during initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Could not read commit log descriptor in file
> /data/cassandra/commitlog/CommitLog-6-1468235564433.log
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:650)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:327)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:148)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:181)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:161)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:289)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:685)
> [apache-cassandra-3.0.8.jar:3.0.8]
> INFO [main] 2016-08-25 17:08:56,944 YamlConfigurationLoader.java:85 -
> Configuration location: file:/etc/cassandra/cassandra.yaml
> {noformat}
> Deleting the empty file fixes the problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12539) Empty CommitLog prevents restart
[
https://issues.apache.org/jira/browse/CASSANDRA-12539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15616594#comment-15616594
]
Arvind Nithrakashyap commented on CASSANDRA-12539:
--
There seems to be a very easy way to reproduce this condition - so it seems
like Cassandra should be able to handle this case without needing manual
intervention at each occurrence. Especially when it gets deployed in an
enterprise setting, when services are expected to auto heal.
> Empty CommitLog prevents restart
>
>
> Key: CASSANDRA-12539
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12539
> Project: Cassandra
> Issue Type: Bug
>Reporter: Stefano Ortolani
>
> A node just crashed (known cause: CASSANDRA-11594) but to my surprise (unlike
> other time) restarting simply fails.
> Checking the logs showed:
> {noformat}
> ERROR [main] 2016-08-25 17:05:22,611 JVMStabilityInspector.java:82 - Exiting
> due to error while processing commit log during initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Could not read commit log descriptor in file
> /data/cassandra/commitlog/CommitLog-6-1468235564433.log
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:650)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:327)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:148)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:181)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:161)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:289)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:685)
> [apache-cassandra-3.0.8.jar:3.0.8]
> INFO [main] 2016-08-25 17:08:56,944 YamlConfigurationLoader.java:85 -
> Configuration location: file:/etc/cassandra/cassandra.yaml
> {noformat}
> Deleting the empty file fixes the problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12539) Empty CommitLog prevents restart
[
https://issues.apache.org/jira/browse/CASSANDRA-12539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15615311#comment-15615311
]
Benjamin Lerer commented on CASSANDRA-12539:
What happened is that the node crashed while a new commit log segment why
Cassandra was trying to get a new file pointer to resize the commit log segment.
Neverthess, it is the expected behaviour that Cassandra should not start if
some commitlog segments are empty (there are some tests that check that
behaviour).
An empty commitlog is a sign of a problem but Cassandra as no way to determine
the root cause of the problem that is why it will not start and notify the
administrator.
> Empty CommitLog prevents restart
>
>
> Key: CASSANDRA-12539
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12539
> Project: Cassandra
> Issue Type: Bug
>Reporter: Stefano Ortolani
>
> A node just crashed (known cause: CASSANDRA-11594) but to my surprise (unlike
> other time) restarting simply fails.
> Checking the logs showed:
> {noformat}
> ERROR [main] 2016-08-25 17:05:22,611 JVMStabilityInspector.java:82 - Exiting
> due to error while processing commit log during initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Could not read commit log descriptor in file
> /data/cassandra/commitlog/CommitLog-6-1468235564433.log
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:650)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:327)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:148)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:181)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:161)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:289)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:685)
> [apache-cassandra-3.0.8.jar:3.0.8]
> INFO [main] 2016-08-25 17:08:56,944 YamlConfigurationLoader.java:85 -
> Configuration location: file:/etc/cassandra/cassandra.yaml
> {noformat}
> Deleting the empty file fixes the problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12539) Empty CommitLog prevents restart
[
https://issues.apache.org/jira/browse/CASSANDRA-12539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15614667#comment-15614667
]
Arvind Nithrakashyap commented on CASSANDRA-12539:
--
It looks like a crash while writing the logfile can lead to a zero byte file.
The following patch which causes a crash at a specific point reliably produces
a commit log full of zeros
{noformat}
diff --git a/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java
b/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java
index 0a03c3c..20cddf8 100644
--- a/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java
+++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java
@@ -153,6 +153,7 @@ public abstract class CommitLogSegment
descriptor = new CommitLogDescriptor(id,
commitLog.configuration.getCompressorClass());
logFile = new File(commitLog.location, descriptor.fileName());
+logger.error("location="+descriptor.fileName());
try
{
channel = FileChannel.open(logFile.toPath(),
StandardOpenOption.WRITE, StandardOpenOption.READ, StandardOpenOption.CREATE);
@@ -164,6 +165,9 @@ public abstract class CommitLogSegment
}
buffer = createBuffer(commitLog);
+if (true) {
+ throw new IllegalArgumentException("Here!");
+}
// write the header
CommitLogDescriptor.writeHeader(buffer, descriptor);
endOfBuffer = buffer.capacity();
{noformat}
> Empty CommitLog prevents restart
>
>
> Key: CASSANDRA-12539
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12539
> Project: Cassandra
> Issue Type: Bug
>Reporter: Stefano Ortolani
>
> A node just crashed (known cause: CASSANDRA-11594) but to my surprise (unlike
> other time) restarting simply fails.
> Checking the logs showed:
> {noformat}
> ERROR [main] 2016-08-25 17:05:22,611 JVMStabilityInspector.java:82 - Exiting
> due to error while processing commit log during initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Could not read commit log descriptor in file
> /data/cassandra/commitlog/CommitLog-6-1468235564433.log
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:650)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:327)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:148)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:181)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:161)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:289)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:685)
> [apache-cassandra-3.0.8.jar:3.0.8]
> INFO [main] 2016-08-25 17:08:56,944 YamlConfigurationLoader.java:85 -
> Configuration location: file:/etc/cassandra/cassandra.yaml
> {noformat}
> Deleting the empty file fixes the problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12539) Empty CommitLog prevents restart
[
https://issues.apache.org/jira/browse/CASSANDRA-12539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15614670#comment-15614670
]
Arvind Nithrakashyap commented on CASSANDRA-12539:
--
The following patch seems to fix the issue
{noformat}
diff --git a/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java
b/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java
index af8efb4..8f11d13 100644
--- a/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java
+++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java
@@ -226,7 +226,7 @@ public class CommitLogReplayer
{
if (end != 0 || filecrc != 0)
{
-handleReplayError(false,
+handleReplayError(tolerateTruncation,
"Encountered bad header at position %d of
commit log %s, with invalid CRC. " +
"The end of segment marker should be zero.",
offset, reader.getPath());
@@ -345,6 +345,14 @@ public class CommitLogReplayer
return;
}
+
+int currentFilePointer = (int) reader.getFilePointer();
+if (readSyncMarker(desc, currentFilePointer, reader, true) < 0) {
+logger.info("Skipping empty logfile {}", file.getName());
+return;
+}
+reader.seek(currentFilePointer);
+
final long segmentId = desc.id;
try
{
{noformat}
> Empty CommitLog prevents restart
>
>
> Key: CASSANDRA-12539
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12539
> Project: Cassandra
> Issue Type: Bug
>Reporter: Stefano Ortolani
>
> A node just crashed (known cause: CASSANDRA-11594) but to my surprise (unlike
> other time) restarting simply fails.
> Checking the logs showed:
> {noformat}
> ERROR [main] 2016-08-25 17:05:22,611 JVMStabilityInspector.java:82 - Exiting
> due to error while processing commit log during initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Could not read commit log descriptor in file
> /data/cassandra/commitlog/CommitLog-6-1468235564433.log
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:650)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:327)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:148)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:181)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:161)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:289)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:685)
> [apache-cassandra-3.0.8.jar:3.0.8]
> INFO [main] 2016-08-25 17:08:56,944 YamlConfigurationLoader.java:85 -
> Configuration location: file:/etc/cassandra/cassandra.yaml
> {noformat}
> Deleting the empty file fixes the problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12539) Empty CommitLog prevents restart
[
https://issues.apache.org/jira/browse/CASSANDRA-12539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437690#comment-15437690
]
Stefano Ortolani commented on CASSANDRA-12539:
--
Maybe related to: CASSANDRA-11995?
> Empty CommitLog prevents restart
>
>
> Key: CASSANDRA-12539
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12539
> Project: Cassandra
> Issue Type: Bug
>Reporter: Stefano Ortolani
>
> A node just crashed (known cause: CASSANDRA-11594) but to my surprise (unlike
> other time) restarting simply fails.
> Checking the logs showed:
> {noformat}
> ERROR [main] 2016-08-25 17:05:22,611 JVMStabilityInspector.java:82 - Exiting
> due to error while processing commit log during initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Could not read commit log descriptor in file
> /data/cassandra/commitlog/CommitLog-6-1468235564433.log
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:650)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:327)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:148)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:181)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:161)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:289)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:685)
> [apache-cassandra-3.0.8.jar:3.0.8]
> INFO [main] 2016-08-25 17:08:56,944 YamlConfigurationLoader.java:85 -
> Configuration location: file:/etc/cassandra/cassandra.yaml
> {noformat}
> Deleting the empty file fixes the problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
