[
https://issues.apache.org/jira/browse/CASSANDRA-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13473100#comment-13473100
]
Fabien Rousseau commented on CASSANDRA-4782:
--------------------------------------------
To solve the first case, it's probably better, when possible, to upgrade to a
newer version, then rewrite all SSTables (or at least all SSTables metadata)
For the second case, just rewrite all SSTables (or at least all SSTables
metadata)
Each SSTable metadata contains the ReplayPosition (and the max is taken to know
which commitlog to replay), thus if System.nanoTime() returned a number which
is a timestamp in the future, previous commitlogs will be ignored),
thus a drain should prevent from losing data (because there is no commitlog to
replay).
And because SSTables are immutables, then rewriting them completely seems a
better option (rather than modifying previously written metadata)
Maybe the simplest to do is to have a new option to nodetool (or a new option
to nodetool upgradesstables) which only changes the metadata using the
following rule :
if the replayPosition.segment of the SSTable is in the future, then reset it to
NONE otherwise, let it to its current value. (NONE is valid value if node was
drained and restarted)
By using this rule :
- if a sstable was generated previously using a higher System.nanoTime() then
it is reset
- if a sstable was generated previously using a lower System.nanoTime() OR
System.currentTimeMillis() then it is left as is
- if a sstable is generated between the start & this rewriting process, then
it is left as is
Thus the upgrade should be :
- drain node
- upgrade
- start
- run the process described above
What do you think ?
> Commitlog not replayed after restart
> ------------------------------------
>
> Key: CASSANDRA-4782
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4782
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5
> Reporter: Fabien Rousseau
>
> It seems that there are two corner cases where commitlog is not replayed
> after a restart :
> - After a reboot of a server + restart of cassandra (1.1.0 to 1.1.4)
> - After doing an upgrade from cassandra 1.1.X to cassandra 1.1.5
> This is due to the fact that the commitlog segment id should always be an
> incrementing number (see this condition :
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L247
> )
> But this assertion can be broken :
> In the first case, it is generated by System.nanoTime() but it seems that
> System.nanoTime() is using the boot time as the base/reference (at least on
> java6 & linux), thus after a reboot, System.nanoTime() can return a lower
> number than before the reboot (and the javadoc says the reference is a
> relative point in time...)
> In the second case, this was introduced by #4601 (which changes
> System.nanoTime() by System.currentTimeMillis() thus people starting with
> 1.1.5 are safe)
> This could explain the following tickets : #4741 and #4481
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
