[
https://issues.apache.org/jira/browse/IGNITE-23595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Puchkovskiy updated IGNITE-23595:
---------------------------------------
Description:
After taking a snapshot, JRaft truncates log to remove indices which are
included in the snapshot. Snapshot metadata is persisted to the underlying
RocksDB storage, and later, when a snapshot is restored, its metadata is read
from the same storage.
Writes to the storage are not flushed to disk forcefully (hence snapshot meta
is not durable), but log truncates are saved durably. As a result, if a node
crashes (or even gets stopped gracefully), it might happen that snapshot
metadata write is lost. If, after that, a snapshot is tried to be restored
(which will happen on each node start), this will fail because an index in the
snapshot meta lags behind the index in the log:
{quote}2024-11-02 19:01:05:384 +0000
[ERROR][%node1%JRaft-FSMCaller-Disruptormetastorage_stripe_0-0][StripedDisruptor]
Handle disruptor event error
[name=%node1%JRaft-FSMCaller-Disruptormetastorage-,
event=org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTask@e69e5ae,
hasHandler=false]
java.lang.IllegalArgumentException: Try to truncate logs before 262622, but the
firstLogIndex is 262661
{quote}
We need to flush the contents of the RocksDB storage to disk after taking a
snapshot (but before doing log truncation) and do it durably (that is, with
fsync).
was:
After taking a snapshot, JRaft truncates log to remove indices which are
included in the snapshot. Snapshot metadata is persisted to the underlying
RocksDB storage, and later, when a snapshot is restored, its metadata is read
from the same storage.
Writes to the storage are not flushed to disk forcefully (hence snapshot meta
is not durable), but log truncates are saved durably. As a result, if a node
crashes (or even gets stopped gracefully), it might happen that snapshot
metadata write is lost. If, after that, a snapshot is tried to be restored
(which will happen on each node start), this will fail because an index in the
snapshot meta lags behind the index in the log:
{quote}2024-11-02 19:01:05:384 +0000
[ERROR][%node1%JRaft-FSMCaller-Disruptormetastorage_stripe_0-0][StripedDisruptor]
Handle disruptor event error
[name=%node1%JRaft-FSMCaller-Disruptormetastorage-,
event=org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTask@e69e5ae,
hasHandler=false]
java.lang.IllegalArgumentException: Try to truncate logs before 262622, but the
firstLogIndex is 262661{quote}
> Flush Metastorage underlying storage to disk after taking snapshot
> ------------------------------------------------------------------
>
> Key: IGNITE-23595
> URL: https://issues.apache.org/jira/browse/IGNITE-23595
> Project: Ignite
> Issue Type: Improvement
> Reporter: Roman Puchkovskiy
> Priority: Major
> Labels: ignite-3
>
> After taking a snapshot, JRaft truncates log to remove indices which are
> included in the snapshot. Snapshot metadata is persisted to the underlying
> RocksDB storage, and later, when a snapshot is restored, its metadata is read
> from the same storage.
> Writes to the storage are not flushed to disk forcefully (hence snapshot meta
> is not durable), but log truncates are saved durably. As a result, if a node
> crashes (or even gets stopped gracefully), it might happen that snapshot
> metadata write is lost. If, after that, a snapshot is tried to be restored
> (which will happen on each node start), this will fail because an index in
> the snapshot meta lags behind the index in the log:
> {quote}2024-11-02 19:01:05:384 +0000
> [ERROR][%node1%JRaft-FSMCaller-Disruptormetastorage_stripe_0-0][StripedDisruptor]
> Handle disruptor event error
> [name=%node1%JRaft-FSMCaller-Disruptormetastorage-,
> event=org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTask@e69e5ae,
> hasHandler=false]
> java.lang.IllegalArgumentException: Try to truncate logs before 262622, but
> the firstLogIndex is 262661
> {quote}
> We need to flush the contents of the RocksDB storage to disk after taking a
> snapshot (but before doing log truncation) and do it durably (that is, with
> fsync).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
