[
https://issues.apache.org/jira/browse/IGNITE-23567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17894223#comment-17894223
]
Roman Puchkovskiy commented on IGNITE-23567:
--------------------------------------------
Thanks!
> Store Metastorage checksums durably
> -----------------------------------
>
> Key: IGNITE-23567
> URL: https://issues.apache.org/jira/browse/IGNITE-23567
> Project: Ignite
> Issue Type: Improvement
> Reporter: Roman Puchkovskiy
> Assignee: Roman Puchkovskiy
> Priority: Major
> Labels: iep-128, ignite-3
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Metastorage internal storage (the state machine state) is stored in a RocksDB
> instance. The WAL inside it is disabled because Raft log plays its role in
> making sure writes to the Metastorage are durable (log writes are fsynced).
> So durability of writes to the Metastorage is guaranteed by the log. This
> currently also relates to Metastorage revision checksums.
> When doing an MG repair and a node does not participate in it, it is later
> 'migrated' to the repaired cluster. Before allowing it to join the cluster,
> we validate that its Metastorage state did not diverge wrt the Metastorage
> leader. Revision checksums are used for this end. But at this moment we
> cannot rely on the log (because we don't know which part of the log is also
> present on the nodes that participated in the MG repair), so checksums'
> durability is not guaranteed. This means that a revision might be applied on
> the node (and its side-effects could be persisted durably; for example, the
> revision corresponds to an ALTER TABLE query and the node already contains
> tuples written in that changed configuration), but its checksum is not
> persisted durably, so, at the moment of Metastorage validation for
> divergency, we'll not see that it's diverged, even though it is.
> To solve this, we should write checksums durably. They are already written to
> their own column family; when doing writes to this column family, we should
> use separate WriteOptions which makes RocksDB use WAL and fsync. All other
> writes still need to be done without a WAL. Also, we need to write the
> checksum out of the batch (more precisely, before the batch).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
