[
https://issues.apache.org/jira/browse/IGNITE-23567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Puchkovskiy updated IGNITE-23567:
---------------------------------------
Description:
Metastorage internal storage (the state machine state) is stored in a RocksDB
instance. The WAL inside it is disabled because Raft log plays its role in
making sure writes to the Metastorage are durable (log writes are fsynced). So
durability of writes to the Metastorage is guaranteed by the log. This
currently also relates to Metastorage revision checksums.
When doing an MG repair and a node does not participate in it, it is later
'migrated' to the repaired cluster. Before allowing it to join the cluster, we
validate that its Metastorage state did not diverge wrt the Metastorage leader.
Revision checksums are used for this end. But at this moment we cannot rely on
the log (because we don't know which part of the log is also present on the
nodes that participated in the MG repair), so checksums' durability is not
guaranteed. This means that a revision might be applied on the node (and its
side-effects could be persisted durably; for example, the revision corresponds
to an ALTER TABLE query and the node already contains tuples written in that
changed configuration), but its checksum is not persisted durably, so, at the
moment of Metastorage validation for divergency, we'll not see that it's
diverged, even though it is.
To solve this, we should write checksums durably. They are already written to
their own column family; when doing writes to this column family, we should use
separate WriteOptions which makes RocksDB use WAL and fsync. All other writes
still need to be done without a WAL. Also, we need to write the checksum out of
the batch (more precisely, before the batch).
> Store Metastorage checksums durably
> -----------------------------------
>
> Key: IGNITE-23567
> URL: https://issues.apache.org/jira/browse/IGNITE-23567
> Project: Ignite
> Issue Type: Improvement
> Reporter: Roman Puchkovskiy
> Assignee: Roman Puchkovskiy
> Priority: Major
> Labels: iep-128, ignite-3
>
> Metastorage internal storage (the state machine state) is stored in a RocksDB
> instance. The WAL inside it is disabled because Raft log plays its role in
> making sure writes to the Metastorage are durable (log writes are fsynced).
> So durability of writes to the Metastorage is guaranteed by the log. This
> currently also relates to Metastorage revision checksums.
> When doing an MG repair and a node does not participate in it, it is later
> 'migrated' to the repaired cluster. Before allowing it to join the cluster,
> we validate that its Metastorage state did not diverge wrt the Metastorage
> leader. Revision checksums are used for this end. But at this moment we
> cannot rely on the log (because we don't know which part of the log is also
> present on the nodes that participated in the MG repair), so checksums'
> durability is not guaranteed. This means that a revision might be applied on
> the node (and its side-effects could be persisted durably; for example, the
> revision corresponds to an ALTER TABLE query and the node already contains
> tuples written in that changed configuration), but its checksum is not
> persisted durably, so, at the moment of Metastorage validation for
> divergency, we'll not see that it's diverged, even though it is.
> To solve this, we should write checksums durably. They are already written to
> their own column family; when doing writes to this column family, we should
> use separate WriteOptions which makes RocksDB use WAL and fsync. All other
> writes still need to be done without a WAL. Also, we need to write the
> checksum out of the batch (more precisely, before the batch).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
