[
https://issues.apache.org/jira/browse/HDDS-17845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz-wo Sze updated HDDS-17845:
------------------------------
Description:
(Assisted by Claude)
When a datanode restart, we need some way to tell the ReplicationType of an
open container in order to recreate the ContainerStateMachine. Two approaches
below are considered, SCM reply vs Per-pipeline RocksDB marker:
|| ||SCM reply||Per-pipeline RocksDB marker||
|On-disk change|none|new column family in the witnessed DB|
|Protocol change|PipelineReport field and handling|none|
|Unknown window after restart|~registration + 1 heartbeat, behaving
permissively|none: known before log replay|
|Source of truth|SCM, so it can't go stale|the datanode, so a crash between
steps can leave a stray marker|
|State machine type|mutable, set once|final, and the subclass split stays
possible|
Although SCM reply does not requires on-disk change, there are may edge cases
listed below. Therefore, we will implement Per-pipeline RocksDB marker
- SCM doesn't know the pipeline: the existing ClosePipelineCommand path
removes the group, so the type never matters.
- Datanode was stale or dead: SCM has already closed the pipeline, which is
the self-healing path from before.
- Old SCM, new datanode: the old SCM ignores the flag, so the type stays
unknown and the datanode keeps allowing both kinds. Safe, just without the
checks.
- New SCM, old datanode: an old datanode never sets the flag, so nothing
changes for it.
- SCM unreachable or not leader: the type stays unknown until the leader
answers.
Regarding to HDDSVersion, see Note 1 in the first comment.
was:
(Assisted by Claude)
When a datanode restart, we need some way to tell the ReplicationType of an
open container in order to recreate the ContainerStateMachine. Two approaches
below are considered, SCM reply vs Per-pipeline RocksDB marker:
|| ||SCM reply||Per-pipeline RocksDB marker||
|On-disk change|none|new column family in the witnessed DB|
|Protocol change|PipelineReport field and handling|none|
|Unknown window after restart|~registration + 1 heartbeat, behaving
permissively|none: known before log replay|
|Source of truth|SCM, so it can't go stale|the datanode, so a crash between
steps can leave a stray marker|
|State machine type|mutable, set once|final, and the subclass split stays
possible|
Although SCM reply does not requires on-disk change, there are may edge cases
listed below. Therefore, we will implement Per-pipeline RocksDB marker
- SCM doesn't know the pipeline: the existing ClosePipelineCommand path
removes the group, so the type never matters.
- Datanode was stale or dead: SCM has already closed the pipeline, which is
the self-healing path from before.
- Old SCM, new datanode: the old SCM ignores the flag, so the type stays
unknown and the datanode keeps allowing both kinds. Safe, just without the
checks.
- New SCM, old datanode: an old datanode never sets the flag, so nothing
changes for it.
- SCM unreachable or not leader: the type stays unknown until the leader
answers.
Regarding to HDDSLayoutFeature and HDDSVersion, see Note 1 & 2 in the first two
comments.
> Restore ReplicationType in ContainerStateMachine when a datanode restart
> ------------------------------------------------------------------------
>
> Key: HDDS-17845
> URL: https://issues.apache.org/jira/browse/HDDS-17845
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Datanode
> Reporter: Tsz-wo Sze
> Assignee: Tsz-wo Sze
> Priority: Major
> Labels: pull-request-available
>
> (Assisted by Claude)
> When a datanode restart, we need some way to tell the ReplicationType of an
> open container in order to recreate the ContainerStateMachine. Two approaches
> below are considered, SCM reply vs Per-pipeline RocksDB marker:
> || ||SCM reply||Per-pipeline RocksDB marker||
> |On-disk change|none|new column family in the witnessed DB|
> |Protocol change|PipelineReport field and handling|none|
> |Unknown window after restart|~registration + 1 heartbeat, behaving
> permissively|none: known before log replay|
> |Source of truth|SCM, so it can't go stale|the datanode, so a crash between
> steps can leave a stray marker|
> |State machine type|mutable, set once|final, and the subclass split stays
> possible|
> Although SCM reply does not requires on-disk change, there are may edge cases
> listed below. Therefore, we will implement Per-pipeline RocksDB marker
> - SCM doesn't know the pipeline: the existing ClosePipelineCommand path
> removes the group, so the type never matters.
> - Datanode was stale or dead: SCM has already closed the pipeline, which is
> the self-healing path from before.
> - Old SCM, new datanode: the old SCM ignores the flag, so the type stays
> unknown and the datanode keeps allowing both kinds. Safe, just without the
> checks.
> - New SCM, old datanode: an old datanode never sets the flag, so nothing
> changes for it.
> - SCM unreachable or not leader: the type stays unknown until the leader
> answers.
> Regarding to HDDSVersion, see Note 1 in the first comment.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]