[ 
https://issues.apache.org/jira/browse/HDDS-17845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18146432#comment-18146432
 ] 

Tsz-wo Sze edited comment on HDDS-17845 at 11/21/26, 5:49 PM:
--------------------------------------------------------------

Note 1:
bq. Does Per-pipeline RocksDB marker need to add a new version in 
HDDSLayoutFeature?

Claude: No, this change doesn't need a new HDDSLayoutFeature. A layout feature 
has to gate a change when software from before finalization (or after a 
downgrade) would misread or lose data the new software wrote. That isn't the 
case here.

What the existing features gate:
- HBASE_SUPPORT (8): "an extra table for the last chunk of blocks". It's gated 
because with incremental chunk lists, part of a block's metadata lives only in 
the new table. An older datanode would ignore that table and see the block as 
truncated, which is data loss. So until finalization, BlockManagerImpl rejects 
partial chunk lists (:259) and KeyValueHandler rejects FinalizeBlock (:749).
- WITNESSED_CONTAINER_DB_PROTO_VALUE (9): HDDS-13176 changed the values of an 
existing table from string to proto, and 2.0.x can't parse the proto values. 
Hence the PreviousVersionTables fallback until finalization.

Why PipelineCreateInfoTable is different:
1. No existing data changes. It's a new column family, and no existing table or 
value format changes.
2. Older versions open it fine. 2.0.0, 2.1.0 and 2.2.0 all open column families 
they don't know about (RocksDatabase.getExtraColumnFamilies), so a downgraded 
datanode still opens the witnessed DB. A new datanode creates the empty column 
family at startup even before finalization, which is harmless.
3. Nothing is lost if an older version ignores it. Entries are only written for 
CHAINED pipelines, and the table only records the type, not data. Committed 
blocks stay in the container DB either way. A downgraded datanode would load 
such a group as RATIS, but released versions have no CHAINED and no 
stream-command PutBlock, so there's nothing CHAINED for them to serve.
4. No CHAINED pipelines exist yet. SCM can't create CHAINED pipelines 
(PipelineFactory has no CHAINED provider), so in practice nothing writes to the 
table today.

Where a layout feature will likely be needed: step 2. When SCM starts creating 
CHAINED pipelines, it must not do so before the cluster is finalized, or a 
datanode that's still downgradable could end up serving a CHAINED pipeline. 
That's the natural place for a new feature such as CHAINED_REPLICATION(11, …), 
checked by SCM before it creates CHAINED pipelines. Separately, SCM should only 
pick datanodes that support CHAINED, using a datanode version check like the 
HDDSVersion checks the stream client already does.

Once that feature exists, the datanode could also reject addGroup(…, CHAINED, 
…) until it's finalized, as a defensive check. That fits better in step 2 than 
in this change.



was (Author: szetszwo):
bq. Does Per-pipeline RocksDB marker need to add a new version in 
HDDSLayoutFeature?

Claude: No, this change doesn't need a new HDDSLayoutFeature. A layout feature 
has to gate a change when software from before finalization (or after a 
downgrade) would misread or lose data the new software wrote. That isn't the 
case here.

What the existing features gate:
- HBASE_SUPPORT (8): "an extra table for the last chunk of blocks". It's gated 
because with incremental chunk lists, part of a block's metadata lives only in 
the new table. An older datanode would ignore that table and see the block as 
truncated, which is data loss. So until finalization, BlockManagerImpl rejects 
partial chunk lists (:259) and KeyValueHandler rejects FinalizeBlock (:749).
- WITNESSED_CONTAINER_DB_PROTO_VALUE (9): HDDS-13176 changed the values of an 
existing table from string to proto, and 2.0.x can't parse the proto values. 
Hence the PreviousVersionTables fallback until finalization.

Why PipelineCreateInfoTable is different:
1. No existing data changes. It's a new column family, and no existing table or 
value format changes.
2. Older versions open it fine. 2.0.0, 2.1.0 and 2.2.0 all open column families 
they don't know about (RocksDatabase.getExtraColumnFamilies), so a downgraded 
datanode still opens the witnessed DB. A new datanode creates the empty column 
family at startup even before finalization, which is harmless.
3. Nothing is lost if an older version ignores it. Entries are only written for 
CHAINED pipelines, and the table only records the type, not data. Committed 
blocks stay in the container DB either way. A downgraded datanode would load 
such a group as RATIS, but released versions have no CHAINED and no 
stream-command PutBlock, so there's nothing CHAINED for them to serve.
4. No CHAINED pipelines exist yet. SCM can't create CHAINED pipelines 
(PipelineFactory has no CHAINED provider), so in practice nothing writes to the 
table today.

Where a layout feature will likely be needed: step 2. When SCM starts creating 
CHAINED pipelines, it must not do so before the cluster is finalized, or a 
datanode that's still downgradable could end up serving a CHAINED pipeline. 
That's the natural place for a new feature such as CHAINED_REPLICATION(11, …), 
checked by SCM before it creates CHAINED pipelines. Separately, SCM should only 
pick datanodes that support CHAINED, using a datanode version check like the 
HDDSVersion checks the stream client already does.

Once that feature exists, the datanode could also reject addGroup(…, CHAINED, 
…) until it's finalized, as a defensive check. That fits better in step 2 than 
in this change.


> Restore ReplicationType in ContainerStateMachine when a datanode restart
> ------------------------------------------------------------------------
>
>                 Key: HDDS-17845
>                 URL: https://issues.apache.org/jira/browse/HDDS-17845
>             Project: Apache Ozone
>          Issue Type: Sub-task
>          Components: Ozone Datanode
>            Reporter: Tsz-wo Sze
>            Assignee: Tsz-wo Sze
>            Priority: Major
>              Labels: pull-request-available
>
> (Assisted by Claude)
> When a datanode restart, we need some way to tell the ReplicationType of an 
> open container in order to recreate the ContainerStateMachine. Two approaches 
> below are considered, SCM reply vs Per-pipeline RocksDB marker:
> || ||SCM reply||Per-pipeline RocksDB marker||
> |On-disk change|none|new column family in the witnessed DB|
> |Protocol change|PipelineReport field and handling|none|
> |Unknown window after restart|~registration + 1 heartbeat, behaving 
> permissively|none: known before log replay|
> |Source of truth|SCM, so it can't go stale|the datanode, so a crash between 
> steps can leave a stray marker|
> |State machine type|mutable, set once|final, and the subclass split stays 
> possible|
> Although SCM reply does not requires on-disk change, there are may edge cases 
> listed below. Therefore, we will implement Per-pipeline RocksDB marker
>  - SCM doesn't know the pipeline: the existing ClosePipelineCommand path 
> removes the group, so the type never matters.
>  - Datanode was stale or dead: SCM has already closed the pipeline, which is 
> the self-healing path from before.
>  - Old SCM, new datanode: the old SCM ignores the flag, so the type stays 
> unknown and the datanode keeps allowing both kinds. Safe, just without the 
> checks.
>  - New SCM, old datanode: an old datanode never sets the flag, so nothing 
> changes for it.
>  - SCM unreachable or not leader: the type stays unknown until the leader 
> answers.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to