[
https://issues.apache.org/jira/browse/HDDS-16057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sadanand Shenoy updated HDDS-16057:
-----------------------------------
Description:
HDDS-15103 added state checks before triggering installSnapshot which caused
this
OM bootstrap fails to add a new node to the Ratis ring when catch-up requires
an OM checkpoint install (leader logs purged). {{installSnapshotFromLeader()}}
aborts because {{{}omState == BOOTSTRAPPING{}}}, causing a deadlock with Ratis
{{setConfiguration}} staging.
h2. Root cause
During {{{}--bootstrap{}}}, the main thread blocks in {{bootstrap()}} while the
leader runs {{setConfiguration}} staging. Staging waits for the new peer to
catch up. The leader triggers OM checkpoint install, but
{{installSnapshotFromLeader()}} rejects the request with {{!isRunning()}} while
{{{}omState == BOOTSTRAPPING{}}}. Catch-up never completes, so the 3-peer
config is never committed to the raft log.
The HDDS-15103 guard is correct for normal restart; it unintentionally blocks
the bootstrap catch-up path.
h2. Timeline
||Time||Node||Event||
|T1|New OM|{{{}ozone om --bootstrap{}}}, {{{}omState=BOOTSTRAPPING{}}}, Ratis
starts with empty peer group|
|T2|New OM|{{start()}} finishes (secret manager, RPC, Ratis up), calls
{{bootstrap()}} — main thread blocks|
|T3|New OM → Leader|Bootstrap RPC sent|
|T4|Leader|{{startSetConfiguration}} (3 peers) — staging begins, conf NOT
written to raft log|
|T5|Leader → New OM|Install snapshot notify (logs purged; e.g. first available
index 213, leader at 542)|
|T6|New OM|{{installSnapshotFromLeader()}} → ABORT: state BOOTSTRAPPING →
{{SNAPSHOT_UNAVAILABLE}}|
|T7|Leader ↔ New OM|Staging loop: leader retries install notify, catch-up never
completes|
|T8|Cluster|Committed Ratis membership stays at 2 peers — new OM never joins
the ring|
h2. Symptoms
* Bootstrap hangs / never completes
* {{ozone admin om roles}} shows 2 OMs
was:
OM bootstrap fails to add a new node to the Ratis ring when catch-up requires
an OM checkpoint install (leader logs purged). {{installSnapshotFromLeader()}}
aborts because {{{}omState == BOOTSTRAPPING{}}}, causing a deadlock with Ratis
{{setConfiguration}} staging.
h2. Root cause
During {{{}--bootstrap{}}}, the main thread blocks in {{bootstrap()}} while the
leader runs {{setConfiguration}} staging. Staging waits for the new peer to
catch up. The leader triggers OM checkpoint install, but
{{installSnapshotFromLeader()}} rejects the request with {{!isRunning()}} while
{{{}omState == BOOTSTRAPPING{}}}. Catch-up never completes, so the 3-peer
config is never committed to the raft log.
The HDDS-15103 guard is correct for normal restart; it unintentionally blocks
the bootstrap catch-up path.
h2. Timeline
||Time||Node||Event||
|T1|New OM|{{{}ozone om --bootstrap{}}}, {{{}omState=BOOTSTRAPPING{}}}, Ratis
starts with empty peer group|
|T2|New OM|{{start()}} finishes (secret manager, RPC, Ratis up), calls
{{bootstrap()}} — main thread blocks|
|T3|New OM → Leader|Bootstrap RPC sent|
|T4|Leader|{{startSetConfiguration}} (3 peers) — staging begins, conf NOT
written to raft log|
|T5|Leader → New OM|Install snapshot notify (logs purged; e.g. first available
index 213, leader at 542)|
|T6|New OM|{{installSnapshotFromLeader()}} → ABORT: state BOOTSTRAPPING →
{{SNAPSHOT_UNAVAILABLE}}|
|T7|Leader ↔ New OM|Staging loop: leader retries install notify, catch-up never
completes|
|T8|Cluster|Committed Ratis membership stays at 2 peers — new OM never joins
the ring|
h2. Symptoms
* Bootstrap hangs / never completes
* {{ozone admin om roles}} shows 2 OMs
> OM bootstrap fails when checkpoint install is rejected during BOOTSTRAPPING
> ---------------------------------------------------------------------------
>
> Key: HDDS-16057
> URL: https://issues.apache.org/jira/browse/HDDS-16057
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Sadanand Shenoy
> Assignee: Sadanand Shenoy
> Priority: Major
>
> HDDS-15103 added state checks before triggering installSnapshot which caused
> this
> OM bootstrap fails to add a new node to the Ratis ring when catch-up requires
> an OM checkpoint install (leader logs purged).
> {{installSnapshotFromLeader()}} aborts because {{{}omState ==
> BOOTSTRAPPING{}}}, causing a deadlock with Ratis {{setConfiguration}} staging.
> h2. Root cause
> During {{{}--bootstrap{}}}, the main thread blocks in {{bootstrap()}} while
> the leader runs {{setConfiguration}} staging. Staging waits for the new peer
> to catch up. The leader triggers OM checkpoint install, but
> {{installSnapshotFromLeader()}} rejects the request with {{!isRunning()}}
> while {{{}omState == BOOTSTRAPPING{}}}. Catch-up never completes, so the
> 3-peer config is never committed to the raft log.
> The HDDS-15103 guard is correct for normal restart; it unintentionally blocks
> the bootstrap catch-up path.
> h2. Timeline
> ||Time||Node||Event||
> |T1|New OM|{{{}ozone om --bootstrap{}}}, {{{}omState=BOOTSTRAPPING{}}}, Ratis
> starts with empty peer group|
> |T2|New OM|{{start()}} finishes (secret manager, RPC, Ratis up), calls
> {{bootstrap()}} — main thread blocks|
> |T3|New OM → Leader|Bootstrap RPC sent|
> |T4|Leader|{{startSetConfiguration}} (3 peers) — staging begins, conf NOT
> written to raft log|
> |T5|Leader → New OM|Install snapshot notify (logs purged; e.g. first
> available index 213, leader at 542)|
> |T6|New OM|{{installSnapshotFromLeader()}} → ABORT: state BOOTSTRAPPING →
> {{SNAPSHOT_UNAVAILABLE}}|
> |T7|Leader ↔ New OM|Staging loop: leader retries install notify, catch-up
> never completes|
> |T8|Cluster|Committed Ratis membership stays at 2 peers — new OM never joins
> the ring|
> h2. Symptoms
> * Bootstrap hangs / never completes
> * {{ozone admin om roles}} shows 2 OMs
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
