Sadanand Shenoy created HDDS-16057:
--------------------------------------
Summary: OM bootstrap fails when checkpoint install is rejected
during BOOTSTRAPPING
Key: HDDS-16057
URL: https://issues.apache.org/jira/browse/HDDS-16057
Project: Apache Ozone
Issue Type: Bug
Reporter: Sadanand Shenoy
Assignee: Sadanand Shenoy
OM bootstrap fails to add a new node to the Ratis ring when catch-up requires
an OM checkpoint install (leader logs purged). {{installSnapshotFromLeader()}}
aborts because {{{}omState == BOOTSTRAPPING{}}}, causing a deadlock with Ratis
{{setConfiguration}} staging.
h2. Root cause
During {{{}--bootstrap{}}}, the main thread blocks in {{bootstrap()}} while the
leader runs {{setConfiguration}} staging. Staging waits for the new peer to
catch up. The leader triggers OM checkpoint install, but
{{installSnapshotFromLeader()}} rejects the request with {{!isRunning()}} while
{{{}omState == BOOTSTRAPPING{}}}. Catch-up never completes, so the 3-peer
config is never committed to the raft log.
The HDDS-15103 guard is correct for normal restart; it unintentionally blocks
the bootstrap catch-up path.
h2. Timeline
||Time||Node||Event||
|T1|New OM|{{{}ozone om --bootstrap{}}}, {{{}omState=BOOTSTRAPPING{}}}, Ratis
starts with empty peer group|
|T2|New OM|{{start()}} finishes (secret manager, RPC, Ratis up), calls
{{bootstrap()}} — main thread blocks|
|T3|New OM → Leader|Bootstrap RPC sent|
|T4|Leader|{{startSetConfiguration}} (3 peers) — staging begins, conf NOT
written to raft log|
|T5|Leader → New OM|Install snapshot notify (logs purged; e.g. first available
index 213, leader at 542)|
|T6|New OM|{{installSnapshotFromLeader()}} → ABORT: state BOOTSTRAPPING →
{{SNAPSHOT_UNAVAILABLE}}|
|T7|Leader ↔ New OM|Staging loop: leader retries install notify, catch-up never
completes|
|T8|Cluster|Committed Ratis membership stays at 2 peers — new OM never joins
the ring|
h2. Symptoms
* Bootstrap hangs / never completes
* {{ozone admin om roles}} shows 2 OMs
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
