[
https://issues.apache.org/jira/browse/HDDS-17114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng reassigned HDDS-17114:
---------------------------------
Assignee: Siyao Meng
> Show startup progress and readiness for OM, SCM and Datanode
> ------------------------------------------------------------
>
> Key: HDDS-17114
> URL: https://issues.apache.org/jira/browse/HDDS-17114
> Project: Apache Ozone
> Issue Type: New Feature
> Components: Ozone Datanode, Ozone Manager, SCM
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
>
> h3. Problem
> OM, SCM and datanodes can take a long time to start on large clusters:
> opening RocksDB, loading containers and pipelines, the OM snapshot chain and
> prefixes, replaying the Raft log, and waiting for SCM safe mode exit. During
> that time operators only have the logs. The services expose no startup phase,
> progress or estimated remaining time, unlike the HDFS NameNode's "Startup
> Progress" page.
> OM and SCM also start their HTTP server near the end of startup, so their web
> UI, /jmx and /prom are unavailable for most of it. There is no endpoint that
> says whether a service is ready, so scripts and Kubernetes probes use an open
> port as a stand-in:
> * The Kubernetes liveness probes use TCP on the OM and SCM RPC ports (9862,
> 9861) with a 30s initial delay. OM binds 9862 only after loading its DB, and
> an HA SCM follower binds 9861 only after catching up with the leader, so a
> slow start can be killed by the probe.
> * The rolling-upgrade test waits on the SCM HTTP port (9876) before
> restarting the next SCM.
> h3. Proposal
> # Track startup in phases with a reusable StartupProgress MXBean
> ({{Hadoop:service=<service>,name=StartupProgress}}). Report status, elapsed
> time, done / total and the estimated remaining time per phase and per step.
> # Show it on a "Startup Progress" page in each service's web UI.
> # Start the OM and SCM HTTP servers right after Kerberos login. Until the
> service is ready, servlets that need service state (/dbCheckpoint,
> /v2/dbCheckpoint, /serviceList, /snapshotList) return 503.
> # Startup is complete once the Raft log has been replayed (OM and SCM) and
> SCM has left safe mode.
> # Add /health/live and /health/ready to OM, SCM and the datanode, following
> the S3 Gateway endpoints from HDDS-16360, and use them in the Kubernetes
> examples and the rolling-upgrade test.
> h3. Related
> * HDDS-16360: S3 Gateway health check endpoints (pattern to reuse)
> * HDDS-17078: OM startup can take hours with many snapshots
> * HDDS-16299, HDDS-16301: guard leadership transfer to an OM or SCM that is
> not ready
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
