Siyao Meng created HDDS-17114:
---------------------------------
Summary: Show startup progress and readiness for OM, SCM and
Datanode
Key: HDDS-17114
URL: https://issues.apache.org/jira/browse/HDDS-17114
Project: Apache Ozone
Issue Type: New Feature
Components: SCM, Ozone Manager, Ozone Datanode
Reporter: Siyao Meng
h3. Problem
OM, SCM and datanodes can take a long time to start on large clusters: opening
RocksDB, loading containers and pipelines, the OM snapshot chain and prefixes,
replaying the Raft log, and waiting for SCM safe mode exit. During that time
operators only have the logs. The services expose no startup phase, progress or
estimated remaining time, unlike the HDFS NameNode's "Startup Progress" page.
OM and SCM also start their HTTP server near the end of startup, so their web
UI, /jmx and /prom are unavailable for most of it. There is no endpoint that
says whether a service is ready, so scripts and Kubernetes probes use an open
port as a stand-in:
* The Kubernetes liveness probes use TCP on the OM and SCM RPC ports (9862,
9861) with a 30s initial delay. OM binds 9862 only after loading its DB, and an
HA SCM follower binds 9861 only after catching up with the leader, so a slow
start can be killed by the probe.
* The rolling-upgrade test waits on the SCM HTTP port (9876) before restarting
the next SCM.
h3. Proposal
# Track startup in phases with a reusable StartupProgress MXBean
({{Hadoop:service=<service>,name=StartupProgress}}). Report status, elapsed
time, done / total and the estimated remaining time per phase and per step.
# Show it on a "Startup Progress" page in each service's web UI.
# Start the OM and SCM HTTP servers right after Kerberos login. Until the
service is ready, servlets that need service state (/dbCheckpoint,
/v2/dbCheckpoint, /serviceList, /snapshotList) return 503.
# Startup is complete once the Raft log has been replayed (OM and SCM) and SCM
has left safe mode.
# Add /health/live and /health/ready to OM, SCM and the datanode, following the
S3 Gateway endpoints from HDDS-16360, and use them in the Kubernetes examples
and the rolling-upgrade test.
h3. Related
* HDDS-16360: S3 Gateway health check endpoints (pattern to reuse)
* HDDS-17078: OM startup can take hours with many snapshots
* HDDS-16299, HDDS-16301: guard leadership transfer to an OM or SCM that is not
ready
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
