Siyao Meng created HDDS-17107:
---------------------------------
Summary: Show startup progress in Datanode web UI
Key: HDDS-17107
URL: https://issues.apache.org/jira/browse/HDDS-17107
Project: Apache Ozone
Issue Type: Improvement
Components: Ozone Datanode
Reporter: Siyao Meng
Assignee: Siyao Meng
h3. Problem
Unlike the HDFS NameNode ("Startup Progress" tab), Ozone services expose no
startup progress. A datanode with many containers can take a long time to
start, and until it does, operators can only follow the logs. They cannot tell
which phase the datanode is in, how long each phase has taken, or how long
startup will still take.
h3. Proposal
Track datanode startup in phases and show it in the Datanode web UI:
# Initializing
# Waiting for SCM (until the version response is received)
# Preparing volumes (formatting volumes and loading volume DBs)
# Loading containers (one step per volume, with done / total containers)
# Checking volumes
# Starting container services (including Ratis pipeline recovery)
For each phase, show its status (pending / running / complete / failed) and
elapsed time. Container loading counts the container directories of each volume
up front, so done / total and an estimated remaining time can be shown per
volume. If startup fails, the running phase and any failed volume are marked as
failed.
The progress is exposed through a {{StartupProgress}} MXBean
({{Hadoop:service=HddsDatanode,name=StartupProgress}}), so it is also available
via {{/jmx}}. A new "Startup Progress" page in the Datanode web UI renders it.
The datanode HTTP server already starts before the slow phases, so their
progress is visible while the datanode is starting.
h3. Follow-ups (out of scope)
The tracker lives in hdds-server-framework so that it can be reused for OM and
SCM. Those services start their HTTP server near the end of startup, so they
would first need the web UI to be available earlier.
h3. Related
* HDDS-12391, HDDS-13331, HDDS-13332: reduce datanode startup time (container
set loading). This page makes the effect of such changes visible.
* HDDS-13896, HDDS-13897: datanode startup can hang on a slow or stuck volume
without logging. Per-volume loading progress shows which volume is stuck.
* HDDS-16425, HDDS-16504: fail datanode startup cleanly, and add a watchdog for
container initialization.
* HDDS-17078: OM startup can take hours with many snapshots (motivation for the
OM follow-up).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
