[
https://issues.apache.org/jira/browse/HDDS-17107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDDS-17107:
-----------------------------------
Fix Version/s: 2.3.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Show startup progress in Datanode web UI
> ----------------------------------------
>
> Key: HDDS-17107
> URL: https://issues.apache.org/jira/browse/HDDS-17107
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Datanode
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.3.0
>
>
> h3. Problem
> Unlike the HDFS NameNode ("Startup Progress" tab), Ozone services expose no
> startup progress. A datanode with many containers can take a long time to
> start, and until it does, operators can only follow the logs. They cannot
> tell which phase the datanode is in, how long each phase has taken, or how
> long startup will still take.
> h3. Proposal
> Track datanode startup in phases and show it in the Datanode web UI:
> # Initializing
> # Waiting for SCM (until the version response is received)
> # Preparing volumes (formatting volumes and loading volume DBs)
> # Loading containers (one step per volume, with done / total containers)
> # Checking volumes
> # Starting container services (including Ratis pipeline recovery)
> For each phase, show its status (pending / running / complete / failed) and
> elapsed time. Container loading counts the container directories of each
> volume up front, so done / total and an estimated remaining time can be shown
> per volume. If startup fails, the running phase and any failed volume are
> marked as failed.
> The progress is exposed through a {{StartupProgress}} MXBean
> ({{Hadoop:service=HddsDatanode,name=StartupProgress}}), so it is also
> available via {{/jmx}}. A new "Startup Progress" page in the Datanode web UI
> renders it.
> The datanode HTTP server already starts before the slow phases, so their
> progress is visible while the datanode is starting.
> h3. Follow-ups (out of scope)
> The tracker lives in hdds-server-framework so that it can be reused for OM
> and SCM. Those services start their HTTP server near the end of startup, so
> they would first need the web UI to be available earlier.
> h3. Related
> * HDDS-12391, HDDS-13331, HDDS-13332: reduce datanode startup time (container
> set loading). This page makes the effect of such changes visible.
> * HDDS-13896, HDDS-13897: datanode startup can hang on a slow or stuck volume
> without logging. Per-volume loading progress shows which volume is stuck.
> * HDDS-16425, HDDS-16504: fail datanode startup cleanly, and add a watchdog
> for container initialization.
> * HDDS-17078: OM startup can take hours with many snapshots (motivation for
> the OM follow-up).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
