jojochuang commented on code in PR #552: URL: https://github.com/apache/ozone-site/pull/552#discussion_r4069321652
########## docs/06-troubleshooting/10-decommission-maintenance.md: ########## @@ -1,8 +1,135 @@ --- -draft: true sidebar_label: Decommissioning and Maintenance --- # Troubleshooting Decommissioning and Maintenance -**TODO:** File a subtask under [HDDS-9860](https://issues.apache.org/jira/browse/HDDS-9860) and complete this page or section. +A Datanode moves from `DECOMMISSIONING` to `DECOMMISSIONED` only after every container it holds is sufficiently replicated on other nodes. +On a dense Datanode this takes hours, and on a large or busy cluster it can take days: the whole node has to be copied over the network while the cluster keeps serving normal traffic. +Slow is normal; stuck is not. +This page shows how to tell the two apart, what usually blocks the workflow, and which settings make it faster. + +The same workflow (close pipelines, replicate containers, change state) is used when a Datanode enters maintenance, so the checks below apply to a node stuck in `ENTERING_MAINTENANCE` as well. + +## How the workflow progresses + +SCM's Datanode admin monitor re-checks every decommissioning node each `ozone.scm.datanode.admin.monitor.interval` (default `30s`). +A node completes only once all of the following hold: + +1. All Ratis pipelines on the node have closed. +2. The Datanode has reported the new operational state in a heartbeat, so it must be alive. +3. Every container on the node is `CLOSED` or `QUASI_CLOSED`, its healthy replicas on `IN_SERVICE` nodes are in that same state, and there are enough of them. Containers in `DELETING` or `DELETED` state are ignored. + +The copying itself is done by the SCM Replication Manager, which sends replicate-container commands to Datanodes. +Those commands are throttled per Datanode and cluster-wide. +SCM copies each container from whichever of its healthy replicas has the fewest replication commands queued. +The decommissioning node is allowed more queued commands than an in-service node, so it stays eligible as a source for longer. +Either way, the throttles set the pace of the whole decommission. + +## Check the progress + +### Decommission status command + +```shell +ozone admin datanode status decommission [--node-id=<uuid> | --ip=<ipAddress>] [--json] +``` + +```text +Decommission Status: DECOMMISSIONING - 1 node(s) + +Datanode: 6f1c...-...-... (/default-rack/10.0.0.12/dn12.example.com) +Decommission Started At : 01/09/2026 02:14:05 UTC +No. of Unclosed Pipelines: 0 +No. of UnderReplicated Containers: 1834 +No. of Unclosed Containers: 3 +{UnderReplicated=[#1001, #1002, ...], UnClosed=[#1200, #1201, #1202]} +``` + +Run the command a few times, some minutes apart: + +- `No. of UnderReplicated Containers` keeps decreasing: the decommission is progressing and just needs time. See [Making it faster](#making-it-faster). +- `No. of Unclosed Pipelines` stays above zero: see [Pipelines that do not close](#pipelines-that-do-not-close). +- `No. of Unclosed Containers` stays above zero while the under-replicated count reaches zero: see [Containers that do not close](#containers-that-do-not-close). +- `No. of UnderReplicated Containers` stays flat across several monitor intervals: replication is not happening. See [Replication is not making progress](#replication-is-not-making-progress). + +### SCM log + +On every monitor run SCM logs one summary line per tracked node: + +```text +<datanode> has 12405 sufficientlyReplicated, 17 deleting, 1834 underReplicated and 3 unclosed containers +``` + +Compare consecutive lines to see whether `underReplicated` moves. +While pipelines are still open the line is `Waiting for pipelines to close for <datanode>. There are N pipelines` instead. +Each blocking container is also logged as `Under Replicated Container <id> <replicas>; <replica details>` or `Unclosed Container <id> <replicas>; <replica details>`, with the state and location of its replicas. +The number of such lines per node and per category is capped by `ozone.scm.datanode.admin.monitor.logging.limit` (default `1000`) unless DEBUG logging is enabled. + +### Metrics + +The SCM `NodeDecommissionMetrics` source (JMX bean `Hadoop:service=StorageContainerManager,name=NodeDecommissionMetrics`) exposes the same counters the status command prints, so they can be graphed in Grafana: Review Comment: Import the Ozone DataNode Decommission and Maintenance dashboard into Grafana: https://github.com/apache/ozone/blob/master/hadoop-ozone/dist/src/main/compose/common/grafana/dashboards/Ozone%20-%20Datanode%20Decommission%20and%20Maintenance.json -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
