jojochuang commented on code in PR #552: URL: https://github.com/apache/ozone-site/pull/552#discussion_r4069249214
########## docs/06-troubleshooting/10-decommission-maintenance.md: ########## @@ -1,8 +1,135 @@ --- -draft: true sidebar_label: Decommissioning and Maintenance --- # Troubleshooting Decommissioning and Maintenance -**TODO:** File a subtask under [HDDS-9860](https://issues.apache.org/jira/browse/HDDS-9860) and complete this page or section. +A Datanode moves from `DECOMMISSIONING` to `DECOMMISSIONED` only after every container it holds is sufficiently replicated on other nodes. +On a dense Datanode this takes hours, and on a large or busy cluster it can take days: the whole node has to be copied over the network while the cluster keeps serving normal traffic. +Slow is normal; stuck is not. +This page shows how to tell the two apart, what usually blocks the workflow, and which settings make it faster. + +The same workflow (close pipelines, replicate containers, change state) is used when a Datanode enters maintenance, so the checks below apply to a node stuck in `ENTERING_MAINTENANCE` as well. + +## How the workflow progresses + +SCM's Datanode admin monitor re-checks every decommissioning node each `ozone.scm.datanode.admin.monitor.interval` (default `30s`). +A node completes only once all of the following hold: + +1. All Ratis pipelines on the node have closed. +2. The Datanode has reported the new operational state in a heartbeat, so it must be alive. +3. Every container on the node is `CLOSED` or `QUASI_CLOSED`, its healthy replicas on `IN_SERVICE` nodes are in that same state, and there are enough of them. Containers in `DELETING` or `DELETED` state are ignored. + +The copying itself is done by the SCM Replication Manager, which sends replicate-container commands to Datanodes. +Those commands are throttled per Datanode and cluster-wide. +SCM copies each container from whichever of its healthy replicas has the fewest replication commands queued. +The decommissioning node is allowed more queued commands than an in-service node, so it stays eligible as a source for longer. +Either way, the throttles set the pace of the whole decommission. + +## Check the progress + +### Decommission status command + +```shell +ozone admin datanode status decommission [--node-id=<uuid> | --ip=<ipAddress>] [--json] Review Comment: no, this is correct. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
