[
https://issues.apache.org/jira/browse/HDDS-15030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
dino2895 reassigned HDDS-15030:
-------------------------------
Assignee: dino2895
> [Website v2][Troubleshooting] Datanode running out of space
> -----------------------------------------------------------
>
> Key: HDDS-15030
> URL: https://issues.apache.org/jira/browse/HDDS-15030
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: documentation
> Reporter: Wei-Chiu Chuang
> Assignee: dino2895
> Priority: Major
> Attachments: Screenshot 2026-04-13 at 5.15.10 PM.png, Screenshot
> 2026-04-13 at 5.15.16 PM.png
>
>
> a cluster will not function well if it reaches storage limit.
> Datanode fails to write a block if the available space is drops below a
> certain threshold. If all Datanodes are out of capacity, the cluster will
> appear slow, because it is unable to quickly assign a Datanode to store the
> data.
> You can find a summary of Datanode space usage across the cluster from SCM
> web UI. From there, you can drill into the Datanode web UI to check out the
> disk usage of all the disks in the Datanode.
> {noformat}
> <property>
> <name>hdds.datanode.storage.utilization.warning.threshold</name>
> <value>0.75</value>
> <tag>OZONE, SCM, MANAGEMENT</tag>
> <description>
> If a datanode overall storage utilization exceeds more than this
> value, a warning will be logged while processing the nodeReport in SCM.
> </description>
> </property>
> <property>
> <name>hdds.datanode.storage.utilization.critical.threshold</name>
> <value>0.95</value>
> <tag>OZONE, SCM, MANAGEMENT</tag>
> <description>
> If a datanode overall storage utilization exceeds more than this
> value, the datanode will be marked out of space.
> </description>
> </property>
> {noformat}
> If a Datanode runs out of capacity, it can add more disks (data volumes) to
> the node. (Check out our 'Cluster expansion' guide, "Add a disk to an
> existing Datanode") However, because new writes will mostly go to the new
> disk, it creates imbalance as more traffic will be served on the new disk.
> Administrators should run disk balancer to alleviate this problem.
> {noformat}
> <property>
> <name>hdds.datanode.volume.choosing.policy</name>
>
> <value>org.apache.hadoop.ozone.container.common.volume.CapacityVolumeChoosingPolicy</value>
> <tag>OZONE, CONTAINER, STORAGE, MANAGEMENT</tag>
> <description>
> The class name of the policy for choosing volumes in the list of
> directories.
> Defaults to
> org.apache.hadoop.ozone.container.common.volume.CapacityVolumeChoosingPolicy.
> This volume choosing policy randomly chooses two volumes with remaining
> space and then picks the one with lower utilization.
> </description>
> </property>
> {noformat}
> Similarly, the Datanode ratis will fail to write more Ratis transaction logs
> if there's no sufficient space available. Reserve at least 1GB free space for
> Ratis.
> {noformat}
> <property>
> <name>ozone.scm.datanode.ratis.volume.free-space.min</name>
> <value>1GB</value>
> <tag>OZONE, DATANODE</tag>
> <description>Minimum amount of storage space required for each ratis
> volume on a datanode to hold a new pipeline.
> Datanodes with all its ratis volumes with space under this value
> will not be allocated a pipeline or container replica.
> </description>
> </property>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
