Gargi-jais11 opened a new pull request, #10312: URL: https://github.com/apache/ozone/pull/10312
## What changes were proposed in this pull request? 1. Make diskBalancer visible on top-level datanode help command which was hidden in this PR [HDDS-13649](https://issues.apache.org/jira/browse/HDDS-13649) 2. Improve diskBalancer help msg to explain **how to terminate stdin input when using -** to provide datanode addresses through standard input. 3. All the values in diskbalancer report should be rounded to 2 decimal places. 4. Only the **IdealUsage: 0.14747954 | Threshold: 10.0% | ThresholdRange: (0.04747954, 0.24747954) Utilization | VolumeDensity** should be in percentage form. 5. In diskbalancer report for each dn storage report also add **ozoneAvailable field** and change **TotalCapacity -> OzoneCapacity and UsedSpace -> OzoneUsed.** 6. Add more clarity about **delta moves** in the diskbalancer status notes. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-15306 ## How was this patch tested? Tested manually: Before Fix: ``` // diskbalancer not appear in datanode cli option Usage: ozone admin datanode [-hV] [--verbose] [COMMAND] Datanode specific operations -h, --help Show this help message and exit. -V, --version Print version information and exit. --verbose More verbose output. Show the stack trace of the errors. Commands: list List info of datanodes decommission Decommission a datanode maintenance Put a datanode into Maintenance Mode recommission Return a datanode to service status Show status of datanodes usageinfo List usage information of a datanode by IP address, hostname or UUID. Legend (bytes): Filesystem Capacity/Used/Available: raw filesystem stats for the Datanode, aggregated across volumes. Ozone Capacity/Used/Available: Ozone-usable stats after reserved-space adjustment, aggregated across volumes (see hdds.datanode.dir.du.reserved / hdds.datanode.dir.du.reserved. percent). Reserved: hdds.datanode.dir.du.reserved: configured reserved space for non-Ozone usage. Committed: space pre-allocated for containers. Free Space To Spare: minimum free space to keep before closing containers. // not much info to use stdin - Get DiskBalancer status [<datanode address>...] Datanode addresses: one or more, separated by spaces. To read from stdin, specify '-' and supply one item per line.Port is optional and defaults to 19864 (CLIENT_RPC port). Examples: 'DN-1', 'DN-1:19864', '192.168.1.10'. -h, --help Show this help message and exit. // Inconsistency in decimals and percentage for Disk Utilisation or Threshold // no ozoneAvailable given dn-1 ~]# ozone admin datanode diskbalancer report --in-service-datanodes --verbose Report result: Datanode: dn1 (10.82.246.167:19864) Aggregate VolumeDataDensity: 0.20786860859385214 IdealUsage: 0.14747954 | Threshold: 10.0% | ThresholdRange: (0.04747954, 0.24747954) Volume Details: StorageID StoragePath TotalCapacity UsedSpace Container Pre-AllocatedSpace EffectiveUsedSpace Utilization VolumeDensity DS-71f33f87-9212-488c-9ae6-4db48c9a48f1 /hadoop-ozone/datanode/data/hdds 349.27 GB 4.71 GB 26.82 GB 75.05 GB 0.21487346 0.06739392 DS-ac73c11c-993a-49f9-b97e-ea61145452e2 /hadoop-ozone/datanode/data2/hdds 195.78 GB 1.11 GB 4.27 GB 16.36 GB 0.08357777 0.06390177 DS-5b27f109-0029-424a-9092-5ae7692b98db hadoop-ozone/datanode/data3/hdds 195.78 GB 2.32 GB 18.06 GB 30.86 GB 0.15760217 0.01012263 DS-3350ed2c-326c-4725-9fe2-755e85d17b02 hadoop-ozone/datanode/data4/hdds 195.78 GB 1.13 GB 4.26 GB 15.86 GB 0.08102925 0.06645028 ``` After Fix: ``` // diskbalancer visible in datanode cli option Usage: ozone admin datanode [-hV] [--verbose] [COMMAND] Datanode specific operations -h, --help Show this help message and exit. -V, --version Print version information and exit. --verbose More verbose output. Show the stack trace of the errors. Commands: list List info of datanodes decommission Decommission a datanode maintenance Put a datanode into Maintenance Mode recommission Return a datanode to service status Show status of datanodes usageinfo List usage information of a datanode by IP address, hostname or UUID. Legend (bytes): Filesystem Capacity/Used/Available: raw filesystem stats for the Datanode, aggregated across volumes. Ozone Capacity/Used/Available: Ozone-usable stats after reserved-space adjustment, aggregated across volumes (see hdds.datanode.dir.du.reserved / hdds.datanode.dir.du.reserved. percent). Reserved: hdds.datanode.dir.du.reserved: configured reserved space for non-Ozone usage. Committed: space pre-allocated for containers. Free Space To Spare: minimum free space to keep before closing containers. diskbalancer DiskBalancer specific operations to ensure even disk utilisation. It is disabled by default. To enable it, set hdds.datanode.disk.balancer.enabled as true. // improved stdin - info in help msg Usage: ozone admin datanode diskbalancer status [-hV] [--in-service-datanodes] [--json] [--verbose] [<datanode address>...] Get DiskBalancer status [<datanode address>...] Datanode addresses: one or more on the command line, OR read from stdin with '-'. Stdin usage: ozone admin datanode diskbalancer <subcommand> - Then type one datanode per line and end input: - Linux/macOS: Ctrl-D - Windows: Ctrl-Z, then Enter Examples: # Piped (recommended for scripts) echo -e "DN-1\nDN-2" | ozone admin datanode diskbalancer status - # From file having list of dns to balance ozone admin datanode diskbalancer report - < datanode-lists. txt Port is optional and defaults to 19864 (CLIENT_RPC port). Address examples: 'DN-1', 'DN-1:19864', '192.168.1.10'. -h, --help Show this help message and exit. --in-service-datanodes If set, the client will send DiskBalancer requests to all available DataNodes in the HEALTHY and IN_SERVICE operational state. --json Format output as JSON -V, --version Print version information and exit. --verbose More verbose output. Show the stack trace of the errors. // round of value to 2 decimal places // *IdealUsage | Threshold | ThresholdRange | Utilization | VolumeDensity** should be in percentage form. // add **ozoneAvailable field** and change **TotalCapacity -> OzoneCapacity and UsedSpace -> OzoneUsed.** // explain delta move for source volume bash-5.1$ ozone admin datanode diskbalancer report - < datanode-list Report result: Datanode: ozone-datanode-3.ozone_default (172.18.0.8:19864) Aggregate VolumeDataDensity: 0.18% IdealUsage: 12.66% | Threshold: 0.01% | ThresholdRange: (12.65%, 12.67%) Volume Details: StorageID StoragePath OzoneCapacity OzoneAvailable OzoneUsed Container Pre-AllocatedSpace EffectiveUsedSpace Utilization VolumeDensity DS-78ca0f8e-6b4f-4639-aa91-ec2389f1c49f /data/hdds1/hdds 1006.75 GB 880.15 GB 5.42 GB 0 B 126.60 GB 12.57% 0.09% DS-45ac9e0c-b254-4585-b535-60467fde3be3 /data/hdds2/hdds 1006.75 GB 880.15 GB 5.64 GB 1.77 GB 128.37 GB 12.75% 0.09% DS-ee5326f5-5933-4560-8b2c-75dcd8900b1b /data/hdds3/hdds 1006.75 GB 880.15 GB 5.53 GB 907 MB 127.48 GB 12.66% 0.00% Note: - Aggregate VolumeDataDensity: Sum of per-volume density (deviation from ideal); higher means more imbalance. - IdealUsage: Target utilization ratio (0-1) when volumes are evenly balanced. - ThresholdRange: Acceptable deviation (percent); volumes within IdealUsage +/- Threshold are considered balanced. - VolumeDensity: Deviation of a particular volume's utilization from IdealUsage. - Utilization: Ratio of actual used space to capacity (0-1) for a particular volume. - OzoneCapacity: Ozone volume capacity. - OzoneAvailable: Ozone available space. - OzoneUsed: Ozone used space. - Container Pre-AllocatedSpace: Space reserved for containers not yet written to disk. - EffectiveUsedSpace: This is the actual used space of volume which is visible to the diskBalancer : (ozoneCapacity minus ozoneAvailable) + containerPreAllocatedSpace + move delta . - move delta: source volume space to be reclaimed after move completion; this value is reflected only when diskBalancer is running else it is 0. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
