Gargi-jais11 opened a new pull request, #9969: URL: https://github.com/apache/ozone/pull/9969
## What changes were proposed in this pull request? The DiskBalancer report today shows only: ``` ozone admin datanode diskbalancer report --in-service-datanodes Report result: Datanode VolumeDensity dn-hostname-3 (10.141.248.70:19864) 0.09267551461620249 dn-hostname-1 (10.141.128.135:19864) 0.06619677701803184 dn-hostname-2 (10.141.126.8:19864) 0.026044182616493772 ``` So users see only a single aggregate VolumeDensity per datanode, with no per-disk breakdown. According to above report if user wants to run diskbalancer at threshold lower that 10% say at 5% it interprets that diskbalancer will start on DN-3 and DN-1. But it does not start and creates confusion that diskbalancer is not working correctly. This is because in reality this threshold value checks wether each volumes utilisation is above or below or within the range. Proposed Solution: We should also show each volume's density along with the details of each volumes utilisation and pre-allocated container bytes ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-14866 ## How was this patch tested? Added a unit test and tested manually as well. **DiskBalancer Report before patch:** ``` ozone admin datanode diskbalancer report --in-service-datanodes Report result: Datanode VolumeDensity dn-hostname-3 (10.141.248.70:19864) 0.09267551461620249 dn-hostname-1 (10.141.128.135:19864) 0.06619677701803184 dn-hostname-2 (10.141.126.8:19864) 0.026044182616493772 ``` **DiskBalancer Report after enhancement:** ``` bash-5.1$ ozone admin datanode diskbalancer report --in-service-datanodes Report result: Datanode: ozone-datanode-1.ozone_default (172.18.0.8:19864) Aggregate VolumeDataDensity: 0.002000812888676179 IdealUsage: 0.1079634239173063 | Threshold: 10.0% | ThresholdRange: (0.007963423917306298, 0.20796342391730632) Volume Details -: StorageID StoragePath VolumeDensity Utilization Pre-Allocated Container Bytes DS-c1e30832-dd36-444d-81fa-949c70b15b22 /data/hdds1/hdds 0.00087753816739934150 0.10884096208470564000 1771044864 DS-d962ecab-8e1c-408a-87b0-b6d53456c86f /data/hdds2/hdds 0.00012286827693874790 0.10808629219424505000 877658112 DS-48547294-c3ef-40ef-a48a-14bf3b2acad8 /data/hdds3/hdds 0.00100040644433808950 0.10696301747296821000 0 ------- Datanode: ozone-datanode-2.ozone_default (172.18.0.7:19864) Aggregate VolumeDataDensity: 1.940025425348213E-6 IdealUsage: 0.10798832091026495 | Threshold: 10.0% | ThresholdRange: (0.00798832091026494, 0.20798832091026495) Volume Details -: StorageID StoragePath VolumeDensity Utilization Pre-Allocated Container Bytes DS-e8505479-f9fd-4705-b59e-c2bca68f0382 /data/hdds1/hdds 0.00000097001271268105 0.10798735089755226000 943718400 DS-d0f5f9cd-920f-4ec9-9248-162ad3e1ad11 /data/hdds2/hdds 0.00000097001271266717 0.10798929092297761000 943718400 DS-212374f2-15bc-4f54-b64e-30f378da456a /data/hdds3/hdds 0.00000000000000000000 0.10798832091026495000 944766976 ------- Datanode: ozone-datanode-3.ozone_default (172.18.0.9:19864) Aggregate VolumeDataDensity: 1.2933502835515975E-6 IdealUsage: 0.10797732743285464 | Threshold: 10.0% | ThresholdRange: (0.007977327432854633, 0.20797732743285463) Volume Details -: StorageID StoragePath VolumeDensity Utilization Pre-Allocated Container Bytes DS-8364a6cc-0d6b-471c-9b3f-fb2b68a6ef27 /data/hdds1/hdds 0.00000064667514178274 0.10797668075771286000 968884224 DS-e9705dfb-f66d-4b21-a619-7c41c53f7cd6 /data/hdds2/hdds 0.00000032333757088443 0.10797765077042552000 966787072 DS-c64998b1-8a10-4506-b358-e1ae278ed71c /data/hdds3/hdds 0.00000032333757088443 0.10797765077042552000 966787072 Note: - Aggregate VolumeDataDensity: Sum of per-volume density (deviation from ideal); higher means more imbalance. - IdealUsage: Target utilization ratio (0-1) when volumes are evenly balanced. - ThresholdRange: Acceptable deviation (percent); volumes within IdealUsage +/- Threshold are considered balanced. - VolumeDensity: Deviation of a particular volume's utilization from IdealUsage. - Utilization: Ratio of actual used space to capacity (0-1) for a particular volume. - Pre-Allocated Container Bytes: Space reserved for containers not yet written to disk. // json output bash-5.1$ ozone admin datanode diskbalancer report --in-service-datanodes --json [ { "datanode" : "ozone-datanode-1.ozone_default (172.18.0.8:19864)", "action" : "report", "status" : "success", "volumeDensity" : 0.002000812888676179, "idealUsage" : 0.1079634239173063, "volumes" : [ { "storageId" : "DS-c1e30832-dd36-444d-81fa-949c70b15b22", "storagePath" : "/data/hdds1/hdds", "volumeDensity" : 8.775381673993415E-4, "utilization" : 0.10884096208470564, "pre-Allocated container bytes" : 1771044864 }, { "storageId" : "DS-d962ecab-8e1c-408a-87b0-b6d53456c86f", "storagePath" : "/data/hdds2/hdds", "volumeDensity" : 1.228682769387479E-4, "utilization" : 0.10808629219424505, "pre-Allocated container bytes" : 877658112 }, { "storageId" : "DS-48547294-c3ef-40ef-a48a-14bf3b2acad8", "storagePath" : "/data/hdds3/hdds", "volumeDensity" : 0.0010004064443380895, "utilization" : 0.10696301747296821, "pre-Allocated container bytes" : 0 } ] }, { "datanode" : "ozone-datanode-3.ozone_default (172.18.0.9:19864)", "action" : "report", "status" : "success", "volumeDensity" : 1.2933502835515975E-6, "idealUsage" : 0.10797732743285464, "volumes" : [ { "storageId" : "DS-8364a6cc-0d6b-471c-9b3f-fb2b68a6ef27", "storagePath" : "/data/hdds1/hdds", "volumeDensity" : 6.466751417827377E-7, "utilization" : 0.10797668075771286, "pre-Allocated container bytes" : 968884224 }, { "storageId" : "DS-e9705dfb-f66d-4b21-a619-7c41c53f7cd6", "storagePath" : "/data/hdds2/hdds", "volumeDensity" : 3.2333757088442994E-7, "utilization" : 0.10797765077042552, "pre-Allocated container bytes" : 966787072 }, { "storageId" : "DS-c64998b1-8a10-4506-b358-e1ae278ed71c", "storagePath" : "/data/hdds3/hdds", "volumeDensity" : 3.2333757088442994E-7, "utilization" : 0.10797765077042552, "pre-Allocated container bytes" : 966787072 } ] }, { "datanode" : "ozone-datanode-2.ozone_default (172.18.0.7:19864)", "action" : "report", "status" : "success", "volumeDensity" : 1.940025425348213E-6, "idealUsage" : 0.10798832091026495, "volumes" : [ { "storageId" : "DS-e8505479-f9fd-4705-b59e-c2bca68f0382", "storagePath" : "/data/hdds1/hdds", "volumeDensity" : 9.700127126810454E-7, "utilization" : 0.10798735089755226, "pre-Allocated container bytes" : 943718400 }, { "storageId" : "DS-d0f5f9cd-920f-4ec9-9248-162ad3e1ad11", "storagePath" : "/data/hdds2/hdds", "volumeDensity" : 9.700127126671676E-7, "utilization" : 0.10798929092297761, "pre-Allocated container bytes" : 943718400 }, { "storageId" : "DS-212374f2-15bc-4f54-b64e-30f378da456a", "storagePath" : "/data/hdds3/hdds", "volumeDensity" : 0.0, "utilization" : 0.10798832091026495, "pre-Allocated container bytes" : 944766976 } ] } ] ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
