Gargi-jais11 opened a new pull request, #9310: URL: https://github.com/apache/ozone/pull/9310
## What changes were proposed in this pull request? After implementing direct client-to-DN communication for DiskBalancer in [HDDS-13598](https://issues.apache.org/jira/browse/HDDS-13598), the SCM part is no longer needed and should be removed. This JIRA covers: - Support **stdin** so that a list of datanodes can be easily passed in from a file. - Support **json** option to gather output from many nodes and just parse out problematic ones. - Remove the **-d / --datanodes** flag and keep it as a list of space separated arguments of datanodes similar to ozone admin container info . - Remove rest of the scm part connection with datanode - Cleanup of obsolete protobuf definitions in `ScmServerDatanodeHeartbeatProtocol.proto` and `ScmAdminProtocol.proto` - Rewriting integration tests (`TestDiskBalancer`, `TestDiskBalancerDuringDecommissionAndMaintenance`) for direct communication - Rewrite `testdiskbalancer.robot` robot test. - Do unit testing of all new diskbalancer subcommands. - Update the design doc and feature doc for diskBalancer (`DiskBalancer.md`). ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-13878 ## How was this patch tested? **Added new Unit Test :** `TestDiskBalancerSubCommands` **Updated Integration Tests :** `TestDiskBalancer.java` and `TestDiskBalancerDuringDecommissioningAndMaintenance.java` **Updated Acceptance Test :** `testdiskbalancer.robot` Below are the manual test output from docker-cluster: - specific dn request: ``` bash-5.1$ ozone admin datanode diskbalancer status ozone-datanode-2 ozone-datanode-3 ozone-datanode-5 Status result: Datanode Status Threshold(%) BandwidthInMB Threads SuccessMove FailureMove BytesMoved(MB) EstBytesToMove(MB) EstTimeLeft(min) ozone-datanode-2.ozone_default STOPPED 10.0000 10 5 0 0 0 0 0 ozone-datanode-3.ozone_default STOPPED 10.0000 10 5 0 0 0 0 0 ozone-datanode-5.ozone_default STOPPED 10.0000 10 5 0 0 0 0 0 Note: Estimated time left is calculated based on the estimated bytes to move and the configured disk bandwidth. bash-5.1$ ozone admin datanode diskbalancer report ozone-datanode-2 ozone-datanode-3 ozone-datanode-5 Report result: Datanode VolumeDensity ozone-datanode-5.ozone_default 5.052149552875473E-9 ozone-datanode-2.ozone_default 5.0521495459365795E-9 ozone-datanode-3.ozone_default 5.052149538997686E-9 bash-5.1$ ozone admin datanode diskbalancer start -t 0.001 -s false ozone-datanode-2 ozone-datanode-3 ozone-datanode-5 Started DiskBalancer on nodes: [ozone-datanode-2, ozone-datanode-3, ozone-datanode-5] bash-5.1$ ozone admin datanode diskbalancer stop ozone-datanode-2 ozone-datanode-3 ozone-datanode-5 Stopped DiskBalancer on nodes: [ozone-datanode-2, ozone-datanode-3, ozone-datanode-5] ``` - json output ``` bash-5.1$ ozone admin datanode diskbalancer report --json --in-service-datanodes [ { "volumeDensity" : 0.0, "datanode" : "ozone-datanode-5.ozone_default" }, { "volumeDensity" : 0.0, "datanode" : "ozone-datanode-3.ozone_default" }, { "volumeDensity" : 0.0, "datanode" : "ozone-datanode-2.ozone_default" }, { "volumeDensity" : 0.0, "datanode" : "ozone-datanode-4.ozone_default" }, { "volumeDensity" : 0.0, "datanode" : "ozone-datanode-1.ozone_default" } ] bash-5.1$ ozone admin datanode diskbalancer status --json ozone-datanode-1 ozone-datanode-2 ozone-datanode-5 [ { "datanode" : "ozone-datanode-1.ozone_default", "status" : "STOPPED", "threshold" : 10.0, "bandwidthInMB" : 10, "threads" : 5, "successMove" : 0, "failureMove" : 0, "bytesMovedMB" : 0, "estBytesToMoveMB" : 0, "estTimeLeftMin" : 0 }, { "datanode" : "ozone-datanode-2.ozone_default", "status" : "STOPPED", "threshold" : 0.001, "bandwidthInMB" : 10, "threads" : 5, "successMove" : 0, "failureMove" : 0, "bytesMovedMB" : 0, "estBytesToMoveMB" : 0, "estTimeLeftMin" : 0 }, { "datanode" : "ozone-datanode-5.ozone_default", "status" : "STOPPED", "threshold" : 0.001, "bandwidthInMB" : 10, "threads" : 5, "successMove" : 0, "failureMove" : 0, "bytesMovedMB" : 0, "estBytesToMoveMB" : 0, "estTimeLeftMin" : 0 } ] bash-5.1$ ozone admin datanode diskbalancer update -t 0.001 -b 20 --json ozone-datanode-1 ozone-datanode-4 [ { "datanode" : "ozone-datanode-1", "action" : "update", "status" : "success", "configuration" : { "threshold" : 0.001, "bandwidthInMB" : 20 } }, { "datanode" : "ozone-datanode-4", "action" : "update", "status" : "success", "configuration" : { "threshold" : 0.001, "bandwidthInMB" : 20 } } ] bash-5.1$ ozone admin datanode diskbalancer start -s false --json ozone-datanode-1 ozone-datanode-4 [ { "datanode" : "ozone-datanode-1", "action" : "start", "status" : "success", "configuration" : { "stopAfterDiskEven" : false } }, { "datanode" : "ozone-datanode-4", "action" : "start", "status" : "success", "configuration" : { "stopAfterDiskEven" : false } } ] bash-5.1$ ozone admin datanode diskbalancer stop --json ozone-datanode-1 [ { "datanode" : "ozone-datanode-1", "action" : "stop", "status" : "success" } ] ``` - stdin output ``` bash-5.1$ cat datanode.txt | ozone admin datanode diskbalancer status - Status result: Datanode Status Threshold(%) BandwidthInMB Threads SuccessMove FailureMove BytesMoved(MB) EstBytesToMove(MB) EstTimeLeft(min) ozone-datanode-1.ozone_default RUNNING 0.0010 20 20 0 0 0 0 0 ozone-datanode-2.ozone_default RUNNING 0.0010 10 20 0 0 0 0 0 Note: Estimated time left is calculated based on the estimated bytes to move and the configured disk bandwidth. bash-5.1$ cat datanode.txt | ozone admin datanode diskbalancer report - Report result: Datanode VolumeDensity ozone-datanode-1.ozone_default 0.0 ozone-datanode-2.ozone_default 0.0 bash-5.1$ cat datanode.txt | ozone admin datanode diskbalancer start -p 20 - Started DiskBalancer on nodes: [ozone-datanode-1, ozone-datanode-2] bash-5.1$ cat datanode.txt | ozone admin datanode diskbalancer status - Status result: Datanode Status Threshold(%) BandwidthInMB Threads SuccessMove FailureMove BytesMoved(MB) EstBytesToMove(MB) EstTimeLeft(min) ozone-datanode-1.ozone_default RUNNING 0.0010 20 20 0 0 0 0 0 ozone-datanode-2.ozone_default RUNNING 0.0010 10 20 0 0 0 0 0 Note: Estimated time left is calculated based on the estimated bytes to move and the configured disk bandwidth. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
