mbrodmann opened a new pull request #86: URL: https://github.com/apache/incubator-crail/pull/86
This PR is the third of three planned PRs for implementing resource elasticity within Crail. More specifically, the goal is to allow to automatically and dynamically adapt the number of running datanodes based on different possible parameters (e.g. resource consumption, available storage capacities, ...). This requires support for starting and terminating running datanodes automatically. This functionality will be implemented mostly within the Crail namenode. Therefore, the Crail namenode will not only maintain information on the current state of the datanodes and the deployment but will also start or terminate datanodes as required. For deciding how to adjust the number of datanodes based on the current state observed in the Crail namenode so-called _policies_ are used. The following changes will be included within the seperate PRs: 1) This first PR implements a new RPC (for both darpc and narpc) that allows to specify a datanode (using IP-address and port number) that should be shutdown. 2) The seconds PR will implement the actual mechanism for shutting down a datanode. 3) This third PR implements a simple policy for implementing elasticity based on the system's storage consumption. It introduces a new `ElasticNameNodeService` extending the existing `NameNodeService` which frequently evaluates the current storage utilization. When it exceeds a certain threshold this will make the namenode start new datanode(s). In case the utilization falls below a certain threshold the namenode tries to terminate unneeded datanodes using the shutdown mechanism. Parameters are configurable. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
