mbrodmann opened a new pull request #85: URL: https://github.com/apache/incubator-crail/pull/85
This PR is the second of three planned PRs for implementing resource elasticity within Crail. More specifically, the goal is to allow to automatically and dynamically adapt the number of running datanodes based on different possible parameters (e.g. resource consumption, available storage capacities, ...). This requires support for starting and terminating running datanodes automatically. This functionality will be implemented mostly within the Crail namenode. Therefore, the Crail namenode will not only maintain information on the current state of the datanodes and the deployment but will also start or terminate datanodes as required. For deciding how to adjust the number of datanodes based on the current state observed in the Crail namenode so-called _policies_ are used. The following changes will be included within the seperate PRs: 1) The first PR implements a new RPC (for both darpc and narpc) that allows to specify a datanode (using IP-address and port number) that should be shutdown. 2) In this second PR the actual mechanism for shutting down a datanode is covered. This includes checking that a datanode can be removed safely (i.e. the datanode does not store any remaining data), otherwise it will only be scheduled for removal. This PR also handles the datanode termination for all of the three available storage layers. 3) The third PR implements a simple example policy for implementing elasticity based on the system's storage consumption. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
