Dear all Crail supports dynamically adding new datanodes, while the Crail cluster is running.
To build an elastic storage service, it would be nice to have mechanisms and protocol extensions, which allow a datanode to gracefully leave the cluster. Especially in serverless environments, it would be great if the Crail cluster could also dynamically grow and shrink according to current storage capacity needs. Since a while I am experimenting with an older version, which is being used in Pocket. I changed and extended it and gained some more experience, what I believe would be good to have. I also believe that adding such functionality natively to Apache Crail (instead of "around" Crail as in Pocket) would help making Crail a storage service choice in serverless environments. Furthermore, having this mechanisms natively in Crail does not harm running Crail the classical way. More concretely, I suggest to add the following: - Add mechanisms to gracefully leave datanodes (with the namenode's help) - Mechanism 1: Datanode leaves when no more blocks are allocated (as in Pocket) - Mechanism 2: Namenode helps to move blocks from the leaving datanode to a remaining datanode. "helps" does not mean, that the namenode has to perform the actual data copying, but only to find new blocks and update the file block lists. - Allow datanodes to express the wish to leave (ask namenode to initiate the process), by sending a message to the namenode. I would also volunteer to add this functionality. I would like to emphasize that I would like to add the mechanisms in a way that they do not get invoked automatically and that performance and current functionality will no be affected in any way, when Crail is used the usual way. Instead, adding the mechanisms allows building a dynamically scaling system based on policy code, which can also run outside of the namenode and datanodes. Thanks Adrian
