Adrian, thanks for providing this info. Looking forward to the implementation.
Best regards, Patrick On Wed, May 27, 2020 at 2:32 AM Adrian Schüpbach < [email protected]> wrote: > Dear all > > Crail supports dynamically adding new datanodes, while the Crail cluster > is running. > > To build an elastic storage service, it would be nice to have > mechanisms and protocol extensions, which allow a datanode > to gracefully leave the cluster. > > Especially in serverless environments, it would be great > if the Crail cluster could also dynamically grow and shrink > according to current storage capacity needs. > > Since a while I am experimenting with an older version, > which is being used in Pocket. I changed and extended it > and gained some more experience, what I believe would be > good to have. I also believe that adding such functionality > natively to Apache Crail (instead of "around" Crail as in Pocket) > would help making Crail a storage service choice in serverless > environments. Furthermore, having this mechanisms natively > in Crail does not harm running Crail the classical way. > > More concretely, I suggest to add the following: > - Add mechanisms to gracefully leave datanodes > (with the namenode's help) > - Mechanism 1: Datanode leaves when no more blocks are allocated > (as in Pocket) > - Mechanism 2: Namenode helps to move blocks from the leaving > datanode to a remaining datanode. "helps" does not mean, > that the namenode has to perform the actual data copying, > but only to find new blocks and update the file block lists. > - Allow datanodes to express the wish to leave > (ask namenode to initiate the process), by sending a message to > the namenode. > > I would also volunteer to add this functionality. > > > > I would like to emphasize that I would like to add > the mechanisms in a way that they do not get invoked > automatically and that performance and current functionality > will no be affected in any way, when Crail is used the usual way. > Instead, adding the mechanisms allows building a dynamically > scaling system based on policy code, which can also run outside > of the namenode and datanodes. > > Thanks > Adrian > >
