Thanks Adrian, this sounds great! Best, Ana
On Thu, May 28, 2020 at 7:40 AM Adrian Schüpbach < adrian.schuepb...@gribex.net> wrote: > Thanks, Patrick! I will post updates to the list > and hopefully have a first part of code ready for > discussion/reviewing soon. > > Best regards > Adrian > > On 27.05.20 18:00, Patrick Stuedi wrote: > > Adrian, thanks for providing this info. Looking forward to the > > implementation. > > > > Best regards, > > Patrick > > > > On Wed, May 27, 2020 at 2:32 AM Adrian Schüpbach < > > adrian.schuepb...@gribex.net> wrote: > > > >> Dear all > >> > >> Crail supports dynamically adding new datanodes, while the Crail cluster > >> is running. > >> > >> To build an elastic storage service, it would be nice to have > >> mechanisms and protocol extensions, which allow a datanode > >> to gracefully leave the cluster. > >> > >> Especially in serverless environments, it would be great > >> if the Crail cluster could also dynamically grow and shrink > >> according to current storage capacity needs. > >> > >> Since a while I am experimenting with an older version, > >> which is being used in Pocket. I changed and extended it > >> and gained some more experience, what I believe would be > >> good to have. I also believe that adding such functionality > >> natively to Apache Crail (instead of "around" Crail as in Pocket) > >> would help making Crail a storage service choice in serverless > >> environments. Furthermore, having this mechanisms natively > >> in Crail does not harm running Crail the classical way. > >> > >> More concretely, I suggest to add the following: > >> - Add mechanisms to gracefully leave datanodes > >> (with the namenode's help) > >> - Mechanism 1: Datanode leaves when no more blocks are allocated > >> (as in Pocket) > >> - Mechanism 2: Namenode helps to move blocks from the leaving > >> datanode to a remaining datanode. "helps" does not mean, > >> that the namenode has to perform the actual data copying, > >> but only to find new blocks and update the file block lists. > >> - Allow datanodes to express the wish to leave > >> (ask namenode to initiate the process), by sending a message to > >> the namenode. > >> > >> I would also volunteer to add this functionality. > >> > >> > >> > >> I would like to emphasize that I would like to add > >> the mechanisms in a way that they do not get invoked > >> automatically and that performance and current functionality > >> will no be affected in any way, when Crail is used the usual way. > >> Instead, adding the mechanisms allows building a dynamically > >> scaling system based on policy code, which can also run outside > >> of the namenode and datanodes. > >> > >> Thanks > >> Adrian > >> > >> > > >
