Dave, NiFi itself does not offer an autoscaling feature for clustering whereby it would add/remove nodes. You'd need to build that over NiFi's API and cluster model which is feasible. It is quite easy to add nodes to the cluster as you need and provided you're using s2s or protocols with queuing semantics (which you'd need for scale anyway) then this works quite well. For scaling down, dropping a node, the key is to disconnect the node from the cluster and stop its sources to let the current data flow through, then you can remove the node.
I do anticipate this being something we will tackle and there are really interesting technologies out there like mesos, yarn, and others that we can learn a great deal from or build on top of. Thanks Joe On Thu, Dec 8, 2016 at 6:49 AM, DAVID SMITH <[email protected]> wrote: > Hi > We are looking to set up some NiFi Clusters, however, we would like them to > scale elastically without having to restart the NCM as we add or remove nodes. > From NiFi 0.7.1 onwards is this possible or is this a feature that is being > looked at or should we be looking at our own solution? > Many thanksDave
