If you are hosting on the cloud, I'd recommend going for dedicated worker nodes for the NiFi cluster. There might be rare (or not) occasions when a worker node is under high load and needs to evict pods. If your NiFi deployment's pod disruption budget allows for eviction of pods then there are always chances that an evicted NiFi pod can be rescheduled on a different node that is tainted (tainted because the node may not meet the pod's volume affinity requirements). Your best case scenario when this happens is that the pod will keep getting rescheduled on different nodes until it starts up again. The worst case scenario is that it'll be stuck in a CrashLoopBackoff limbo.
Disclaimer - I speak from my experience on a non production environment. Our NiFi clusters will be deployed to a production k8s environment in a few weeks from now. I am only sharing some learnings I've had w.r.t. k8s statefulsets along the way. Hope this helps, Swarup. On Mon, Oct 21, 2019, 9:32 PM Wyllys Ingersoll < wyllys.ingers...@keepertech.com> wrote: > > We had success running a 3-node cluster in kubernetes using modified > configuration scripts from the AlexJones github repo - > https://github.com/AlexsJones/nifi > Ours is on an internal bare-metal k8s lab configuration, not in a public > cloud at this time, but the basics are the same either way. > > - setup nifi as a stateful set so you can scale up or down as needed. When > a pod fails, k8s will spawn another to take its place and zookeeper will > manage the election of the master during transitions. > - manage your certs as K8S secrets. > - you also need to also have a stateful set of zookeeper pods for managing > the nifi servers. > - use persistent volume mounts to hold the flowfile, database, content, > and provenance _repository directories > > > > On Mon, Oct 21, 2019 at 11:21 AM Joe Gresock <jgres...@gmail.com> wrote: > >> Apologies if this has been answered on the list already.. >> >> Does anyone have knowledge of the latest in the realm of nifi kubernetes >> support? I see some pages like https://hub.helm.sh/charts/cetic/nifi, >> and https://github.com/AlexsJones/nifi but am unsure which example to >> pick to start with. >> >> I'm curious how well kubernetes maintains the nifi cluster state with pod >> failures. I.e., do any of the k8s implementations play well with the nifi >> cluster list so that we don't have dangling downed nodes in the cluster? >> Also, I'm wondering how certs are managed in a secured cluster. >> >> Appreciate any nudge in the right direction, >> Joe >> > On Mon, Oct 21, 2019, 9:32 PM Wyllys Ingersoll < wyllys.ingers...@keepertech.com> wrote: > > We had success running a 3-node cluster in kubernetes using modified > configuration scripts from the AlexJones github repo - > https://github.com/AlexsJones/nifi > Ours is on an internal bare-metal k8s lab configuration, not in a public > cloud at this time, but the basics are the same either way. > > - setup nifi as a stateful set so you can scale up or down as needed. When > a pod fails, k8s will spawn another to take its place and zookeeper will > manage the election of the master during transitions. > - manage your certs as K8S secrets. > - you also need to also have a stateful set of zookeeper pods for managing > the nifi servers. > - use persistent volume mounts to hold the flowfile, database, content, > and provenance _repository directories > > > > On Mon, Oct 21, 2019 at 11:21 AM Joe Gresock <jgres...@gmail.com> wrote: > >> Apologies if this has been answered on the list already.. >> >> Does anyone have knowledge of the latest in the realm of nifi kubernetes >> support? I see some pages like https://hub.helm.sh/charts/cetic/nifi, >> and https://github.com/AlexsJones/nifi but am unsure which example to >> pick to start with. >> >> I'm curious how well kubernetes maintains the nifi cluster state with pod >> failures. I.e., do any of the k8s implementations play well with the nifi >> cluster list so that we don't have dangling downed nodes in the cluster? >> Also, I'm wondering how certs are managed in a secured cluster. >> >> Appreciate any nudge in the right direction, >> Joe >> >
