Just kinda checking in again, to see if anyone is working on a more standardized Kubernetes Nifi integration, like a NiFi "CRD" for Kubernetes, or a helm chart. I have tried a couple of the charts out there, but none seem to be able to produce a fully secure NiFi cluster in Kubernetes.
Thanks! Joe On Mon, Oct 21, 2019 at 12:19 PM Swarup Karavadi <[email protected]> wrote: > If you are hosting on the cloud, I'd recommend going for dedicated worker > nodes for the NiFi cluster. There might be rare (or not) occasions when a > worker node is under high load and needs to evict pods. If your NiFi > deployment's pod disruption budget allows for eviction of pods then there > are always chances that an evicted NiFi pod can be rescheduled on a > different node that is tainted (tainted because the node may not meet the > pod's volume affinity requirements). Your best case scenario when this > happens is that the pod will keep getting rescheduled on different nodes > until it starts up again. The worst case scenario is that it'll be stuck in > a CrashLoopBackoff limbo. > > Disclaimer - I speak from my experience on a non production environment. > Our NiFi clusters will be deployed to a production k8s environment in a few > weeks from now. I am only sharing some learnings I've had w.r.t. k8s > statefulsets along the way. > > Hope this helps, > Swarup. > > On Mon, Oct 21, 2019, 9:32 PM Wyllys Ingersoll < > [email protected]> wrote: > >> >> We had success running a 3-node cluster in kubernetes using modified >> configuration scripts from the AlexJones github repo - >> https://github.com/AlexsJones/nifi >> Ours is on an internal bare-metal k8s lab configuration, not in a public >> cloud at this time, but the basics are the same either way. >> >> - setup nifi as a stateful set so you can scale up or down as needed. >> When a pod fails, k8s will spawn another to take its place and zookeeper >> will manage the election of the master during transitions. >> - manage your certs as K8S secrets. >> - you also need to also have a stateful set of zookeeper pods for >> managing the nifi servers. >> - use persistent volume mounts to hold the flowfile, database, content, >> and provenance _repository directories >> >> >> >> On Mon, Oct 21, 2019 at 11:21 AM Joe Gresock <[email protected]> wrote: >> >>> Apologies if this has been answered on the list already.. >>> >>> Does anyone have knowledge of the latest in the realm of nifi kubernetes >>> support? I see some pages like https://hub.helm.sh/charts/cetic/nifi, >>> and https://github.com/AlexsJones/nifi but am unsure which example to >>> pick to start with. >>> >>> I'm curious how well kubernetes maintains the nifi cluster state with >>> pod failures. I.e., do any of the k8s implementations play well with the >>> nifi cluster list so that we don't have dangling downed nodes in the >>> cluster? Also, I'm wondering how certs are managed in a secured cluster. >>> >>> Appreciate any nudge in the right direction, >>> Joe >>> >> > On Mon, Oct 21, 2019, 9:32 PM Wyllys Ingersoll < > [email protected]> wrote: > >> >> We had success running a 3-node cluster in kubernetes using modified >> configuration scripts from the AlexJones github repo - >> https://github.com/AlexsJones/nifi >> Ours is on an internal bare-metal k8s lab configuration, not in a public >> cloud at this time, but the basics are the same either way. >> >> - setup nifi as a stateful set so you can scale up or down as needed. >> When a pod fails, k8s will spawn another to take its place and zookeeper >> will manage the election of the master during transitions. >> - manage your certs as K8S secrets. >> - you also need to also have a stateful set of zookeeper pods for >> managing the nifi servers. >> - use persistent volume mounts to hold the flowfile, database, content, >> and provenance _repository directories >> >> >> >> On Mon, Oct 21, 2019 at 11:21 AM Joe Gresock <[email protected]> wrote: >> >>> Apologies if this has been answered on the list already.. >>> >>> Does anyone have knowledge of the latest in the realm of nifi kubernetes >>> support? I see some pages like https://hub.helm.sh/charts/cetic/nifi, >>> and https://github.com/AlexsJones/nifi but am unsure which example to >>> pick to start with. >>> >>> I'm curious how well kubernetes maintains the nifi cluster state with >>> pod failures. I.e., do any of the k8s implementations play well with the >>> nifi cluster list so that we don't have dangling downed nodes in the >>> cluster? Also, I'm wondering how certs are managed in a secured cluster. >>> >>> Appreciate any nudge in the right direction, >>> Joe >>> >> -- Be on your guard; stand firm in the faith; be courageous; be strong. Do everything in love. -*1 Corinthians 16:13-14*
