Hi Mark, We are also looking at spinning up nifi on k8s. If you could share more information on your architecture of Nifi on k8s, that would be very helpful. Also any best practice that you would suggest. I am interested in understanding the auto scaling of nifi on k8s. Before I deep dive, wanted to check if you have explored the best way to scale up or down and what parameters would qualify in triggering that action.
Thanks Anil On Mon, Apr 16, 2018 at 1:36 PM, Mark Payne <[email protected]> wrote: > Jonathan, > > I've spent some time in the last few weeks playing around with getting > some clusters running on Kubernetes > as well. I know from the mailing lists that some others have been > venturing into this also. I'm by no means > a Kubernetes expert myself, but I've gotten it up & running on a few-node > cluster in GCE, so I do have some > familiarity. > > One thing to note is that the log file that you shared appears to be from > an instance that's been running for a while, not > one that was newly started, so it's possible that there were some > important details regarding why the clustering is not > working as expected at the beginning of the logs that were missed. > > I also notice that you're setting the "nifi.web.http.host" and > "nifi.cluster.node.address" properties to "nifi-test". > I'm not sure that this will work. Perhaps it will, I've never tried that, > but I don't know if the NiFi process will be able to listen > on that hostname. What I ended up doing was to update my docker image > locally to update the "hostname" variable to be > the fully-qualified hostname: > > export hostname=$(hostname -f) > > And then set the http.host and node.address properties to that. If setting > to nifi-test works, though, then that would be great, > certainly a lot easier. But I suspect we'll need to take a closer look at > the logs just after startup to see if they reveal anything > interesting. > > Thanks > -Mark > > > On Apr 16, 2018, at 12:26 PM, Jonathan Kosgei <[email protected]< > mailto:[email protected]>> wrote: > > Hi, > > > > I'm trying to cluster nifi on kubernetes but haven't been able to get the > pods to connect to each other. > > > > I have the following statefulset and services; > > https://gist.github.com/jonathan-kosgei/cdf7c9ec882948eac12beb6b28ffa748 > > > > I'm running nifi 1.6.0 on Kubernetes v1.8.8-gke.0. > > > > My nifi-app.log: > > https://gist.github.com/jonathan-kosgei/527952976ec18cf3957cec4d1e186f68 > > > > I set every log option to DEBUG in logback.xml > > > > I've noticed a lot of restarts and several files in the logs folder > > > > nifi@nifi2-0:/opt/nifi/nifi-1.6.0/logs$ ls > > nifi-app.log nifi-app_2018-04-16_03.0.log > nifi-app_2018-04-16_06.0.log nifi-app_2018-04-16_09.0.log > nifi-app_2018-04-16_12.0.log nifi-app_2018-04-16_15.0.log > > nifi-app_2018-04-16_01.0.log nifi-app_2018-04-16_04.0.log > nifi-app_2018-04-16_07.0.log nifi-app_2018-04-16_10.0.log > nifi-app_2018-04-16_13.0.log nifi-bootstrap.log > > nifi-app_2018-04-16_02.0.log nifi-app_2018-04-16_05.0.log > nifi-app_2018-04-16_08.0.log nifi-app_2018-04-16_11.0.log > nifi-app_2018-04-16_14.0.log nifi-user.log > > > > Any ideas what I might be missing? > > > > The cluster configuration seemed very straightforward. > > > > Thanks! > > >
