Yes, to further clarify what I meant... If you are trying to change the Initial Admin or Node Identities in authorizers.xml, these will only be used when there are no other users/group/policies present. People frequently make a mistake during initial config and then try to edit authorizers.xml and try again, but it won't actually do anything unless you remove the users.xml and authorizations.xml to start over.
In your case it sounds like you are trying to convert and existing standalone node to a cluster, given that I would do the following... - In standalone mode, use the UI to add users for the DN's of the server certificates (CN=nifi-node-1, OU=NIFI, CN=nifi-node-2, OU=NIFI) - In the UI, grant those users Write access to "Proxy" - Convert to a cluster and keep your same authorizers.xml, users.xml, and authorizations.xml when you setup your cluster, this way all your users and policies are already setup and the Initial Admin and Node Identities are not needed On Mon, Oct 22, 2018 at 1:06 PM Saip, Alexander (NIH/CC/BTRIS) [C] <alexander.s...@nih.gov> wrote: > > Thanks again, Bryan. Just a quick follow-up question: does removing users.xml > and authorizations.xml mean that we will need to re-create all users and > groups that we had in the original standalone NiFi instance? > > -----Original Message----- > From: Bryan Bende <bbe...@gmail.com> > Sent: Monday, October 22, 2018 12:48 PM > To: users@nifi.apache.org > Subject: Re: NiFi fails on cluster nodes > > Sorry I was confused when you said two 1 node clusters and I assumed they > each had their own ZooKeeper. > > You don't need to run ZK on both nodes, you can create a 2 node cluster using > the embedded ZK on the first node. > > This blog post shows how to setup a secure 2 node cluster: > > https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy > > The only difference is that the authorizers.xml has changed slightly, so > instead of: > > <authorizer> > <identifier>file-provider</identifier> > <class>org.apache.nifi.authorization.FileAuthorizer</class> > <property name="Authorizations File">./conf/authorizations.xml</property> > <property name="Users File">./conf/users.xml</property> > <property name="Initial Admin Identity">CN=bbende, > OU=ApacheNiFi</property> > <property name="Legacy Authorized Users File"></property> > <property name="Node Identity 1">CN=localhost, OU=NIFI</property> > </authorizer> > > You need to add the the users to the user-group-provider and then to the > access-policy-provider... > > <userGroupProvider> > <identifier>file-user-group-provider</identifier> > <class>org.apache.nifi.authorization.FileUserGroupProvider</class> > <property name="Users File">./conf/users.xml</property> > <property name="Legacy Authorized Users File"></property> > <property name="Initial User Identity 1">CN=bbende, OU=Apache > NiFI</property> > <property name="Initial User Identity 2">CN=nifi-host-1, > OU=NIFI</property> > <property name="Initial User Identity 2">CN=nifi-host-2, > OU=NIFI</property> > </userGroupProvider> > > <accessPolicyProvider> > <identifier>file-access-policy-provider</identifier> > <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class> > <property name="User Group > Provider">composite-configurable-user-group-provider</property> > <property name="Authorizations > File">./conf/authorizations.xml</property> > <property name="Initial Admin Identity">CN=bbende, OU=Apache > NiFI</property> > <property name="Legacy Authorized Users File"></property> > <property name="Node Identity 1">CN=nifi-host-1, OU=NIFI</property> > <property name="Node Identity 1">CN=nifi-host-2, OU=NIFI</property> > </accessPolicyProvider> > > > Also, whenever you change any config in the authorizers.xml related to the > file-based providers, then you will need to remove users.xml and > authorizations.xml On Mon, Oct 22, 2018 at 12:20 PM Saip, Alexander > (NIH/CC/BTRIS) [C] <alexander.s...@nih.gov> wrote: > > > > Hi Bryan, > > > > > > > > At this point, we don't want to run ZooKeeper on both nodes (as far as I > > understand, it prefers an odd number of members in the ensemble). Actually, > > the ZooKeeper running on one of them, sees both NiFi instances, but they > > don't talk to each other. When we try to make them do so by using a > > different authorizers.xml file, which is very much just a customized > > version of the “composite” example from the NiFi Admin Guide, then none of > > the nodes is able to start at all, throwing the error I mentioned in my > > previous post. > > > > > > > > Are you saying that we have to run ZooKeeper on both nodes? BTW, do we > > still need > > > > > > > > nifi.login.identity.provider.configuration.file=./conf/login-identity- > > providers.xml > > > > > > > > in the nifi.properties file when we use that new authorizers.xml? I’m > > asking since we have the same LDAP authentication/authorization settings in > > the latter. > > > > > > > > Thank you, > > > > > > > > Alexander > > > > > > > > -----Original Message----- > > From: Bryan Bende <bbe...@gmail.com> > > Sent: Monday, October 22, 2018 11:55 AM > > To: users@nifi.apache.org > > Subject: Re: NiFi fails on cluster nodes > > > > > > > > If you are getting separate clusters then each node is likely only using > > it's own ZooKeeper and therefore doesn't know about the other node. > > > > > > > > In nifi.properties the ZK connect string would need to be something like > > nifi-node1-hostname:2181,nifi-node2-hostname:2181 and in zoo.properties you > > would need entries for both ZooKeepers: > > > > > > > > server.1=nifi-node1-hostname:2888:3888 > > > > server.2=nifi-node2-hostname:2888:3888 > > > > On Mon, Oct 22, 2018 at 11:28 AM Saip, Alexander (NIH/CC/BTRIS) [C] > > <alexander.s...@nih.gov> wrote: > > > > > > > > > > I wonder if anyone has run into the same problem when trying to > > > > > configure composite authentication/authorization (LDAP and local > > > > > file)? When we use the “stand-alone” authorizers.xml file with the > > > > > addition of two extra properties > > > > > > > > > > > > > > > > > > > > <property name="Node Identity 1">… > > > > > > > > > > <property name="Node Identity 2">… > > > > > > > > > > > > > > > > > > > > and let ZooKeeper start on one on the nodes, we end up with two > > > > > one-node clusters, since apparently, the NiFi instances don’t talk > > > to > > > > > each other, but at least, they come alive… > > > > > > > > > > > > > > > > > > > > From: Saip, Alexander (NIH/CC/BTRIS) [C] <alexander.s...@nih.gov> > > > > > Sent: Friday, October 19, 2018 11:18 AM > > > > > To: users@nifi.apache.org > > > > > Subject: RE: NiFi fails on cluster nodes > > > > > > > > > > > > > > > > > > > > We have managed to get past that error by installing the CA cert in the > > > truststore. So, we can get a one-node cluster up and running. In order to > > > add another node, I edited the authorizers.xml file, basically, using the > > > “example composite implementation loading users and groups from LDAP and > > > a local file” from the Admin guide as a template. When I re-started the > > > node running ZooKeeper, though, it crashed with the following error > > > written into the nifi-app.log file: > > > > > > > > > > > > > > > > > > > > 2018-10-19 08:09:26,992 ERROR [main] o.s.web.context.ContextLoader > > > > > Context initialization failed > > > > > > > > > > org.springframework.beans.factory.UnsatisfiedDependencyException: > > > > > Error creating bean with name > > > > > 'org.springframework.security.config.annotation.web.configuration.We > > > bS > > > > > ecurityConfiguration': Unsatisfied dependency expressed through > > > method > > > > > 'setFilterChainProxySecurityConfigurer' parameter 1; nested > > > exception > > > > > is org.springframework.beans.factory.BeanExpressionException: > > > > > Expression parsing failed; nested exception is > > > > > org.springframework.beans.factory.UnsatisfiedDependencyException: > > > > > Error creating bean with name > > > > > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied > > > > > dependency expressed through method 'setJwtAuthenticationProvider' > > > > > parameter 0; nested exception is > > > > > org.springframework.beans.factory.BeanCreationException: Error > > > > > creating bean with name 'jwtAuthenticationProvider' defined in class > > > > > path resource [nifi-web-security-context.xml]: Cannot resolve > > > > > reference to bean 'authorizer' while setting constructor argument; > > > > > nested exception is > > > > > org.springframework.beans.factory.BeanCreationException: Error > > > > > creating bean with name 'authorizer': FactoryBean threw exception on > > > > > object creation; nested exception is java.lang.NullPointerException: > > > > > Name is null > > > > > > > > > > at > > > > > org.springframework.beans.factory.annotation.AutowiredAnnotationBean > > > Po > > > > > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPos > > > tP > > > > > rocessor.java:667) > > > > > > > > > > at > > > > > org.springframework.beans.factory.annotation.InjectionMetadata.injec > > > t( > > > > > InjectionMetadata.java:88) > > > > > > > > > > at > > > > > org.springframework.beans.factory.annotation.AutowiredAnnotationBean > > > Po > > > > > stProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostPro > > > ce > > > > > ssor.java:366) > > > > > > > > > > at > > > > > org.springframework.beans.factory.support.AbstractAutowireCapableBea > > > nF > > > > > actory.populateBean(AbstractAutowireCapableBeanFactory.java:1264) > > > > > > > > > > at > > > > > org.springframework.beans.factory.support.AbstractAutowireCapableBea > > > nF > > > > > actory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553) > > > > > > > > > > at > > > > > org.springframework.beans.factory.support.AbstractAutowireCapableBea > > > nF > > > > > actory.createBean(AbstractAutowireCapableBeanFactory.java:483) > > > > > > > > > > at > > > > > org.springframework.beans.factory.support.AbstractBeanFactory$1.getO > > > bj > > > > > ect(AbstractBeanFactory.java:306) > > > > > > > > > > at > > > > > org.springframework.beans.factory.support.DefaultSingletonBeanRegist > > > ry > > > > > .getSingleton(DefaultSingletonBeanRegistry.java:230) > > > > > > > > > > at > > > > > org.springframework.beans.factory.support.AbstractBeanFactory.doGetB > > > ea > > > > > n(AbstractBeanFactory.java:302) > > > > > > > > > > at > > > > > org.springframework.beans.factory.support.AbstractBeanFactory.getBea > > > n( > > > > > AbstractBeanFactory.java:197) > > > > > > > > > > at > > > > > org.springframework.beans.factory.support.DefaultListableBeanFactory > > > .p > > > > > reInstantiateSingletons(DefaultListableBeanFactory.java:761) > > > > > > > > > > at > > > > > org.springframework.context.support.AbstractApplicationContext.finis > > > hB > > > > > eanFactoryInitialization(AbstractApplicationContext.java:867) > > > > > > > > > > at > > > > > org.springframework.context.support.AbstractApplicationContext.refre > > > sh > > > > > (AbstractApplicationContext.java:543) > > > > > > > > > > at > > > > > org.springframework.web.context.ContextLoader.configureAndRefreshWeb > > > Ap > > > > > plicationContext(ContextLoader.java:443) > > > > > > > > > > at > > > > > org.springframework.web.context.ContextLoader.initWebApplicationCont > > > ex > > > > > t(ContextLoader.java:325) > > > > > > > > > > at > > > > > org.springframework.web.context.ContextLoaderListener.contextInitial > > > iz > > > > > ed(ContextLoaderListener.java:107) > > > > > > > > > > at > > > > > org.eclipse.jetty.server.handler.ContextHandler.callContextInitializ > > > ed > > > > > (ContextHandler.java:876) > > > > > > > > > > at > > > > > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitializ > > > ed > > > > > (ServletContextHandler.java:532) > > > > > > > > > > at > > > > > org.eclipse.jetty.server.handler.ContextHandler.startContext(Context > > > Ha > > > > > ndler.java:839) > > > > > > > > > > at > > > > > org.eclipse.jetty.servlet.ServletContextHandler.startContext(Servlet > > > Co > > > > > ntextHandler.java:344) > > > > > > > > > > at > > > > > org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java: > > > > > 1480) > > > > > > > > > > at > > > > > org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.ja > > > va > > > > > :1442) > > > > > > > > > > at > > > > > org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandl > > > er > > > > > .java:799) > > > > > > > > > > at > > > > > org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletConte > > > xt > > > > > Handler.java:261) > > > > > > > > > > at > > > > > org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:54 > > > 0) > > > > > > > > > > at > > > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLif > > > eC > > > > > ycle.java:68) > > > > > > > > > > at > > > > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerL > > > if > > > > > eCycle.java:131) > > > > > > > > > > at > > > > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Containe > > > rL > > > > > ifeCycle.java:113) > > > > > > > > > > at > > > > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHan > > > dl > > > > > er.java:113) > > > > > > > > > > at > > > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLif > > > eC > > > > > ycle.java:68) > > > > > > > > > > at > > > > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerL > > > if > > > > > eCycle.java:131) > > > > > > > > > > at > > > > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Containe > > > rL > > > > > ifeCycle.java:105) > > > > > > > > > > at > > > > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHan > > > dl > > > > > er.java:113) > > > > > > > > > > at > > > > > org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler. > > > > > java:290) > > > > > > > > > > at > > > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLif > > > eC > > > > > ycle.java:68) > > > > > > > > > > at > > > > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerL > > > if > > > > > eCycle.java:131) > > > > > > > > > > at > > > > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Containe > > > rL > > > > > ifeCycle.java:113) > > > > > > > > > > at > > > > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHan > > > dl > > > > > er.java:113) > > > > > > > > > > at > > > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLif > > > eC > > > > > ycle.java:68) > > > > > > > > > > at > > > > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerL > > > if > > > > > eCycle.java:131) > > > > > > > > > > at org.eclipse.jetty.server.Server.start(Server.java:452) > > > > > > > > > > at > > > > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Containe > > > rL > > > > > ifeCycle.java:105) > > > > > > > > > > at > > > > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHan > > > dl > > > > > er.java:113) > > > > > > > > > > at org.eclipse.jetty.server.Server.doStart(Server.java:419) > > > > > > > > > > at > > > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLif > > > eC > > > > > ycle.java:68) > > > > > > > > > > at > > > > > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:838) > > > > > > > > > > at org.apache.nifi.NiFi.<init>(NiFi.java:157) > > > > > > > > > > at org.apache.nifi.NiFi.<init>(NiFi.java:71) > > > > > > > > > > at org.apache.nifi.NiFi.main(NiFi.java:292) > > > > > > > > > > Caused by: org.springframework.beans.factory.BeanExpressionException: > > > > > Expression parsing failed; nested exception is > > > > > org.springframework.beans.factory.UnsatisfiedDependencyException: > > > > > Error creating bean with name > > > > > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied > > > > > dependency expressed through method 'setJwtAuthenticationProvider' > > > > > parameter 0; nested exception is > > > > > org.springframework.beans.factory.BeanCreationException: Error > > > > > creating bean with name 'jwtAuthenticationProvider' defined in class > > > > > path resource [nifi-web-security-context.xml]: Cannot resolve > > > > > reference to bean 'authorizer' while setting constructor argument; > > > > > nested exception is > > > > > org.springframework.beans.factory.BeanCreationException: Error > > > > > creating bean with name 'authorizer': FactoryBean threw exception on > > > > > object creation; nested exception is java.lang.NullPointerException: > > > > > Name is null > > > > > > > > > > at > > > > > org.springframework.context.expression.StandardBeanExpressionResolver. > > > > > evaluate(StandardBeanExpressionResolver.java:164) > > > > > > > > > > at > > > > > org.springframework.beans.factory.support.AbstractBeanFactory.evalua > > > te > > > > > BeanDefinitionString(AbstractBeanFactory.java:1448) > > > > > > > > > > at > > > > > org.springframework.beans.factory.support.DefaultListableBeanFactory > > > .d > > > > > oResolveDependency(DefaultListableBeanFactory.java:1088) > > > > > > > > > > at > > > > > org.springframework.beans.factory.support.DefaultListableBeanFactory > > > .r > > > > > esolveDependency(DefaultListableBeanFactory.java:1066) > > > > > > > > > > at > > > > > org.springframework.beans.factory.annotation.AutowiredAnnotationBean > > > Po > > > > > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPos > > > tP > > > > > rocessor.java:659) > > > > > > > > > > ... 48 common frames omitted > > > > > > > > > > Caused by: > > > > > org.springframework.beans.factory.UnsatisfiedDependencyException: > > > > > Error creating bean with name > > > > > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied > > > > > dependency expressed through method 'setJwtAuthenticationProvider' > > > > > parameter 0; nested exception is > > > > > org.springframework.beans.factory.BeanCreationException: Error > > > > > creating bean with name 'jwtAuthenticationProvider' defined in class > > > > > path resource [nifi-web-security-context.xml]: Cannot resolve > > > > > reference to bean 'authorizer' while setting constructor argument; > > > > > nested exception is > > > > > org.springframework.beans.factory.BeanCreationException: Error > > > > > creating bean with name 'authorizer': FactoryBean threw exception on > > > > > object creation; nested exception is java.lang.NullPointerException: > > > > > Name is null > > > > > > > > > > at > > > > > org.springframework.beans.factory.annotation.AutowiredAnnotationBean > > > Po > > > > > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPos > > > tP > > > > > rocessor.java:667) > > > > > > > > > > at > > > > > org.springframework.beans.factory.annotation.InjectionMetadata.injec > > > t( > > > > > InjectionMetadata.java:88) > > > > > > > > > > at > > > > > org.springframework.beans.factory.annotation.AutowiredAnnotationBean > > > Po > > > > > stProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostPro > > > ce > > > > > ssor.java:366) > > > > > > > > > > at > > > > > org.springframework.beans.factory.support.AbstractAutowireCapableBea > > > nF > > > > > actory.populateBean(AbstractAutowireCapableBeanFactory.java:1264) > > > > > > > > > > at > > > > > org.springframework.beans.factory.support.AbstractAutowireCapableBea > > > nF > > > > > actory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553) > > > > > > > > > > at > > > > > org.springframework.beans.factory.support.AbstractAutowireCapableBea > > > nF > > > > > actory.createBean(AbstractAutowireCapableBeanFactory.java:483) > > > > > > > > > > at > > > > > org.springframework.beans.factory.support.AbstractBeanFactory$1.getO > > > bj > > > > > ect(AbstractBeanFactory.java:306) > > > > > > > > > > at > > > > > org.springframework.beans.factory.support.DefaultSingletonBeanRegist > > > ry > > > > > .getSingleton(DefaultSingletonBeanRegistry.java:230) > > > > > > > > > > at > > > > > org.springframework.beans.factory.support.AbstractBeanFactory.doGetB > > > ea > > > > > n(AbstractBeanFactory.java:302) > > > > > > > > > > at > > > > > org.springframework.beans.factory.support.AbstractBeanFactory.getBea > > > n( > > > > > AbstractBeanFactory.java:202) > > > > > > > > > > at > > > > > org.springframework.beans.factory.support.DefaultListableBeanFactory > > > .g > > > > > etBeansOfType(DefaultListableBeanFactory.java:519) > > > > > > > > > > at > > > > > org.springframework.beans.factory.support.DefaultListableBeanFactory > > > .g > > > > > etBeansOfType(DefaultListableBeanFactory.java:508) > > > > > > > > > > at > > > > > org.springframework.security.config.annotation.web.configuration.Aut > > > ow > > > > > iredWebSecurityConfigurersIgnoreParents.getWebSecurityConfigurers(Au > > > to > > > > > wiredWebSecurityConfigurersIgnoreParents.java:53) > > > > > > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > > > Method) > > > > > > > > > > at > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl > > > .j > > > > > ava:62) > > > > > > > > > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcce > > > ss > > > > > orImpl.java:43) > > > > > > > > > > at java.lang.reflect.Method.invoke(Method.java:498) > > > > > > > > > > at > > > > > org.springframework.expression.spel.support.ReflectiveMethodExecutor > > > .e > > > > > xecute(ReflectiveMethodExecutor.java:113) > > > > > > > > > > at > > > > > org.springframework.expression.spel.ast.MethodReference.getValueInte > > > rn > > > > > al(MethodReference.java:129) > > > > > > > > > > at > > > > > org.springframework.expression.spel.ast.MethodReference.access$000(M > > > et > > > > > hodReference.java:49) > > > > > > > > > > at > > > > > org.springframework.expression.spel.ast.MethodReference$MethodValueR > > > ef > > > > > .getValue(MethodReference.java:347) > > > > > > > > > > at > > > > > org.springframework.expression.spel.ast.CompoundExpression.getValueI > > > nt > > > > > ernal(CompoundExpression.java:88) > > > > > > > > > > at > > > > > org.springframework.expression.spel.ast.SpelNodeImpl.getValue(SpelNo > > > de > > > > > Impl.java:120) > > > > > > > > > > at > > > > > org.springframework.expression.spel.standard.SpelExpression.getValue > > > (S > > > > > pelExpression.java:262) > > > > > > > > > > at > > > > > org.springframework.context.expression.StandardBeanExpressionResolver. > > > > > evaluate(StandardBeanExpressionResolver.java:161) > > > > > > > > > > ... 52 common frames omitted > > > > > > > > > > > > > > > > > > > > I tried to Google for possible clues, but so far, there hasn’t been > > > > > any luck… > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: Bryan Bende <bbe...@gmail.com> > > > > > Sent: Monday, October 15, 2018 10:27 AM > > > > > To: users@nifi.apache.org > > > > > Subject: Re: NiFi fails on cluster nodes > > > > > > > > > > > > > > > > > > > > I'm not really sure, the error message is indicating that either a > > > certificate was not sent during cluster communications, or possibly the > > > cert was not valid/trusted. > > > > > > > > > > > > > > > > > > > > In this case since it is only 1 node, it is the same node talking back to > > > itself, so the only parts involved here are the keystore and truststore > > > of that node, and the config in nifi.properties. > > > > > > > > > > > > > > > > > > > > Maybe your truststore is not setup correctly to trust certs signed by the > > > CA that created the server cert? > > > > > > > > > > On Mon, Oct 15, 2018 at 9:53 AM Saip, Alexander (NIH/CC/BTRIS) [C] > > > <alexander.s...@nih.gov> wrote: > > > > > > > > > > > > > > > > > > > > > > Yes, 'nifi.cluster.protocol.is.secure' is set to 'true', since > > > > otherwise, NiFi would require values for 'nifi.web.http.host' and > > > > 'nifi.web.http.port'. We have a cert that is used to serve HTTPS > > > > requests to the NiFi web UI, and it works just fine. > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > > From: Bryan Bende <bbe...@gmail.com> > > > > > > > > > > > Sent: Monday, October 15, 2018 9:43 AM > > > > > > > > > > > To: users@nifi.apache.org > > > > > > > > > > > Subject: Re: NiFi fails on cluster nodes > > > > > > > > > > > > > > > > > > > > > > This is not related to ZooKeeper... I think you are missing something > > > > related to TLS/SSL configuration, maybe you set cluster protocol to be > > > > secure, but then you didn't configure NiFi with a keystore/truststore? > > > > > > > > > > > > > > > > > > > > > > On Mon, Oct 15, 2018 at 9:41 AM Mike Thomsen <mikerthom...@gmail.com> > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > Not sure what's going on here, but NiFi does not require a cert to > > > > > setup ZooKeeper. > > > > > > > > > > > > > > > > > > > > > > > > Mike > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Oct 15, 2018 at 9:39 AM Saip, Alexander (NIH/CC/BTRIS) [C] > > > > > <alexander.s...@nih.gov> wrote: > > > > > > > > > > > >> > > > > > > > > > > > >> Hi Mike and Bryan, > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> I’ve installed and started ZooKeeper 3.4.13 and re-started a single > > > > >> NiFi node so far. Here is the error from the NiFi log: > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> 2018-10-15 09:19:48,371 ERROR [Process Cluster Protocol > > > > > > >> Request-1] > > > > > > > > > > > >> o.a.nifi.security.util.CertificateUtils The incoming request > > > > >> did > > > > > > > > > > > >> not contain client certificates and thus the DN cannot be extracted. > > > > > > > > > > > >> Check that the other endpoint is providing a complete client > > > > > > > > > > > >> certificate chain > > > > > > > > > > > >> > > > > > > > > > > > >> 2018-10-15 09:19:48,425 INFO [main] > > > > > > > > > > > >> o.a.nifi.controller.StandardFlowService Connecting Node: > > > > > > > > > > > >> 0.0.0.0:8008 > > > > > > > > > > > >> > > > > > > > > > > > >> 2018-10-15 09:19:48,452 ERROR [Process Cluster Protocol > > > > > > >> Request-2] > > > > > > > > > > > >> o.a.nifi.security.util.CertificateUtils The incoming request > > > > >> did > > > > > > > > > > > >> not contain client certificates and thus the DN cannot be extracted. > > > > > > > > > > > >> Check that the other endpoint is providing a complete client > > > > > > > > > > > >> certificate chain > > > > > > > > > > > >> > > > > > > > > > > > >> 2018-10-15 09:19:48,456 WARN [main] > > > > > > > > > > > >> o.a.nifi.controller.StandardFlowService Failed to connect to > > > > > > > > > > > >> cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: > > > > > > > > > > > >> Failed marshalling 'CONNECTION_REQUEST' protocol message due to: > > > > > > > > > > > >> javax.net.ssl.SSLHandshakeException: Received fatal alert: > > > > > > > > > > > >> bad_certificate > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> It is likely extraneous to NiFi, but does this mean that we need > > > > >> install a cert into ZooKeeper? Right now, both apps are running on > > > > >> the same box. > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> Thank you. > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> From: Mike Thomsen <mikerthom...@gmail.com> > > > > > > > > > > > >> Sent: Monday, October 15, 2018 9:02 AM > > > > > > > > > > > >> To: users@nifi.apache.org > > > > > > > > > > > >> Subject: Re: NiFi fails on cluster nodes > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> http://nifi.apache.org/docs/nifi-docs/html/administration-guide > > > > >> .h > > > > > > >> tm > > > > > > > > > > > >> l > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> See the properties that start with "nifi.zookeeper." > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> On Mon, Oct 15, 2018 at 8:58 AM Saip, Alexander (NIH/CC/BTRIS) [C] > > > > >> <alexander.s...@nih.gov> wrote: > > > > > > > > > > > >> > > > > > > > > > > > >> Mike, > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> I wonder if you could point me to instructions how to configure a > > > > >> cluster with an external instance of ZooKeeper? The NiFi Admin Guide > > > > >> talks exclusively about the embedded one. > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> Thanks again. > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> From: Mike Thomsen <mikerthom...@gmail.com> > > > > > > > > > > > >> Sent: Friday, October 12, 2018 10:17 AM > > > > > > > > > > > >> To: users@nifi.apache.org > > > > > > > > > > > >> Subject: Re: NiFi fails on cluster nodes > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> It very well could become a problem down the road. The reason > > > > >> ZooKeeper is usually on a dedicated machine is that you want it to > > > > >> be able to have enough resources to always communicate within a > > > > >> quorum to reconcile configuration changes and feed configuration > > > > >> details to clients. > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> That particular message is just a warning message. From what I can > > > > >> tell, it's just telling you that no cluster coordinator has been > > > > >> elected and it's going to try to do something about that. It's > > > > >> usually a problem with embedded ZooKeeper because each node by > > > > >> default points to the version of ZooKeeper it fires up. > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> For a development environment, a VM with 2GB of RAM and 1-2 CPU > > > > >> cores should be enough to run an external ZooKeeper. > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> On Fri, Oct 12, 2018 at 9:47 AM Saip, Alexander (NIH/CC/BTRIS) [C] > > > > >> <alexander.s...@nih.gov> wrote: > > > > > > > > > > > >> > > > > > > > > > > > >> Thanks Mike. We will get an external ZooKeeper instance deployed. I > > > > >> guess co-locating it with one of the NiFi nodes shouldn’t be an > > > > >> issue, or will it? We are chronically short of hardware. BTW, does > > > > >> the following message in the logs point to some sort of problem with > > > > >> the embedded ZooKeeper? > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> 2018-10-12 08:21:35,838 WARN [main] > > > > > > > > > > > >> o.a.nifi.controller.StandardFlowService There is currently no > > > > > > > > > > > >> Cluster Coordinator. This often happens upon restart of NiFi > > > > >> when > > > > > > > > > > > >> running an embedded ZooKeeper. Will register this node to > > > > >> become > > > > > > > > > > > >> the active Cluster Coordinator and will attempt to connect to > > > > > > > > > > > >> cluster again > > > > > > > > > > > >> > > > > > > > > > > > >> 2018-10-12 08:21:35,838 INFO [main] > > > > > > > > > > > >> o.a.n.c.l.e.CuratorLeaderElectionManager > > > > > > > > > > > >> CuratorLeaderElectionManager[stopped=false] Attempted to > > > > >> register > > > > > > > > > > > >> Leader Election for role 'Cluster Coordinator' but this role is > > > > > > > > > > > >> already registered > > > > > > > > > > > >> > > > > > > > > > > > >> 2018-10-12 08:21:42,090 INFO [Curator-Framework-0] > > > > > > > > > > > >> o.a.c.f.state.ConnectionStateManager State change: SUSPENDED > > > > > > > > > > > >> > > > > > > > > > > > >> 2018-10-12 08:21:42,092 INFO [Curator-ConnectionStateManager-0] > > > > > > > > > > > >> o.a.n.c.l.e.CuratorLeaderElectionManager > > > > > > > > > > > >> org.apache.nifi.controller.leader.election.CuratorLeaderElectio > > > > >> nM > > > > > > >> an > > > > > > > > > > > >> ag er$ElectionListener@17900f5b Connection State changed to > > > > > > > > > > > >> SUSPENDED > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> From: Mike Thomsen <mikerthom...@gmail.com> > > > > > > > > > > > >> Sent: Friday, October 12, 2018 8:33 AM > > > > > > > > > > > >> To: users@nifi.apache.org > > > > > > > > > > > >> Subject: Re: NiFi fails on cluster nodes > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> Also, in a production environment NiFi should have its own dedicated > > > > >> ZooKeeper cluster to be on the safe side. You should not reuse > > > > >> ZooKeeper quora (ex. have HBase and NiFi point to the same quorum). > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> On Fri, Oct 12, 2018 at 8:29 AM Mike Thomsen > > > > >> <mikerthom...@gmail.com> wrote: > > > > > > > > > > > >> > > > > > > > > > > > >> Alexander, > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> I am pretty sure your problem is here: > > > > > > > > > > > >> nifi.state.management.embedded.zookeeper.start=true > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> That spins up an embedded ZooKeeper, which is generally intended to > > > > >> be used for local development. For example, HBase provides the same > > > > >> feature, but it is intended to allow you to test a real HBase client > > > > >> application against a single node of HBase running locally. > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> What you need to try is these steps: > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> 1. Set up an external ZooKeeper instance (or set up 3 in a > > > > > > >> quorum; > > > > > > > > > > > >> must be odd numbers) > > > > > > > > > > > >> > > > > > > > > > > > >> 2. Update nifi.properties on each node to use the external ZooKeeper > > > > >> setup. > > > > > > > > > > > >> > > > > > > > > > > > >> 3. Restart all of them. > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> See if that works. > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> Mike > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> On Fri, Oct 12, 2018 at 8:13 AM Saip, Alexander (NIH/CC/BTRIS) [C] > > > > >> <alexander.s...@nih.gov> wrote: > > > > > > > > > > > >> > > > > > > > > > > > >> nifi.cluster.node.protocol.port=11443 by default on all nodes, I > > > > >> haven’t touched that property. Yesterday, we discovered some issues > > > > >> preventing two of the boxes from communicating. Now, they can talk > > > > >> okay. Ports 11443, 2181 and 3888 are explicitly open in iptables, > > > > >> but clustering still doesn’t happen. The log files are filled up > > > > >> with errors like this: > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> 2018-10-12 07:59:08,494 ERROR [Curator-Framework-0] > > > > > > > > > > > >> o.a.c.f.imps.CuratorFrameworkImpl Background operation retry > > > > >> gave > > > > > > > > > > > >> up > > > > > > > > > > > >> > > > > > > > > > > > >> org.apache.zookeeper.KeeperException$ConnectionLossException: > > > > > > > > > > > >> KeeperErrorCode = ConnectionLoss > > > > > > > > > > > >> > > > > > > > > > > > >> at > > > > > > > > > > > >> org.apache.zookeeper.KeeperException.create(KeeperException.java: > > > > > > >> 99 > > > > > > > > > > > >> ) > > > > > > > > > > > >> > > > > > > > > > > > >> at > > > > > > > > > > > >> org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBac > > > > >> kg > > > > > > >> ro > > > > > > > > > > > >> un > > > > > > > > > > > >> dRetry(CuratorFrameworkImpl.java:728) > > > > > > > > > > > >> > > > > > > > > > > > >> at > > > > > > > > > > > >> org.apache.curator.framework.imps.CuratorFrameworkImpl.performB > > > > >> ac > > > > > > >> kg > > > > > > > > > > > >> ro > > > > > > > > > > > >> undOperation(CuratorFrameworkImpl.java:857) > > > > > > > > > > > >> > > > > > > > > > > > >> at > > > > > > > > > > > >> org.apache.curator.framework.imps.CuratorFrameworkImpl.backgrou > > > > >> nd > > > > > > >> Op > > > > > > > > > > > >> er > > > > > > > > > > > >> ationsLoop(CuratorFrameworkImpl.java:809) > > > > > > > > > > > >> > > > > > > > > > > > >> at > > > > > > > > > > > >> org.apache.curator.framework.imps.CuratorFrameworkImpl.access$3 > > > > >> 00 > > > > > > >> (C > > > > > > > > > > > >> ur > > > > > > > > > > > >> atorFrameworkImpl.java:64) > > > > > > > > > > > >> > > > > > > > > > > > >> at > > > > > > > > > > > >> org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(C > > > > >> ur > > > > > > >> at > > > > > > > > > > > >> or > > > > > > > > > > > >> FrameworkImpl.java:267) > > > > > > > > > > > >> > > > > > > > > > > > >> at > > > > > > >> java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > > > > > > > > > >> > > > > > > > > > > > >> at > > > > > > > > > > > >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask. > > > > > > > > > > > >> access$201(ScheduledThreadPoolExecutor.java:180) > > > > > > > > > > > >> > > > > > > > > > > > >> at > > > > > > > > > > > >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask. > > > > > > > > > > > >> run(ScheduledThreadPoolExecutor.java:293) > > > > > > > > > > > >> > > > > > > > > > > > >> at > > > > > > > > > > > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. > > > > > > > > > > > >> java:1149) > > > > > > > > > > > >> > > > > > > > > > > > >> at > > > > > > > > > > > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolEx > > > > >> ec > > > > > > >> ut > > > > > > > > > > > >> or > > > > > > > > > > > >> .java:624) > > > > > > > > > > > >> > > > > > > > > > > > >> at java.lang.Thread.run(Thread.java:748) > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> Is there anything else we should check? > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> From: Nathan Gough <thena...@gmail.com> > > > > > > > > > > > >> Sent: Thursday, October 11, 2018 9:12 AM > > > > > > > > > > > >> To: users@nifi.apache.org > > > > > > > > > > > >> Subject: Re: NiFi fails on cluster nodes > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> You may also need to explicitly open > > > > >> ‘nifi.cluster.node.protocol.port’ on all nodes to allow cluster > > > > >> communication for cluster heartbeats etc. > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> From: ashmeet kandhari <ashmeetkandhar...@gmail.com> > > > > > > > > > > > >> Reply-To: <users@nifi.apache.org> > > > > > > > > > > > >> Date: Thursday, October 11, 2018 at 9:09 AM > > > > > > > > > > > >> To: <users@nifi.apache.org> > > > > > > > > > > > >> Subject: Re: NiFi fails on cluster nodes > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> Hi Alexander, > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> Can you verify by pinging if the 3 nodes (tcp ping) or run nifi in > > > > >> standalone mode and see if you can ping them from other 2 servers > > > > >> just to be sure if they can communicate with one another. > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> On Thu, Oct 11, 2018 at 11:49 AM Saip, Alexander (NIH/CC/BTRIS) [C] > > > > >> <alexander.s...@nih.gov> wrote: > > > > > > > > > > > >> > > > > > > > > > > > >> How do I do that? The nifi.properties file on each node includes > > > > >> ‘nifi.state.management.embedded.zookeeper.start=true’, so I assume > > > > >> Zookeeper does start. > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> From: ashmeet kandhari <ashmeetkandhar...@gmail.com> > > > > > > > > > > > >> Sent: Thursday, October 11, 2018 4:36 AM > > > > > > > > > > > >> To: users@nifi.apache.org > > > > > > > > > > > >> Subject: Re: NiFi fails on cluster nodes > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> Can you see if zookeeper node is up and running and can connect > > > > > > >> to > > > > > > > > > > > >> the nifi nodes > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> On Wed, Oct 10, 2018 at 7:34 PM Saip, Alexander (NIH/CC/BTRIS) [C] > > > > >> <alexander.s...@nih.gov> wrote: > > > > > > > > > > > >> > > > > > > > > > > > >> Hello, > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> We have three NiFi 1.7.1 nodes originally configured as independent > > > > >> instances, each on its own server. There is no firewall between > > > > >> them. When I tried to build a cluster following instructions here, > > > > >> NiFi failed to start on all of them, despite the fact that I even > > > > >> set nifi.cluster.protocol.is.secure=false in the nifi.properties > > > > >> file on each node. Here is the error in the log files: > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> 2018-10-10 13:57:07,506 INFO [main] org.apache.nifi.NiFi Launching > > > > >> NiFi... > > > > > > > > > > > >> > > > > > > > > > > > >> 2018-10-10 13:57:07,745 INFO [main] > > > > >> o.a.nifi.properties.NiFiPropertiesLoader Determined default > > > > >> nifi.properties path to be '/opt/nifi-1.7.1/./conf/nifi.properties' > > > > > > > > > > > >> > > > > > > > > > > > >> 2018-10-10 13:57:07,748 INFO [main] > > > > > > > > > > > >> o.a.nifi.properties.NiFiPropertiesLoader Loaded 125 properties > > > > > > >> from > > > > > > > > > > > >> /opt/nifi-1.7.1/./conf/nifi.properties > > > > > > > > > > > >> > > > > > > > > > > > >> 2018-10-10 13:57:07,755 INFO [main] org.apache.nifi.NiFi Loaded > > > > > > >> 125 > > > > > > > > > > > >> properties > > > > > > > > > > > >> > > > > > > > > > > > >> 2018-10-10 13:57:07,762 INFO [main] > > > > > > > > > > > >> org.apache.nifi.BootstrapListener Started Bootstrap Listener, > > > > > > > > > > > >> Listening for incoming requests on port > > > > > > > > > > > >> 43744 > > > > > > > > > > > >> > > > > > > > > > > > >> 2018-10-10 13:59:15,056 ERROR [main] org.apache.nifi.NiFi > > > > >> Failure > > > > > > > > > > > >> to launch NiFi due to java.net.ConnectException: Connection > > > > >> timed > > > > > > > > > > > >> out (Connection timed out) > > > > > > > > > > > >> > > > > > > > > > > > >> java.net.ConnectException: Connection timed out (Connection > > > > >> timed > > > > > > > > > > > >> out) > > > > > > > > > > > >> > > > > > > > > > > > >> at java.net.PlainSocketImpl.socketConnect(Native > > > > >> Method) > > > > > > > > > > > >> > > > > > > > > > > > >> at > > > > > > > > > > > >> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl. > > > > > > > > > > > >> ja > > > > > > > > > > > >> va:350) > > > > > > > > > > > >> > > > > > > > > > > > >> at > > > > > > > > > > > >> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlain > > > > >> So > > > > > > >> ck > > > > > > > > > > > >> et > > > > > > > > > > > >> Impl.java:206) > > > > > > > > > > > >> > > > > > > > > > > > >> at > > > > > > > > > > > >> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl. > > > > > > >> ja > > > > > > > > > > > >> va > > > > > > > > > > > >> :188) > > > > > > > > > > > >> > > > > > > > > > > > >> at > > > > > > > > > > > >> java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > > > > > > > > > > > >> > > > > > > > > > > > >> at java.net.Socket.connect(Socket.java:589) > > > > > > > > > > > >> > > > > > > > > > > > >> at java.net.Socket.connect(Socket.java:538) > > > > > > > > > > > >> > > > > > > > > > > > >> at > > > > > > > > > > > >> org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java: > > > > > > > > > > > >> 100) > > > > > > > > > > > >> > > > > > > > > > > > >> at > > > > > > > > > > > >> org.apache.nifi.BootstrapListener.start(BootstrapListener.java: > > > > >> 83 > > > > > > >> ) > > > > > > > > > > > >> > > > > > > > > > > > >> at org.apache.nifi.NiFi.<init>(NiFi.java:102) > > > > > > > > > > > >> > > > > > > > > > > > >> at org.apache.nifi.NiFi.<init>(NiFi.java:71) > > > > > > > > > > > >> > > > > > > > > > > > >> at org.apache.nifi.NiFi.main(NiFi.java:292) > > > > > > > > > > > >> > > > > > > > > > > > >> 2018-10-10 13:59:15,058 INFO [Thread-1] org.apache.nifi.NiFi > > > > >> Initiating shutdown of Jetty web server... > > > > > > > > > > > >> > > > > > > > > > > > >> 2018-10-10 13:59:15,059 INFO [Thread-1] org.apache.nifi.NiFi Jetty > > > > >> web server shutdown completed (nicely or otherwise). > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> Without clustering, the instances had no problem starting. Since > > > > >> this is our first experiment building a cluster, I’m not sure where > > > > >> to look for clues. > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> Thanks in advance, > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> Alexander >
