No worries, thanks for following up and letting us know! Thanks -Mark
On Aug 5, 2020, at 3:42 PM, dan young <[email protected]<mailto:[email protected]>> wrote: Hello, Sorry for all the noise...dooh....was due to the realm in the jaas.conf being lowercase...i'm a knucklehead... Dano On Wed, Aug 5, 2020 at 1:12 PM Bryan Bende <[email protected]<mailto:[email protected]>> wrote: I don't see how this would relate to the problem, but shouldn't the ACL be set to "Creator" when Sasl/Kerberos is setup correctly? In addition to the nifi configs you showed, you would also need a jaas conf file specified in bootstrap.conf and in that file you would need the jaas entry for the ZK client. On Wed, Aug 5, 2020 at 3:02 PM dan young <[email protected]<mailto:[email protected]>> wrote: Hello Mark, Attached is a dump from one of the nodes....I replaced the domain related entries with X/x. I'm not sure if it's relevant or not, but I did notice that in the log there's entries "Looking for keys for [email protected]<mailto:[email protected]>" the x (domain) is lowercase whereas in the keytab file it's uppercase X. Also not sure if the Found unsupported keytype (1) is meaningful. Not that when I delete the znode in zookeeper=, at least the initial znode is created /nifi, but we never see the other typical suspect, i.e Coordinator, Primary, etc... Seems to be something stuck in Curator??? Regards. Dano On Wed, Aug 5, 2020 at 12:20 PM Mark Payne <[email protected]<mailto:[email protected]>> wrote: Dan, Can you grab a thread dump and provide that? Specifically, the “main” thread is the important one with startup. The note that the role is already registered is normal. It probably could be changed to a DEBUG level, really. It should not be concerning. A thread dump, though, would show us exactly where it’s at. Thanks -Mark On Aug 5, 2020, at 2:02 PM, dan young <[email protected]<mailto:[email protected]>> wrote: Hello, Running nifi 1.11.4, 3 X secure cluster mode and have enabled kerberos/sasl, upon trying to startup the cluster, they seem to get stuck in : 2020-08-05 17:10:18,907 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again 2020-08-05 17:10:18,907 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered I've checked zookeeper and I can see that the /nifi znode has been created, although empty, and the ACL seem to look correct zk: nifi1-5.X.net:2181<http://nifi1-5.x.net:2181/>(CONNECTED) 3] getAcl /nifi 'sasl,'[email protected]<mailto:[email protected]> : cdrwa 'world,'anyone : r relevant Nifi config settings nifi.properties: nifi.zookeeper.auth.type=sasl nifi.zookeeper.kerberos.removeHostFromPrincipal=true nifi.zookeeper.kerberos.removeRealmFromPrincipal=false # kerberos # nifi.kerberos.krb5.file=/etc/krb5.conf # kerberos service principal # [email protected]<mailto:[email protected]> nifi.kerberos.service.keytab.location=/opt/nifi/conf/nifi.keytab state-management.xml <cluster-provider> <id>zk-provider</id> <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class> <property name="Root Node">/nifi</property> <property name="Session Timeout">30 seconds</property> <property name="Access Control">CreatorOnly</property> <property name="Connect String">X:2181,Y:2181,Z:2181</property> </cluster-provider> KRB5_TRACE=/dev/stdout kinit -k -t /opt/nifi/conf/nifi.keytab [email protected]<mailto:[email protected]> ... ... klist Ticket cache: FILE:/tmp/krb5cc_2004 Default principal: [email protected]<mailto:[email protected]> Valid starting Expires Service principal 08/05/2020 17:57:02 08/06/2020 03:57:02 krbtgt/[email protected]<mailto:[email protected]> renew until 08/06/2020 17:57:02 As a side note, secure NiFi was working fine before the kerberos bit, I've been beating my head against the wall with it for the day, but the kerberos/zookeeper stuff seems to be working now.... do we need to have Server-Server zookeeper auth working for this? Appreciate any insight.... Regards, Dano
