Hello Mark, Attached is a dump from one of the nodes....I replaced the domain related entries with X/x. I'm not sure if it's relevant or not, but I did notice that in the log there's entries "Looking for keys for [email protected]" the x (domain) is lowercase whereas in the keytab file it's uppercase X. Also not sure if the Found unsupported keytype (1) is meaningful. Not that when I delete the znode in zookeeper=, at least the initial znode is created /nifi, but we never see the other typical suspect, i.e Coordinator, Primary, etc...
Seems to be something stuck in Curator??? Regards. Dano On Wed, Aug 5, 2020 at 12:20 PM Mark Payne <[email protected]> wrote: > Dan, > > Can you grab a thread dump and provide that? Specifically, the “main” > thread is the important one with startup. The note that the role is already > registered is normal. It probably could be changed to a DEBUG level, > really. It should not be concerning. A thread dump, though, would show us > exactly where it’s at. > > Thanks > -Mark > > > On Aug 5, 2020, at 2:02 PM, dan young <[email protected]> wrote: > > Hello, > Running nifi 1.11.4, 3 X secure cluster mode and have enabled > kerberos/sasl, upon trying to startup the cluster, they seem to get stuck > in : > > 2020-08-05 17:10:18,907 WARN [main] > o.a.nifi.controller.StandardFlowService There is currently no Cluster > Coordinator. This often happens upon restart of NiFi > when running an embedded ZooKeeper. Will register this node to become the > active Cluster Coordinator and will attempt to connect to cluster again > 2020-08-05 17:10:18,907 INFO [main] > o.a.n.c.l.e.CuratorLeaderElectionManager > CuratorLeaderElectionManager[stopped=false] Attempted to register Leader > Election > for role 'Cluster Coordinator' but this role is already registered > > > > I've checked zookeeper and I can see that the /nifi znode has been > created, although empty, and the ACL seem to look correct > zk: nifi1-5.X.net:2181 <http://nifi1-5.x.net:2181>(CONNECTED) 3] getAcl > /nifi > 'sasl,'[email protected] > : cdrwa > 'world,'anyone > : r > > > relevant Nifi config settings > > nifi.properties: > > nifi.zookeeper.auth.type=sasl > nifi.zookeeper.kerberos.removeHostFromPrincipal=true > nifi.zookeeper.kerberos.removeRealmFromPrincipal=false > > # kerberos # > nifi.kerberos.krb5.file=/etc/krb5.conf > > # kerberos service principal # > [email protected] > nifi.kerberos.service.keytab.location=/opt/nifi/conf/nifi.keytab > > > state-management.xml > <cluster-provider> > <id>zk-provider</id> > > <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class> > <property name="Root Node">/nifi</property> > <property name="Session Timeout">30 seconds</property> > <property name="Access Control">CreatorOnly</property> > <property name="Connect String">X:2181,Y:2181,Z:2181</property> > </cluster-provider> > > > > KRB5_TRACE=/dev/stdout kinit -k -t /opt/nifi/conf/nifi.keytab [email protected] > ... > ... > > klist > Ticket cache: FILE:/tmp/krb5cc_2004 > Default principal: [email protected] > > Valid starting Expires Service principal > 08/05/2020 17:57:02 08/06/2020 03:57:02 krbtgt/[email protected] > renew until 08/06/2020 17:57:02 > > > > > As a side note, secure NiFi was working fine before the kerberos bit, I've > been beating my head against the wall with it for the day, but the > kerberos/zookeeper stuff seems to be working now.... > do we need to have Server-Server zookeeper auth working for this? > > > Appreciate any insight.... > > Regards, > > Dano > > >
bootstrap-dump.log.gz
Description: GNU Zip compressed data
