Dan, Thanks for reporting this. Case-sensitivity in these kinds of things is important but it also seems like low-hanging fruit for us to at least detect & alert on when errors occur. “Failed to connect to external service X with principal [email protected] <mailto:[email protected]> …. Did you mean [email protected] <mailto:[email protected]>?” Or even potentially trying to do case-conversion internally as a fallback.
Andy LoPresto [email protected] [email protected] He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Aug 5, 2020, at 2:50 PM, dan young <[email protected]> wrote: > > On a related note, I noticed that the ACL are getting set, but also for each > znode under the /nifi, the Read ACL for world is being set. Is there a way > to have nifi only set with the sasl? > > zk: nifi1-5.X.net:2181(CONNECTED) 12] getAcl /nifi > 'sasl,'[email protected] <mailto:[email protected]> > : cdrwa > 'world,'anyone > : r > > On Wed, Aug 5, 2020 at 1:56 PM Mark Payne <[email protected] > <mailto:[email protected]>> wrote: > No worries, thanks for following up and letting us know! > > Thanks > -Mark > > >> On Aug 5, 2020, at 3:42 PM, dan young <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hello, >> >> Sorry for all the noise...dooh....was due to the realm in the jaas.conf >> being lowercase...i'm a knucklehead... >> >> Dano >> >> On Wed, Aug 5, 2020 at 1:12 PM Bryan Bende <[email protected] >> <mailto:[email protected]>> wrote: >> I don't see how this would relate to the problem, but shouldn't the ACL be >> set to "Creator" when Sasl/Kerberos is setup correctly? >> >> In addition to the nifi configs you showed, you would also need a jaas conf >> file specified in bootstrap.conf and in that file you would need the jaas >> entry for the ZK client. >> >> On Wed, Aug 5, 2020 at 3:02 PM dan young <[email protected] >> <mailto:[email protected]>> wrote: >> Hello Mark, >> >> Attached is a dump from one of the nodes....I replaced the domain related >> entries with X/x. I'm not sure if it's relevant or not, but I did notice >> that in the log there's entries "Looking for keys for [email protected] >> <mailto:[email protected]>" the x (domain) is lowercase whereas in the keytab >> file it's uppercase X. Also not sure if the Found unsupported keytype (1) >> is meaningful. Not that when I delete the znode in zookeeper=, at least the >> initial znode is created /nifi, but we never see the other typical suspect, >> i.e Coordinator, Primary, etc... >> >> Seems to be something stuck in Curator??? >> >> Regards. >> >> Dano >> >> On Wed, Aug 5, 2020 at 12:20 PM Mark Payne <[email protected] >> <mailto:[email protected]>> wrote: >> Dan, >> >> Can you grab a thread dump and provide that? Specifically, the “main” thread >> is the important one with startup. The note that the role is already >> registered is normal. It probably could be changed to a DEBUG level, really. >> It should not be concerning. A thread dump, though, would show us exactly >> where it’s at. >> >> Thanks >> -Mark >> >> >>> On Aug 5, 2020, at 2:02 PM, dan young <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Hello, >>> Running nifi 1.11.4, 3 X secure cluster mode and have enabled >>> kerberos/sasl, upon trying to startup the cluster, they seem to get stuck >>> in : >>> >>> 2020-08-05 17:10:18,907 WARN [main] o.a.nifi.controller.StandardFlowService >>> There is currently no Cluster Coordinator. This often happens upon restart >>> of NiFi >>> when running an embedded ZooKeeper. Will register this node to become the >>> active Cluster Coordinator and will attempt to connect to cluster again >>> 2020-08-05 17:10:18,907 INFO [main] >>> o.a.n.c.l.e.CuratorLeaderElectionManager >>> CuratorLeaderElectionManager[stopped=false] Attempted to register Leader >>> Election >>> for role 'Cluster Coordinator' but this role is already registered >>> >>> >>> >>> I've checked zookeeper and I can see that the /nifi znode has been created, >>> although empty, and the ACL seem to look correct >>> zk: nifi1-5.X.net:2181 <http://nifi1-5.x.net:2181/>(CONNECTED) 3] getAcl >>> /nifi >>> 'sasl,'[email protected] <mailto:[email protected]> >>> : cdrwa >>> 'world,'anyone >>> : r >>> >>> >>> relevant Nifi config settings >>> >>> nifi.properties: >>> >>> nifi.zookeeper.auth.type=sasl >>> nifi.zookeeper.kerberos.removeHostFromPrincipal=true >>> nifi.zookeeper.kerberos.removeRealmFromPrincipal=false >>> >>> # kerberos # >>> nifi.kerberos.krb5.file=/etc/krb5.conf >>> >>> # kerberos service principal # >>> [email protected] <mailto:[email protected]> >>> nifi.kerberos.service.keytab.location=/opt/nifi/conf/nifi.keytab >>> >>> >>> state-management.xml >>> <cluster-provider> >>> <id>zk-provider</id> >>> >>> <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class> >>> <property name="Root Node">/nifi</property> >>> <property name="Session Timeout">30 seconds</property> >>> <property name="Access Control">CreatorOnly</property> >>> <property name="Connect String">X:2181,Y:2181,Z:2181</property> >>> </cluster-provider> >>> >>> >>> >>> KRB5_TRACE=/dev/stdout kinit -k -t /opt/nifi/conf/nifi.keytab [email protected] >>> <mailto:[email protected]> >>> ... >>> ... >>> >>> klist >>> Ticket cache: FILE:/tmp/krb5cc_2004 >>> Default principal: [email protected] <mailto:[email protected]> >>> >>> Valid starting Expires Service principal >>> 08/05/2020 17:57:02 08/06/2020 03:57:02 krbtgt/[email protected] >>> <mailto:[email protected]> >>> renew until 08/06/2020 17:57:02 >>> >>> >>> >>> >>> As a side note, secure NiFi was working fine before the kerberos bit, I've >>> been beating my head against the wall with it for the day, but the >>> kerberos/zookeeper stuff seems to be working now.... >>> do we need to have Server-Server zookeeper auth working for this? >>> >>> >>> Appreciate any insight.... >>> >>> Regards, >>> >>> Dano >> >
