Re: cluster stick in "Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered"

[Andy LoPresto]

Dan, 

Thanks for reporting this. Case-sensitivity in these kinds of things is 
important but it also seems like low-hanging fruit for us to at least detect & 
alert on when errors occur. “Failed to connect to external service X with 
principal n...@x.net  …. Did you mean n...@x.net 
?” Or even potentially trying to do case-conversion 
internally as a fallback. 

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Aug 5, 2020, at 2:50 PM, dan young  wrote:
> 
> On a related note, I noticed that the ACL are getting set, but also for each 
> znode under the /nifi, the Read ACL for world is being set.  Is there a way 
> to have nifi only set with the sasl?
> 
> zk: nifi1-5.X.net:2181(CONNECTED) 12] getAcl /nifi
> 'sasl,'n...@x.net 
> : cdrwa
> 'world,'anyone
> : r
> 
> On Wed, Aug 5, 2020 at 1:56 PM Mark Payne  > wrote:
> No worries, thanks for following up and letting us know!
> 
> Thanks
> -Mark
> 
> 
>> On Aug 5, 2020, at 3:42 PM, dan young > > wrote:
>> 
>> Hello,
>> 
>> Sorry for all the noise...doohwas due to the realm in the jaas.conf 
>> being lowercase...i'm a knucklehead...
>> 
>> Dano
>> 
>> On Wed, Aug 5, 2020 at 1:12 PM Bryan Bende > > wrote:
>> I don't see how this would relate to the problem, but shouldn't the ACL be 
>> set to "Creator" when Sasl/Kerberos is setup correctly?
>> 
>> In addition to the nifi configs you showed, you would also need a jaas conf 
>> file specified in bootstrap.conf and in that file you would need the jaas 
>> entry for the ZK client.
>> 
>> On Wed, Aug 5, 2020 at 3:02 PM dan young > > wrote:
>> Hello Mark,
>> 
>> Attached is a dump from one of the nodesI replaced the domain related 
>> entries with X/x.  I'm not sure if it's relevant or not, but I did notice 
>> that in the log there's entries "Looking for keys for n...@x.net 
>> "  the x (domain)  is lowercase whereas in the keytab 
>> file it's uppercase X.  Also not sure if the Found unsupported keytype (1) 
>> is meaningful.  Not that when I delete the znode in zookeeper=, at least the 
>> initial znode is created /nifi, but we never see the other typical suspect, 
>> i.e Coordinator, Primary, etc...
>> 
>> Seems to be something stuck in Curator???
>> 
>> Regards.
>> 
>> Dano
>> 
>> On Wed, Aug 5, 2020 at 12:20 PM Mark Payne > > wrote:
>> Dan,
>> 
>> Can you grab a thread dump and provide that? Specifically, the “main” thread 
>> is the important one with startup. The note that the role is already 
>> registered is normal. It probably could be changed to a DEBUG level, really. 
>> It should not be concerning. A thread dump, though, would show us exactly 
>> where it’s at.
>> 
>> Thanks
>> -Mark
>> 
>> 
>>> On Aug 5, 2020, at 2:02 PM, dan young >> > wrote:
>>> 
>>> Hello,
>>> Running nifi 1.11.4, 3 X secure cluster mode and have enabled 
>>> kerberos/sasl, upon trying to startup the cluster, they seem to get stuck 
>>> in :
>>> 
>>> 2020-08-05 17:10:18,907 WARN [main] o.a.nifi.controller.StandardFlowService 
>>> There is currently no Cluster Coordinator. This often happens upon restart 
>>> of NiFi
>>>  when running an embedded ZooKeeper. Will register this node to become the 
>>> active Cluster Coordinator and will attempt to connect to cluster again
>>> 2020-08-05 17:10:18,907 INFO [main] 
>>> o.a.n.c.l.e.CuratorLeaderElectionManager 
>>> CuratorLeaderElectionManager[stopped=false] Attempted to register Leader 
>>> Election
>>>  for role 'Cluster Coordinator' but this role is already registered
>>> 
>>> 
>>> 
>>> I've checked zookeeper and I can see that the /nifi znode has been created, 
>>> although empty, and the ACL seem to look correct
>>> zk: nifi1-5.X.net:2181 (CONNECTED) 3] getAcl 
>>> /nifi
>>> 'sasl,'n...@x.net 
>>> : cdrwa
>>> 'world,'anyone
>>> : r
>>> 
>>> 
>>> relevant Nifi config settings
>>> 
>>> nifi.properties:
>>> 
>>> nifi.zookeeper.auth.type=sasl
>>> nifi.zookeeper.kerberos.removeHostFromPrincipal=true
>>> nifi.zookeeper.kerberos.removeRealmFromPrincipal=false
>>> 
>>> # kerberos #
>>> nifi.kerberos.krb5.file=/etc/krb5.conf
>>> 
>>> # kerberos service principal #
>>> nifi.kerberos.service.principal=n...@x.net 
>>> nifi.kerberos.service.keytab.location=/opt/nifi/conf/nifi.keytab
>>> 
>>> 
>>> state-management.xml
>>> 
>>> zk-provider
>>> 
>>> org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider
>>> /nifi
>>> 30 seconds
>>> CreatorOnly
>>> X:2181,Y:2181,Z:2181
>>>  
>>> 
>>> 
>>> 
>>> KRB5_TRACE=/dev/stdout kinit -k -t /opt/nifi/conf/nifi.keytab n...@x.net 
>>> 
>>> ...
>>> ...
>>> 
>>> klist
>>> Ticket cache: 

Re: cluster stick in "Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered"

[dan young]

On a related note, I noticed that the ACL are getting set, but also for
each znode under the /nifi, the Read ACL for world is being set.  Is there
a way to have nifi only set with the sasl?

zk: nifi1-5.X.net:2181(CONNECTED) 12] getAcl /nifi
'sasl,'n...@x.net
: cdrwa
'world,'anyone
: r

On Wed, Aug 5, 2020 at 1:56 PM Mark Payne  wrote:

> No worries, thanks for following up and letting us know!
>
> Thanks
> -Mark
>
>
> On Aug 5, 2020, at 3:42 PM, dan young  wrote:
>
> Hello,
>
> Sorry for all the noise...doohwas due to the realm in the
> jaas.conf being lowercase...i'm a knucklehead...
>
> Dano
>
> On Wed, Aug 5, 2020 at 1:12 PM Bryan Bende  wrote:
>
>> I don't see how this would relate to the problem, but shouldn't the ACL
>> be set to "Creator" when Sasl/Kerberos is setup correctly?
>>
>> In addition to the nifi configs you showed, you would also need a jaas
>> conf file specified in bootstrap.conf and in that file you would need the
>> jaas entry for the ZK client.
>>
>> On Wed, Aug 5, 2020 at 3:02 PM dan young  wrote:
>>
>>> Hello Mark,
>>>
>>> Attached is a dump from one of the nodesI replaced the domain
>>> related entries with X/x.  I'm not sure if it's relevant or not, but I did
>>> notice that in the log there's entries "Looking for keys for n...@x.net"
>>> the x (domain)  is lowercase whereas in the keytab file it's uppercase X.
>>> Also not sure if the Found unsupported keytype (1) is meaningful.  Not that
>>> when I delete the znode in zookeeper=, at least the initial znode is
>>> created /nifi, but we never see the other typical suspect, i.e Coordinator,
>>> Primary, etc...
>>>
>>> Seems to be something stuck in Curator???
>>>
>>> Regards.
>>>
>>> Dano
>>>
>>> On Wed, Aug 5, 2020 at 12:20 PM Mark Payne  wrote:
>>>
 Dan,

 Can you grab a thread dump and provide that? Specifically, the “main”
 thread is the important one with startup. The note that the role is already
 registered is normal. It probably could be changed to a DEBUG level,
 really. It should not be concerning. A thread dump, though, would show us
 exactly where it’s at.

 Thanks
 -Mark


 On Aug 5, 2020, at 2:02 PM, dan young  wrote:

 Hello,
 Running nifi 1.11.4, 3 X secure cluster mode and have enabled
 kerberos/sasl, upon trying to startup the cluster, they seem to get stuck
 in :

 2020-08-05 17:10:18,907 WARN [main]
 o.a.nifi.controller.StandardFlowService There is currently no Cluster
 Coordinator. This often happens upon restart of NiFi
  when running an embedded ZooKeeper. Will register this node to become
 the active Cluster Coordinator and will attempt to connect to cluster again
 2020-08-05 17:10:18,907 INFO [main]
 o.a.n.c.l.e.CuratorLeaderElectionManager
 CuratorLeaderElectionManager[stopped=false] Attempted to register Leader
 Election
  for role 'Cluster Coordinator' but this role is already registered



 I've checked zookeeper and I can see that the /nifi znode has been
 created, although empty, and the ACL seem to look correct
 zk: nifi1-5.X.net:2181 (CONNECTED) 3]
 getAcl /nifi
 'sasl,'n...@x.net
 : cdrwa
 'world,'anyone
 : r


 relevant Nifi config settings

 nifi.properties:

 nifi.zookeeper.auth.type=sasl
 nifi.zookeeper.kerberos.removeHostFromPrincipal=true
 nifi.zookeeper.kerberos.removeRealmFromPrincipal=false

 # kerberos #
 nifi.kerberos.krb5.file=/etc/krb5.conf

 # kerberos service principal #
 nifi.kerberos.service.principal=n...@x.net
 nifi.kerberos.service.keytab.location=/opt/nifi/conf/nifi.keytab


 state-management.xml
 
 zk-provider

 org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider
 /nifi
 30 seconds
 CreatorOnly
 X:2181,Y:2181,Z:2181
  



 KRB5_TRACE=/dev/stdout kinit -k -t /opt/nifi/conf/nifi.keytab
 n...@x.net
 ...
 ...

 klist
 Ticket cache: FILE:/tmp/krb5cc_2004
 Default principal: n...@x.net

 Valid starting   Expires  Service principal
 08/05/2020 17:57:02  08/06/2020 03:57:02  krbtgt/x@x.net
 renew until 08/06/2020 17:57:02




 As a side note, secure NiFi was working fine before the kerberos bit,
 I've been beating my head against the wall with it for the day, but the
 kerberos/zookeeper stuff seems to be working now
 do we need to have Server-Server zookeeper auth working for this?


 Appreciate any insight

 Regards,

 Dano



>

Re: cluster stick in "Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered"

[Mark Payne]

No worries, thanks for following up and letting us know!

Thanks
-Mark


On Aug 5, 2020, at 3:42 PM, dan young 
mailto:danoyo...@gmail.com>> wrote:

Hello,

Sorry for all the noise...doohwas due to the realm in the jaas.conf being 
lowercase...i'm a knucklehead...

Dano

On Wed, Aug 5, 2020 at 1:12 PM Bryan Bende 
mailto:bbe...@gmail.com>> wrote:
I don't see how this would relate to the problem, but shouldn't the ACL be set 
to "Creator" when Sasl/Kerberos is setup correctly?

In addition to the nifi configs you showed, you would also need a jaas conf 
file specified in bootstrap.conf and in that file you would need the jaas entry 
for the ZK client.

On Wed, Aug 5, 2020 at 3:02 PM dan young 
mailto:danoyo...@gmail.com>> wrote:
Hello Mark,

Attached is a dump from one of the nodesI replaced the domain related 
entries with X/x.  I'm not sure if it's relevant or not, but I did notice that 
in the log there's entries "Looking for keys for n...@x.net" 
 the x (domain)  is lowercase whereas in the keytab file it's uppercase X.  
Also not sure if the Found unsupported keytype (1) is meaningful.  Not that 
when I delete the znode in zookeeper=, at least the initial znode is created 
/nifi, but we never see the other typical suspect, i.e Coordinator, Primary, 
etc...

Seems to be something stuck in Curator???

Regards.

Dano

On Wed, Aug 5, 2020 at 12:20 PM Mark Payne 
mailto:marka...@hotmail.com>> wrote:
Dan,

Can you grab a thread dump and provide that? Specifically, the “main” thread is 
the important one with startup. The note that the role is already registered is 
normal. It probably could be changed to a DEBUG level, really. It should not be 
concerning. A thread dump, though, would show us exactly where it’s at.

Thanks
-Mark


On Aug 5, 2020, at 2:02 PM, dan young 
mailto:danoyo...@gmail.com>> wrote:

Hello,
Running nifi 1.11.4, 3 X secure cluster mode and have enabled kerberos/sasl, 
upon trying to startup the cluster, they seem to get stuck in :

2020-08-05 17:10:18,907 WARN [main] o.a.nifi.controller.StandardFlowService 
There is currently no Cluster Coordinator. This often happens upon restart of 
NiFi
 when running an embedded ZooKeeper. Will register this node to become the 
active Cluster Coordinator and will attempt to connect to cluster again
2020-08-05 17:10:18,907 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager 
CuratorLeaderElectionManager[stopped=false] Attempted to register Leader 
Election
 for role 'Cluster Coordinator' but this role is already registered



I've checked zookeeper and I can see that the /nifi znode has been created, 
although empty, and the ACL seem to look correct
zk: nifi1-5.X.net:2181(CONNECTED) 3] getAcl /nifi
'sasl,'n...@x.net
: cdrwa
'world,'anyone
: r


relevant Nifi config settings

nifi.properties:

nifi.zookeeper.auth.type=sasl
nifi.zookeeper.kerberos.removeHostFromPrincipal=true
nifi.zookeeper.kerberos.removeRealmFromPrincipal=false

# kerberos #
nifi.kerberos.krb5.file=/etc/krb5.conf

# kerberos service principal #
nifi.kerberos.service.principal=n...@x.net
nifi.kerberos.service.keytab.location=/opt/nifi/conf/nifi.keytab


state-management.xml

zk-provider

org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider
/nifi
30 seconds
CreatorOnly
X:2181,Y:2181,Z:2181
 



KRB5_TRACE=/dev/stdout kinit -k -t /opt/nifi/conf/nifi.keytab 
n...@x.net
...
...

klist
Ticket cache: FILE:/tmp/krb5cc_2004
Default principal: n...@x.net

Valid starting   Expires  Service principal
08/05/2020 17:57:02  08/06/2020 03:57:02  krbtgt/x@x.net
renew until 08/06/2020 17:57:02




As a side note, secure NiFi was working fine before the kerberos bit, I've been 
beating my head against the wall with it for the day, but the 
kerberos/zookeeper stuff seems to be working now
do we need to have Server-Server zookeeper auth working for this?


Appreciate any insight

Regards,

Dano

Re: cluster stick in "Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered"

[dan young]

Hello,

Sorry for all the noise...doohwas due to the realm in the
jaas.conf being lowercase...i'm a knucklehead...

Dano

On Wed, Aug 5, 2020 at 1:12 PM Bryan Bende  wrote:

> I don't see how this would relate to the problem, but shouldn't the ACL be
> set to "Creator" when Sasl/Kerberos is setup correctly?
>
> In addition to the nifi configs you showed, you would also need a jaas
> conf file specified in bootstrap.conf and in that file you would need the
> jaas entry for the ZK client.
>
> On Wed, Aug 5, 2020 at 3:02 PM dan young  wrote:
>
>> Hello Mark,
>>
>> Attached is a dump from one of the nodesI replaced the domain related
>> entries with X/x.  I'm not sure if it's relevant or not, but I did notice
>> that in the log there's entries "Looking for keys for n...@x.net"  the x
>> (domain)  is lowercase whereas in the keytab file it's uppercase X.  Also
>> not sure if the Found unsupported keytype (1) is meaningful.  Not that when
>> I delete the znode in zookeeper=, at least the initial znode is created
>> /nifi, but we never see the other typical suspect, i.e Coordinator,
>> Primary, etc...
>>
>> Seems to be something stuck in Curator???
>>
>> Regards.
>>
>> Dano
>>
>> On Wed, Aug 5, 2020 at 12:20 PM Mark Payne  wrote:
>>
>>> Dan,
>>>
>>> Can you grab a thread dump and provide that? Specifically, the “main”
>>> thread is the important one with startup. The note that the role is already
>>> registered is normal. It probably could be changed to a DEBUG level,
>>> really. It should not be concerning. A thread dump, though, would show us
>>> exactly where it’s at.
>>>
>>> Thanks
>>> -Mark
>>>
>>>
>>> On Aug 5, 2020, at 2:02 PM, dan young  wrote:
>>>
>>> Hello,
>>> Running nifi 1.11.4, 3 X secure cluster mode and have enabled
>>> kerberos/sasl, upon trying to startup the cluster, they seem to get stuck
>>> in :
>>>
>>> 2020-08-05 17:10:18,907 WARN [main]
>>> o.a.nifi.controller.StandardFlowService There is currently no Cluster
>>> Coordinator. This often happens upon restart of NiFi
>>>  when running an embedded ZooKeeper. Will register this node to become
>>> the active Cluster Coordinator and will attempt to connect to cluster again
>>> 2020-08-05 17:10:18,907 INFO [main]
>>> o.a.n.c.l.e.CuratorLeaderElectionManager
>>> CuratorLeaderElectionManager[stopped=false] Attempted to register Leader
>>> Election
>>>  for role 'Cluster Coordinator' but this role is already registered
>>>
>>>
>>>
>>> I've checked zookeeper and I can see that the /nifi znode has been
>>> created, although empty, and the ACL seem to look correct
>>> zk: nifi1-5.X.net:2181 (CONNECTED) 3] getAcl
>>> /nifi
>>> 'sasl,'n...@x.net
>>> : cdrwa
>>> 'world,'anyone
>>> : r
>>>
>>>
>>> relevant Nifi config settings
>>>
>>> nifi.properties:
>>>
>>> nifi.zookeeper.auth.type=sasl
>>> nifi.zookeeper.kerberos.removeHostFromPrincipal=true
>>> nifi.zookeeper.kerberos.removeRealmFromPrincipal=false
>>>
>>> # kerberos #
>>> nifi.kerberos.krb5.file=/etc/krb5.conf
>>>
>>> # kerberos service principal #
>>> nifi.kerberos.service.principal=n...@x.net
>>> nifi.kerberos.service.keytab.location=/opt/nifi/conf/nifi.keytab
>>>
>>>
>>> state-management.xml
>>> 
>>> zk-provider
>>>
>>> org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider
>>> /nifi
>>> 30 seconds
>>> CreatorOnly
>>> X:2181,Y:2181,Z:2181
>>>  
>>>
>>>
>>>
>>> KRB5_TRACE=/dev/stdout kinit -k -t /opt/nifi/conf/nifi.keytab n...@x.net
>>> ...
>>> ...
>>>
>>> klist
>>> Ticket cache: FILE:/tmp/krb5cc_2004
>>> Default principal: n...@x.net
>>>
>>> Valid starting   Expires  Service principal
>>> 08/05/2020 17:57:02  08/06/2020 03:57:02  krbtgt/x@x.net
>>> renew until 08/06/2020 17:57:02
>>>
>>>
>>>
>>>
>>> As a side note, secure NiFi was working fine before the kerberos bit,
>>> I've been beating my head against the wall with it for the day, but the
>>> kerberos/zookeeper stuff seems to be working now
>>> do we need to have Server-Server zookeeper auth working for this?
>>>
>>>
>>> Appreciate any insight
>>>
>>> Regards,
>>>
>>> Dano
>>>
>>>
>>>

Re: cluster stick in "Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered"

[dan young]

Hello Bryan,

Same issue.  I have a jaas.conf and config in the bootstrap.cof

Client {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  keyTab="/opt/nifi/conf/nifi.keytab"
  storeKey=true
  useTicketCache=false
  principal="n...@x.net ";
};


java.arg.16=-Djavax.security.auth.useSubjectCredsOnly=true

# Zookeeper 3.5 now includes an Admin Server that starts on port 8080,
since NiFi is already using that port disable by default.
# Please see
https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_adminserver_config
for
configuration options.
java.arg.17=-Dzookeeper.admin.enableServer=false
java.arg.18=-Djava.security.auth.login.config=/opt/nifi/conf/jaas.conf
java.arg.19=-Dsun.security.krb5.debug=true



I just noticed that the realm here is in lowercase, let me change
that.maybe that's an issue
ReplyForward


On Wed, Aug 5, 2020 at 1:12 PM Bryan Bende  wrote:

> I don't see how this would relate to the problem, but shouldn't the ACL be
> set to "Creator" when Sasl/Kerberos is setup correctly?
>
> In addition to the nifi configs you showed, you would also need a jaas
> conf file specified in bootstrap.conf and in that file you would need the
> jaas entry for the ZK client.
>
> On Wed, Aug 5, 2020 at 3:02 PM dan young  wrote:
>
>> Hello Mark,
>>
>> Attached is a dump from one of the nodesI replaced the domain related
>> entries with X/x.  I'm not sure if it's relevant or not, but I did notice
>> that in the log there's entries "Looking for keys for n...@x.net"  the x
>> (domain)  is lowercase whereas in the keytab file it's uppercase X.  Also
>> not sure if the Found unsupported keytype (1) is meaningful.  Not that when
>> I delete the znode in zookeeper=, at least the initial znode is created
>> /nifi, but we never see the other typical suspect, i.e Coordinator,
>> Primary, etc...
>>
>> Seems to be something stuck in Curator???
>>
>> Regards.
>>
>> Dano
>>
>> On Wed, Aug 5, 2020 at 12:20 PM Mark Payne  wrote:
>>
>>> Dan,
>>>
>>> Can you grab a thread dump and provide that? Specifically, the “main”
>>> thread is the important one with startup. The note that the role is already
>>> registered is normal. It probably could be changed to a DEBUG level,
>>> really. It should not be concerning. A thread dump, though, would show us
>>> exactly where it’s at.
>>>
>>> Thanks
>>> -Mark
>>>
>>>
>>> On Aug 5, 2020, at 2:02 PM, dan young  wrote:
>>>
>>> Hello,
>>> Running nifi 1.11.4, 3 X secure cluster mode and have enabled
>>> kerberos/sasl, upon trying to startup the cluster, they seem to get stuck
>>> in :
>>>
>>> 2020-08-05 17:10:18,907 WARN [main]
>>> o.a.nifi.controller.StandardFlowService There is currently no Cluster
>>> Coordinator. This often happens upon restart of NiFi
>>>  when running an embedded ZooKeeper. Will register this node to become
>>> the active Cluster Coordinator and will attempt to connect to cluster again
>>> 2020-08-05 17:10:18,907 INFO [main]
>>> o.a.n.c.l.e.CuratorLeaderElectionManager
>>> CuratorLeaderElectionManager[stopped=false] Attempted to register Leader
>>> Election
>>>  for role 'Cluster Coordinator' but this role is already registered
>>>
>>>
>>>
>>> I've checked zookeeper and I can see that the /nifi znode has been
>>> created, although empty, and the ACL seem to look correct
>>> zk: nifi1-5.X.net:2181 (CONNECTED) 3] getAcl
>>> /nifi
>>> 'sasl,'n...@x.net
>>> : cdrwa
>>> 'world,'anyone
>>> : r
>>>
>>>
>>> relevant Nifi config settings
>>>
>>> nifi.properties:
>>>
>>> nifi.zookeeper.auth.type=sasl
>>> nifi.zookeeper.kerberos.removeHostFromPrincipal=true
>>> nifi.zookeeper.kerberos.removeRealmFromPrincipal=false
>>>
>>> # kerberos #
>>> nifi.kerberos.krb5.file=/etc/krb5.conf
>>>
>>> # kerberos service principal #
>>> nifi.kerberos.service.principal=n...@x.net
>>> nifi.kerberos.service.keytab.location=/opt/nifi/conf/nifi.keytab
>>>
>>>
>>> state-management.xml
>>> 
>>> zk-provider
>>>
>>> org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider
>>> /nifi
>>> 30 seconds
>>> CreatorOnly
>>> X:2181,Y:2181,Z:2181
>>>  
>>>
>>>
>>>
>>> KRB5_TRACE=/dev/stdout kinit -k -t /opt/nifi/conf/nifi.keytab n...@x.net
>>> ...
>>> ...
>>>
>>> klist
>>> Ticket cache: FILE:/tmp/krb5cc_2004
>>> Default principal: n...@x.net
>>>
>>> Valid starting   Expires  Service principal
>>> 08/05/2020 17:57:02  08/06/2020 03:57:02  krbtgt/x@x.net
>>> renew until 08/06/2020 17:57:02
>>>
>>>
>>>
>>>
>>> As a side note, secure NiFi was working fine before the kerberos bit,
>>> I've been beating my head against the wall with it for the day, but the
>>> kerberos/zookeeper stuff seems to be working now
>>> do we need to have Server-Server zookeeper auth working for this?
>>>
>>>
>>> Appreciate any insight
>>>
>>> Regards,
>>>
>>> Dano
>>>
>>>
>>>

Re: cluster stick in "Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered"

[dan young]

I'll make that change, was going off the comments in the
state-management.xml

-Open
-CreatorOnly

Let me try Creator...

On Wed, Aug 5, 2020 at 1:12 PM Bryan Bende  wrote:

> I don't see how this would relate to the problem, but shouldn't the ACL be
> set to "Creator" when Sasl/Kerberos is setup correctly?
>
> In addition to the nifi configs you showed, you would also need a jaas
> conf file specified in bootstrap.conf and in that file you would need the
> jaas entry for the ZK client.
>
> On Wed, Aug 5, 2020 at 3:02 PM dan young  wrote:
>
>> Hello Mark,
>>
>> Attached is a dump from one of the nodesI replaced the domain related
>> entries with X/x.  I'm not sure if it's relevant or not, but I did notice
>> that in the log there's entries "Looking for keys for n...@x.net"  the x
>> (domain)  is lowercase whereas in the keytab file it's uppercase X.  Also
>> not sure if the Found unsupported keytype (1) is meaningful.  Not that when
>> I delete the znode in zookeeper=, at least the initial znode is created
>> /nifi, but we never see the other typical suspect, i.e Coordinator,
>> Primary, etc...
>>
>> Seems to be something stuck in Curator???
>>
>> Regards.
>>
>> Dano
>>
>> On Wed, Aug 5, 2020 at 12:20 PM Mark Payne  wrote:
>>
>>> Dan,
>>>
>>> Can you grab a thread dump and provide that? Specifically, the “main”
>>> thread is the important one with startup. The note that the role is already
>>> registered is normal. It probably could be changed to a DEBUG level,
>>> really. It should not be concerning. A thread dump, though, would show us
>>> exactly where it’s at.
>>>
>>> Thanks
>>> -Mark
>>>
>>>
>>> On Aug 5, 2020, at 2:02 PM, dan young  wrote:
>>>
>>> Hello,
>>> Running nifi 1.11.4, 3 X secure cluster mode and have enabled
>>> kerberos/sasl, upon trying to startup the cluster, they seem to get stuck
>>> in :
>>>
>>> 2020-08-05 17:10:18,907 WARN [main]
>>> o.a.nifi.controller.StandardFlowService There is currently no Cluster
>>> Coordinator. This often happens upon restart of NiFi
>>>  when running an embedded ZooKeeper. Will register this node to become
>>> the active Cluster Coordinator and will attempt to connect to cluster again
>>> 2020-08-05 17:10:18,907 INFO [main]
>>> o.a.n.c.l.e.CuratorLeaderElectionManager
>>> CuratorLeaderElectionManager[stopped=false] Attempted to register Leader
>>> Election
>>>  for role 'Cluster Coordinator' but this role is already registered
>>>
>>>
>>>
>>> I've checked zookeeper and I can see that the /nifi znode has been
>>> created, although empty, and the ACL seem to look correct
>>> zk: nifi1-5.X.net:2181 (CONNECTED) 3] getAcl
>>> /nifi
>>> 'sasl,'n...@x.net
>>> : cdrwa
>>> 'world,'anyone
>>> : r
>>>
>>>
>>> relevant Nifi config settings
>>>
>>> nifi.properties:
>>>
>>> nifi.zookeeper.auth.type=sasl
>>> nifi.zookeeper.kerberos.removeHostFromPrincipal=true
>>> nifi.zookeeper.kerberos.removeRealmFromPrincipal=false
>>>
>>> # kerberos #
>>> nifi.kerberos.krb5.file=/etc/krb5.conf
>>>
>>> # kerberos service principal #
>>> nifi.kerberos.service.principal=n...@x.net
>>> nifi.kerberos.service.keytab.location=/opt/nifi/conf/nifi.keytab
>>>
>>>
>>> state-management.xml
>>> 
>>> zk-provider
>>>
>>> org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider
>>> /nifi
>>> 30 seconds
>>> CreatorOnly
>>> X:2181,Y:2181,Z:2181
>>>  
>>>
>>>
>>>
>>> KRB5_TRACE=/dev/stdout kinit -k -t /opt/nifi/conf/nifi.keytab n...@x.net
>>> ...
>>> ...
>>>
>>> klist
>>> Ticket cache: FILE:/tmp/krb5cc_2004
>>> Default principal: n...@x.net
>>>
>>> Valid starting   Expires  Service principal
>>> 08/05/2020 17:57:02  08/06/2020 03:57:02  krbtgt/x@x.net
>>> renew until 08/06/2020 17:57:02
>>>
>>>
>>>
>>>
>>> As a side note, secure NiFi was working fine before the kerberos bit,
>>> I've been beating my head against the wall with it for the day, but the
>>> kerberos/zookeeper stuff seems to be working now
>>> do we need to have Server-Server zookeeper auth working for this?
>>>
>>>
>>> Appreciate any insight
>>>
>>> Regards,
>>>
>>> Dano
>>>
>>>
>>>

Re: cluster stick in "Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered"

[Bryan Bende]

I don't see how this would relate to the problem, but shouldn't the ACL be
set to "Creator" when Sasl/Kerberos is setup correctly?

In addition to the nifi configs you showed, you would also need a jaas conf
file specified in bootstrap.conf and in that file you would need the jaas
entry for the ZK client.

On Wed, Aug 5, 2020 at 3:02 PM dan young  wrote:

> Hello Mark,
>
> Attached is a dump from one of the nodesI replaced the domain related
> entries with X/x.  I'm not sure if it's relevant or not, but I did notice
> that in the log there's entries "Looking for keys for n...@x.net"  the x
> (domain)  is lowercase whereas in the keytab file it's uppercase X.  Also
> not sure if the Found unsupported keytype (1) is meaningful.  Not that when
> I delete the znode in zookeeper=, at least the initial znode is created
> /nifi, but we never see the other typical suspect, i.e Coordinator,
> Primary, etc...
>
> Seems to be something stuck in Curator???
>
> Regards.
>
> Dano
>
> On Wed, Aug 5, 2020 at 12:20 PM Mark Payne  wrote:
>
>> Dan,
>>
>> Can you grab a thread dump and provide that? Specifically, the “main”
>> thread is the important one with startup. The note that the role is already
>> registered is normal. It probably could be changed to a DEBUG level,
>> really. It should not be concerning. A thread dump, though, would show us
>> exactly where it’s at.
>>
>> Thanks
>> -Mark
>>
>>
>> On Aug 5, 2020, at 2:02 PM, dan young  wrote:
>>
>> Hello,
>> Running nifi 1.11.4, 3 X secure cluster mode and have enabled
>> kerberos/sasl, upon trying to startup the cluster, they seem to get stuck
>> in :
>>
>> 2020-08-05 17:10:18,907 WARN [main]
>> o.a.nifi.controller.StandardFlowService There is currently no Cluster
>> Coordinator. This often happens upon restart of NiFi
>>  when running an embedded ZooKeeper. Will register this node to become
>> the active Cluster Coordinator and will attempt to connect to cluster again
>> 2020-08-05 17:10:18,907 INFO [main]
>> o.a.n.c.l.e.CuratorLeaderElectionManager
>> CuratorLeaderElectionManager[stopped=false] Attempted to register Leader
>> Election
>>  for role 'Cluster Coordinator' but this role is already registered
>>
>>
>>
>> I've checked zookeeper and I can see that the /nifi znode has been
>> created, although empty, and the ACL seem to look correct
>> zk: nifi1-5.X.net:2181 (CONNECTED) 3] getAcl
>> /nifi
>> 'sasl,'n...@x.net
>> : cdrwa
>> 'world,'anyone
>> : r
>>
>>
>> relevant Nifi config settings
>>
>> nifi.properties:
>>
>> nifi.zookeeper.auth.type=sasl
>> nifi.zookeeper.kerberos.removeHostFromPrincipal=true
>> nifi.zookeeper.kerberos.removeRealmFromPrincipal=false
>>
>> # kerberos #
>> nifi.kerberos.krb5.file=/etc/krb5.conf
>>
>> # kerberos service principal #
>> nifi.kerberos.service.principal=n...@x.net
>> nifi.kerberos.service.keytab.location=/opt/nifi/conf/nifi.keytab
>>
>>
>> state-management.xml
>> 
>> zk-provider
>>
>> org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider
>> /nifi
>> 30 seconds
>> CreatorOnly
>> X:2181,Y:2181,Z:2181
>>  
>>
>>
>>
>> KRB5_TRACE=/dev/stdout kinit -k -t /opt/nifi/conf/nifi.keytab n...@x.net
>> ...
>> ...
>>
>> klist
>> Ticket cache: FILE:/tmp/krb5cc_2004
>> Default principal: n...@x.net
>>
>> Valid starting   Expires  Service principal
>> 08/05/2020 17:57:02  08/06/2020 03:57:02  krbtgt/x@x.net
>> renew until 08/06/2020 17:57:02
>>
>>
>>
>>
>> As a side note, secure NiFi was working fine before the kerberos bit,
>> I've been beating my head against the wall with it for the day, but the
>> kerberos/zookeeper stuff seems to be working now
>> do we need to have Server-Server zookeeper auth working for this?
>>
>>
>> Appreciate any insight
>>
>> Regards,
>>
>> Dano
>>
>>
>>

Re: cluster stick in "Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered"

[dan young]

Hello Mark,

Attached is a dump from one of the nodesI replaced the domain related
entries with X/x.  I'm not sure if it's relevant or not, but I did notice
that in the log there's entries "Looking for keys for n...@x.net"  the x
(domain)  is lowercase whereas in the keytab file it's uppercase X.  Also
not sure if the Found unsupported keytype (1) is meaningful.  Not that when
I delete the znode in zookeeper=, at least the initial znode is created
/nifi, but we never see the other typical suspect, i.e Coordinator,
Primary, etc...

Seems to be something stuck in Curator???

Regards.

Dano

On Wed, Aug 5, 2020 at 12:20 PM Mark Payne  wrote:

> Dan,
>
> Can you grab a thread dump and provide that? Specifically, the “main”
> thread is the important one with startup. The note that the role is already
> registered is normal. It probably could be changed to a DEBUG level,
> really. It should not be concerning. A thread dump, though, would show us
> exactly where it’s at.
>
> Thanks
> -Mark
>
>
> On Aug 5, 2020, at 2:02 PM, dan young  wrote:
>
> Hello,
> Running nifi 1.11.4, 3 X secure cluster mode and have enabled
> kerberos/sasl, upon trying to startup the cluster, they seem to get stuck
> in :
>
> 2020-08-05 17:10:18,907 WARN [main]
> o.a.nifi.controller.StandardFlowService There is currently no Cluster
> Coordinator. This often happens upon restart of NiFi
>  when running an embedded ZooKeeper. Will register this node to become the
> active Cluster Coordinator and will attempt to connect to cluster again
> 2020-08-05 17:10:18,907 INFO [main]
> o.a.n.c.l.e.CuratorLeaderElectionManager
> CuratorLeaderElectionManager[stopped=false] Attempted to register Leader
> Election
>  for role 'Cluster Coordinator' but this role is already registered
>
>
>
> I've checked zookeeper and I can see that the /nifi znode has been
> created, although empty, and the ACL seem to look correct
> zk: nifi1-5.X.net:2181 (CONNECTED) 3] getAcl
> /nifi
> 'sasl,'n...@x.net
> : cdrwa
> 'world,'anyone
> : r
>
>
> relevant Nifi config settings
>
> nifi.properties:
>
> nifi.zookeeper.auth.type=sasl
> nifi.zookeeper.kerberos.removeHostFromPrincipal=true
> nifi.zookeeper.kerberos.removeRealmFromPrincipal=false
>
> # kerberos #
> nifi.kerberos.krb5.file=/etc/krb5.conf
>
> # kerberos service principal #
> nifi.kerberos.service.principal=n...@x.net
> nifi.kerberos.service.keytab.location=/opt/nifi/conf/nifi.keytab
>
>
> state-management.xml
> 
> zk-provider
>
> org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider
> /nifi
> 30 seconds
> CreatorOnly
> X:2181,Y:2181,Z:2181
>  
>
>
>
> KRB5_TRACE=/dev/stdout kinit -k -t /opt/nifi/conf/nifi.keytab n...@x.net
> ...
> ...
>
> klist
> Ticket cache: FILE:/tmp/krb5cc_2004
> Default principal: n...@x.net
>
> Valid starting   Expires  Service principal
> 08/05/2020 17:57:02  08/06/2020 03:57:02  krbtgt/x@x.net
> renew until 08/06/2020 17:57:02
>
>
>
>
> As a side note, secure NiFi was working fine before the kerberos bit, I've
> been beating my head against the wall with it for the day, but the
> kerberos/zookeeper stuff seems to be working now
> do we need to have Server-Server zookeeper auth working for this?
>
>
> Appreciate any insight
>
> Regards,
>
> Dano
>
>
>


bootstrap-dump.log.gz
Description: GNU Zip compressed data

Re: cluster stick in "Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered"

[Mark Payne]

Dan,

Can you grab a thread dump and provide that? Specifically, the “main” thread is 
the important one with startup. The note that the role is already registered is 
normal. It probably could be changed to a DEBUG level, really. It should not be 
concerning. A thread dump, though, would show us exactly where it’s at.

Thanks
-Mark


On Aug 5, 2020, at 2:02 PM, dan young 
mailto:danoyo...@gmail.com>> wrote:

Hello,
Running nifi 1.11.4, 3 X secure cluster mode and have enabled kerberos/sasl, 
upon trying to startup the cluster, they seem to get stuck in :

2020-08-05 17:10:18,907 WARN [main] o.a.nifi.controller.StandardFlowService 
There is currently no Cluster Coordinator. This often happens upon restart of 
NiFi
 when running an embedded ZooKeeper. Will register this node to become the 
active Cluster Coordinator and will attempt to connect to cluster again
2020-08-05 17:10:18,907 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager 
CuratorLeaderElectionManager[stopped=false] Attempted to register Leader 
Election
 for role 'Cluster Coordinator' but this role is already registered



I've checked zookeeper and I can see that the /nifi znode has been created, 
although empty, and the ACL seem to look correct
zk: nifi1-5.X.net:2181(CONNECTED) 3] getAcl /nifi
'sasl,'n...@x.net
: cdrwa
'world,'anyone
: r


relevant Nifi config settings

nifi.properties:

nifi.zookeeper.auth.type=sasl
nifi.zookeeper.kerberos.removeHostFromPrincipal=true
nifi.zookeeper.kerberos.removeRealmFromPrincipal=false

# kerberos #
nifi.kerberos.krb5.file=/etc/krb5.conf

# kerberos service principal #
nifi.kerberos.service.principal=n...@x.net
nifi.kerberos.service.keytab.location=/opt/nifi/conf/nifi.keytab


state-management.xml

zk-provider

org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider
/nifi
30 seconds
CreatorOnly
X:2181,Y:2181,Z:2181
 



KRB5_TRACE=/dev/stdout kinit -k -t /opt/nifi/conf/nifi.keytab 
n...@x.net
...
...

klist
Ticket cache: FILE:/tmp/krb5cc_2004
Default principal: n...@x.net

Valid starting   Expires  Service principal
08/05/2020 17:57:02  08/06/2020 03:57:02  krbtgt/x@x.net
renew until 08/06/2020 17:57:02




As a side note, secure NiFi was working fine before the kerberos bit, I've been 
beating my head against the wall with it for the day, but the 
kerberos/zookeeper stuff seems to be working now
do we need to have Server-Server zookeeper auth working for this?


Appreciate any insight

Regards,

Dano

cluster stick in "Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered"

[dan young]

Hello,
Running nifi 1.11.4, 3 X secure cluster mode and have enabled
kerberos/sasl, upon trying to startup the cluster, they seem to get stuck
in :

2020-08-05 17:10:18,907 WARN [main] o.a.nifi.controller.StandardFlowService
There is currently no Cluster Coordinator. This often happens upon restart
of NiFi
 when running an embedded ZooKeeper. Will register this node to become the
active Cluster Coordinator and will attempt to connect to cluster again
2020-08-05 17:10:18,907 INFO [main]
o.a.n.c.l.e.CuratorLeaderElectionManager
CuratorLeaderElectionManager[stopped=false] Attempted to register Leader
Election
 for role 'Cluster Coordinator' but this role is already registered



I've checked zookeeper and I can see that the /nifi znode has been created,
although empty, and the ACL seem to look correct
zk: nifi1-5.X.net:2181(CONNECTED) 3] getAcl /nifi
'sasl,'n...@x.net
: cdrwa
'world,'anyone
: r


relevant Nifi config settings

nifi.properties:

nifi.zookeeper.auth.type=sasl
nifi.zookeeper.kerberos.removeHostFromPrincipal=true
nifi.zookeeper.kerberos.removeRealmFromPrincipal=false

# kerberos #
nifi.kerberos.krb5.file=/etc/krb5.conf

# kerberos service principal #
nifi.kerberos.service.principal=n...@x.net
nifi.kerberos.service.keytab.location=/opt/nifi/conf/nifi.keytab


state-management.xml

zk-provider

org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider
/nifi
30 seconds
CreatorOnly
X:2181,Y:2181,Z:2181
 



KRB5_TRACE=/dev/stdout kinit -k -t /opt/nifi/conf/nifi.keytab n...@x.net
...
...

klist
Ticket cache: FILE:/tmp/krb5cc_2004
Default principal: n...@x.net

Valid starting   Expires  Service principal
08/05/2020 17:57:02  08/06/2020 03:57:02  krbtgt/x@x.net
renew until 08/06/2020 17:57:02




As a side note, secure NiFi was working fine before the kerberos bit, I've
been beating my head against the wall with it for the day, but the
kerberos/zookeeper stuff seems to be working now
do we need to have Server-Server zookeeper auth working for this?


Appreciate any insight

Regards,

Dano

Re: IBM MQ message selector

[Peter Turcsanyi]

Hi Marc,

Great, thanks for the feedback.

Best,
Peter

On Mon, Aug 3, 2020 at 3:16 PM Marc Recht  wrote:

> Hi Peter,
>
> I tested it now quite a bit and it works like a charm! Thanks for
> implementing it so swiftly.
>
> Best,
>
> Marc
> On 27/07/2020 13:40, Peter Turcsanyi wrote:
>
> Hi Marc,
>
> ConsumeJMS message selector support has been implemented and will be
> released in the upcoming 1.12 version.
>
> Best,
> Peter
>
> On Fri, Jul 24, 2020 at 7:40 AM Marc Recht  wrote:
>
>> Hi Peter,
>>
>> OK, thanks! Will do.
>>
>> Best,
>>
>> Marc
>> On 23/07/2020 20:30, Peter Turcsanyi wrote:
>>
>> Hi Marc,
>>
>> Message selectors are not supported by ConsumeJMS currently.
>> Feel free to file a jira ticket (https://issues.apache.org/jira) to
>> request for this improvement.
>>
>> Regards,
>> Peter
>>
>> On Wed, Jul 22, 2020 at 5:51 PM Marc Recht 
>> wrote:
>>
>>>
>>>
>>> Hi,
>>>
>>> I try to use a message selector with NiFi and IBM MQ and the ConsumeJMS
>>> processor. Unfortunately, I am not sure how... Is there an example
>>> available somewhere how to set this up?
>>>
>>> Any help appreciated,
>>> Marc
>>>
>>>