Pierre. Ok, I keep this in mind. For now, I have an other "issue" configuring secure cluster and registry.
I study this and will give a try to push documentation. May be throw a Jira to just expose my difficulties. Writing the documentation by my side may not be well formed ;) Regards Etienne Le jeu. 26 nov. 2020 à 12:14, Pierre Villard <pierre.villard...@gmail.com> a écrit : > Etienne, > > If you feel like there would be nice additions to have in the > documentation, please feel free to share your suggestions to improve it > (though a JIRA / pull request). We don't want NiFi users/administrators to > have a painful process when configuring all of this. > > Thanks, > Pierre > > Le jeu. 26 nov. 2020 à 12:01, Etienne Jouvin <lapinoujou...@gmail.com> a > écrit : > >> Hello all. >> >> And finally I got it. >> When I am not able to find in the documentation, because I was not >> correctly searching offcourse, I read the source code and do remote debug. >> >> And the conclusion that it is all about identity mapping. >> >> For reminder, the objective was to have a secured cluster and everything >> managed in the LDAP. >> I follow the walk throughs documentation. But in it, it is only matter of >> nodes managed in userGroupIdentity, based on files. >> >> In my case, users and groups are managed in the LDAP, even nodes for >> cluster. >> >> With remote debugging, I touched the class X509AuthenticationProvider and >> specially the function authenticate. By the way, this is the only class >> that fires an exception with message Untrusted proxy ... >> In it, identities are extracted from the request and linked certificates. >> Ids extracted were : >> >> - uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch , for the "real" user >> - CN=mig1.amexio.ch <http://mig1.assura.ch/>, OU=users, OU=nifi, >> DC=amexio, DC=ch, for the "proxy" node >> >> >> But in the LDAP, the node as the entry DN : cn=mig1.amexio.ch >> ,ou=users,ou=nifi;dc=amexio,dc=ch >> >> So when the provider validate identities, the NiFi identity "CN= >> mig1.amexio.ch <http://mig1.assura.ch/>, OU=users, OU=nifi, DC=amexio, >> DC=ch" is not found. No user/group, and does not belongs to any group. Then >> the validation on policy "proxy", with "write" access, failed and the >> connection was rejected. >> >> What I found also in this function, there is a identity mapping, >> function mapIdentity from class NiFiAuthenticationProvider. >> This will use function mapIdentity in class IdentityMappingUtil. >> With this, the identity is normalized and this is the trick to transform >> the idnetity retrieved from the certificate, with space and upper case, to >> an identity matching the one retrieved from LDAP. >> >> >> That was the point. Si in the file nifi.properties, for all nodes, here >> is modifications : >> nifi.security.identity.mapping.pattern.ldapCertNode=^CN=(.*?), OU=(.*?), >> OU=(.*?), DC=(.*?), DC=(.*?)$ >> >> nifi.security.identity.mapping.value.ldapCertNode=cn=$1,ou=$2,ou=$3,dc=$4,dc=$5 >> nifi.security.identity.mapping.transform.ldapCertNode=NONE >> >> Like this, the identity retrieved from certificate, for the proxy node, >> is CN=mig1.amexio.ch <http://mig1.assura.ch/>, OU=users, OU=nifi, >> DC=amexio, DC=ch >> After identity mapping it is cn=mig1.amexio.ch <http://mig1.assura.ch/> >> ,ou=users,ou=nifi,dc=amexio,dc=ch >> And it is exactly matching identity retrieved from the LDAP for the node. >> >> Et voilà. >> >> Hard work (for me) and happy to find the way to manage this. >> >> Thanks all for your help. >> >> >> Etienne >> >> >> >> Le mer. 25 nov. 2020 à 19:50, Etienne Jouvin <lapinoujou...@gmail.com> a >> écrit : >> >>> That's what I suspect for now. >>> >>> But this is strange to have a transformation with space introduced. >>> In the certificate (I use KeyStore Explorer to visualize the >>> certificate) there is no space. >>> >>> But I will give a try to use the identity mapping from here >>> # Identity Mapping Properties # >>> # These properties allow normalizing user identities such that >>> identities coming from different identity providers >>> # (certificates, LDAP, Kerberos) can be treated the same internally in >>> NiFi. The following example demonstrates normalizing >>> # DNs from certificates and principals from Kerberos into a common >>> identity string: >>> # >>> # nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?), >>> O=(.*?), L=(.*?), ST=(.*?), C=(.*?)$ >>> # nifi.security.identity.mapping.value.dn=$1@$2 >>> # nifi.security.identity.mapping.transform.dn=NONE >>> # nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$ >>> # nifi.security.identity.mapping.value.kerb=$1@$2 >>> # nifi.security.identity.mapping.transform.kerb=UPPER >>> >>> # Group Mapping Properties # >>> # These properties allow normalizing group names coming from external >>> sources like LDAP. The following example >>> # lowercases any group name. >>> # >>> # nifi.security.group.mapping.pattern.anygroup=^(.*)$ >>> # nifi.security.group.mapping.value.anygroup=$1 >>> # nifi.security.group.mapping.transform.anygroup=LOWER >>> >>> for now, I do not know what to put ;) >>> >>> Etienne >>> >>> >>> >>> Le mer. 25 nov. 2020 à 19:45, Bryan Bende <bbe...@gmail.com> a écrit : >>> >>>> The values have to match exactly, it’s case and white space sensitive. >>>> There are identity transforms in Nifi properties that possibly help. >>>> >>>> The value from where it says “Attempting request for...” is the value >>>> nifi is getting from the certificate. You can also use key tool to list the >>>> contents of the keystore jks and it will probably be the same as the log. >>>> >>>> On Wed, Nov 25, 2020 at 1:30 PM Etienne Jouvin <lapinoujou...@gmail.com> >>>> wrote: >>>> >>>>> Hello all. >>>>> >>>>> Still no luck. >>>>> I follow steps from here : >>>>> https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html#creating-and-securing-a-nifi-cluster-with-the-tls-toolkit >>>>> >>>>> Reminder, everything is currently in LDAP, I do not have any file >>>>> user/group provider. >>>>> When only one node, I can connect, this is correct. >>>>> >>>>> I can see the group having nodes "users" from the LDAP and the group >>>>> has the proxy policy. >>>>> >>>>> But one thing strange, may be not the cause. >>>>> In the LDAP, the DN is something like : >>>>> cn=mig1.amexio.ch,ou=users,ou=nifi,dc=amexio,dc=ch >>>>> >>>>> And when I try to connect, I can see thoses errors : >>>>> In nifi-user.log for the node where I connect : >>>>> INFO [NiFi Web Server-22] o.a.n.w.s.NiFiAuthenticationFilter >>>>> Authentication success for uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch >>>>> >>>>> In nifi-user.log on the second node : >>>>> INFO [NiFi Web Server-29] o.a.n.w.s.NiFiAuthenticationFilter >>>>> Attempting request for >>>>> (<uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch><CN= >>>>> mig1.amexio.ch <http://mig1.assura.ch>, OU=users, OU=nifi, DC=amexio, >>>>> DC=ch>) GET https://mig2.amexio.ch:9442/nifi-api/flow/current-user >>>>> (source ip: 127.0.0.1) >>>>> WARN [NiFi Web Server-29] o.a.n.w.s.NiFiAuthenticationFilter Rejecting >>>>> access to web api: Untrusted proxy CN=mig1.amexio.ch, OU=users, >>>>> OU=nifi, DC=amexio, DC=ch >>>>> >>>>> The doubt is on the message, why the node DN has spaces between each >>>>> level ? >>>>> That does not match the value in the certificate and not the DN in >>>>> LDAP. >>>>> >>>>> Maybe this is not the good route to investigate. Still searching. I >>>>> have to find a way to "debug" the NiFiAuthenticationFilter and check the >>>>> certificate comparison. >>>>> >>>>> Etienne >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Le mer. 25 nov. 2020 à 14:18, David Handermann < >>>>> exceptionfact...@gmail.com> a écrit : >>>>> >>>>>> I am not as familiar with the LDAP user group provider, but based on >>>>>> the "Untrusted proxy" message you are seeing, it sounds like the nodes >>>>>> are >>>>>> not being identified properly as members of the "nodes" group from LDAP. >>>>>> Just for testing purposes, you could try specifying the node >>>>>> distinguished >>>>>> names in the "Node Identity N" properties of the access policy provider, >>>>>> using "Node Identity 1", "Node Identity 2" and "Node Identity 3" to >>>>>> specify >>>>>> each node DN. If that works, then it sounds like a configuration issue >>>>>> with the Node Group, either on the LDAP server, or in the way NiFi is >>>>>> attempting to query LDAP. >>>>>> >>>>>> Regards, >>>>>> David Handermann >>>>>> >>>>>> On Wed, Nov 25, 2020 at 5:19 AM Etienne Jouvin < >>>>>> lapinoujou...@gmail.com> wrote: >>>>>> >>>>>>> Just for information, did not have time to test it from now. >>>>>>> I was not able to get this Walk Throughs documentation. >>>>>>> https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html >>>>>>> >>>>>>> Hope I will find the error I have about certificate (I have a little >>>>>>> idea) >>>>>>> >>>>>>> Etienne >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Le mer. 25 nov. 2020 à 08:36, Etienne Jouvin < >>>>>>> lapinoujou...@gmail.com> a écrit : >>>>>>> >>>>>>>> Hello. >>>>>>>> >>>>>>>> I made some progress yesterday. >>>>>>>> I did setup in LDAP groups and person >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Groups : >>>>>>>> cn=administrators,ou=groups,ou=nifi,dc=amexio,dc=ch : for >>>>>>>> administrators >>>>>>>> cn=supervisors,ou=groups,ou=nifi,dc=amexio,dc=ch : for supervisors >>>>>>>> cn=nodes,ou=groups,ou=nifi,dc=amexio,dc=ch : group where I put all >>>>>>>> "person" representing NiFi nodes. >>>>>>>> >>>>>>>> Users : >>>>>>>> cn=migX.amexio.ch,ou=users,ou=nifi,dc=amexio,dc=ch : for each >>>>>>>> node, replacing X by the index, and with object class person >>>>>>>> uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch : example of "real" >>>>>>>> user used to connect on the platform, with object class inetOrgperson >>>>>>>> >>>>>>>> In NiFi configuration. >>>>>>>> I did activate a userGroupProvider linked to the LDAP >>>>>>>> <userGroupProvider> >>>>>>>> <identifier>amexio-ldap-user-group-provider</identifier> >>>>>>>> >>>>>>>> <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class> >>>>>>>> <property name="Authentication Strategy">SIMPLE</property> >>>>>>>> >>>>>>>> <property name="Manager DN">uid=admin,ou=system</property> >>>>>>>> <property name="Manager Password">secret</property> >>>>>>>> >>>>>>>> <property name="TLS - Keystore"></property> >>>>>>>> <property name="TLS - Keystore Password"></property> >>>>>>>> <property name="TLS - Keystore Type"></property> >>>>>>>> <property name="TLS - Truststore"></property> >>>>>>>> <property name="TLS - Truststore Password"></property> >>>>>>>> <property name="TLS - Truststore Type"></property> >>>>>>>> <property name="TLS - Client Auth"></property> >>>>>>>> <property name="TLS - Protocol"></property> >>>>>>>> <property name="TLS - Shutdown Gracefully"></property> >>>>>>>> >>>>>>>> <property name="Referral Strategy">FOLLOW</property> >>>>>>>> <property name="Connect Timeout">10 secs</property> >>>>>>>> <property name="Read Timeout">10 secs</property> >>>>>>>> >>>>>>>> <property name="Url">ldap://localhost:10389</property> >>>>>>>> <property name="Page Size">50</property> >>>>>>>> <!-- <property name="Sync Interval">30 mins</property> --> >>>>>>>> <property name="Sync Interval">30 seconds</property> >>>>>>>> <property name="Group Membership - Enforce Case >>>>>>>> Sensitivity">false</property> >>>>>>>> >>>>>>>> <property name="User Search >>>>>>>> Base">ou=users,ou=nifi,dc=amexio,dc=ch</property> >>>>>>>> <property name="User Object Class">person</property> >>>>>>>> <property name="User Search Scope">ONE_LEVEL</property> >>>>>>>> <property name="User Search Filter"></property> >>>>>>>> <property name="User Identity Attribute"></property> >>>>>>>> <property name="User Group Name Attribute"></property> >>>>>>>> <property name="User Group Name Attribute - Referenced >>>>>>>> Group Attribute"></property> >>>>>>>> >>>>>>>> <property name="Group Search >>>>>>>> Base">ou=groups,ou=nifi,dc=amexio,dc=ch</property> >>>>>>>> <property name="Group Object Class">groupOfNames</property> >>>>>>>> <property name="Group Search Scope">ONE_LEVEL</property> >>>>>>>> <property name="Group Search Filter"></property> >>>>>>>> <property name="Group Name Attribute">cn</property> >>>>>>>> <property name="Group Member Attribute">member</property> >>>>>>>> <property name="Group Member Attribute - Referenced User >>>>>>>> Attribute"></property> >>>>>>>> </userGroupProvider> >>>>>>>> >>>>>>>> Of course, register it inside the accessPolicyProvider >>>>>>>> <accessPolicyProvider> >>>>>>>> <identifier>file-access-policy-provider</identifier> >>>>>>>> >>>>>>>> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class> >>>>>>>> <!-- <property name="User Group >>>>>>>> Provider">file-user-group-provider</property> --> >>>>>>>> <property name="User Group >>>>>>>> Provider">amexio-ldap-user-group-provider</property> >>>>>>>> <property name="Authorizations >>>>>>>> File">./conf/authorizations.xml</property> >>>>>>>> <!-- <property name="Initial Admin Identity"></property> --> >>>>>>>> <property name="Initial Admin >>>>>>>> Identity">uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch</property> >>>>>>>> <property name="Legacy Authorized Users File"></property> >>>>>>>> <property name="Node Identity 1"></property> >>>>>>>> <property name="Node Group">nodes</property> >>>>>>>> </accessPolicyProvider> >>>>>>>> >>>>>>>> I am able to connect with the initial administrator account, when >>>>>>>> the first node is started. >>>>>>>> And all nodes are synchronized in the NiFi instance. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> As soon as I start an additional node, I can not connect to the >>>>>>>> first node >>>>>>>> Erreur message >>>>>>>> Untrusted proxy CN=mig1.amexio.ch, OU=users, OU=nifi, DC=amexio, >>>>>>>> DC=ch >>>>>>>> >>>>>>>> But I can connect on the second node. >>>>>>>> >>>>>>>> >>>>>>>> So all this is about the certificate I guess. >>>>>>>> for reminder, I use nls-toolkit to generate certificate on all >>>>>>>> nodes with something like : >>>>>>>> tls-toolkit.bat standalone -f >>>>>>>> "C:\nifi-1.12.1\node1\conf\nifi.properties" -o >>>>>>>> "C:\nifi-1.12.1\node1\conf\secured" -n mig1.amexio.ch >>>>>>>> --nifiDnPrefix cn= --nifiDnSuffix ou=users,ou=nifi,dc=amexio,dc=ch >>>>>>>> >>>>>>>> Proxy is untrusted, ok fine. So may be I should not use the >>>>>>>> standalone function of toolkit, but using server and client. In that >>>>>>>> case, >>>>>>>> I have to stay alive the server from toolkit ? >>>>>>>> Also, it seems I did not add certificate from node1 inside node2 >>>>>>>> trutstore, and node2 certificate inside node1 truststore ? >>>>>>>> But in this case, if I have to add a new node, let's say node4, I >>>>>>>> would have to push the certificate from node4 inside all existing >>>>>>>> nodes ? >>>>>>>> >>>>>>>> I continue to search, but any idea / input will be appreciated. >>>>>>>> >>>>>>>> Etienne >>>>>>>> >>>>>>>> >>>>>>>> Le lun. 23 nov. 2020 à 18:39, Bryan Bende <bbe...@gmail.com> a >>>>>>>> écrit : >>>>>>>> >>>>>>>>> Yes it will be the DN of the server's certificate which comes from >>>>>>>>> the keystore. >>>>>>>>> >>>>>>>>> NiFi will get an incoming request, see that there is an X509 cert, >>>>>>>>> take the DN and go to the user group provider and ask for the user >>>>>>>>> with this identity. >>>>>>>>> >>>>>>>>> On Mon, Nov 23, 2020 at 12:01 PM Etienne Jouvin < >>>>>>>>> lapinoujou...@gmail.com> wrote: >>>>>>>>> > >>>>>>>>> > Hum OK, >>>>>>>>> > >>>>>>>>> > I will give it a try. >>>>>>>>> > But one more thing... >>>>>>>>> > >>>>>>>>> > If I only set the group node; >>>>>>>>> > How NiFi will connect the node with the nodeId in the LDAP ? >>>>>>>>> > Where does it take the nodeid value ? >>>>>>>>> > Is it the value we set in the keystore / truststore, by default >>>>>>>>> cn=localhost, dc=NIFI (something like this) ? >>>>>>>>> > >>>>>>>>> > Etienne >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > Le lun. 23 nov. 2020 à 17:54, Bryan Bende <bbe...@gmail.com> a >>>>>>>>> écrit : >>>>>>>>> >> >>>>>>>>> >> I don't really know the LDAP specifics too well, so I'm not >>>>>>>>> actually sure. >>>>>>>>> >> >>>>>>>>> >> You just need the nodes to come back from the LDAP >>>>>>>>> UserGroupProvider >>>>>>>>> >> as if they were regular users and members of some group "foo", >>>>>>>>> which >>>>>>>>> >> you then put "foo" into the "Node Group". >>>>>>>>> >> >>>>>>>>> >> On Mon, Nov 23, 2020 at 11:50 AM Etienne Jouvin < >>>>>>>>> lapinoujou...@gmail.com> wrote: >>>>>>>>> >> > >>>>>>>>> >> > Thanks Bryan. >>>>>>>>> >> > >>>>>>>>> >> > With your answer.... I will go to the Node Group and assign >>>>>>>>> node identities. >>>>>>>>> >> > Better for deployment and setup on the fly, I guess. >>>>>>>>> >> > >>>>>>>>> >> > One more point, you said "creating ldap entries for your >>>>>>>>> nodes and assigning them group membership in ldap". What type of >>>>>>>>> objectClass would you assign to the node in LDAP ? >>>>>>>>> >> > This is not inetOrgPerson. The node should not have password. >>>>>>>>> >> > If I create groupOfMembers for each node, is it correct ? >>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> >> > Thanks >>>>>>>>> >> > >>>>>>>>> >> > Etienne >>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> >> > Le lun. 23 nov. 2020 à 17:27, Bryan Bende <bbe...@gmail.com> >>>>>>>>> a écrit : >>>>>>>>> >> >> >>>>>>>>> >> >> Hello, >>>>>>>>> >> >> >>>>>>>>> >> >> "Node Identity" is similar to the "Initial Admin" concept, >>>>>>>>> in that it >>>>>>>>> >> >> sets up the policies for the initial nodes to have >>>>>>>>> permissions to >>>>>>>>> >> >> proxy. >>>>>>>>> >> >> >>>>>>>>> >> >> If you are creating ldap entries for your nodes and >>>>>>>>> assigning them >>>>>>>>> >> >> group membership in ldap, then yes you could put that group >>>>>>>>> name as >>>>>>>>> >> >> the "Node Group" and then you don't need to specify the "Node >>>>>>>>> >> >> Identities". >>>>>>>>> >> >> >>>>>>>>> >> >> If you are creating the node users in NiFi's file-based user >>>>>>>>> group >>>>>>>>> >> >> provider then you need to use node identities, and when >>>>>>>>> adding a new >>>>>>>>> >> >> node to the cluster you'd have to add the user first through >>>>>>>>> the >>>>>>>>> >> >> UI/REST API and grant it proxy, then actually connect it to >>>>>>>>> the >>>>>>>>> >> >> cluster. >>>>>>>>> >> >> >>>>>>>>> >> >> Thanks, >>>>>>>>> >> >> >>>>>>>>> >> >> Bryan >>>>>>>>> >> >> >>>>>>>>> >> >> >>>>>>>>> >> >> On Mon, Nov 23, 2020 at 7:58 AM Etienne Jouvin < >>>>>>>>> lapinoujou...@gmail.com> wrote: >>>>>>>>> >> >> > >>>>>>>>> >> >> > Hello all. >>>>>>>>> >> >> > >>>>>>>>> >> >> > >>>>>>>>> >> >> > I am currently setting up a NiFi, 1.12.1, Cluster with >>>>>>>>> LDAP authentication. >>>>>>>>> >> >> > For now the accessPolicyProvider is the default one with >>>>>>>>> the configuration template : >>>>>>>>> >> >> > <accessPolicyProvider> >>>>>>>>> >> >> > >>>>>>>>> <identifier>file-access-policy-provider</identifier> >>>>>>>>> >> >> > >>>>>>>>> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class> >>>>>>>>> >> >> > <property name="User Group >>>>>>>>> Provider">file-user-group-provider</property> >>>>>>>>> >> >> > <property name="Authorizations >>>>>>>>> File">./conf/authorizations.xml</property> >>>>>>>>> >> >> > <property name="Initial Admin Identity"></property> >>>>>>>>> >> >> > <property name="Legacy Authorized Users >>>>>>>>> File"></property> >>>>>>>>> >> >> > <property name="Node Identity 1"></property> >>>>>>>>> >> >> > <property name="Node Group"></property> >>>>>>>>> >> >> > </accessPolicyProvider> >>>>>>>>> >> >> > >>>>>>>>> >> >> > But I do not really understand the purpose of the Node >>>>>>>>> Identity X property. >>>>>>>>> >> >> > If I well understood, all nodes should have the same >>>>>>>>> configuration file, and I should register all nodes identity. >>>>>>>>> >> >> > >>>>>>>>> >> >> > But what about if I want to add a new node in the cluster >>>>>>>>> on the fly ? >>>>>>>>> >> >> > Should I register a new node identity, and then I should >>>>>>>>> change all nodes configurations ? >>>>>>>>> >> >> > The comment, in the configuration file, mentions the >>>>>>>>> configuration Node Group, The name of a group containing NiFi cluster >>>>>>>>> nodes. The typical use for this is when nodes are dynamically >>>>>>>>> added/removed >>>>>>>>> from the cluster. >>>>>>>>> >> >> > Should I just put a Node group name and this will do the >>>>>>>>> trick ? >>>>>>>>> >> >> > >>>>>>>>> >> >> > What should I put ? At the following link, >>>>>>>>> https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.0.3/bk_administration/content/cluster-node-identities.html, >>>>>>>>> it is said something like : cn=nifi-1,ou=people,dc=example,dc=com >>>>>>>>> >> >> > In that case, what should be the obejct class for the node >>>>>>>>> cn=nifi-1 in the LDAP ? >>>>>>>>> >> >> > >>>>>>>>> >> >> > Any documentation links will be appreciated. >>>>>>>>> >> >> > >>>>>>>>> >> >> > Regards. >>>>>>>>> >> >> > >>>>>>>>> >> >> > Etienne Jouvin >>>>>>>>> >>>>>>>> -- >>>> Sent from Gmail Mobile >>>> >>>
