Re: LDAP/AD User groups
Bingo! Thank you so much Kevin, In the end not using a Group Search Filter is what made this work for AD. Here is the relevant snippet: OU=groups,OU=Dev,DC=xxx,DC=,DC=net group ONE_LEVEL cn member Thanks again! Martijn On 6 June 2018 at 05:58, Kevin Doran wrote: > Whoops, in my example, disregard the class for the > ldap-user-group-provider > (org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider). > I took that snippet from one of my NiFi Registry configuration files, which > is the same as NiFi except for the class name ☺. Just mentioning that so > it doesn’t confuse anyone or you don’t copy/paste from my example. > > > > *From: *Kevin Doran > *Reply-To: * > *Date: *Tuesday, June 5, 2018 at 23:52 > *To: * > *Subject: *Re: LDAP/AD User groups > > > > Hi Martijn, > > > > You’re really close to having this configured correctly. Basically, you > just need to set a few more Group properties in your > ldap-user-group-provider config. > > > > The way user and group loading works in NiFI: > > > >1. The users are synced from LDAP using the “User Search *” (and >related) properties >2. The groups are synced from LDAP using the “Group Search * (and >related) properties >3. The user <-> group relationships are created using the “User Group >Name Attribute” or “Group Member Attribute” properties > > > > The full description of these properties is available in the NiFi Admin > Guide [1]. > > > > It looks like you’ve fully specified the User related properties and it > sound like that is working the way you expect. > > > > Next you’ll want to specify the Group properties that NiFi needs to sync > the group records (search base, search filter, etc.) This will be very > similar to your user search, just change the base and filter. This will > create the groups in NiFi, but not the information about which users are in > each group. > > > > For that, you’ll need to specify the User Group Name or Group Member > Attribute properties. Which ones to set and how to configure them depends > on the structure of your LDAP/AD. From the NiFi Admin Guide documentation > [1]: > > > > *User Group Name Attribute - Attribute to use to define group membership > (i.e. memberof). Optional. If not set group membership will not be > calculated through the users. Will rely on group membership being defined > through 'Group Member Attribute' if set. The value of this property is the > name of the attribute in the user ldap entry that associates them with a > group. The value of that user attribute could be a dn or group name for > instance. What value is expected is configured in the 'User Group Name > Attribute - Referenced Group Attribute'.* > > > > *Group Member Attribute - Attribute to use to define group membership > (i.e. member). Optional. If not set group membership will not be calculated > through the groups. Will rely on group membership being defined through > 'User Group Name Attribute' if set. The value of this property is the name > of the attribute in the group ldap entry that associates them with a user. > The value of that group attribute could be a dn or memberUid for instance. > What value is expected is configured in the 'Group Member Attribute - > Referenced User Attribute'. (i.e. member: cn=User 1,ou=users,o=nifi vs. > memberUid: user1)* > > > > [1] https://nifi.apache.org/docs/nifi-docs/html/administration- > guide.html#authorizers-setup > > > > --- > > > > Here is just one example of how this might work. > > > > For this directory structure (in LDIF format): > > > # define dc=example,dc=com, etc ... > > > > dn: uid=user1,dc=example,dc=com > sn: User1 > objectClass: inetOrgPerson > objectClass: organizationalPerson > objectClass: person > objectClass: top > uid: user1 > cn: User 1 > > > > dn: uid=user2,dc=example,dc=com > sn: User2 > objectClass: inetOrgPerson > objectClass: organizationalPerson > objectClass: person > objectClass: top > uid: user1 > cn: User 2 > > > dn: ou=group1,dc=example,dc=com > uniqueMember: uid=user1,dc=example,dc=com > uniqueMember: uid=user2,dc=example,dc=com > ou: group1 > cn: Group 1 > objectClass: groupOfUniqueNames > objectClass: top > > > > The ldap-user-group-provider in authorizers.xml would look like this: > > > > <*userGroupProvider*> > <*identifier*>ldap-user-group-provider > > <*class*>org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider > > <*property **name**="User Search Base"*>dc=example,dc=com > <*property **name**="User Object Class"*>person > <*property **name**="User Search Scope"*>ONE_LEVEL > <*property **name**="User Search Filter"*>(uid=*) > <*property **name**="User Identity Attribute"*>uid > > > **<*property > **name**="Group Search Base"*>dc=example,dc=com > <*property **name**="Group Object Class"*>groupOfUniqueNames > <*property **name**="Group Search Scope"*>ONE_LEVEL > <*property
Re: LDAP/AD User groups
Whoops, in my example, disregard the class for the ldap-user-group-provider (org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider). I took that snippet from one of my NiFi Registry configuration files, which is the same as NiFi except for the class name ☺. Just mentioning that so it doesn’t confuse anyone or you don’t copy/paste from my example. From: Kevin Doran Reply-To: Date: Tuesday, June 5, 2018 at 23:52 To: Subject: Re: LDAP/AD User groups Hi Martijn, You’re really close to having this configured correctly. Basically, you just need to set a few more Group properties in your ldap-user-group-provider config. The way user and group loading works in NiFI: The users are synced from LDAP using the “User Search *” (and related) properties The groups are synced from LDAP using the “Group Search * (and related) properties The user <-> group relationships are created using the “User Group Name Attribute” or “Group Member Attribute” properties The full description of these properties is available in the NiFi Admin Guide [1]. It looks like you’ve fully specified the User related properties and it sound like that is working the way you expect. Next you’ll want to specify the Group properties that NiFi needs to sync the group records (search base, search filter, etc.) This will be very similar to your user search, just change the base and filter. This will create the groups in NiFi, but not the information about which users are in each group. For that, you’ll need to specify the User Group Name or Group Member Attribute properties. Which ones to set and how to configure them depends on the structure of your LDAP/AD. From the NiFi Admin Guide documentation [1]: User Group Name Attribute - Attribute to use to define group membership (i.e. memberof). Optional. If not set group membership will not be calculated through the users. Will rely on group membership being defined through 'Group Member Attribute' if set. The value of this property is the name of the attribute in the user ldap entry that associates them with a group. The value of that user attribute could be a dn or group name for instance. What value is expected is configured in the 'User Group Name Attribute - Referenced Group Attribute'. Group Member Attribute - Attribute to use to define group membership (i.e. member). Optional. If not set group membership will not be calculated through the groups. Will rely on group membership being defined through 'User Group Name Attribute' if set. The value of this property is the name of the attribute in the group ldap entry that associates them with a user. The value of that group attribute could be a dn or memberUid for instance. What value is expected is configured in the 'Group Member Attribute - Referenced User Attribute'. (i.e. member: cn=User 1,ou=users,o=nifi vs. memberUid: user1) [1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#authorizers-setup --- Here is just one example of how this might work. For this directory structure (in LDIF format): # define dc=example,dc=com, etc ... dn: uid=user1,dc=example,dc=com sn: User1 objectClass: inetOrgPerson objectClass: organizationalPerson objectClass: person objectClass: top uid: user1 cn: User 1 dn: uid=user2,dc=example,dc=com sn: User2 objectClass: inetOrgPerson objectClass: organizationalPerson objectClass: person objectClass: top uid: user1 cn: User 2 dn: ou=group1,dc=example,dc=com uniqueMember: uid=user1,dc=example,dc=com uniqueMember: uid=user2,dc=example,dc=com ou: group1 cn: Group 1 objectClass: groupOfUniqueNames objectClass: top The ldap-user-group-provider in authorizers.xml would look like this: ldap-user-group-provider org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider dc=example,dc=com person ONE_LEVEL (uid=*) uid dc=example,dc=com groupOfUniqueNames ONE_LEVEL (ou=*) ou uniqueMember This is just an example of course. Using this properties depends on your directory. There will be some unused properties (this is because the configuration has to be flexible enough to support almost any directory structure), but make sure something is defining group membership. Hope this helps solve your issue! Regards, Kevin From: Martijn Dekkers Reply-To: Date: Tuesday, June 5, 2018 at 22:01 To: Subject: LDAP/AD User groups Hello all, I have set up a secure NiFi that works using an AD LDAP server. I can log in with different users, and authenticate, assign policies - all great. The one thing I am not managing to get to work at all is groups and group membership. Users that are part of the NiFiUsers group show up, and no others - as I want it to be. Unfortunately, the groups defined in AD (in the correct OU) just don't show, and in the userlist in NiFi "Member of:" is empty for the listed users.
Re: LDAP/AD User groups
Hi Martijn, You’re really close to having this configured correctly. Basically, you just need to set a few more Group properties in your ldap-user-group-provider config. The way user and group loading works in NiFI: The users are synced from LDAP using the “User Search *” (and related) properties The groups are synced from LDAP using the “Group Search * (and related) properties The user <-> group relationships are created using the “User Group Name Attribute” or “Group Member Attribute” properties The full description of these properties is available in the NiFi Admin Guide [1]. It looks like you’ve fully specified the User related properties and it sound like that is working the way you expect. Next you’ll want to specify the Group properties that NiFi needs to sync the group records (search base, search filter, etc.) This will be very similar to your user search, just change the base and filter. This will create the groups in NiFi, but not the information about which users are in each group. For that, you’ll need to specify the User Group Name or Group Member Attribute properties. Which ones to set and how to configure them depends on the structure of your LDAP/AD. From the NiFi Admin Guide documentation [1]: User Group Name Attribute - Attribute to use to define group membership (i.e. memberof). Optional. If not set group membership will not be calculated through the users. Will rely on group membership being defined through 'Group Member Attribute' if set. The value of this property is the name of the attribute in the user ldap entry that associates them with a group. The value of that user attribute could be a dn or group name for instance. What value is expected is configured in the 'User Group Name Attribute - Referenced Group Attribute'. Group Member Attribute - Attribute to use to define group membership (i.e. member). Optional. If not set group membership will not be calculated through the groups. Will rely on group membership being defined through 'User Group Name Attribute' if set. The value of this property is the name of the attribute in the group ldap entry that associates them with a user. The value of that group attribute could be a dn or memberUid for instance. What value is expected is configured in the 'Group Member Attribute - Referenced User Attribute'. (i.e. member: cn=User 1,ou=users,o=nifi vs. memberUid: user1) [1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#authorizers-setup --- Here is just one example of how this might work. For this directory structure (in LDIF format): # define dc=example,dc=com, etc ... dn: uid=user1,dc=example,dc=com sn: User1 objectClass: inetOrgPerson objectClass: organizationalPerson objectClass: person objectClass: top uid: user1 cn: User 1 dn: uid=user2,dc=example,dc=com sn: User2 objectClass: inetOrgPerson objectClass: organizationalPerson objectClass: person objectClass: top uid: user1 cn: User 2 dn: ou=group1,dc=example,dc=com uniqueMember: uid=user1,dc=example,dc=com uniqueMember: uid=user2,dc=example,dc=com ou: group1 cn: Group 1 objectClass: groupOfUniqueNames objectClass: top The ldap-user-group-provider in authorizers.xml would look like this: ldap-user-group-provider org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider dc=example,dc=com person ONE_LEVEL (uid=*) uid dc=example,dc=com groupOfUniqueNames ONE_LEVEL (ou=*) ou uniqueMember This is just an example of course. Using this properties depends on your directory. There will be some unused properties (this is because the configuration has to be flexible enough to support almost any directory structure), but make sure something is defining group membership. Hope this helps solve your issue! Regards, Kevin From: Martijn Dekkers Reply-To: Date: Tuesday, June 5, 2018 at 22:01 To: Subject: LDAP/AD User groups Hello all, I have set up a secure NiFi that works using an AD LDAP server. I can log in with different users, and authenticate, assign policies - all great. The one thing I am not managing to get to work at all is groups and group membership. Users that are part of the NiFiUsers group show up, and no others - as I want it to be. Unfortunately, the groups defined in AD (in the correct OU) just don't show, and in the userlist in NiFi "Member of:" is empty for the listed users. Any suggestions appreciated. I followed Pierre Villard's excellent guide. Many thanks. Martijn My authorizers.xml: file-user-group-provider org.apache.nifi.authorization.FileUserGroupProvider ./conf/users.xml ldap-user-group-provider org.apache.nifi.ldap.tenants.LdapUserGroupProvider START_TLS CN=LDAP MGR,OU=people,OU=Dev,DC=xxx,DC=,DC=net
LDAP/AD User groups
Hello all, I have set up a secure NiFi that works using an AD LDAP server. I can log in with different users, and authenticate, assign policies - all great. The one thing I am not managing to get to work at all is groups and group membership. Users that are part of the NiFiUsers group show up, and no others - as I want it to be. Unfortunately, the groups defined in AD (in the correct OU) just don't show, and in the userlist in NiFi "Member of:" is empty for the listed users. Any suggestions appreciated. I followed Pierre Villard's excellent guide. Many thanks. Martijn My authorizers.xml: file-user-group-provider org.apache.nifi.authorization.FileUserGroupProvider ./conf/users.xml ldap-user-group-provider org.apache.nifi.ldap.tenants.LdapUserGroupProvider START_TLS CN=LDAP MGR,OU=people,OU=Dev,DC=xxx,DC=,DC=net x FOLLOW 10 secs 10 secs ldap://xxx.xxx..net:389 5 mins OU=people,OU=Dev,DC=xxx,DC=,DC=net person ONE_LEVEL (memberOf=CN=NiFiUsers,OU=groups,OU=Dev,DC=xxx,DC=,DC=net) sAMAccountName group ONE_LEVEL cn composite-configurable-user-group-provider org.apache.nifi.authorization.CompositeConfigurableUserGroupProvider file-user-group-provider ldap-user-group-provider file-access-policy-provider org.apache.nifi.authorization.FileAccessPolicyProvider composite-configurable-user-group-provider ./conf/authorizations.xml mdekkers managed-authorizer org.apache.nifi.authorization.StandardManagedAuthorizer file-access-policy-provider
Re: Only get file when a set exists.
Hi Koji,
Some more information from debugging.
I have today deployed Redis since that gives me an easy interface to check
the existence of keys, and found that for files that end up stuck in the
wait queues, the provenance in the Notify queue shows the relevant flowfile
as having arrived, but the relevant key in Redis shows as (nil)
Files that have been processed successfully show a "good" key in Redis.
Thanks,
Martijn
On 5 June 2018 at 06:27, Martijn Dekkers wrote:
> Hello Koji,
>
> Many thanks, apologies for the delay in responding - I had to work on some
> different tasks.
>
> I have followed your advice and have configured a flow accordingly, and on
> the whole the logic works. However, I still see the issue where a set will
> be stuck in the wait queue. I have tracked it down to the instance where
> there is a longer delay between the arrival of ext1 and ext2 files. If I
> pause the appropriate processor that gates the ext2 files, that set will
> get stuck. If all files come through roughly at a similar time, we see no
> issues, and the flow happily runs.
>
> I am not quite sure about the best way to debug this. I have looked at the
> attributes in provenance, and notice that the relevant counter for the
> specific wait processor isn't updated. I am not sure how I can check the
> status of the distributed map cache to see if this might be responsible.
>
> I can share my flowfile, but would have to email it to you directly,
> unfortunately I cannot share the flowfile publicly, and sanitising it to
> the extent that I can publicly share it would be difficult.
>
> Oh, we are using 1.6
>
> Many thanks,
>
> Martijn
>
> On 31 May 2018 at 09:57, Koji Kawamura wrote:
>
>> BTW, which version are you using? I hope it is 1.4.0 or higher. There
>> was an issue having effects to your usage.
>> https://issues.apache.org/jira/browse/NIFI-4028
>>
>> On Thu, May 31, 2018 at 4:51 PM, Koji Kawamura
>> wrote:
>> > HI Martijn,
>> >
>> > I used the filename increment pattern based on your first filename
>> example.
>> > file_123_456_ab.ex1
>> > I increment the 456 part. If it changed, that's fine.
>> >
>> > Your current configurations look like below:
>> > - Given a filename: file_123_type3.ext1
>> > - matched with a RegEx: ^file_(\d{3}_)(\w+)\.ext1$
>> > - groupID will be: 123_ (including the underscore)
>> > - counterName will be: type3
>> >
>> > I was suggesting include the extension to the counterName.
>> > How about changing the RegEx as:
>> > - RegEx: ^file_(\d+)_(\w+\.ext\d)$
>> > - groupID will be: 123
>> > - counterName will be: type3.ext1
>> >
>> > Then you can route type1.ext1 to Wait branch and other 7 to Notify.
>> > In Wait branch, you need 7 Wait processors.
>> >
>> > It would fast to debug if you can share your flow template..
>> >
>> > Thanks,
>> > Koji
>> >
>> > On Thu, May 31, 2018 at 3:15 PM, Martijn Dekkers <
>> mart...@dekkers.org.uk> wrote:
>> >> Thank you Koji,
>> >>
>> >> I have tried once again, using your updated example. Unfortunately,
>> things
>> >> still get stuck at the first Wait processors' wait queue.
>> >> I did notice that the format of the files your example generates is
>> >> different. I will try to clarify:
>> >>
>> >> - 8 files in total:
>> >>
>> >> -- file_123_type1.ext1
>> >> -- file_123_type1.ext2
>> >>
>> >> -- file_123_type2.ext1
>> >> -- file_123_type2.ext2
>> >>
>> >> -- file_123_type3.ext1
>> >> -- file_123_type3.ext2
>> >>
>> >> -- file_123_type4.ext1
>> >> -- file_123_type4.ext2
>> >>
>> >> For each set of 8 files, "file_123" increments, so the first set of 8
>> is
>> >> "file_123", and the next set is "file_124" and so on.
>> >>
>> >> When I look at your example, I notice that at the final step
>> (LogAttribute
>> >> after the FetchFile set) the filenames are file_123_> >> number>.ex(1|2)
>> >>
>> >> My UpdateAttribute before the Notify branch is configured as:
>> >> groupID - ${fileName:replaceFirst('^file_(\d{3}_)(\w+)\.ext1$','$1')}
>> >> counterName - ${fileName:replaceFirst('^file
>> _(\d{3}_)(\w+)\.ext1$','$2')}
>> >>
>> >> The UpdateAttribute before the Wait branch is configured as:
>> >> groupID - ${fileName:replaceFirst('^file_(\d{3}_)(\w+)\.ext1$','$1')}
>> >>
>> >> The 4 Wait processors in the Wait branch are configured as:
>> >> All Wait processors:
>> >> Release Signal Identifier - ${groupID}
>> >>
>> >> For each individual Wait processor:
>> >> Signal Counter Name - type1
>> >> Signal Counter Name - type2
>> >> Signal Counter Name - type3
>> >> Signal Counter Name - type4
>> >>
>> >> I am a bit stumped. The best success we had was a configuration with a
>> >> RouteonAttribute sending each of type1|type2|type3|type4 to their own
>> Wait
>> >> processor, and a similar config for the Notify branch, followed by a
>> final
>> >> Wait/Notify pair that simply ensures we have the correct amount of
>> sets.
>> >>
>> >> This configuration did exactly what we want, but unfortunately we had
>> random
>> >> flowfiles stuck in the waitqueue for no apparent
