smolnar82 opened a new pull request, #1247: URL: https://github.com/apache/knox/pull/1247
[KNOX-3337](https://issues.apache.org/jira/browse/KNOX-3337) - Introduce LDAP Role Lookup Service ## What changes were proposed in this pull request? This PR introduces a new LDAPRolesLookupService to Apache Knox, allowing for dynamic mapping of users and groups to specific roles during the authentication process. This is particularly useful for downstream services that require fine-grained role information (e.g., `platform:admin`, `workspace:viewer`) rather than just raw LDAP groups. Key changes include: - New Service Architecture: Implementation of `LDAPRolesLookupService` with a pluggable strategy pattern. - Lookup Strategies: - `FileBasedLdapRolesLookup`: Maps roles based on a local JSON configuration file. - `RestApiLdapRolesLookup`: Queries an external REST API to retrieve role assignments. - `KnoxLDAPService` Integration: Updated the embedded LDAP service to optionally utilize the role lookup service, allowing it to return resolved roles in place of groups. - `Header Propagation`: Updated AbstractAuthResource (used by `PreAuth` and `ExtAuthz` resources) to populate a new `X-Knox-Actor-Roles` HTTP header when role lookups are successful. - Configuration: Added new `GatewayConfig` properties to configure the lookup strategy, file paths, and API endpoints. ## How was this patch tested? The changes were verified through both existing and newly added unit tests: 1. Unit Tests for Lookup Logic: - `FileBasedLdapRolesLookupTest`: Verified JSON mapping logic for users and groups. - `RestApiLdapRolesLookupTest`: Verified REST API integration using mocked HTTP clients. 2. Service Integration Tests: - `KnoxLDAPServiceTest`: Updated to ensure `getUserGroups` correctly integrates with the lookup service when enabled. 3. Authentication Resource Tests: - `PreAuthResourceTest`: Added `testPopulatingRolesHeader` to verify `X-Knox-Actor-Roles` is set correctly. - `ExtAuthzResourceTest`: Added `testPopulatingRolesHeader` to verify role propagation in external authorization flows. ## Integration Tests No automated test added this time, but I configured Knox with [MockServer](https://www.mock-server.com/where/docker.html) as role lookup provider ``` $ curl -X PUT "http://localhost:55000/mockserver/expectation"; \ -H "Content-Type: application/json" \ -d '{ "httpRequest": { "method": "POST", "path": "/auth/roles" }, "httpResponse": { "statusCode": 200, "headers": { "Content-Type": ["application/json"] }, "body": { "user_id": "alice", "roles": [ { "scope": "platform", "name": "awc-admin" }, { "scope": "ml-workspace-abc", "name": "viewer" } ] } } }' ``` and set the following configs in `gateway-reloadable.xml`: ``` <configuration> <property> <name>gateway.ldap.roles.lookup.strategy</name> <value>rest</value> </property> <property> <name>gateway.ldap.roles.lookup.rest.api.endpoint</name> <value>http://localhost:55000/auth/roles</value> </property> </configuration> ``` Verified that the LDAP roles loookup service (as well as the Knox LDAP service) was reconfigured properly: ``` 2026-06-03 11:31:25,224 INFO knox.gateway (GatewayServer.java:refreshGatewayConfig(275)) - Refreshed gateway config 2026-06-03 11:31:25,225 INFO services.ldap (DefaultLDAPRolesLookupService.java:onGatewayConfigChanged(66)) - Reloading LDAP roles lookup configuration... 2026-06-03 11:31:25,235 INFO services.ldap (DefaultLDAPRolesLookupService.java:logStatus(50)) - LDAP roles lookup is disabled 2026-06-03 11:31:25,235 INFO services.ldap (KnoxLDAPService.java:onGatewayConfigChanged(99)) - Reloading LDAP configuration 2026-06-03 11:31:25,235 INFO services.ldap (KnoxLDAPServerManager.java:stop(193)) - Stopping LDAP service on port 33,390 2026-06-03 11:31:25,246 INFO services.ldap (KnoxLDAPServerManager.java:stop(211)) - LDAP service stopped successfully 2026-06-03 11:31:25,247 INFO services.ldap (BackendFactory.java:createBackend(39)) - Loading backend: ldap (via ServiceLoader) 2026-06-03 11:31:25,247 INFO services.ldap (LdapProxyBackend.java:initialize(148)) - Loading backend: ldap (via Proxying dc=hadoop,dc=apache,dc=org to ldap://localhost:33389 (dc=hadoop,dc=apache,dc=org) with uid attribute using group searches with recursive group resolution (max depth: 3)) ... 2026-06-03 11:32:25,223 INFO knox.gateway (GatewayServer.java:refreshGatewayConfig(275)) - Refreshed gateway config 2026-06-03 11:32:29,362 INFO services.ldap (DefaultLDAPRolesLookupService.java:onGatewayConfigChanged(66)) - Reloading LDAP roles lookup configuration... 2026-06-03 11:32:53,041 INFO services.ldap (DefaultLDAPRolesLookupService.java:logStatus(48)) - LDAP roles lookup is enabled with strategy: rest 2026-06-03 11:32:55,655 INFO services.ldap (KnoxLDAPService.java:onGatewayConfigChanged(99)) - Reloading LDAP configuration 2026-06-03 11:32:55,655 INFO services.ldap (KnoxLDAPServerManager.java:stop(193)) - Stopping LDAP service on port 33,390 2026-06-03 11:32:55,659 INFO services.ldap (KnoxLDAPServerManager.java:stop(211)) - LDAP service stopped successfully 2026-06-03 11:32:55,661 INFO services.ldap (BackendFactory.java:createBackend(39)) - Loading backend: ldap (via ServiceLoader) 2026-06-03 11:32:55,661 INFO services.ldap (LdapProxyBackend.java:initialize(148)) - Loading backend: ldap (via Proxying dc=hadoop,dc=apache,dc=org to ldap://localhost:33389 (dc=hadoop,dc=apache,dc=org) with uid attribute using group searches with recursive group resolution (max depth: 3)) ``` Then issued the following `curl` commands: ``` $ curl -iu recursiveUser:recursiveUser-password http://localhost:8443/gateway/sandbox/auth/api/v1/pre HTTP/1.1 200 OK Date: Wed, 03 Jun 2026 09:33:14 GMT Set-Cookie: KNOXSESSIONID=node016p2r5x9xtju1sxddl59m3a210.node0; Path=/gateway/sandbox; Secure; HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT Set-Cookie: rememberMe=deleteMe; Path=/gateway/sandbox; Max-Age=0; Expires=Tue, 02-Jun-2026 09:33:14 GMT; SameSite=lax X-Knox-Actor-ID: recursiveUser X-Knox-Actor-Roles: platform:awc-admin X-Knox-Actor-Roles: ml-workspace-abc:viewer Content-Length: 0 $ curl -iku admin:admin-password http://localhost:8443/gateway/sandbox/auth/api/v1/pre HTTP/1.1 200 OK Date: Wed, 03 Jun 2026 09:36:34 GMT Set-Cookie: KNOXSESSIONID=node01tgpz4zn4diz91vp1eiwyymtud1.node0; Path=/gateway/sandbox; Secure; HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT Set-Cookie: rememberMe=deleteMe; Path=/gateway/sandbox; Max-Age=0; Expires=Tue, 02-Jun-2026 09:36:34 GMT; SameSite=lax X-Knox-Actor-ID: admin X-Knox-Actor-Roles: platform:awc-admin X-Knox-Actor-Roles: ml-workspace-abc:viewer Content-Length: 0 ``` Checked the logs: ``` 2026-06-03 11:33:14,804 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG knox.gateway (GatewayFilter.java:doFilter(130)) - Received request: GET /auth/api/v1/pre 2026-06-03 11:33:14,813 58981090-581a-40cb-b5c8-782b8fa6455d INFO knox.gateway (KnoxLdapRealm.java:getUserDn(688)) - Computed userDn: uid=recursiveUser,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for principal: recursiveUser 2026-06-03 11:33:14,839 DEBUG services.ldap (GroupLookupInterceptor.java:bind(144)) - LDAP Bind: uid=recursiveUser,ou=people,dc=hadoop,dc=apache,dc=org 2026-06-03 11:33:14,848 DEBUG services.ldap (LdapProxyBackend.java:authenticate(278)) - LDAP authentication succeeded for user: uid=recursiveUser,ou=people,dc=hadoop,dc=apache,dc=org 2026-06-03 11:33:19,007 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG knox.gateway (AuthFilterUtils.java:saveImpersonationProvider(138)) - Creating impersonation provider in sandbox / identity-assertion with prefix hadoop.proxyuser and config {admin.groups=*, admin.hosts=*, impersonation.enabled=true} 2026-06-03 11:33:19,009 58981090-581a-40cb-b5c8-782b8fa6455d INFO knox.gateway (HadoopGroupProviderFilter.java:hadoopGroups(135)) - Using Knox LDAP service to fetch groups... 2026-06-03 11:33:19,018 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG services.ldap (LdapProxyBackend.java:resolveGroupsRecursive(400)) - Recursive group search enabled: true, max depth: 3 2026-06-03 11:33:19,018 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG services.ldap (LdapProxyBackend.java:logRecursiveSearchProgress(515)) - Recursive group search for user recursiveUser found 1 group(s) (level1) at depth 0 2026-06-03 11:33:19,019 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG services.ldap (LdapProxyBackend.java:updateCache(488)) - Added parent cn=level2,ou=groups,dc=hadoop,dc=apache,dc=org to cache for group cn=level1,ou=groups,dc=hadoop,dc=apache,dc=org 2026-06-03 11:33:19,020 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG services.ldap (LdapProxyBackend.java:logRecursiveSearchProgress(515)) - Recursive group search for user recursiveUser found 1 group(s) (level2) at depth 1 2026-06-03 11:33:19,020 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG services.ldap (LdapProxyBackend.java:updateCache(488)) - Added parent cn=level3,ou=groups,dc=hadoop,dc=apache,dc=org to cache for group cn=level2,ou=groups,dc=hadoop,dc=apache,dc=org 2026-06-03 11:33:19,021 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG services.ldap (LdapProxyBackend.java:logRecursiveSearchProgress(515)) - Recursive group search for user recursiveUser found 1 group(s) (level3) at depth 2 2026-06-03 11:33:19,021 58981090-581a-40cb-b5c8-782b8fa6455d WARN services.ldap (LdapProxyBackend.java:resolveGroupsRecursive(458)) - Recursive group search for user recursiveUser reached max depth 3 2026-06-03 11:33:19,021 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG services.ldap (LdapProxyBackend.java:resolveGroupsRecursive(462)) - Recursive group search for user recursiveUser completed. Total groups found: 3 2026-06-03 11:33:19,086 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG services.ldap (KnoxLDAPService.java:getUserGroups(149)) - LDAP roles lookup for user recursiveUser and groups level1,level2,level3 returned roles: platform:awc-admin,ml-workspace-abc:viewer 2026-06-03 11:33:19,086 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG knox.gateway (HadoopGroupProviderFilter.java:mapGroupPrincipals(115)) - Found groups for principal recursiveUser : [platform:awc-admin, ml-workspace-abc:viewer] 2026-06-03 11:33:19,087 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG knox.gateway (VirtualGroupMapper.java:mapGroups(59)) - User recursiveUser (with group(s) [platform:awc-admin, ml-workspace-abc:viewer]) added to group(s) [] 2026-06-03 11:34:07,954 DEBUG services.ldap (GroupLookupInterceptor.java:bind(144)) - LDAP Bind: uid=recursiveUser,ou=people,dc=hadoop,dc=apache,dc=org 2026-06-03 11:34:07,956 DEBUG services.ldap (LdapProxyBackend.java:authenticate(278)) - LDAP authentication succeeded for user: uid=recursiveUser,ou=people,dc=hadoop,dc=apache,dc=org 2026-06-03 11:34:07,958 DEBUG services.ldap (GroupLookupInterceptor.java:search(79)) - LDAP Search: dc=hadoop,dc=apache,dc=org | (|(uid=recursiveUser)(objectClass=referral)) 2026-06-03 11:34:07,959 INFO services.ldap (GroupLookupInterceptor.java:search(119)) - Loaded user from backend: recursiveUser 2026-06-03 11:34:07,962 DEBUG services.ldap (LdapProxyBackend.java:resolveGroupsRecursive(400)) - Recursive group search enabled: true, max depth: 3 2026-06-03 11:34:07,962 DEBUG services.ldap (LdapProxyBackend.java:logRecursiveSearchProgress(515)) - Recursive group search for user recursiveUser found 1 group(s) (level1) at depth 0 2026-06-03 11:34:07,962 DEBUG services.ldap (LdapProxyBackend.java:updateCache(488)) - Added parent cn=level2,ou=groups,dc=hadoop,dc=apache,dc=org to cache for group cn=level1,ou=groups,dc=hadoop,dc=apache,dc=org 2026-06-03 11:34:07,963 DEBUG services.ldap (LdapProxyBackend.java:logRecursiveSearchProgress(515)) - Recursive group search for user recursiveUser found 1 group(s) (level2) at depth 1 2026-06-03 11:34:07,963 DEBUG services.ldap (LdapProxyBackend.java:updateCache(488)) - Added parent cn=level3,ou=groups,dc=hadoop,dc=apache,dc=org to cache for group cn=level2,ou=groups,dc=hadoop,dc=apache,dc=org 2026-06-03 11:34:07,963 DEBUG services.ldap (LdapProxyBackend.java:logRecursiveSearchProgress(515)) - Recursive group search for user recursiveUser found 1 group(s) (level3) at depth 2 2026-06-03 11:34:07,963 WARN services.ldap (LdapProxyBackend.java:resolveGroupsRecursive(458)) - Recursive group search for user recursiveUser reached max depth 3 2026-06-03 11:34:07,963 DEBUG services.ldap (LdapProxyBackend.java:resolveGroupsRecursive(462)) - Recursive group search for user recursiveUser completed. Total groups found: 3 ... 2026-06-03 11:36:34,540 e394d849-9a41-4313-9e48-2505599077c4 DEBUG knox.gateway (GatewayFilter.java:doFilter(130)) - Received request: GET /auth/api/v1/pre 2026-06-03 11:36:34,541 e394d849-9a41-4313-9e48-2505599077c4 INFO knox.gateway (KnoxLdapRealm.java:getUserDn(688)) - Computed userDn: uid=admin,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for principal: admin 2026-06-03 11:36:34,543 DEBUG services.ldap (GroupLookupInterceptor.java:bind(144)) - LDAP Bind: uid=admin,ou=people,dc=hadoop,dc=apache,dc=org 2026-06-03 11:36:34,546 DEBUG services.ldap (LdapProxyBackend.java:authenticate(278)) - LDAP authentication succeeded for user: uid=admin,ou=people,dc=hadoop,dc=apache,dc=org 2026-06-03 11:36:34,547 e394d849-9a41-4313-9e48-2505599077c4 INFO knox.gateway (HadoopGroupProviderFilter.java:hadoopGroups(135)) - Using Knox LDAP service to fetch groups... 2026-06-03 11:36:34,549 e394d849-9a41-4313-9e48-2505599077c4 DEBUG services.ldap (LdapProxyBackend.java:resolveGroupsRecursive(400)) - Recursive group search enabled: true, max depth: 3 2026-06-03 11:36:34,549 e394d849-9a41-4313-9e48-2505599077c4 DEBUG services.ldap (LdapProxyBackend.java:logRecursiveSearchProgress(515)) - Recursive group search for user admin found 1 group(s) (admin) at depth 0 2026-06-03 11:36:34,550 e394d849-9a41-4313-9e48-2505599077c4 DEBUG services.ldap (LdapProxyBackend.java:logRecursiveSearchProgress(515)) - Recursive group search for user admin found 0 group(s) () at depth 1 2026-06-03 11:36:34,550 e394d849-9a41-4313-9e48-2505599077c4 DEBUG services.ldap (LdapProxyBackend.java:resolveGroupsRecursive(462)) - Recursive group search for user admin completed. Total groups found: 1 2026-06-03 11:36:34,560 e394d849-9a41-4313-9e48-2505599077c4 DEBUG services.ldap (KnoxLDAPService.java:getUserGroups(149)) - LDAP roles lookup for user admin and groups admin returned roles: platform:awc-admin,ml-workspace-abc:viewer 2026-06-03 11:36:34,561 e394d849-9a41-4313-9e48-2505599077c4 DEBUG knox.gateway (HadoopGroupProviderFilter.java:mapGroupPrincipals(115)) - Found groups for principal admin : [platform:awc-admin, ml-workspace-abc:viewer] 2026-06-03 11:36:34,561 e394d849-9a41-4313-9e48-2505599077c4 DEBUG knox.gateway (VirtualGroupMapper.java:mapGroups(59)) - User admin (with group(s) [platform:awc-admin, ml-workspace-abc:viewer]) added to group(s) [] ``` I also ran the new KnoxCLI command: ``` $ bin/knoxcli.sh ldap-user-groups-test --u recursiveUser Querying KnoxLDAPService for groups of user: recursiveUser recursiveUser is a member of: platform:awc-admin, ml-workspace-abc:viewer ``` ## UI changes N/A -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
