[
https://issues.apache.org/jira/browse/KNOX-3337?focusedWorklogId=1023496&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-1023496
]
ASF GitHub Bot logged work on KNOX-3337:
----------------------------------------
Author: ASF GitHub Bot
Created on: 03/Jun/26 12:24
Start Date: 03/Jun/26 12:24
Worklog Time Spent: 10m
Work Description: smolnar82 opened a new pull request, #1247:
URL: https://github.com/apache/knox/pull/1247
[KNOX-3337](https://issues.apache.org/jira/browse/KNOX-3337) - Introduce
LDAP Role Lookup Service
## What changes were proposed in this pull request?
This PR introduces a new LDAPRolesLookupService to Apache Knox, allowing for
dynamic mapping of users and groups to specific roles during the authentication
process. This is particularly useful for downstream services that require
fine-grained role information (e.g., `platform:admin`, `workspace:viewer`)
rather than just raw LDAP groups.
Key changes include:
- New Service Architecture: Implementation of `LDAPRolesLookupService`
with a pluggable strategy pattern.
- Lookup Strategies:
- `FileBasedLdapRolesLookup`: Maps roles based on a local JSON
configuration file.
- `RestApiLdapRolesLookup`: Queries an external REST API to retrieve
role assignments.
- `KnoxLDAPService` Integration: Updated the embedded LDAP service to
optionally utilize the role lookup service, allowing it to return resolved
roles in place of groups.
- `Header Propagation`: Updated AbstractAuthResource (used by `PreAuth`
and `ExtAuthz` resources) to populate a new `X-Knox-Actor-Roles` HTTP header
when role lookups are successful.
- Configuration: Added new `GatewayConfig` properties to configure the
lookup strategy, file paths, and API endpoints.
## How was this patch tested?
The changes were verified through both existing and newly added unit tests:
1. Unit Tests for Lookup Logic:
- `FileBasedLdapRolesLookupTest`: Verified JSON mapping logic for users
and groups.
- `RestApiLdapRolesLookupTest`: Verified REST API integration using mocked
HTTP clients.
2. Service Integration Tests:
- `KnoxLDAPServiceTest`: Updated to ensure `getUserGroups` correctly
integrates with the lookup service when enabled.
3. Authentication Resource Tests:
- `PreAuthResourceTest`: Added `testPopulatingRolesHeader` to verify
`X-Knox-Actor-Roles` is set correctly.
- `ExtAuthzResourceTest`: Added `testPopulatingRolesHeader` to verify role
propagation in external authorization flows.
## Integration Tests
No automated test added this time, but I configured Knox with
[MockServer](https://www.mock-server.com/where/docker.html) as role lookup
provider
```
$ curl -X PUT "http://localhost:55000/mockserver/expectation"; \
-H "Content-Type: application/json" \
-d '{
"httpRequest": {
"method": "POST",
"path": "/auth/roles"
},
"httpResponse": {
"statusCode": 200,
"headers": {
"Content-Type": ["application/json"]
},
"body": {
"user_id": "alice",
"roles": [
{
"scope": "platform",
"name": "awc-admin"
},
{
"scope": "ml-workspace-abc",
"name": "viewer"
}
]
}
}
}'
```
and set the following configs in `gateway-reloadable.xml`:
```
<configuration>
<property>
<name>gateway.ldap.roles.lookup.strategy</name>
<value>rest</value>
</property>
<property>
<name>gateway.ldap.roles.lookup.rest.api.endpoint</name>
<value>http://localhost:55000/auth/roles</value>
</property>
</configuration>
```
Verified that the LDAP roles loookup service (as well as the Knox LDAP
service) was reconfigured properly:
```
2026-06-03 11:31:25,224 INFO knox.gateway
(GatewayServer.java:refreshGatewayConfig(275)) - Refreshed gateway config
2026-06-03 11:31:25,225 INFO services.ldap
(DefaultLDAPRolesLookupService.java:onGatewayConfigChanged(66)) - Reloading
LDAP roles lookup configuration...
2026-06-03 11:31:25,235 INFO services.ldap
(DefaultLDAPRolesLookupService.java:logStatus(50)) - LDAP roles lookup is
disabled
2026-06-03 11:31:25,235 INFO services.ldap
(KnoxLDAPService.java:onGatewayConfigChanged(99)) - Reloading LDAP configuration
2026-06-03 11:31:25,235 INFO services.ldap
(KnoxLDAPServerManager.java:stop(193)) - Stopping LDAP service on port 33,390
2026-06-03 11:31:25,246 INFO services.ldap
(KnoxLDAPServerManager.java:stop(211)) - LDAP service stopped successfully
2026-06-03 11:31:25,247 INFO services.ldap
(BackendFactory.java:createBackend(39)) - Loading backend: ldap (via
ServiceLoader)
2026-06-03 11:31:25,247 INFO services.ldap
(LdapProxyBackend.java:initialize(148)) - Loading backend: ldap (via Proxying
dc=hadoop,dc=apache,dc=org to ldap://localhost:33389
(dc=hadoop,dc=apache,dc=org) with uid attribute using group searches with
recursive group resolution (max depth: 3))
...
2026-06-03 11:32:25,223 INFO knox.gateway
(GatewayServer.java:refreshGatewayConfig(275)) - Refreshed gateway config
2026-06-03 11:32:29,362 INFO services.ldap
(DefaultLDAPRolesLookupService.java:onGatewayConfigChanged(66)) - Reloading
LDAP roles lookup configuration...
2026-06-03 11:32:53,041 INFO services.ldap
(DefaultLDAPRolesLookupService.java:logStatus(48)) - LDAP roles lookup is
enabled with strategy: rest
2026-06-03 11:32:55,655 INFO services.ldap
(KnoxLDAPService.java:onGatewayConfigChanged(99)) - Reloading LDAP configuration
2026-06-03 11:32:55,655 INFO services.ldap
(KnoxLDAPServerManager.java:stop(193)) - Stopping LDAP service on port 33,390
2026-06-03 11:32:55,659 INFO services.ldap
(KnoxLDAPServerManager.java:stop(211)) - LDAP service stopped successfully
2026-06-03 11:32:55,661 INFO services.ldap
(BackendFactory.java:createBackend(39)) - Loading backend: ldap (via
ServiceLoader)
2026-06-03 11:32:55,661 INFO services.ldap
(LdapProxyBackend.java:initialize(148)) - Loading backend: ldap (via Proxying
dc=hadoop,dc=apache,dc=org to ldap://localhost:33389
(dc=hadoop,dc=apache,dc=org) with uid attribute using group searches with
recursive group resolution (max depth: 3))
```
Then issued the following `curl` commands:
```
$ curl -iu recursiveUser:recursiveUser-password
http://localhost:8443/gateway/sandbox/auth/api/v1/pre
HTTP/1.1 200 OK
Date: Wed, 03 Jun 2026 09:33:14 GMT
Set-Cookie: KNOXSESSIONID=node016p2r5x9xtju1sxddl59m3a210.node0;
Path=/gateway/sandbox; Secure; HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Set-Cookie: rememberMe=deleteMe; Path=/gateway/sandbox; Max-Age=0;
Expires=Tue, 02-Jun-2026 09:33:14 GMT; SameSite=lax
X-Knox-Actor-ID: recursiveUser
X-Knox-Actor-Roles: platform:awc-admin
X-Knox-Actor-Roles: ml-workspace-abc:viewer
Content-Length: 0
$ curl -iku admin:admin-password
http://localhost:8443/gateway/sandbox/auth/api/v1/pre
HTTP/1.1 200 OK
Date: Wed, 03 Jun 2026 09:36:34 GMT
Set-Cookie: KNOXSESSIONID=node01tgpz4zn4diz91vp1eiwyymtud1.node0;
Path=/gateway/sandbox; Secure; HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Set-Cookie: rememberMe=deleteMe; Path=/gateway/sandbox; Max-Age=0;
Expires=Tue, 02-Jun-2026 09:36:34 GMT; SameSite=lax
X-Knox-Actor-ID: admin
X-Knox-Actor-Roles: platform:awc-admin
X-Knox-Actor-Roles: ml-workspace-abc:viewer
Content-Length: 0
```
Checked the logs:
```
2026-06-03 11:33:14,804 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG
knox.gateway (GatewayFilter.java:doFilter(130)) - Received request: GET
/auth/api/v1/pre
2026-06-03 11:33:14,813 58981090-581a-40cb-b5c8-782b8fa6455d INFO
knox.gateway (KnoxLdapRealm.java:getUserDn(688)) - Computed userDn:
uid=recursiveUser,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for
principal: recursiveUser
2026-06-03 11:33:14,839 DEBUG services.ldap
(GroupLookupInterceptor.java:bind(144)) - LDAP Bind:
uid=recursiveUser,ou=people,dc=hadoop,dc=apache,dc=org
2026-06-03 11:33:14,848 DEBUG services.ldap
(LdapProxyBackend.java:authenticate(278)) - LDAP authentication succeeded for
user: uid=recursiveUser,ou=people,dc=hadoop,dc=apache,dc=org
2026-06-03 11:33:19,007 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG
knox.gateway (AuthFilterUtils.java:saveImpersonationProvider(138)) - Creating
impersonation provider in sandbox / identity-assertion with prefix
hadoop.proxyuser and config {admin.groups=*, admin.hosts=*,
impersonation.enabled=true}
2026-06-03 11:33:19,009 58981090-581a-40cb-b5c8-782b8fa6455d INFO
knox.gateway (HadoopGroupProviderFilter.java:hadoopGroups(135)) - Using Knox
LDAP service to fetch groups...
2026-06-03 11:33:19,018 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG
services.ldap (LdapProxyBackend.java:resolveGroupsRecursive(400)) - Recursive
group search enabled: true, max depth: 3
2026-06-03 11:33:19,018 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG
services.ldap (LdapProxyBackend.java:logRecursiveSearchProgress(515)) -
Recursive group search for user recursiveUser found 1 group(s) (level1) at
depth 0
2026-06-03 11:33:19,019 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG
services.ldap (LdapProxyBackend.java:updateCache(488)) - Added parent
cn=level2,ou=groups,dc=hadoop,dc=apache,dc=org to cache for group
cn=level1,ou=groups,dc=hadoop,dc=apache,dc=org
2026-06-03 11:33:19,020 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG
services.ldap (LdapProxyBackend.java:logRecursiveSearchProgress(515)) -
Recursive group search for user recursiveUser found 1 group(s) (level2) at
depth 1
2026-06-03 11:33:19,020 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG
services.ldap (LdapProxyBackend.java:updateCache(488)) - Added parent
cn=level3,ou=groups,dc=hadoop,dc=apache,dc=org to cache for group
cn=level2,ou=groups,dc=hadoop,dc=apache,dc=org
2026-06-03 11:33:19,021 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG
services.ldap (LdapProxyBackend.java:logRecursiveSearchProgress(515)) -
Recursive group search for user recursiveUser found 1 group(s) (level3) at
depth 2
2026-06-03 11:33:19,021 58981090-581a-40cb-b5c8-782b8fa6455d WARN
services.ldap (LdapProxyBackend.java:resolveGroupsRecursive(458)) - Recursive
group search for user recursiveUser reached max depth 3
2026-06-03 11:33:19,021 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG
services.ldap (LdapProxyBackend.java:resolveGroupsRecursive(462)) - Recursive
group search for user recursiveUser completed. Total groups found: 3
2026-06-03 11:33:19,086 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG
services.ldap (KnoxLDAPService.java:getUserGroups(149)) - LDAP roles lookup for
user recursiveUser and groups level1,level2,level3 returned roles:
platform:awc-admin,ml-workspace-abc:viewer
2026-06-03 11:33:19,086 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG
knox.gateway (HadoopGroupProviderFilter.java:mapGroupPrincipals(115)) - Found
groups for principal recursiveUser : [platform:awc-admin,
ml-workspace-abc:viewer]
2026-06-03 11:33:19,087 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG
knox.gateway (VirtualGroupMapper.java:mapGroups(59)) - User recursiveUser (with
group(s) [platform:awc-admin, ml-workspace-abc:viewer]) added to group(s) []
2026-06-03 11:34:07,954 DEBUG services.ldap
(GroupLookupInterceptor.java:bind(144)) - LDAP Bind:
uid=recursiveUser,ou=people,dc=hadoop,dc=apache,dc=org
2026-06-03 11:34:07,956 DEBUG services.ldap
(LdapProxyBackend.java:authenticate(278)) - LDAP authentication succeeded for
user: uid=recursiveUser,ou=people,dc=hadoop,dc=apache,dc=org
2026-06-03 11:34:07,958 DEBUG services.ldap
(GroupLookupInterceptor.java:search(79)) - LDAP Search:
dc=hadoop,dc=apache,dc=org | (|(uid=recursiveUser)(objectClass=referral))
2026-06-03 11:34:07,959 INFO services.ldap
(GroupLookupInterceptor.java:search(119)) - Loaded user from backend:
recursiveUser
2026-06-03 11:34:07,962 DEBUG services.ldap
(LdapProxyBackend.java:resolveGroupsRecursive(400)) - Recursive group search
enabled: true, max depth: 3
2026-06-03 11:34:07,962 DEBUG services.ldap
(LdapProxyBackend.java:logRecursiveSearchProgress(515)) - Recursive group
search for user recursiveUser found 1 group(s) (level1) at depth 0
2026-06-03 11:34:07,962 DEBUG services.ldap
(LdapProxyBackend.java:updateCache(488)) - Added parent
cn=level2,ou=groups,dc=hadoop,dc=apache,dc=org to cache for group
cn=level1,ou=groups,dc=hadoop,dc=apache,dc=org
2026-06-03 11:34:07,963 DEBUG services.ldap
(LdapProxyBackend.java:logRecursiveSearchProgress(515)) - Recursive group
search for user recursiveUser found 1 group(s) (level2) at depth 1
2026-06-03 11:34:07,963 DEBUG services.ldap
(LdapProxyBackend.java:updateCache(488)) - Added parent
cn=level3,ou=groups,dc=hadoop,dc=apache,dc=org to cache for group
cn=level2,ou=groups,dc=hadoop,dc=apache,dc=org
2026-06-03 11:34:07,963 DEBUG services.ldap
(LdapProxyBackend.java:logRecursiveSearchProgress(515)) - Recursive group
search for user recursiveUser found 1 group(s) (level3) at depth 2
2026-06-03 11:34:07,963 WARN services.ldap
(LdapProxyBackend.java:resolveGroupsRecursive(458)) - Recursive group search
for user recursiveUser reached max depth 3
2026-06-03 11:34:07,963 DEBUG services.ldap
(LdapProxyBackend.java:resolveGroupsRecursive(462)) - Recursive group search
for user recursiveUser completed. Total groups found: 3
...
2026-06-03 11:36:34,540 e394d849-9a41-4313-9e48-2505599077c4 DEBUG
knox.gateway (GatewayFilter.java:doFilter(130)) - Received request: GET
/auth/api/v1/pre
2026-06-03 11:36:34,541 e394d849-9a41-4313-9e48-2505599077c4 INFO
knox.gateway (KnoxLdapRealm.java:getUserDn(688)) - Computed userDn:
uid=admin,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for principal:
admin
2026-06-03 11:36:34,543 DEBUG services.ldap
(GroupLookupInterceptor.java:bind(144)) - LDAP Bind:
uid=admin,ou=people,dc=hadoop,dc=apache,dc=org
2026-06-03 11:36:34,546 DEBUG services.ldap
(LdapProxyBackend.java:authenticate(278)) - LDAP authentication succeeded for
user: uid=admin,ou=people,dc=hadoop,dc=apache,dc=org
2026-06-03 11:36:34,547 e394d849-9a41-4313-9e48-2505599077c4 INFO
knox.gateway (HadoopGroupProviderFilter.java:hadoopGroups(135)) - Using Knox
LDAP service to fetch groups...
2026-06-03 11:36:34,549 e394d849-9a41-4313-9e48-2505599077c4 DEBUG
services.ldap (LdapProxyBackend.java:resolveGroupsRecursive(400)) - Recursive
group search enabled: true, max depth: 3
2026-06-03 11:36:34,549 e394d849-9a41-4313-9e48-2505599077c4 DEBUG
services.ldap (LdapProxyBackend.java:logRecursiveSearchProgress(515)) -
Recursive group search for user admin found 1 group(s) (admin) at depth 0
2026-06-03 11:36:34,550 e394d849-9a41-4313-9e48-2505599077c4 DEBUG
services.ldap (LdapProxyBackend.java:logRecursiveSearchProgress(515)) -
Recursive group search for user admin found 0 group(s) () at depth 1
2026-06-03 11:36:34,550 e394d849-9a41-4313-9e48-2505599077c4 DEBUG
services.ldap (LdapProxyBackend.java:resolveGroupsRecursive(462)) - Recursive
group search for user admin completed. Total groups found: 1
2026-06-03 11:36:34,560 e394d849-9a41-4313-9e48-2505599077c4 DEBUG
services.ldap (KnoxLDAPService.java:getUserGroups(149)) - LDAP roles lookup for
user admin and groups admin returned roles:
platform:awc-admin,ml-workspace-abc:viewer
2026-06-03 11:36:34,561 e394d849-9a41-4313-9e48-2505599077c4 DEBUG
knox.gateway (HadoopGroupProviderFilter.java:mapGroupPrincipals(115)) - Found
groups for principal admin : [platform:awc-admin, ml-workspace-abc:viewer]
2026-06-03 11:36:34,561 e394d849-9a41-4313-9e48-2505599077c4 DEBUG
knox.gateway (VirtualGroupMapper.java:mapGroups(59)) - User admin (with
group(s) [platform:awc-admin, ml-workspace-abc:viewer]) added to group(s) []
```
I also ran the new KnoxCLI command:
```
$ bin/knoxcli.sh ldap-user-groups-test --u recursiveUser
Querying KnoxLDAPService for groups of user: recursiveUser
recursiveUser is a member of: platform:awc-admin, ml-workspace-abc:viewer
```
## UI changes
N/A
Issue Time Tracking
-------------------
Worklog Id: (was: 1023496)
Remaining Estimate: 0h
Time Spent: 10m
> Enhance KnoxLdapService with Pluggable Role Lookup Support
> ----------------------------------------------------------
>
> Key: KNOX-3337
> URL: https://issues.apache.org/jira/browse/KNOX-3337
> Project: Apache Knox
> Issue Type: Improvement
> Components: Server
> Affects Versions: 3.0.0
> Reporter: Sandor Molnar
> Assignee: Sandor Molnar
> Priority: Critical
> Fix For: 3.0.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, the KnoxLdapService provides user and group information directly
> from its configured LDAP backend. In many deployment scenarios, particularly
> in modern cloud-native environments, there is a requirement to map these
> LDAP-authenticated identities and their group memberships to higher-level
> application roles managed by an external source.
> This JIRA introduces a pluggable role lookup mechanism within KnoxLdapService
> that allows Knox to intercept group resolution and instead populate roles
> from either a local configuration file (for testing purposes) or a remote
> REST API.
> *Key Features*
> 1. *Pluggable Interface:* Introduction of LdapRolesLookup to allow for
> extensible lookup strategies.
> 2. *REST Implementation:* A client implementation that POSTs user ID and
> group lists to a configured endpoint to retrieve role mappings, following a
> specific OpenAPI schema.
> 3. *File-based Implementation:* A JSON-based lookup strategy for static
> environments, using the same data structure as the REST API.
> 4. *Configuration-driven:* New gateway-site.xml properties to toggle
> lookup types and configure endpoints or file paths.
> *Proposed Configuration*
> * {{{}gateway.ldap.roles.lookup.strategy{}}}: Enables the lookup (file or
> rest).
> * {{{}gateway.ldap.roles.lookup.rest.api.endpoint{}}}: The destination URL
> for REST lookups.
> * {{{}gateway.ldap.roles.lookup.file.path{}}}: the file path which points to
> the JSON mapping file.
> *Data Contract:*
> The implementation will exchange JSON payloads containing a user_id and an
> array of groups, receiving a response containing a list of RoleAssignment
> objects (consisting of scope and name).
>
> This enhancement will allow for more dynamic and flexible authorization
> workflows in Knox-managed environments.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
