[
https://issues.apache.org/jira/browse/KNOX-3554?focusedWorklogId=1052971&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-1052971
]
ASF GitHub Bot logged work on KNOX-3554:
----------------------------------------
Author: ASF GitHub Bot
Created on: 06/Nov/26 10:38
Start Date: 06/Nov/26 10:38
Worklog Time Spent: 10m
Work Description: smolnar82 opened a new pull request, #1494:
URL: https://github.com/apache/knox/pull/1494
[KNOX-3554](https://issues.apache.org/jira/browse/KNOX-3554) -
HadoopGroupProvider: skip the group lookup for principals known not to exist in
LDAP
## What changes were proposed in this pull request?
Some authenticated principals are known in advance not to exist in LDAP
(e.g. Kubernetes service accounts or other workload identities arriving as
JWTs). `HadoopGroupProvider` still ran a group lookup for them. With
`use.ldap.service=true`, that meant an embedded KnoxLDAP search plus the
backend LDAP round trips, all of which always returned nothing.
This PR adds an optional `HadoopGroupProvider` parameter,
`group.lookup.skip.principal.pattern`:
- It is a regex that must match the whole mapped principal name, i.e. the
name after the principal and lowercase mapping. Principals matching it get no
groups and no LDAP call. This applies to both the Knox LDAP service path and
the Hadoop `LdapGroupsMapping` path.
- `ROLES_LOOKUP_EXECUTED` is not set for skipped principals, so downstream
consumers (e.g. `KNOX-AUTH-SERVICE`) still do their own roles lookup.
- A blank value is ignored. An invalid regex makes the filter's `init()`
fail with a `ServletException`.
- It can be set in the topology or centrally in gateway-site via
`CENTRAL_GROUP_CONFIG_PREFIX`, e.g.
`gateway.group.config.group.lookup.skip.principal.pattern`.
- Logging: an INFO line with the configured pattern when the filter is
initialized, and a DEBUG line for every skipped principal.
Example:
```xml
<param>
<name>group.lookup.skip.principal.pattern</name>
<value>system:serviceaccount:.*</value>
</param>
```
## How was this patch tested?
Unit tests (`HadoopGroupProviderFilterTest`):
- A matching principal skips the lookup on the Knox LDAP service path (a
strict `KnoxLDAPService` mock fails on any call) and on the Hadoop groups path.
In both cases, no `ROLES_LOOKUP_EXECUTED` is set.
- A pattern that only partially matches the principal does not skip the
lookup.
- A blank pattern is ignored.
- An invalid regex fails `init()`.
Manual test on a local Knox install (same as in #1493):
- Setup:
- the embedded LDAP with the `rolesLookup` interceptor and the REST
strategy, pointing to the mock roles API from
`.github/workflows/compose/role-lookup-rest/mock_roles_api.py`;
- a topology with `HeaderPreAuth`, `HadoopGroupProvider`
(`use.ldap.service=true`, `group.lookup.skip.principal.pattern=k8s-sa-.*`) and
`KNOX-AUTH-SERVICE`.
- `k8s-sa-test`: no KnoxLDAP or backend LDAP search was made.
`KNOX-AUTH-SERVICE` did the roles lookup (`"groups":[]`) and returned
`platform:sa-role`.
- `guest` (no match): the LDAP lookup and the interceptor roles lookup ran
as before, and `KNOX-AUTH-SERVICE` skipped its own lookup.
```
$ curl -sik -H 'SM_USER: k8s-sa-test'
https://localhost:8443/gateway/preauth-roles/auth/api/v1/pre | grep -i x-awc
x-awc-username: k8s-sa-test
x-awc-roles: platform:sa-role
$ curl -sik -H 'SM_USER: guest'
https://localhost:8443/gateway/preauth-roles/auth/api/v1/pre | grep -i x-awc
x-awc-username: guest
x-awc-roles: platform:guest-role
2026-11-06 11:33:46,174 a05cc992-5b1b-4504-b374-29a3a090481d DEBUG
knox.gateway (GatewayFilter.java:doFilter(130)) - Received request: GET
/auth/api/v1/pre
2026-11-06 11:33:46,176 a05cc992-5b1b-4504-b374-29a3a090481d INFO
knox.gateway (HadoopGroupProviderFilter.java:init(92)) - The group lookup is
skipped for principals matching k8s-sa-.*
2026-11-06 11:33:46,176 a05cc992-5b1b-4504-b374-29a3a090481d DEBUG
knox.gateway (HadoopGroupProviderFilter.java:mapGroupPrincipals(133)) -
Skipping the group lookup of principal k8s-sa-test: it matches k8s-sa-.*
2026-11-06 11:33:46,176 a05cc992-5b1b-4504-b374-29a3a090481d DEBUG
knox.gateway (VirtualGroupMapper.java:mapGroups(59)) - User k8s-sa-test (with
group(s) []) added to group(s) []
2026-11-06 11:33:46,214 a05cc992-5b1b-4504-b374-29a3a090481d DEBUG
service.auth (AbstractAuthResource.java:lookupRoles(340)) - Looking up the
roles of user k8s-sa-test with groups []
2026-11-06 11:33:46,215 a05cc992-5b1b-4504-b374-29a3a090481d DEBUG
services.ldap (RestApiLdapRolesLookup.java:lookupRoles(59)) - Sending role
lookup request to http://localhost:8090/api/v1/roles/lookup:
{"user_id":"k8s-sa-test","groups":[]}
2026-11-06 11:33:46,218 a05cc992-5b1b-4504-b374-29a3a090481d DEBUG
services.ldap (RestApiLdapRolesLookup.java:lookupRoles(68)) - Received role
lookup response from http://localhost:8090/api/v1/roles/lookup: HTTP 200, body:
{"user_id": "k8s-sa-test", "roles": [{"scope": "platform", "name": "sa-role"}]}
2026-11-06 11:33:46,219 a05cc992-5b1b-4504-b374-29a3a090481d DEBUG
service.auth (AbstractAuthResource.java:lookupRoles(342)) - Found roles for
user k8s-sa-test: [platform:sa-role]
2026-11-06 11:34:20,573 607fd52e-d2bc-4824-82e5-f919e9d03855 DEBUG
knox.gateway (GatewayFilter.java:doFilter(130)) - Received request: GET
/auth/api/v1/pre
2026-11-06 11:34:20,573 607fd52e-d2bc-4824-82e5-f919e9d03855 INFO
knox.gateway (HadoopGroupProviderFilter.java:hadoopGroups(161)) - Using Knox
LDAP service to fetch groups...
2026-11-06 11:34:20,574 607fd52e-d2bc-4824-82e5-f919e9d03855 DEBUG
services.ldap (UserSearchInterceptor.java:search(112)) - LDAP Search:
dc=hadoop,dc=apache,dc=org | (uid=guest) | [*]
2026-11-06 11:34:20,576 607fd52e-d2bc-4824-82e5-f919e9d03855 DEBUG
services.ldap (LdapProxyBackend.java:performPagedSearch(1055)) - LDAP Paged
Search: dc=hadoop,dc=apache,dc=org | (uid=guest), page size 1,000, page 1
2026-11-06 11:34:20,578 607fd52e-d2bc-4824-82e5-f919e9d03855 DEBUG
services.ldap (LdapProxyBackend.java:performPagedSearch(1095)) - LDAP Paged
Search Completed: dc=hadoop,dc=apache,dc=org | (uid=guest). Found 1 entries
2026-11-06 11:34:20,578 607fd52e-d2bc-4824-82e5-f919e9d03855 DEBUG
services.ldap (LdapProxyBackend.java:performPagedSearch(1055)) - LDAP Paged
Search: ou=groups,dc=hadoop,dc=apache,dc=org |
(|(member=uid=guest,ou=people,dc=hadoop,dc=apache,dc=org)(uniqueMember=uid=guest,ou=people,dc=hadoop,dc=apache,dc=org)),
page size 1,000, page 1
2026-11-06 11:34:20,585 607fd52e-d2bc-4824-82e5-f919e9d03855 DEBUG
services.ldap (LdapProxyBackend.java:performPagedSearch(1095)) - LDAP Paged
Search Completed: ou=groups,dc=hadoop,dc=apache,dc=org |
(|(member=uid=guest,ou=people,dc=hadoop,dc=apache,dc=org)(uniqueMember=uid=guest,ou=people,dc=hadoop,dc=apache,dc=org)).
Found 0 entries
2026-11-06 11:34:20,586 607fd52e-d2bc-4824-82e5-f919e9d03855 DEBUG
services.ldap (RestApiLdapRolesLookup.java:lookupRoles(59)) - Sending role
lookup request to http://localhost:8090/api/v1/roles/lookup:
{"user_id":"guest","groups":[]}
2026-11-06 11:34:20,589 607fd52e-d2bc-4824-82e5-f919e9d03855 DEBUG
services.ldap (RestApiLdapRolesLookup.java:lookupRoles(68)) - Received role
lookup response from http://localhost:8090/api/v1/roles/lookup: HTTP 200, body:
{"user_id": "guest", "roles": [{"scope": "platform", "name": "guest-role"}]}
2026-11-06 11:34:20,589 607fd52e-d2bc-4824-82e5-f919e9d03855 DEBUG
knox.gateway (HadoopGroupProviderFilter.java:hadoopGroups(166)) - The Knox LDAP
service looked up the roles of principal guest; marking the roles lookup as
executed for this request
2026-11-06 11:34:20,589 607fd52e-d2bc-4824-82e5-f919e9d03855 DEBUG
knox.gateway (HadoopGroupProviderFilter.java:mapGroupPrincipals(141)) - Found
groups for principal guest : [platform:guest-role]
2026-11-06 11:34:20,589 607fd52e-d2bc-4824-82e5-f919e9d03855 DEBUG
knox.gateway (VirtualGroupMapper.java:mapGroups(59)) - User guest (with
group(s) [platform:guest-role]) added to group(s) []
2026-11-06 11:34:20,590 607fd52e-d2bc-4824-82e5-f919e9d03855 DEBUG
service.auth (AbstractAuthResource.java:lookupRoles(338)) - The roles of user
guest were already looked up by the identity assertion provider; skipping the
roles lookup
```
## Integration Tests
No integration test added. The unit tests cover the new parameter, and it
was also verified manually against the mock roles REST API.
Issue Time Tracking
-------------------
Worklog Id: (was: 1052971)
Remaining Estimate: 0h
Time Spent: 10m
> HadoopGroupProvider: skip the group lookup for principals known not to exist
> in LDAP
> ------------------------------------------------------------------------------------
>
> Key: KNOX-3554
> URL: https://issues.apache.org/jira/browse/KNOX-3554
> Project: Apache Knox
> Issue Type: Improvement
> Components: HadoopGroupProvider
> Affects Versions: 3.0.0
> Reporter: Sandor Molnar
> Assignee: Sandor Molnar
> Priority: Major
> Fix For: 3.1.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Some authenticated principals are known in advance not to exist in LDAP (e.g.
> Kubernetes service accounts or other workload identities arriving as JWTs).
> {{HadoopGroupProvider}} still runs a group lookup for them. With
> {{use.ldap.service=true}} that means an embedded KnoxLDAP search plus a round
> trip to the backend LDAP, which always returns nothing.
> Add a new optional {{HadoopGroupProvider}} parameter,
> {{group.lookup.skip.principal.pattern}} (a regex, full match against the
> mapped principal name). Principals matching it get no groups and no LDAP
> call, on both the Knox LDAP service and the Hadoop {{LdapGroupsMapping}}
> paths. Like any other provider parameter, it can be set centrally in
> gateway-site via {{CENTRAL_GROUP_CONFIG_PREFIX}}, e.g.
> {{gateway.group.config.group.lookup.skip.principal.pattern}}.
> {{ROLES_LOOKUP_EXECUTED}} is not set for skipped principals, so downstream
> consumers (e.g. {{KNOX-AUTH-SERVICE}}) still do their own roles lookup.
> Example:
> {code:xml}
> <param>
> <name>group.lookup.skip.principal.pattern</name>
> <value>system:serviceaccount:.*</value>
> </param>
> {code}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
