Sandor Molnar created KNOX-3520:
-----------------------------------
Summary: Token exchange: support multi-hop RFC 8693 delegation by
binding re-exchanged subject tokens to the actor (delegation policy
subjectAudiences)
Key: KNOX-3520
URL: https://issues.apache.org/jira/browse/KNOX-3520
Project: Apache Knox
Issue Type: Bug
Components: JWT, Server
Affects Versions: 3.1.0
Reporter: Sandor Molnar
Assignee: Sandor Molnar
Fix For: 3.1.0
h2. Summary
The RFC 8693 token exchange topology validates every presented JWT (both
{{subject_token}} and {{{}actor_token{}}}) against a single
{{knox.token.audiences}} list. Tokens minted by a delegated exchange carry only
the requested target audience, so they can never be presented again as a
{{{}subject_token{}}}. As a result, multi-hop on-behalf-of chains (A -> B -> C
-> D) stop after the first delegated hop, even though
{{delegation.max.actor.chain.depth}} suggests deeper chains are supported.
The two available workarounds either do not scale (adding the token-service
audience to every downstream token) or remove an important protection (dropping
{{{}knox.token.audiences{}}}, which lets any stolen service-account token,
minted for any audience, act as a delegation actor).
This JIRA proposes per-role audience rules: actor tokens stay strictly bound to
the token-service audience, while a re-exchanged {{subject_token}} is accepted
only when it is addressed to the actor presenting it. The actor's own audiences
are declared in a new delegation policy field, {{{}subjectAudiences{}}}.
h2. Current configuration
{code:xml}
<topology>
<gateway>
<provider>
<role>federation</role>
<name>JWTProvider</name>
<enabled>true</enabled>
<param>
<name>jwt.expected.issuer</name>
<value>http://knox.knox.svc.cluster.local:8443/gateway/knox-knoxidf-tokenexchange</value>
</param>
<param>
<name>knox.token.audiences</name>
<value>knoxidf</value>
</param>
<param>
<name>jwt.instance.key.fallback</name>
<value>true</value>
</param>
<param>
<name>delegation.server.enabled</name>
<value>true</value>
</param>
<param>
<name>delegation.requested.subject.enabled</name>
<value>true</value>
</param>
<param>
<name>delegation.enforce.requested.audience.required</name>
<value>true</value>
</param>
<param>
<name>delegation.enforce.requested.audience.max.one</name>
<value>true</value>
</param>
<param>
<name>token.exchange.same.subject.requested.audience.enabled</name>
<value>true</value>
</param>
</provider>
<provider>
<role>identity-assertion</role>
<name>Default</name>
<enabled>true</enabled>
</provider>
</gateway>
<service>
<role>KNOXIDF</role>
<param>
<name>knoxidf.knox.token.ttl</name>
<value>3600000</value>
</param>
<param>
<name>delegation.max.actor.chain.depth</name>
<value>6</value>
</param>
<param>
<name>knoxidf.knox.token.limit.per.user</name>
<value>500</value>
</param>
<param>
<name>knoxidf.knox.token.issuer</name>
<value>http://knox.knox.svc.cluster.local:8443/gateway/knox-knoxidf-tokenexchange</value>
</param>
<param>
<name>knoxidf.knox.token.audiences</name>
<value>knoxidf</value>
</param>
<param>
<name>knoxidf.knox.token.enable.delegated.auth</name>
<value>true</value>
</param>
<param>
<name>knoxidf.knox.token.audience.validator</name>
<value>passthrough</value>
</param>
</service>
</topology>
{code}
* First-hop subject tokens (user access tokens) are minted by another Knox
topology with {{{}aud=knoxidf{}}}.
* Actor tokens are Kubernetes projected service-account (SA) tokens with
{{{}audience: knoxidf{}}}, from an issuer registered in the Trusted OIDC Issuer
registry.
* The {{passthrough}} audience validator makes the minted token's {{aud}}
exactly the requested {{{}resource{}}}/{{{}audience{}}} values.
h2. Problem: multi-hop delegation does not work
Example chain: user {{alice}} -> {{dataservice}} (SA {{{}ds-sa{}}}) ->
{{mcp-server}} (SA {{{}mcp-sa{}}}) -> {{search-api}} (SA {{{}search-sa{}}}) ->
{{{}vector-db{}}}.
# *Hop 1* ({{{}dataservice{}}} exchanges {{{}alice{}}}'s token,
{{{}resource=[https://svc.cluster.local/tools/mcp-server]{}}}): succeeds. The
minted token T1 has {{{}sub=alice{}}},
{{{}act.sub=system:serviceaccount:dataservice:ds-sa{}}}, and {{aud}} contains
only {{{}[https://svc.cluster.local/tools/mcp-server]{}}}.
# *Hop 2* ({{{}mcp-server{}}} presents T1 as {{subject_token}} and its own SA
token as {{{}actor_token{}}}): rejected with {{{}401{}}}.
{{TokenExchangeHandler#handle}} validates the {{subject_token}} through
{{{}JWTFederationFilter#parseAndValidateJWT{}}}, which applies
{{AbstractJWTFilter#matchesConfiguredAudiences}} against
{{{}knox.token.audiences=knoxidf{}}}. T1 does not carry {{{}knoxidf{}}}.
The same check runs on the {{actor_token}} in
{{{}TokenExchangeHandler#handleDelegationExchange{}}}, so one list governs two
tokens with very different provenance.
h3. Workaround 1: add {{knoxidf}} to every downstream token (does not scale,
weakens binding)
The caller requests {{resource=<target>}} plus {{audience=knoxidf}} on every
hop whose token may be exchanged again. This requires:
* {{{}delegation.enforce.requested.audience.max.one=false{}}};
* {{knoxidf}} as a key in every actor's {{resourcePolicy}} (otherwise
{{evaluate()}} denies with {{{}resource_not_allowed{}}});
* every caller to know in advance whether its callee will re-exchange;
* every downstream service, including third-party tools and MCP servers, to
accept a multi-valued {{aud}} that names the token service.
It also removes recipient binding. Every downstream token becomes a valid
{{subject_token}} for *any* actor whose policy allows the user and the next
resource, not only for the service it was addressed to. A token leaked from
logs, a proxy or a misbehaving client can be exchanged by a different,
authorized-for-that-user workload.
h3. Workaround 2: remove {{knox.token.audiences}} (opens the gate)
With the parameter unset, {{configuredAudiences == null}} and
{{matchesConfiguredAudiences}} accepts *any* audience, for both subject and
actor tokens. Hop 2 then succeeds, but:
* *Any SA token from a trusted issuer, minted for any audience, becomes a
valid {{{}actor_token{}}}.* This includes the automounted token
({{{}aud=[https://kubernetes.default.svc|https://kubernetes.default.svc/]...{}}}),
and tokens a workload sends to Vault or to any other service that accepts SA
tokens. Whoever obtains such a token (the receiving service, its logs, an
intermediary) can exchange it for tokens for every user the SA's policy allows.
* *The same applies to headless exchanges* ({{{}requested_subject{}}}). There
the {{subject_token}} is the actor, and no user token is needed at all. With
{{{}allowHeadlessExchange=true{}}}, a single stolen SA token is enough to mint
tokens for any user in {{{}canActForUsers{}}}, to any resource in
{{{}resourcePolicy{}}}.
* *Same-subject exchanges* accept any SA token and convert it into a
Knox-issued token.
* *Re-exchanged subject tokens are still not bound to their recipient* (same
issue as workaround 1).
Delegation policies cannot compensate. A policy is keyed by actor identity
({{{}ActorIdentity{}}}: {{iss:namespace:serviceaccount}} for {{{}K8S_SA{}}}),
so an SA token minted for Vault and one minted for {{knoxidf}} resolve to the
same policy. Neither {{DelegationPolicy}} nor {{TrustedOidcIssuer}} has any
notion of the presented token's audience.
Stealing the actor's own {{{}knoxidf{}}}-audience token remains a full
compromise of that actor. That case is unavoidable, and the mitigation is
short-lived projected tokens. The goal is to keep every other token useless at
the token endpoint.
h2. Proposal
h3. Audience rules per token role
||#||Token||Rule||Status||Why||
|1|{{{}actor_token{}}}, and the {{subject_token}} of a headless exchange (it is
the actor)|{{aud}} must contain a {{knox.token.audiences}}
value|Existing|Blocks replay of SA tokens minted for other audiences
(kube-apiserver, Vault, ...). Only tokens deliberately minted for the token
service can act.|
|2|{{subject_token}} of a delegated exchange, first hop (e.g. a user access
token)|{{aud}} must contain a {{knox.token.audiences}} value|Existing|Entry
into the delegation system is limited to tokens deliberately minted for the
token service.|
|3|{{subject_token}} of a delegated exchange, later hops (a token minted by
this topology for the actor's service)|Accepted without a
{{knox.token.audiences}} value only if: an {{actor_token}} is present, the
token's {{iss}} is in
{{{}token.exchange.subject.token.policy.audiences.issuers{}}}, and its {{aud}}
intersects the actor policy's {{subjectAudiences}}|*New*|The only relaxation;
it enables multi-hop. A service can re-exchange only a token addressed to
itself, and only while proving its identity with its own rule-1 actor token.
Other actors, even ones authorized for the same user, cannot use it.|
|4|{{subject_token}} of a same-subject exchange|{{aud}} must contain a
{{knox.token.audiences}} value (no exception)|Existing|No policy is consulted
in same-subject exchanges, so there is no recipient binding to fall back on.|
|5|{{subject_token}} of a headless exchange|Must not carry an {{act}} claim
({{{}delegation.headless.reject.subject.with.act{}}})|*New*|Defense in depth: a
delegated token, however it got a {{knoxidf}} audience, must never become a
headless actor, which would mean acting as anyone in the policy without a user
token.|
Applied per exchange type:
* *Delegated* ({{{}actor_token{}}} present): rule 1 on the actor, rule 2 or
rule 3 on the subject.
* *Headless* ({{{}requested_subject{}}}, no actor): rules 1 and 5 on the
{{{}subject_token{}}}. Rule 3 never applies.
* {*}Same-subject{*}: rule 4.
h3. New delegation policy field: {{subjectAudiences}}
An optional set of audiences that identify the actor {*}as a recipient{*}, i.e.
the audiences under which other actors mint tokens for it. Defaults to empty,
which means rule 3 never matches for that actor (fail-safe, backward
compatible).
{code:yaml}
{
"actorAuthority": "K8S_SA",
"actorId": "https://kubernetes.default.svc.cluster.local:tools:mcp-sa";,
"name": "mcp-sa-delegation",
"status": "active",
"allowHeadlessExchange": false,
"canActForUsers": ["alice"],
"canActForGroups": [],
"subjectAudiences": ["https://svc.cluster.local/tools/mcp-server";],
"resourcePolicy": {
"https://svc.cluster.local/search/search-api": []
}
}
{code}
h3. Why {{resourcePolicy}} cannot be reused for this check
{{resourcePolicy}} and {{subjectAudiences}} answer opposite questions:
||Field||Direction||Question||Example for {{mcp-sa}}||
|{{resourcePolicy}}|Outbound|Which services may this actor mint tokens
{_}for{_}?|{{search-api}} (the services it calls)|
|{{subjectAudiences}}|Inbound|Which tokens addressed _to_ this actor may it
present as {{{}subject_token{}}}?|{{mcp-server}} (the service it is)|
# *The legitimate flow fails.* T1's {{aud}} is {{{}mcp-server{}}}, but
{{{}mcp-sa{}}}'s {{resourcePolicy}} lists {{{}search-api{}}}. A service rarely
lists itself as a target.
# *It binds the token to the wrong parties.* The actors that list
{{mcp-server}} in {{resourcePolicy}} are its {_}callers{_}. An "aud in
resourcePolicy" check would let any caller of {{mcp-server}} that obtains T1
exchange it. That is a confused-deputy gap: it proves the presenter may _send
to_ the service, not that it _is_ the service.
# *Making #1 pass widens outbound permissions.* Adding {{mcp-server}} to
{{{}mcp-sa{}}}'s own {{resourcePolicy}} also authorizes {{mcp-sa}} to mint
tokens addressed to itself for any user in its policy, including headless
exchanges when {{{}allowHeadlessExchange=true{}}}.
# *Different administrative constraints.* {{resourcePolicy}} keys must be able
to name services in other namespaces or tenants, because services call across
them. {{subjectAudiences}} can be restricted by admission control or the admin
API to the actor's own namespace, which prevents team A from claiming team B's
audience. One shared field would allow no such restriction.
# {{resourcePolicy}} values are scopes for the outbound target; they have no
meaning for an inbound check.
Nothing in the current policy model links the actor identity to the audience it
serves. {{subjectAudiences}} is that link.
h3. Proposed topology
Unchanged except for the two new JWTProvider parameters (both default to unset
or {{{}false{}}}, which keeps today's behavior):
{code:xml}
<topology>
<gateway>
<provider>
<role>federation</role>
<name>JWTProvider</name>
<enabled>true</enabled>
<param>
<name>jwt.expected.issuer</name>
<value>http://knox.knox.svc.cluster.local:8443/gateway/knox-knoxidf-tokenexchange</value>
</param>
<!-- Rules 1, 2, 4: unchanged, still applies to every actor token and to
non re-exchanged subject tokens. -->
<param>
<name>knox.token.audiences</name>
<value>knoxidf</value>
</param>
<param>
<name>jwt.instance.key.fallback</name>
<value>true</value>
</param>
<!-- NEW, rule 3: on a delegated exchange (actor_token present), a
subject_token issued by one of these
issuers that lacks knox.token.audiences is accepted if its aud
intersects the actor policy's
subjectAudiences. Unset: rule 3 is disabled (today's behavior).
Never applies to actor tokens,
headless or same-subject exchanges. -->
<param>
<name>token.exchange.subject.token.policy.audiences.issuers</name>
<value>http://knox.knox.svc.cluster.local:8443/gateway/knox-knoxidf-tokenexchange</value>
</param>
<!-- NEW, rule 5: reject a headless subject_token that carries an act
claim. -->
<param>
<name>delegation.headless.reject.subject.with.act</name>
<value>true</value>
</param>
<param>
<name>delegation.server.enabled</name>
<value>true</value>
</param>
<param>
<name>delegation.requested.subject.enabled</name>
<value>true</value>
</param>
<param>
<name>delegation.enforce.requested.audience.required</name>
<value>true</value>
</param>
<!-- Can stay true: minted tokens no longer need the token-service
audience to be re-exchanged. -->
<param>
<name>delegation.enforce.requested.audience.max.one</name>
<value>true</value>
</param>
<param>
<name>token.exchange.same.subject.requested.audience.enabled</name>
<value>true</value>
</param>
</provider>
<provider>
<role>identity-assertion</role>
<name>Default</name>
<enabled>true</enabled>
</provider>
</gateway>
<service>
<role>KNOXIDF</role>
<param>
<name>knoxidf.knox.token.ttl</name>
<value>3600000</value>
</param>
<param>
<name>delegation.max.actor.chain.depth</name>
<value>6</value>
</param>
<param>
<name>knoxidf.knox.token.limit.per.user</name>
<value>500</value>
</param>
<param>
<name>knoxidf.knox.token.issuer</name>
<value>http://knox.knox.svc.cluster.local:8443/gateway/knox-knoxidf-tokenexchange</value>
</param>
<param>
<name>knoxidf.knox.token.audiences</name>
<value>knoxidf</value>
</param>
<param>
<name>knoxidf.knox.token.enable.delegated.auth</name>
<value>true</value>
</param>
<param>
<name>knoxidf.knox.token.audience.validator</name>
<value>passthrough</value>
</param>
</service>
</topology>
{code}
h3. Implementation notes
* {{{}TokenExchangeHandler#handle{}}}: the {{subject_token}} audience can no
longer be decided when it is parsed, because the actor and its policy are only
known later. Validate signature, issuer and expiry with the
{{JWTFederationFilter#parseAndValidateJWT}} overload that takes a
{{{}RequestAudienceValidator{}}}, and record whether the
{{knox.token.audiences}} check passed.
** Same-subject or headless and the check failed: reject as today
({{{}401{}}}).
** Delegated and the check failed: only continue if the token's {{iss}} is in
{{{}token.exchange.subject.token.policy.audiences.issuers{}}}. Otherwise reject
as today.
* {{actor_token}} validation is unchanged (rule 1).
* {{PolicyCheckRequest}} carries the subject token's audiences and whether
rule 3 is needed. {{JdbcDelegationPolicyService#evaluate}} denies with a new
reason {{subject_audience_not_allowed}} when they do not intersect
{{{}subjectAudiences{}}}. The client still gets the generic {{{}400
invalid_request{}}}; the reason is audited.
* {{{}DelegationPolicy{}}}: new {{subjectAudiences}} field (DB schema
migration, admin API request/response, validation of non-blank values).
* Headless: when {{{}delegation.headless.reject.subject.with.act=true{}}}, a
{{subject_token}} with an {{act}} claim is rejected (new audit reason
{{{}headless_subject_has_act{}}}).
* Fail-safe defaults: no new parameter set and empty {{subjectAudiences}}
keeps today's behavior.
h3. Related (optional, could be a separate JIRA)
With the {{passthrough}} validator, a same-subject exchange that requests no
audience mints a token with *no* {{aud}} claim ({{{}TokenResource#getJWT{}}}
only sets audiences when the list is non-empty). Such a token cannot be
exchanged again and is rejected by most receivers. Suggestion: {{passthrough}}
should fall back to the configured {{knox.token.audiences}} when the request
names none.
h2. 3-hop example once implemented
Chain: {{alice}} -> {{dataservice}} ({{{}ds-sa{}}}) -> {{mcp-server}}
({{{}mcp-sa{}}}) -> {{search-api}} ({{{}search-sa{}}}) -> {{{}vector-db{}}}.
Delegation policies (only the relevant fields):
{code:yaml}
// ds-sa: the first hop needs no subjectAudiences (the subject token carries
knoxidf, rule 2)
{ "actorId": "https://kubernetes.default.svc.cluster.local:dataservice:ds-sa";,
"canActForUsers": ["alice"],
"subjectAudiences": [],
"resourcePolicy": { "https://svc.cluster.local/tools/mcp-server": [] } }
// mcp-sa
{ "actorId": "https://kubernetes.default.svc.cluster.local:tools:mcp-sa";,
"canActForUsers": ["alice"],
"subjectAudiences": ["https://svc.cluster.local/tools/mcp-server";],
"resourcePolicy": { "https://svc.cluster.local/search/search-api": [] } }
// search-sa
{ "actorId": "https://kubernetes.default.svc.cluster.local:search:search-sa";,
"canActForUsers": ["alice"],
"subjectAudiences": ["https://svc.cluster.local/search/search-api";],
"resourcePolicy": { "https://svc.cluster.local/data/vector-db": [] } }
{code}
h3. Hop 1: {{dataservice}} -> token for {{mcp-server}}
{code:bash}
POST /gateway/knox-knoxidf-tokenexchange/knoxidf/api/v1/token
grant_type=urn:ietf:params:oauth:grant-type:token-exchange
subject_token=<T0> subject_token_type=urn:ietf:params:oauth:token-type:jwt
actor_token=<ds-sa> actor_token_type=urn:ietf:params:oauth:token-type:jwt
resource=https://svc.cluster.local/tools/mcp-server
// subject_token T0 (user access token) -> rule 2: aud contains knoxidf
{ "iss": "http://knox.knox.svc.cluster.local:8443/gateway/knox-tokenexchange";,
"sub": "alice", "aud": ["knoxidf"] }
// actor_token (ds-sa projected SA token) -> rule 1: aud contains knoxidf
{ "iss": "https://kubernetes.default.svc.cluster.local";,
"sub": "system:serviceaccount:dataservice:ds-sa", "aud": ["knoxidf"] }
// policy ds-sa: alice allowed, mcp-server in resourcePolicy -> minted T1
{ "iss":
"http://knox.knox.svc.cluster.local:8443/gateway/knox-knoxidf-tokenexchange";,
"sub": "alice",
"aud": ["https://svc.cluster.local/tools/mcp-server";],
"act": { "sub": "system:serviceaccount:dataservice:ds-sa" } }
{code}
{{dataservice}} calls {{mcp-server}} with T1. {{mcp-server}} performs an
ordinary audience check: its own audience is in {{{}aud{}}}. It does not need
{{knoxidf}} in it.
h3. Hop 2: {{mcp-server}} -> token for {{search-api}}
{code:yaml}
subject_token=<T1> actor_token=<mcp-sa>
resource=https://svc.cluster.local/search/search-api
// subject_token T1: aud lacks knoxidf -> rule 3:
// actor_token present, iss = knox-knoxidf-tokenexchange (listed), aud
intersects mcp-sa subjectAudiences
// actor_token (mcp-sa projected SA token) -> rule 1
{ "iss": "https://kubernetes.default.svc.cluster.local";,
"sub": "system:serviceaccount:tools:mcp-sa", "aud": ["knoxidf"] }
// policy mcp-sa: alice allowed, search-api in resourcePolicy, chain depth 2 <=
6 -> minted T2
{ "iss":
"http://knox.knox.svc.cluster.local:8443/gateway/knox-knoxidf-tokenexchange";,
"sub": "alice",
"aud": ["https://svc.cluster.local/search/search-api";],
"act": { "sub": "system:serviceaccount:tools:mcp-sa",
"act": { "sub": "system:serviceaccount:dataservice:ds-sa" } } }
{code}
h3. Hop 3: {{search-api}} -> token for {{vector-db}}
{code:yaml}
subject_token=<T2> actor_token=<search-sa>
resource=https://svc.cluster.local/data/vector-db
// subject_token T2 -> rule 3: aud intersects search-sa subjectAudiences
// actor_token (search-sa projected SA token, aud knoxidf) -> rule 1
// policy search-sa: alice allowed, vector-db in resourcePolicy, chain depth 3
<= 6 -> minted T3
{ "iss":
"http://knox.knox.svc.cluster.local:8443/gateway/knox-knoxidf-tokenexchange";,
"sub": "alice",
"aud": ["https://svc.cluster.local/data/vector-db";],
"act": { "sub": "system:serviceaccount:search:search-sa",
"act": { "sub": "system:serviceaccount:tools:mcp-sa",
"act": { "sub": "system:serviceaccount:dataservice:ds-sa" }
} } }
{code}
h3. Attack cases
||Attacker has||Result||Rule||
|{{ds-sa}} automounted SA token
({{{}aud=[https://kubernetes.default.svc|https://kubernetes.default.svc/]...{}}})
or one minted for Vault|Rejected as {{actor_token}} and as headless
{{subject_token}}|1|
|T1, presented with another workload's {{knoxidf}} actor token (even one
allowed to act for {{{}alice{}}})|Rejected: {{mcp-server}} is not in that
actor's {{subjectAudiences}}|3|
|T1 as a headless {{subject_token}}|Rejected: no {{knoxidf}} audience, and it
carries {{act}}|1, 5|
|T1 in a same-subject exchange|Rejected: no {{knoxidf}} audience|4|
|An SA token with {{aud=[https://svc.cluster.local/tools/mcp-server]}} sent
directly to {{{}mcp-server{}}}, used as {{subject_token}}|Rejected: its issuer
is not in {{token.exchange.subject.token.policy.audiences.issuers}}|3|
|{{{}mcp-sa{}}}'s own {{knoxidf}} projected SA token|Attacker acts as
{{{}mcp-sa{}}}, within its policy. Unavoidable; mitigate with short
{{expirationSeconds}}|-|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
