Hi Solr Devs,
This is a follow-up to a Slack post of mine:
https://apachesolr.slack.com/archives/C01GVPZSSK0/p1760562172938599
Looking for guidance (and maybe a feature/Doc enhancement) around JWT
authN/Z in SolrCloud.
Context (Solr 9.9), running on k8s, using Solr Operator:
* We use MultiAuth with *JWT* (for end-users) + *Basic* (for admin/ops).
* Collection-scoped authZ via RuleBasedAuthorizationPlugin (e.g., read on
restricted_collection requires role data-read-only derived from a nested
claim roles.my-app-name).
* Everything works on the *coordinator* node: JWTAuthPlugin Authentication
SUCCESS and the role is extracted.
*Problem (distributed fan-out):*
* On the coordinator, the distributed request to replicas is sent with *PKI*
(SolrAuthV2) and _forwardedCount=1. The *original JWT is not forwarded*.
* On the replica, auth context shows the *end-user principal* (sub), but *no
roles* (since there’s no bearer header to re-extract), so collection-scoped
read rules requiring data-read-only *403*.
* This was consistent and reproducible; logs on replicas show:
AuthorizationUtils ... userPrincipal: [[principal: myuserid]] ...
auth header null context
RuleBasedAuthorizationPluginBase ... permission {
"name":"read","collection":"restricted_collection","role":["data-read-only",...]}
... principal ... does not have the right role
*Workaround that finally fixed it:*
We added a *collection-scoped* permission that only matches *internal shard
hops* by checking _forwardedCount:
{
"name": "restricted-collection-forwarded-read",
"collection": "restricted_collection",
"path": ["/select", "/get", "/terms", "/export", "/spell", "/sql"],
"method": ["GET", "POST"],
"params": { "_forwardedCount": ["1","2","3","4","5","6","7","8","9","10"] },
"role": ["*"]
}
... while keeping the existing *role-gated* read rule for *external*
requests. This works (replicas now allow the coordinator’s trusted, signed
fan-out read), but it feels surprising and under-documented.
*Questions:*
1. *Design intent:* Is it by design that JWT credentials are *not*
forwarded to replicas during distributed queries? If so, what’s the
recommended pattern for collection-scoped role checks on replicas?
2. *Forwarding JWT:* Would you consider supporting a forwardCredentials-style
option for JWTAuthPlugin (similar to Basic), or another standard way for
replicas to receive/verifiably reuse end-user roles? (Our attempt to set
forwardCredentials on JWT fails with Invalid JwtAuth configuration
parameter [forwardCredentials].)
3. *Doc enhancements:* If the current model is “authorize on the
coordinator, replicas rely on PKI trust,” could the docs explicitly call
out the need for a *forwarded-read* permission (like above) when doing
per-collection, per-role JWT gating? This would have saved us a lot of time.
4. *Alternative approaches:* Would you prefer a design where the
coordinator conveys an *internal, signed role assertion* (e.g., piggybacked
on SolrAuthV2) that replicas can trust, instead of forwarding the full JWT?
If yes, happy to draft a JIRA with a proposal.
*Minimal repro sketch:*
* JWTAuthPlugin with rolesClaim: "roles.my-app-name", claimsMatch.iss,
jwksUrl, and a collection rule:
{ "name":"read", "collection":"restricted_collection",
"role":["data-read-only"] }
* Query restricted_collection with a valid JWT that includes
"data-read-only".
* Coordinator returns 403 from *replica* unless you allow forwarded reads
as above.
Happy to provide full security.json snippets and logs if helpful. Thanks
for any guidance on recommended practice and whether we should open a JIRA
for forwarding or doc changes [image: :pray:]
