smolnar82 opened a new pull request, #1451: URL: https://github.com/apache/knox/pull/1451
[KNOX-3512](https://issues.apache.org/jira/browse/KNOX-3512) - Per-issuer CA bundle (jwksCa) for trusted OIDC issuers ## What changes were proposed in this pull request? Today, Knox fetches a trusted OIDC issuer's discovery document and JWKS using the JVM default truststore. To trust a cluster whose API server uses a self-signed CA (e.g. a k8s/k3s issuer), you have to import that CA into the global truststore. That widens trust for every other TLS client in the gateway, and every CA rotation needs a restart. This PR adds an optional, per-issuer **`jwksCa`** to the trusted OIDC issuer registry: - **Two forms** - inline PEM bundle (≤ 65,000 chars) - Kubernetes Secret reference `secret:[<namespace>/]<name>#<key>`. The namespace defaults to `knox`; name and key are required. - **Pinned trust:** when set, that issuer's discovery and JWKS connections trust *only* that bundle. The global truststore is never consulted or modified. Hostname verification stays on. - **Fail-closed:** if the issuer has a `jwksCa` that cannot be loaded (missing Secret or key, bad PEM), its tokens are rejected (401). Knox never falls back to the default truststore. - **Unchanged when unset:** issuers without a `jwksCa` behave exactly as before. - **No restart needed for CA changes** - Inline bundle: re-registering (delete + create, as the operator does) applies immediately. - Secret reference: the Secret is read lazily on first use, with no background refresh. It is re-read: - after `gateway.trusted.oidc.issuer.ca.secret.max.age.secs` (default `1800`) - on `POST .../trusted-oidc-issuers/refresh-jwks` - after a TLS trust failure on a discovery or JWKS fetch. Knox then retries the fetch once. This is rate-limited to once per 30s per issuer and is not triggered by timeouts, refused connections or non-200 responses. A re-read with unchanged content keeps the existing SSL context. A failed periodic re-read keeps the last good bundle. ### Changes by area - **Model / persistence** (`gateway-spi`, `gateway-server`) - `TrustedOidcIssuer` gains `jwksCa`. - A new `jwks_ca` column: `TEXT`, and `CLOB` on Oracle. It is added in the CREATE scripts only, because the table is unreleased. - New `JwksCa` utility: secret-reference parsing and PEM bundle validation. - **REST API** (`gateway-service-knoxidf`) - `POST` accepts `jwksCa`; `GET` returns it. CA certificates are public, and the operator needs the value to detect drift. - Returns `400 invalid_request` when: - `jwksCa` is set with `dynamicJwks=false` - the value is oversized - a secret reference has bad syntax - an inline bundle has no parseable X.509 certificate The syntax check alone is enough to register a Secret reference; the Secret itself may be created later. - **Trust material** (`IssuerTrustProvider`) - Builds an `SSLContext` from an in-memory keystore holding only the bundle's certificates. - Reads Secrets through the fabric8 client (the same kubeconfig/in-cluster config as the ServiceAccount validator). - Logs SHA-256 fingerprints only, never PEM or Secret contents. - **Discovery and JWKS fetch** - `OIDCDiscoveryHelper` uses a pinned HTTP client for issuers that have a `jwksCa`. - `DefaultTokenAuthorityService` builds the Nimbus `JWKSource` with a new `IssuerTrustResourceRetriever` that wraps a `DefaultResourceRetriever` with the issuer's socket factory. On a TLS trust failure it performs the Secret re-read and single retry. The re-read lives here because Nimbus's retrying and outage-tolerant layers can swallow the TLS cause. - **SPI changes** - `TrustedOidcIssuerService#getJwksTrust(String)`: new method returning `Optional<IssuerJwksTrust>`. - `JWTokenAuthority#verifyToken(..., IssuerJwksTrust)`: new default method, so existing implementations still compile. - `AbstractJWTFilter#resolveRegisteredIssuerJwks` now returns a `RegisteredIssuerJwks(uris, jwksTrust)` record instead of `Set<URI>`. The URIs and trust come from a single lookup, so they always belong together. - **Signing-key rotation:** works without any config change. A token whose `kid` is unknown makes the JWK source re-fetch the JWKS. - **Config:** `gateway.trusted.oidc.issuer.ca.secret.max.age.secs` (default `1800`). - **Docs** (`knox-site/docs/knoxidf/`) - `endpoints.md`: the new "Per-issuer CA bundle (`jwksCa`)" section covers both forms, the 400 rules and the reload behaviour. - `configuration.md`: the new property. - `security.md`: new "Trusting self-signed issuer CAs" section. It covers: - pinning - the RBAC Knox's ServiceAccount needs: `get` on `secrets` in each referenced namespace - the shared-root option - the HA caveat - `federation.md`: a pointer to the new section. ### Follow-ups (other repos) - knox-operator: add `jwksCa` to the `TrustedOidcIssuer` CRD and include it in the drift check, so a CA change triggers delete + create. - knox-helm: add a Role/RoleBinding granting the Knox ServiceAccount `get` on Secrets in the referenced namespaces. This replaces the cacerts import and the restart-on-rotation approach. ## How was this patch tested? Unit and integration tests, run with `mvn install` on `gateway-util-common`, `gateway-spi`, `gateway-spi-common`, `gateway-provider-security-jwt`, `gateway-server` and `gateway-service-knoxidf`. Checkstyle, PMD and SpotBugs all pass. Totals: gateway-spi 364, gateway-provider-security-jwt 578, gateway-server 889 (1 skipped), gateway-service-knoxidf 246, all green. The downstream modules (`gateway-service-knoxsso`, `gateway-service-knoxtoken`) also compile. New and extended tests: - `JwksCaTest`: secret-reference parsing (with and without a namespace; malformed values) and PEM bundles with 1 certificate, 2 certificates, or garbage. - `IssuerTrustProviderTest`: uses a stubbed Secret reader and an injectable clock. Covers: - the Secret is read once on first use and not again before max-age - re-read after max-age keeps the same SSL context when the content is unchanged and swaps it when it changed - re-reads after TLS failures are rate-limited to one per 30s - a missing Secret or key fails closed - a failed periodic re-read keeps the old bundle - the fabric8 reader handles `data` and `stringData` - `IssuerTrustResourceRetrieverTest`: a TLS trust failure leads to one re-read and one retry; a timeout or 404 does not. - `IssuerJwksCaAcceptanceTest`: real Jetty HTTPS JWKS servers with two independent self-signed CAs. Covers: - issuers A and B each verify with their own CA - CA rotation works without a restart - A's CA cannot reach B, and neither the default truststore nor a plain `HttpsURLConnection` trusts them, so no trust leaks - after signing-key rotation, a token with a new `kid` verifies with no config change - `OIDCDiscoveryHelperTest`, `JdbcTrustedOidcIssuerServiceTest` (H2 round-trip, `getJwksTrust`, invalidation on register/deregister/refresh), `TrustedOidcIssuersSchemaTest`, `TrustedOidcIssuersResourceTest` (the 400 cases and list output), `JWTFederationFilterTokenExchangeTest` (trust passed through, and a load failure leads to 401). ## Integration Tests **New suite: `.github/workflows/tests/test_k8s_jwks_ca.py`** (11 tests). It runs against the throwaway k3s cluster in the compose stack, a real OIDC issuer at `https://k3s:6443` with a self-signed CA, and covers: - invalid `jwksCa` values are rejected at registration with 400 and nothing is stored - inline CA: the exchange is accepted; with an unrelated CA it is rejected (pinned) - the CA does not leak into the default truststore: once the issuer has been trusted through its `jwksCa`, the same issuer registered without one is still rejected - re-registering an inline CA applies without a restart - a Secret reference works in the default (`knox`) namespace and in an explicitly named one - a missing Secret fails closed and works once the Secret is created and `refresh-jwks` is called - a missing Secret key fails closed - a rotated Secret applies on `refresh-jwks` - a rotated Secret alone also applies, after a TLS failure triggers the re-read **CI change:** the k3s CA is **no longer imported into the JVM cacerts**. The custom `gateway-k8s-delegation.sh` entrypoint did nothing else, so it is removed and the knox service runs plain `/gateway.sh` again. `test_k8s_delegation.py` now registers the k3s issuer with its CA as `jwksCa`. As a result, every positive k8s exchange in CI goes through the pinned per-issuer path, including the AC4/AC5 delegation tests that verify real signatures. Supporting fixture: a `knox` namespace manifest (`k8s/manifests/knox-namespace.yaml`). Result of the full compose run: `pylint` 10.00/10, **136 passed**: ``` tests-1 | ------------------------------------ tests-1 | Your code has been rated at 10.00/10 tests-1 | tests-1 | Waiting for knox... tests-1 | ============================= test session starts ============================== tests-1 | platform linux -- Python 3.10.20, pytest-9.0.3, pluggy-1.6.0 tests-1 | rootdir: /tests tests-1 | plugins: platformdirs-4.12.4 tests-1 | collected 136 items tests-1 | tests-1 | test_clientid_credentials.py ....... [ 5%] tests-1 | test_delegation.py ......... [ 11%] tests-1 | test_health.py ..... [ 15%] tests-1 | test_k8s_delegation.py ........ [ 21%] tests-1 | test_k8s_jwks_ca.py ........... [ 29%] tests-1 | test_k8s_serviceaccount_validation.py ...... [ 33%] tests-1 | test_knox_admin_path_traversal.py ... [ 36%] tests-1 | test_knox_auth_service_and_ldap.py ... [ 38%] tests-1 | test_knox_configs.py . [ 38%] tests-1 | test_knox_ldap_cache.py ... [ 41%] tests-1 | test_knox_ldap_injection.py ....... [ 46%] tests-1 | test_knox_ldap_proxy_search.py ......... [ 52%] tests-1 | test_knoxauth_preauth_and_paths.py ...... [ 57%] tests-1 | test_knoxauth_token_forwarding.py ........... [ 65%] tests-1 | test_knoxidf.py ....... [ 70%] tests-1 | test_knoxsso_redirect.py . [ 71%] tests-1 | test_knoxtoken_jwt.py .................... [ 86%] tests-1 | test_remote_auth.py ... [ 88%] tests-1 | test_remoteauth_extauthz_additional_path.py .... [ 91%] tests-1 | test_token_exchange.py ............ [100%] tests-1 | tests-1 | =============================== warnings summary =============================== ... tests-1 | -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html tests-1 | ----------------- generated xml file: /tests/test-results.xml ------------------ tests-1 | ================= 136 passed, 128 warnings in 71.13s (0:01:11) ================= tests-1 exited with code 0 Aborting on container exit... Container compose-tests-1 Stopping Container compose-tests-1 Stopped Config w Enable Watch d Detach ``` ### Opt-in test suites (PR labels) None needed. The new suite runs in the default compose job. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
