smolnar82 opened a new pull request, #1516: URL: https://github.com/apache/knox/pull/1516
[KNOX-3579](https://issues.apache.org/jira/browse/KNOX-3579) - DefaultKeystoreService reloads keystores from disk under a global lock on every request, capping SSO login and cookie verification throughput ## What changes were proposed in this pull request? Every SSO cookie verification, WebSSO login and LDAP bind took the single `DefaultKeystoreService` monitor and loaded a keystore from disk while holding it. For BCFKS, each load runs PBKDF2-HMAC-SHA512, so a gateway could do only one load at a time, whatever CPU it had free. This PR caches loaded stores and replaces the global monitor. It is limited to the signing keystore and the credential stores. - **Cache for loaded stores.** `getSigningKeystore()` and `getCredentialStoreForCluster()` return a cached `KeyStore`. - Each lookup checks the file's stamp: modification time, size and file key, following symlinks. The store is loaded again only when the stamp changes, so changes made outside the gateway (Knox CLI, a rotated Kubernetes secret) are picked up. - A store loaded within 2 s of its last change is checked again once that window has passed. This catches a second change that falls within the file system's timestamp granularity. - A store loaded with a different type or password is never reused, so a wrong password still fails as before. The password is compared by its SHA-256 digest. - A cache hit costs one `stat` and takes no lock. - **Missing aliases are cheap.** Looking up an alias the credential store doesn't hold no longer reloads the store. Such lookups happen on every request: the HMAC secret in `WebSSOResource.init`, the signing keystore password alias, and the topology-then-gateway fallback for the LDAP system password. - **Locking.** - `synchronized` is replaced by a `ReentrantReadWriteLock`: reads share it, writes take it exclusively. - Loads of the same file are serialized by a lock per path. Two file locks on one file within a JVM throw `OverlappingFileLockException`. - Writers (add or remove credentials, add a cert, create a keystore) change a private copy. `writeKeyStoreToFile` removes the written file's cache entry under the write lock. - **Not cached: private-key reads.** This covers `getSigningKey`, the identity keystore, and the HTTP client keystore and truststore. Once a BCFKS store has handed out a private key, it returns that key for any passphrase, so a shared instance would skip the passphrase check. These paths still load from disk, but different files now load in parallel. - **New setting:** `gateway.keystore.file.cache.enabled`, default `true`. Set it to `false` to load the stores from disk on every access, as before; reads still run in parallel. It is documented in `knox-site/docs/config.md`. `getSigningKeystore()` and `getCredentialStoreForCluster()` now return shared instances that callers must not modify. All current callers only read from them (`DefaultTokenAuthorityService`, `JWKSResource`, `KnoxMetadataResource`, `DefaultAliasService`, `SimpleDescriptorHandler`). The `KeystoreService` javadoc now says so. ## How was this patch tested? **Unit tests** added to `DefaultKeystoreServiceTest`: - The signing keystore is loaded once, and loaded again after the file is changed outside the service and after the service's own write. - With `gateway.keystore.file.cache.enabled=false`, the stores are loaded on every access. - A cached credential store is not returned when the master secret changes, and the load fails. - 16 threads loading the same keystore at once succeed, with the cache on and off. `DefaultTokenAuthorityServiceTest` mocks now stub the new setting. All 1,070 `gateway-server` tests pass, as do the `gateway-test` functional tests `GatewayLocalServiceFuncTest` and `GatewayLdapDynamicGroupFuncTest`. Checkstyle, PMD, SpotBugs and forbidden-apis are clean. **Benchmark.** The new `DefaultKeystoreServiceBenchmarkTest` is skipped unless `-Dknox.keystore.benchmark=true` is set: ``` mvn -pl gateway-server test -Dtest=DefaultKeystoreServiceBenchmarkTest -Dknox.keystore.benchmark=true \ -Dknox.keystore.benchmark.threads=1,4,16 -Dknox.keystore.benchmark.seconds=5 ``` It measures the keystore work on the two hot paths from the JIRA, with BCFKS signing keystores and credential stores: - **SSO cookie verification:** `getSigningKeystore().getCertificate(...)`, as in `DefaultTokenAuthorityService.selectVerificationKey`. - **WebSSO login:** the HMAC secret lookup in `WebSSOResource.init`, then the LDAP system password lookup (topology credential store first, then the gateway credential store). Operations per second (5 s per row after warm-up; Apple M4 Pro, 14 cores, macOS, JDK 17): | Operation | Threads | master | This PR, cache off | This PR, cache on | |---|---:|---:|---:|---:| | SSO cookie verification | 1 | 2.6 | 2.4 | 18,590 | | | 4 | 2.4 | 4.0 | 31,249 | | | 16 | 2.6 | 4.0 | 16,314 | | WebSSO login (keystore part) | 1 | 2.6 | 2.4 | 29,678 | | | 4 | 2.6 | 4.0 | 33,799 | | | 16 | 2.6 | 4.0 | 16,764 | Notes on the numbers: - **Master** does not scale with threads: every operation loads BCFKS stores while holding the global monitor. - **Cache off** reaches 4/s because the credential store and the signing keystore now load in parallel. - **The table measures only the keystore work, not a full request.** A full login also does the LDAP bind, signs a JWT, and handles HTTP and redirects. Those costs now set the limit instead of the keystore. - **Cache on, 4 to 16 threads:** throughput drops, but not because of locking. In thread dumps all threads are RUNNABLE, in BouncyCastle ASN.1 decoding of the stored certificate and in the per-lookup `stat`. The machine has 14 cores, 4 of them efficiency cores, so 16 busy threads exceed its fast cores. ## Integration Tests No new integration tests. The change is internal to `DefaultKeystoreService` and adds no endpoint or topology. The existing Docker Compose suite runs WebSSO login, SSO cookie verification and LDAP authentication through the changed code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
