smolnar82 opened a new pull request, #1516:
URL: https://github.com/apache/knox/pull/1516

   [KNOX-3579](https://issues.apache.org/jira/browse/KNOX-3579) - 
DefaultKeystoreService reloads keystores from disk under a global lock on every 
request, capping SSO login and cookie verification throughput
   
   ## What changes were proposed in this pull request?
   
   Every SSO cookie verification, WebSSO login and LDAP bind took the single 
`DefaultKeystoreService` monitor and loaded a keystore from disk while holding 
it. For BCFKS, each load runs PBKDF2-HMAC-SHA512, so a gateway could do only 
one load at a time, whatever CPU it had free.
   
   This PR caches loaded stores and replaces the global monitor. It is limited 
to the signing keystore and the credential stores.
   
   - **Cache for loaded stores.** `getSigningKeystore()` and 
`getCredentialStoreForCluster()` return a cached `KeyStore`.
     - Each lookup checks the file's stamp: modification time, size and file 
key, following symlinks. The store is loaded again only when the stamp changes, 
so changes made outside the gateway (Knox CLI, a rotated Kubernetes secret) are 
picked up.
     - A store loaded within 2 s of its last change is checked again once that 
window has passed. This catches a second change that falls within the file 
system's timestamp granularity.
     - A store loaded with a different type or password is never reused, so a 
wrong password still fails as before. The password is compared by its SHA-256 
digest.
     - A cache hit costs one `stat` and takes no lock.
   - **Missing aliases are cheap.** Looking up an alias the credential store 
doesn't hold no longer reloads the store. Such lookups happen on every request: 
the HMAC secret in `WebSSOResource.init`, the signing keystore password alias, 
and the topology-then-gateway fallback for the LDAP system password.
   - **Locking.**
     - `synchronized` is replaced by a `ReentrantReadWriteLock`: reads share 
it, writes take it exclusively.
     - Loads of the same file are serialized by a lock per path. Two file locks 
on one file within a JVM throw `OverlappingFileLockException`.
     - Writers (add or remove credentials, add a cert, create a keystore) 
change a private copy. `writeKeyStoreToFile` removes the written file's cache 
entry under the write lock.
   - **Not cached: private-key reads.** This covers `getSigningKey`, the 
identity keystore, and the HTTP client keystore and truststore. Once a BCFKS 
store has handed out a private key, it returns that key for any passphrase, so 
a shared instance would skip the passphrase check. These paths still load from 
disk, but different files now load in parallel.
   - **New setting:** `gateway.keystore.file.cache.enabled`, default `true`. 
Set it to `false` to load the stores from disk on every access, as before; 
reads still run in parallel. It is documented in `knox-site/docs/config.md`.
   
   `getSigningKeystore()` and `getCredentialStoreForCluster()` now return 
shared instances that callers must not modify. All current callers only read 
from them (`DefaultTokenAuthorityService`, `JWKSResource`, 
`KnoxMetadataResource`, `DefaultAliasService`, `SimpleDescriptorHandler`). The 
`KeystoreService` javadoc now says so.
   
   ## How was this patch tested?
   
   **Unit tests** added to `DefaultKeystoreServiceTest`:
   - The signing keystore is loaded once, and loaded again after the file is 
changed outside the service and after the service's own write.
   - With `gateway.keystore.file.cache.enabled=false`, the stores are loaded on 
every access.
   - A cached credential store is not returned when the master secret changes, 
and the load fails.
   - 16 threads loading the same keystore at once succeed, with the cache on 
and off.
   
   `DefaultTokenAuthorityServiceTest` mocks now stub the new setting.
   
   All 1,070 `gateway-server` tests pass, as do the `gateway-test` functional 
tests `GatewayLocalServiceFuncTest` and `GatewayLdapDynamicGroupFuncTest`. 
Checkstyle, PMD, SpotBugs and forbidden-apis are clean.
   
   **Benchmark.** The new `DefaultKeystoreServiceBenchmarkTest` is skipped 
unless `-Dknox.keystore.benchmark=true` is set:
   
   ```
   mvn -pl gateway-server test -Dtest=DefaultKeystoreServiceBenchmarkTest 
-Dknox.keystore.benchmark=true \
       -Dknox.keystore.benchmark.threads=1,4,16 
-Dknox.keystore.benchmark.seconds=5
   ```
   
   It measures the keystore work on the two hot paths from the JIRA, with BCFKS 
signing keystores and credential stores:
   - **SSO cookie verification:** `getSigningKeystore().getCertificate(...)`, 
as in `DefaultTokenAuthorityService.selectVerificationKey`.
   - **WebSSO login:** the HMAC secret lookup in `WebSSOResource.init`, then 
the LDAP system password lookup (topology credential store first, then the 
gateway credential store).
   
   Operations per second (5 s per row after warm-up; Apple M4 Pro, 14 cores, 
macOS, JDK 17):
   
   | Operation | Threads | master | This PR, cache off | This PR, cache on |
   |---|---:|---:|---:|---:|
   | SSO cookie verification | 1 | 2.6 | 2.4 | 18,590 |
   | | 4 | 2.4 | 4.0 | 31,249 |
   | | 16 | 2.6 | 4.0 | 16,314 |
   | WebSSO login (keystore part) | 1 | 2.6 | 2.4 | 29,678 |
   | | 4 | 2.6 | 4.0 | 33,799 |
   | | 16 | 2.6 | 4.0 | 16,764 |
   
   Notes on the numbers:
   - **Master** does not scale with threads: every operation loads BCFKS stores 
while holding the global monitor.
   - **Cache off** reaches 4/s because the credential store and the signing 
keystore now load in parallel.
   - **The table measures only the keystore work, not a full request.** A full 
login also does the LDAP bind, signs a JWT, and handles HTTP and redirects. 
Those costs now set the limit instead of the keystore.
   - **Cache on, 4 to 16 threads:** throughput drops, but not because of 
locking. In thread dumps all threads are RUNNABLE, in BouncyCastle ASN.1 
decoding of the stored certificate and in the per-lookup `stat`. The machine 
has 14 cores, 4 of them efficiency cores, so 16 busy threads exceed its fast 
cores.
   
   ## Integration Tests
   
   No new integration tests. The change is internal to `DefaultKeystoreService` 
and adds no endpoint or topology. The existing Docker Compose suite runs WebSSO 
login, SSO cookie verification and LDAP authentication through the changed code.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to