[ 
https://issues.apache.org/jira/browse/KNOX-3579?focusedWorklogId=1055155&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-1055155
 ]

ASF GitHub Bot logged work on KNOX-3579:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 17/Nov/26 01:26
            Start Date: 17/Nov/26 01:26
    Worklog Time Spent: 10m 
      Work Description: smolnar82 opened a new pull request, #1516:
URL: https://github.com/apache/knox/pull/1516

   [KNOX-3579](https://issues.apache.org/jira/browse/KNOX-3579) - 
DefaultKeystoreService reloads keystores from disk under a global lock on every 
request, capping SSO login and cookie verification throughput
   
   ## What changes were proposed in this pull request?
   
   Every SSO cookie verification, WebSSO login and LDAP bind took the single 
`DefaultKeystoreService` monitor and loaded a keystore from disk while holding 
it. For BCFKS, each load runs PBKDF2-HMAC-SHA512, so a gateway could do only 
one load at a time, whatever CPU it had free.
   
   This PR caches loaded stores and replaces the global monitor. It is limited 
to the signing keystore and the credential stores.
   
   - **Cache for loaded stores.** `getSigningKeystore()` and 
`getCredentialStoreForCluster()` return a cached `KeyStore`.
     - Each lookup checks the file's stamp: modification time, size and file 
key, following symlinks. The store is loaded again only when the stamp changes, 
so changes made outside the gateway (Knox CLI, a rotated Kubernetes secret) are 
picked up.
     - A store loaded within 2 s of its last change is checked again once that 
window has passed. This catches a second change that falls within the file 
system's timestamp granularity.
     - A store loaded with a different type or password is never reused, so a 
wrong password still fails as before. The password is compared by its SHA-256 
digest.
     - A cache hit costs one `stat` and takes no lock.
   - **Missing aliases are cheap.** Looking up an alias the credential store 
doesn't hold no longer reloads the store. Such lookups happen on every request: 
the HMAC secret in `WebSSOResource.init`, the signing keystore password alias, 
and the topology-then-gateway fallback for the LDAP system password.
   - **Locking.**
     - `synchronized` is replaced by a `ReentrantReadWriteLock`: reads share 
it, writes take it exclusively.
     - Loads of the same file are serialized by a lock per path. Two file locks 
on one file within a JVM throw `OverlappingFileLockException`.
     - Writers (add or remove credentials, add a cert, create a keystore) 
change a private copy. `writeKeyStoreToFile` removes the written file's cache 
entry under the write lock.
   - **Not cached: private-key reads.** This covers `getSigningKey`, the 
identity keystore, and the HTTP client keystore and truststore. Once a BCFKS 
store has handed out a private key, it returns that key for any passphrase, so 
a shared instance would skip the passphrase check. These paths still load from 
disk, but different files now load in parallel.
   - **New setting:** `gateway.keystore.file.cache.enabled`, default `true`. 
Set it to `false` to load the stores from disk on every access, as before; 
reads still run in parallel. It is documented in `knox-site/docs/config.md`.
   
   `getSigningKeystore()` and `getCredentialStoreForCluster()` now return 
shared instances that callers must not modify. All current callers only read 
from them (`DefaultTokenAuthorityService`, `JWKSResource`, 
`KnoxMetadataResource`, `DefaultAliasService`, `SimpleDescriptorHandler`). The 
`KeystoreService` javadoc now says so.
   
   ## How was this patch tested?
   
   **Unit tests** added to `DefaultKeystoreServiceTest`:
   - The signing keystore is loaded once, and loaded again after the file is 
changed outside the service and after the service's own write.
   - With `gateway.keystore.file.cache.enabled=false`, the stores are loaded on 
every access.
   - A cached credential store is not returned when the master secret changes, 
and the load fails.
   - 16 threads loading the same keystore at once succeed, with the cache on 
and off.
   
   `DefaultTokenAuthorityServiceTest` mocks now stub the new setting.
   
   All 1,070 `gateway-server` tests pass, as do the `gateway-test` functional 
tests `GatewayLocalServiceFuncTest` and `GatewayLdapDynamicGroupFuncTest`. 
Checkstyle, PMD, SpotBugs and forbidden-apis are clean.
   
   **Benchmark.** The new `DefaultKeystoreServiceBenchmarkTest` is skipped 
unless `-Dknox.keystore.benchmark=true` is set:
   
   ```
   mvn -pl gateway-server test -Dtest=DefaultKeystoreServiceBenchmarkTest 
-Dknox.keystore.benchmark=true \
       -Dknox.keystore.benchmark.threads=1,4,16 
-Dknox.keystore.benchmark.seconds=5
   ```
   
   It measures the keystore work on the two hot paths from the JIRA, with BCFKS 
signing keystores and credential stores:
   - **SSO cookie verification:** `getSigningKeystore().getCertificate(...)`, 
as in `DefaultTokenAuthorityService.selectVerificationKey`.
   - **WebSSO login:** the HMAC secret lookup in `WebSSOResource.init`, then 
the LDAP system password lookup (topology credential store first, then the 
gateway credential store).
   
   Operations per second (5 s per row after warm-up; Apple M4 Pro, 14 cores, 
macOS, JDK 17):
   
   | Operation | Threads | master | This PR, cache off | This PR, cache on |
   |---|---:|---:|---:|---:|
   | SSO cookie verification | 1 | 2.6 | 2.4 | 18,590 |
   | | 4 | 2.4 | 4.0 | 31,249 |
   | | 16 | 2.6 | 4.0 | 16,314 |
   | WebSSO login (keystore part) | 1 | 2.6 | 2.4 | 29,678 |
   | | 4 | 2.6 | 4.0 | 33,799 |
   | | 16 | 2.6 | 4.0 | 16,764 |
   
   Notes on the numbers:
   - **Master** does not scale with threads: every operation loads BCFKS stores 
while holding the global monitor.
   - **Cache off** reaches 4/s because the credential store and the signing 
keystore now load in parallel.
   - **The table measures only the keystore work, not a full request.** A full 
login also does the LDAP bind, signs a JWT, and handles HTTP and redirects. 
Those costs now set the limit instead of the keystore.
   - **Cache on, 4 to 16 threads:** throughput drops, but not because of 
locking. In thread dumps all threads are RUNNABLE, in BouncyCastle ASN.1 
decoding of the stored certificate and in the per-lookup `stat`. The machine 
has 14 cores, 4 of them efficiency cores, so 16 busy threads exceed its fast 
cores.
   
   ## Integration Tests
   
   No new integration tests. The change is internal to `DefaultKeystoreService` 
and adds no endpoint or topology. The existing Docker Compose suite runs WebSSO 
login, SSO cookie verification and LDAP authentication through the changed code.
   




Issue Time Tracking
-------------------

            Worklog Id:     (was: 1055155)
    Remaining Estimate: 0h
            Time Spent: 10m

> DefaultKeystoreService reloads keystores from disk under a global lock on 
> every request, capping SSO login and cookie verification throughput
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: KNOX-3579
>                 URL: https://issues.apache.org/jira/browse/KNOX-3579
>             Project: Apache Knox
>          Issue Type: Improvement
>          Components: Server
>    Affects Versions: 2.0.0, 1.6.0, 1.6.1, 2.1.0, 3.0.0, 2.1.1
>            Reporter: Sandor Molnar
>            Assignee: Sandor Molnar
>            Priority: Major
>             Fix For: 3.1.0
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> h3. Problem
> Under load, Knox serves WebSSO logins and SSO cookie verifications only up to 
> a fixed rate, about 25 logins/s and 12 cookie verifications/s on one gateway 
> instance, while the gateway still has CPU free. Requests beyond that rate 
> queue up and eventually time out. With 1,500 or more users holding sessions 
> and polling every 10 s, 64-96% of requests timed out at 60 s.
> Environment: BCFKS keystores (FIPS), one gateway instance, LDAP-backed WebSSO.
> Thread dumps taken under load show almost all Jetty request threads (245 of 
> 254) BLOCKED on the single DefaultKeystoreService monitor, waiting in:
> * {{DefaultKeystoreService.getSigningKeystore}} (SSO cookie / JWT 
> verification)
> * {{KnoxLdapContextFactory.getSystemPassword}} (LDAP bind)
> * {{WebSSOResource.init}} -> {{TokenUtils.getSignatureAlgorithm}} (WebSSO 
> login)
> In every dump, the thread holding the monitor is in 
> {{DefaultKeystoreService.loadKeyStore}} -> {{KeyStore.load}} -> BCFKS 
> {{engineLoad}} -> {{PKCS5S2ParametersGenerator}} (PBKDF2-HMAC-SHA512).
> h3. Root cause
> # *Loaded keystores are never cached.* Every {{getSigningKeystore()}} and 
> {{getCredentialStoreForCluster()}} call reads the file and calls 
> {{KeyStore.load()}}. For BCFKS, that runs PBKDF2 for the integrity check, 
> which costs tens of milliseconds of CPU per load. Only credential values 
> (alias -> password) are cached, and only for aliases that exist. A lookup for 
> an alias that doesn't exist reloads the whole credential store every time.
> # *One global lock serializes all loads.* {{getKeystore}}, {{loadKeyStore}}, 
> {{getCredentialForCluster}} and the write methods are all synchronized on the 
> service instance. The gateway therefore runs one PBKDF2 at a time, however 
> many cores it has.
> Hot paths that reload a store on every request:
> ||Path||Store(s) reloaded on every request||
> |SSO cookie / JWT verification 
> ({{DefaultTokenAuthorityService.selectVerificationKey}})|The signing 
> keystore, always. Also the gateway-level credential store when the signing 
> keystore password alias (default: gateway-identity-keystore-password) is not 
> defined.|
> |LDAP bind ({{KnoxLdapContextFactory.getSystemPassword}})|The topology 
> credential store, when the system password alias is defined only at gateway 
> level.|
> |WebSSO login ({{WebSSOResource.init}} -> 
> {{TokenUtils.getSignatureAlgorithm}})|The gateway-level credential store, 
> when gateway.signing.hmac.secret is not defined.|
> Throughput is therefore capped at roughly 1 / (PBKDF2 time x loads per 
> request).
> h3. Proposed fix
> # Cache the loaded signing keystore and credential stores in memory, keyed by 
> path and type. On every access, check the cached copy against a cheap file 
> stamp (last-modified time, size, file key) and reload only when the file has 
> changed. External updates, such as {{knoxcli create-alias}} or re-mounted 
> secrets, are still picked up. Keep a digest of the store password with each 
> entry, so a wrong password still fails as it does today. Re-check once any 
> entry that was loaded within the mtime granularity window of its file's last 
> change.
> # Replace the global monitor with a {{ReentrantReadWriteLock}}. Reads share 
> the lock. Writes (add/remove alias, create store, add self-signed cert) take 
> it exclusively. Loads of the same file run one at a time. Within one JVM this 
> is required anyway: concurrent {{FileChannel.lock()}} calls on the same file 
> throw {{OverlappingFileLockException}}. The credential-cache hit path becomes 
> lock-free.
> # Writers ({{addCredentialsForCluster}}, {{removeCredentialsForCluster}}, 
> {{addCertForGateway}}) load a private copy under the write lock and drop the 
> cached entry after writing, so a shared cached instance is never modified.
> # Private-key lookups ({{getSigningKey}}, {{getKeyForGateway}}) and the 
> gateway identity keystore, truststore and HTTP-client keystore keep loading a 
> fresh copy, but no longer under a global lock. Sharing BCFKS instances for 
> private-key access is unsafe: {{BcFKSKeyStoreSpi.engineGetKey}} fills an 
> unsynchronized {{privateKeyCache}} HashMap, and once a key is cached it is 
> returned whatever password is supplied.
> # Add {{gateway.keystore.file.cache.enabled}} (default: true) to 
> gateway-site.xml, so the cache can be turned off without a rebuild.
> h3. Expected result
> On the hot paths, each request costs an in-memory lookup plus a stat instead 
> of 1-2 PBKDF2 runs under a global lock. Login and cookie verification 
> throughput grows with the CPU available, and no request thread waits on 
> DefaultKeystoreService for a store that is already loaded.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to