[
https://issues.apache.org/jira/browse/KNOX-3579?focusedWorklogId=1055155&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-1055155
]
ASF GitHub Bot logged work on KNOX-3579:
----------------------------------------
Author: ASF GitHub Bot
Created on: 17/Nov/26 01:26
Start Date: 17/Nov/26 01:26
Worklog Time Spent: 10m
Work Description: smolnar82 opened a new pull request, #1516:
URL: https://github.com/apache/knox/pull/1516
[KNOX-3579](https://issues.apache.org/jira/browse/KNOX-3579) -
DefaultKeystoreService reloads keystores from disk under a global lock on every
request, capping SSO login and cookie verification throughput
## What changes were proposed in this pull request?
Every SSO cookie verification, WebSSO login and LDAP bind took the single
`DefaultKeystoreService` monitor and loaded a keystore from disk while holding
it. For BCFKS, each load runs PBKDF2-HMAC-SHA512, so a gateway could do only
one load at a time, whatever CPU it had free.
This PR caches loaded stores and replaces the global monitor. It is limited
to the signing keystore and the credential stores.
- **Cache for loaded stores.** `getSigningKeystore()` and
`getCredentialStoreForCluster()` return a cached `KeyStore`.
- Each lookup checks the file's stamp: modification time, size and file
key, following symlinks. The store is loaded again only when the stamp changes,
so changes made outside the gateway (Knox CLI, a rotated Kubernetes secret) are
picked up.
- A store loaded within 2 s of its last change is checked again once that
window has passed. This catches a second change that falls within the file
system's timestamp granularity.
- A store loaded with a different type or password is never reused, so a
wrong password still fails as before. The password is compared by its SHA-256
digest.
- A cache hit costs one `stat` and takes no lock.
- **Missing aliases are cheap.** Looking up an alias the credential store
doesn't hold no longer reloads the store. Such lookups happen on every request:
the HMAC secret in `WebSSOResource.init`, the signing keystore password alias,
and the topology-then-gateway fallback for the LDAP system password.
- **Locking.**
- `synchronized` is replaced by a `ReentrantReadWriteLock`: reads share
it, writes take it exclusively.
- Loads of the same file are serialized by a lock per path. Two file locks
on one file within a JVM throw `OverlappingFileLockException`.
- Writers (add or remove credentials, add a cert, create a keystore)
change a private copy. `writeKeyStoreToFile` removes the written file's cache
entry under the write lock.
- **Not cached: private-key reads.** This covers `getSigningKey`, the
identity keystore, and the HTTP client keystore and truststore. Once a BCFKS
store has handed out a private key, it returns that key for any passphrase, so
a shared instance would skip the passphrase check. These paths still load from
disk, but different files now load in parallel.
- **New setting:** `gateway.keystore.file.cache.enabled`, default `true`.
Set it to `false` to load the stores from disk on every access, as before;
reads still run in parallel. It is documented in `knox-site/docs/config.md`.
`getSigningKeystore()` and `getCredentialStoreForCluster()` now return
shared instances that callers must not modify. All current callers only read
from them (`DefaultTokenAuthorityService`, `JWKSResource`,
`KnoxMetadataResource`, `DefaultAliasService`, `SimpleDescriptorHandler`). The
`KeystoreService` javadoc now says so.
## How was this patch tested?
**Unit tests** added to `DefaultKeystoreServiceTest`:
- The signing keystore is loaded once, and loaded again after the file is
changed outside the service and after the service's own write.
- With `gateway.keystore.file.cache.enabled=false`, the stores are loaded on
every access.
- A cached credential store is not returned when the master secret changes,
and the load fails.
- 16 threads loading the same keystore at once succeed, with the cache on
and off.
`DefaultTokenAuthorityServiceTest` mocks now stub the new setting.
All 1,070 `gateway-server` tests pass, as do the `gateway-test` functional
tests `GatewayLocalServiceFuncTest` and `GatewayLdapDynamicGroupFuncTest`.
Checkstyle, PMD, SpotBugs and forbidden-apis are clean.
**Benchmark.** The new `DefaultKeystoreServiceBenchmarkTest` is skipped
unless `-Dknox.keystore.benchmark=true` is set:
```
mvn -pl gateway-server test -Dtest=DefaultKeystoreServiceBenchmarkTest
-Dknox.keystore.benchmark=true \
-Dknox.keystore.benchmark.threads=1,4,16
-Dknox.keystore.benchmark.seconds=5
```
It measures the keystore work on the two hot paths from the JIRA, with BCFKS
signing keystores and credential stores:
- **SSO cookie verification:** `getSigningKeystore().getCertificate(...)`,
as in `DefaultTokenAuthorityService.selectVerificationKey`.
- **WebSSO login:** the HMAC secret lookup in `WebSSOResource.init`, then
the LDAP system password lookup (topology credential store first, then the
gateway credential store).
Operations per second (5 s per row after warm-up; Apple M4 Pro, 14 cores,
macOS, JDK 17):
| Operation | Threads | master | This PR, cache off | This PR, cache on |
|---|---:|---:|---:|---:|
| SSO cookie verification | 1 | 2.6 | 2.4 | 18,590 |
| | 4 | 2.4 | 4.0 | 31,249 |
| | 16 | 2.6 | 4.0 | 16,314 |
| WebSSO login (keystore part) | 1 | 2.6 | 2.4 | 29,678 |
| | 4 | 2.6 | 4.0 | 33,799 |
| | 16 | 2.6 | 4.0 | 16,764 |
Notes on the numbers:
- **Master** does not scale with threads: every operation loads BCFKS stores
while holding the global monitor.
- **Cache off** reaches 4/s because the credential store and the signing
keystore now load in parallel.
- **The table measures only the keystore work, not a full request.** A full
login also does the LDAP bind, signs a JWT, and handles HTTP and redirects.
Those costs now set the limit instead of the keystore.
- **Cache on, 4 to 16 threads:** throughput drops, but not because of
locking. In thread dumps all threads are RUNNABLE, in BouncyCastle ASN.1
decoding of the stored certificate and in the per-lookup `stat`. The machine
has 14 cores, 4 of them efficiency cores, so 16 busy threads exceed its fast
cores.
## Integration Tests
No new integration tests. The change is internal to `DefaultKeystoreService`
and adds no endpoint or topology. The existing Docker Compose suite runs WebSSO
login, SSO cookie verification and LDAP authentication through the changed code.
Issue Time Tracking
-------------------
Worklog Id: (was: 1055155)
Remaining Estimate: 0h
Time Spent: 10m
> DefaultKeystoreService reloads keystores from disk under a global lock on
> every request, capping SSO login and cookie verification throughput
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: KNOX-3579
> URL: https://issues.apache.org/jira/browse/KNOX-3579
> Project: Apache Knox
> Issue Type: Improvement
> Components: Server
> Affects Versions: 2.0.0, 1.6.0, 1.6.1, 2.1.0, 3.0.0, 2.1.1
> Reporter: Sandor Molnar
> Assignee: Sandor Molnar
> Priority: Major
> Fix For: 3.1.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h3. Problem
> Under load, Knox serves WebSSO logins and SSO cookie verifications only up to
> a fixed rate, about 25 logins/s and 12 cookie verifications/s on one gateway
> instance, while the gateway still has CPU free. Requests beyond that rate
> queue up and eventually time out. With 1,500 or more users holding sessions
> and polling every 10 s, 64-96% of requests timed out at 60 s.
> Environment: BCFKS keystores (FIPS), one gateway instance, LDAP-backed WebSSO.
> Thread dumps taken under load show almost all Jetty request threads (245 of
> 254) BLOCKED on the single DefaultKeystoreService monitor, waiting in:
> * {{DefaultKeystoreService.getSigningKeystore}} (SSO cookie / JWT
> verification)
> * {{KnoxLdapContextFactory.getSystemPassword}} (LDAP bind)
> * {{WebSSOResource.init}} -> {{TokenUtils.getSignatureAlgorithm}} (WebSSO
> login)
> In every dump, the thread holding the monitor is in
> {{DefaultKeystoreService.loadKeyStore}} -> {{KeyStore.load}} -> BCFKS
> {{engineLoad}} -> {{PKCS5S2ParametersGenerator}} (PBKDF2-HMAC-SHA512).
> h3. Root cause
> # *Loaded keystores are never cached.* Every {{getSigningKeystore()}} and
> {{getCredentialStoreForCluster()}} call reads the file and calls
> {{KeyStore.load()}}. For BCFKS, that runs PBKDF2 for the integrity check,
> which costs tens of milliseconds of CPU per load. Only credential values
> (alias -> password) are cached, and only for aliases that exist. A lookup for
> an alias that doesn't exist reloads the whole credential store every time.
> # *One global lock serializes all loads.* {{getKeystore}}, {{loadKeyStore}},
> {{getCredentialForCluster}} and the write methods are all synchronized on the
> service instance. The gateway therefore runs one PBKDF2 at a time, however
> many cores it has.
> Hot paths that reload a store on every request:
> ||Path||Store(s) reloaded on every request||
> |SSO cookie / JWT verification
> ({{DefaultTokenAuthorityService.selectVerificationKey}})|The signing
> keystore, always. Also the gateway-level credential store when the signing
> keystore password alias (default: gateway-identity-keystore-password) is not
> defined.|
> |LDAP bind ({{KnoxLdapContextFactory.getSystemPassword}})|The topology
> credential store, when the system password alias is defined only at gateway
> level.|
> |WebSSO login ({{WebSSOResource.init}} ->
> {{TokenUtils.getSignatureAlgorithm}})|The gateway-level credential store,
> when gateway.signing.hmac.secret is not defined.|
> Throughput is therefore capped at roughly 1 / (PBKDF2 time x loads per
> request).
> h3. Proposed fix
> # Cache the loaded signing keystore and credential stores in memory, keyed by
> path and type. On every access, check the cached copy against a cheap file
> stamp (last-modified time, size, file key) and reload only when the file has
> changed. External updates, such as {{knoxcli create-alias}} or re-mounted
> secrets, are still picked up. Keep a digest of the store password with each
> entry, so a wrong password still fails as it does today. Re-check once any
> entry that was loaded within the mtime granularity window of its file's last
> change.
> # Replace the global monitor with a {{ReentrantReadWriteLock}}. Reads share
> the lock. Writes (add/remove alias, create store, add self-signed cert) take
> it exclusively. Loads of the same file run one at a time. Within one JVM this
> is required anyway: concurrent {{FileChannel.lock()}} calls on the same file
> throw {{OverlappingFileLockException}}. The credential-cache hit path becomes
> lock-free.
> # Writers ({{addCredentialsForCluster}}, {{removeCredentialsForCluster}},
> {{addCertForGateway}}) load a private copy under the write lock and drop the
> cached entry after writing, so a shared cached instance is never modified.
> # Private-key lookups ({{getSigningKey}}, {{getKeyForGateway}}) and the
> gateway identity keystore, truststore and HTTP-client keystore keep loading a
> fresh copy, but no longer under a global lock. Sharing BCFKS instances for
> private-key access is unsafe: {{BcFKSKeyStoreSpi.engineGetKey}} fills an
> unsynchronized {{privateKeyCache}} HashMap, and once a key is cached it is
> returned whatever password is supplied.
> # Add {{gateway.keystore.file.cache.enabled}} (default: true) to
> gateway-site.xml, so the cache can be turned off without a rebuild.
> h3. Expected result
> On the hot paths, each request costs an in-memory lookup plus a stat instead
> of 1-2 PBKDF2 runs under a global lock. Login and cookie verification
> throughput grows with the CPU available, and no request thread waits on
> DefaultKeystoreService for a store that is already loaded.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)