Sandor Molnar created KNOX-3579:
-----------------------------------

             Summary: DefaultKeystoreService reloads keystores from disk under 
a global lock on every request, capping SSO login and cookie verification 
throughput
                 Key: KNOX-3579
                 URL: https://issues.apache.org/jira/browse/KNOX-3579
             Project: Apache Knox
          Issue Type: Improvement
          Components: Server
    Affects Versions: 2.1.1, 3.0.0, 2.1.0, 1.6.1, 1.6.0, 2.0.0
            Reporter: Sandor Molnar
            Assignee: Sandor Molnar
             Fix For: 3.1.0


h3. Problem
Under load, Knox serves WebSSO logins and SSO cookie verifications only up to a 
fixed rate, about 25 logins/s and 12 cookie verifications/s on one gateway 
instance, while the gateway still has CPU free. Requests beyond that rate queue 
up and eventually time out. With 1,500 or more users holding sessions and 
polling every 10 s, 64-96% of requests timed out at 60 s.

Environment: BCFKS keystores (FIPS), one gateway instance, LDAP-backed WebSSO.

Thread dumps taken under load show almost all Jetty request threads (245 of 
254) BLOCKED on the single DefaultKeystoreService monitor, waiting in:
* {{DefaultKeystoreService.getSigningKeystore}} (SSO cookie / JWT verification)
* {{KnoxLdapContextFactory.getSystemPassword}} (LDAP bind)
* {{WebSSOResource.init}} -> {{TokenUtils.getSignatureAlgorithm}} (WebSSO login)

In every dump, the thread holding the monitor is in 
{{DefaultKeystoreService.loadKeyStore}} -> {{KeyStore.load}} -> BCFKS 
{{engineLoad}} -> {{PKCS5S2ParametersGenerator}} (PBKDF2-HMAC-SHA512).

h3. Root cause
# *Loaded keystores are never cached.* Every {{getSigningKeystore()}} and 
{{getCredentialStoreForCluster()}} call reads the file and calls 
{{KeyStore.load()}}. For BCFKS, that runs PBKDF2 for the integrity check, which 
costs tens of milliseconds of CPU per load. Only credential values (alias -> 
password) are cached, and only for aliases that exist. A lookup for an alias 
that doesn't exist reloads the whole credential store every time.
# *One global lock serializes all loads.* {{getKeystore}}, {{loadKeyStore}}, 
{{getCredentialForCluster}} and the write methods are all synchronized on the 
service instance. The gateway therefore runs one PBKDF2 at a time, however many 
cores it has.

Hot paths that reload a store on every request:
||Path||Store(s) reloaded on every request||
|SSO cookie / JWT verification 
({{DefaultTokenAuthorityService.selectVerificationKey}})|The signing keystore, 
always. Also the gateway-level credential store when the signing keystore 
password alias (default: gateway-identity-keystore-password) is not defined.|
|LDAP bind ({{KnoxLdapContextFactory.getSystemPassword}})|The topology 
credential store, when the system password alias is defined only at gateway 
level.|
|WebSSO login ({{WebSSOResource.init}} -> 
{{TokenUtils.getSignatureAlgorithm}})|The gateway-level credential store, when 
gateway.signing.hmac.secret is not defined.|

Throughput is therefore capped at roughly 1 / (PBKDF2 time x loads per request).

h3. Proposed fix
# Cache the loaded signing keystore and credential stores in memory, keyed by 
path and type. On every access, check the cached copy against a cheap file 
stamp (last-modified time, size, file key) and reload only when the file has 
changed. External updates, such as {{knoxcli create-alias}} or re-mounted 
secrets, are still picked up. Keep a digest of the store password with each 
entry, so a wrong password still fails as it does today. Re-check once any 
entry that was loaded within the mtime granularity window of its file's last 
change.
# Replace the global monitor with a {{ReentrantReadWriteLock}}. Reads share the 
lock. Writes (add/remove alias, create store, add self-signed cert) take it 
exclusively. Loads of the same file run one at a time. Within one JVM this is 
required anyway: concurrent {{FileChannel.lock()}} calls on the same file throw 
{{OverlappingFileLockException}}. The credential-cache hit path becomes 
lock-free.
# Writers ({{addCredentialsForCluster}}, {{removeCredentialsForCluster}}, 
{{addCertForGateway}}) load a private copy under the write lock and drop the 
cached entry after writing, so a shared cached instance is never modified.
# Private-key lookups ({{getSigningKey}}, {{getKeyForGateway}}) and the gateway 
identity keystore, truststore and HTTP-client keystore keep loading a fresh 
copy, but no longer under a global lock. Sharing BCFKS instances for 
private-key access is unsafe: {{BcFKSKeyStoreSpi.engineGetKey}} fills an 
unsynchronized {{privateKeyCache}} HashMap, and once a key is cached it is 
returned whatever password is supplied.
# Add {{gateway.keystore.file.cache.enabled}} (default: true) to 
gateway-site.xml, so the cache can be turned off without a rebuild.

h3. Expected result
On the hot paths, each request costs an in-memory lookup plus a stat instead of 
1-2 PBKDF2 runs under a global lock. Login and cookie verification throughput 
grows with the CPU available, and no request thread waits on 
DefaultKeystoreService for a store that is already loaded.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to