[ 
https://issues.apache.org/jira/browse/KNOX-3579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sandor Molnar resolved KNOX-3579.
---------------------------------
    Resolution: Fixed

> DefaultKeystoreService reloads keystores from disk under a global lock on 
> every request, capping SSO login and cookie verification throughput
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: KNOX-3579
>                 URL: https://issues.apache.org/jira/browse/KNOX-3579
>             Project: Apache Knox
>          Issue Type: Improvement
>          Components: Server
>    Affects Versions: 2.0.0, 1.6.0, 1.6.1, 2.1.0, 3.0.0, 2.1.1
>            Reporter: Sandor Molnar
>            Assignee: Sandor Molnar
>            Priority: Major
>             Fix For: 3.1.0
>
>          Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> h3. Problem
> Under load, Knox serves WebSSO logins and SSO cookie verifications only up to 
> a fixed rate, about 25 logins/s and 12 cookie verifications/s on one gateway 
> instance, while the gateway still has CPU free. Requests beyond that rate 
> queue up and eventually time out. With 1,500 or more users holding sessions 
> and polling every 10 s, 64-96% of requests timed out at 60 s.
> Environment: BCFKS keystores (FIPS), one gateway instance, LDAP-backed WebSSO.
> Thread dumps taken under load show almost all Jetty request threads (245 of 
> 254) BLOCKED on the single DefaultKeystoreService monitor, waiting in:
> * {{DefaultKeystoreService.getSigningKeystore}} (SSO cookie / JWT 
> verification)
> * {{KnoxLdapContextFactory.getSystemPassword}} (LDAP bind)
> * {{WebSSOResource.init}} -> {{TokenUtils.getSignatureAlgorithm}} (WebSSO 
> login)
> In every dump, the thread holding the monitor is in 
> {{DefaultKeystoreService.loadKeyStore}} -> {{KeyStore.load}} -> BCFKS 
> {{engineLoad}} -> {{PKCS5S2ParametersGenerator}} (PBKDF2-HMAC-SHA512).
> h3. Root cause
> # *Loaded keystores are never cached.* Every {{getSigningKeystore()}} and 
> {{getCredentialStoreForCluster()}} call reads the file and calls 
> {{KeyStore.load()}}. For BCFKS, that runs PBKDF2 for the integrity check, 
> which costs tens of milliseconds of CPU per load. Only credential values 
> (alias -> password) are cached, and only for aliases that exist. A lookup for 
> an alias that doesn't exist reloads the whole credential store every time.
> # *One global lock serializes all loads.* {{getKeystore}}, {{loadKeyStore}}, 
> {{getCredentialForCluster}} and the write methods are all synchronized on the 
> service instance. The gateway therefore runs one PBKDF2 at a time, however 
> many cores it has.
> Hot paths that reload a store on every request:
> ||Path||Store(s) reloaded on every request||
> |SSO cookie / JWT verification 
> ({{DefaultTokenAuthorityService.selectVerificationKey}})|The signing 
> keystore, always. Also the gateway-level credential store when the signing 
> keystore password alias (default: gateway-identity-keystore-password) is not 
> defined.|
> |LDAP bind ({{KnoxLdapContextFactory.getSystemPassword}})|The topology 
> credential store, when the system password alias is defined only at gateway 
> level.|
> |WebSSO login ({{WebSSOResource.init}} -> 
> {{TokenUtils.getSignatureAlgorithm}})|The gateway-level credential store, 
> when gateway.signing.hmac.secret is not defined.|
> Throughput is therefore capped at roughly 1 / (PBKDF2 time x loads per 
> request).
> h3. Proposed fix
> # Cache the loaded signing keystore and credential stores in memory, keyed by 
> path and type. On every access, check the cached copy against a cheap file 
> stamp (last-modified time, size, file key) and reload only when the file has 
> changed. External updates, such as {{knoxcli create-alias}} or re-mounted 
> secrets, are still picked up. Keep a digest of the store password with each 
> entry, so a wrong password still fails as it does today. Re-check once any 
> entry that was loaded within the mtime granularity window of its file's last 
> change.
> # Replace the global monitor with a {{ReentrantReadWriteLock}}. Reads share 
> the lock. Writes (add/remove alias, create store, add self-signed cert) take 
> it exclusively. Loads of the same file run one at a time. Within one JVM this 
> is required anyway: concurrent {{FileChannel.lock()}} calls on the same file 
> throw {{OverlappingFileLockException}}. The credential-cache hit path becomes 
> lock-free.
> # Writers ({{addCredentialsForCluster}}, {{removeCredentialsForCluster}}, 
> {{addCertForGateway}}) load a private copy under the write lock and drop the 
> cached entry after writing, so a shared cached instance is never modified.
> # Private-key lookups ({{getSigningKey}}, {{getKeyForGateway}}) and the 
> gateway identity keystore, truststore and HTTP-client keystore keep loading a 
> fresh copy, but no longer under a global lock. Sharing BCFKS instances for 
> private-key access is unsafe: {{BcFKSKeyStoreSpi.engineGetKey}} fills an 
> unsynchronized {{privateKeyCache}} HashMap, and once a key is cached it is 
> returned whatever password is supplied.
> # Add {{gateway.keystore.file.cache.enabled}} (default: true) to 
> gateway-site.xml, so the cache can be turned off without a rebuild.
> h3. Expected result
> On the hot paths, each request costs an in-memory lookup plus a stat instead 
> of 1-2 PBKDF2 runs under a global lock. Login and cookie verification 
> throughput grows with the CPU available, and no request thread waits on 
> DefaultKeystoreService for a store that is already loaded.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to