[
https://issues.apache.org/jira/browse/KNOX-3579?focusedWorklogId=1055157&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-1055157
]
ASF GitHub Bot logged work on KNOX-3579:
----------------------------------------
Author: ASF GitHub Bot
Created on: 17/Nov/26 01:39
Start Date: 17/Nov/26 01:39
Worklog Time Spent: 10m
Work Description: github-actions[bot] commented on PR #1516:
URL: https://github.com/apache/knox/pull/1516#issuecomment-6935183206
## Test Results
5 files 5 suites 1m 41s ⏱️
222 tests 221 ✅ 1 💤 0 ❌
263 runs 262 ✅ 1 💤 0 ❌
Results for commit 6c9d9057.
[test-results]:data:application/gzip;base64,H4sIAEyw+2oC/02Myw6DIBQFf8Ww7oILBUt/phG4JKQqDY+V6b8XiUWXMydnNuL8jIk8B3EbSCo+d7AlTtmHtSJQqKJOeR8ZY396pWJMU3Cqt//sny7c5OcqaBcYY4iHiWVtTckP6EnJujmLjS/BxteeCcvicwUijbKKihHxIQRQze6Ao3AUQSvLjOMOqOWoyfcHXfSh7gUBAAA=
Issue Time Tracking
-------------------
Worklog Id: (was: 1055157)
Time Spent: 20m (was: 10m)
> DefaultKeystoreService reloads keystores from disk under a global lock on
> every request, capping SSO login and cookie verification throughput
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: KNOX-3579
> URL: https://issues.apache.org/jira/browse/KNOX-3579
> Project: Apache Knox
> Issue Type: Improvement
> Components: Server
> Affects Versions: 2.0.0, 1.6.0, 1.6.1, 2.1.0, 3.0.0, 2.1.1
> Reporter: Sandor Molnar
> Assignee: Sandor Molnar
> Priority: Major
> Fix For: 3.1.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> h3. Problem
> Under load, Knox serves WebSSO logins and SSO cookie verifications only up to
> a fixed rate, about 25 logins/s and 12 cookie verifications/s on one gateway
> instance, while the gateway still has CPU free. Requests beyond that rate
> queue up and eventually time out. With 1,500 or more users holding sessions
> and polling every 10 s, 64-96% of requests timed out at 60 s.
> Environment: BCFKS keystores (FIPS), one gateway instance, LDAP-backed WebSSO.
> Thread dumps taken under load show almost all Jetty request threads (245 of
> 254) BLOCKED on the single DefaultKeystoreService monitor, waiting in:
> * {{DefaultKeystoreService.getSigningKeystore}} (SSO cookie / JWT
> verification)
> * {{KnoxLdapContextFactory.getSystemPassword}} (LDAP bind)
> * {{WebSSOResource.init}} -> {{TokenUtils.getSignatureAlgorithm}} (WebSSO
> login)
> In every dump, the thread holding the monitor is in
> {{DefaultKeystoreService.loadKeyStore}} -> {{KeyStore.load}} -> BCFKS
> {{engineLoad}} -> {{PKCS5S2ParametersGenerator}} (PBKDF2-HMAC-SHA512).
> h3. Root cause
> # *Loaded keystores are never cached.* Every {{getSigningKeystore()}} and
> {{getCredentialStoreForCluster()}} call reads the file and calls
> {{KeyStore.load()}}. For BCFKS, that runs PBKDF2 for the integrity check,
> which costs tens of milliseconds of CPU per load. Only credential values
> (alias -> password) are cached, and only for aliases that exist. A lookup for
> an alias that doesn't exist reloads the whole credential store every time.
> # *One global lock serializes all loads.* {{getKeystore}}, {{loadKeyStore}},
> {{getCredentialForCluster}} and the write methods are all synchronized on the
> service instance. The gateway therefore runs one PBKDF2 at a time, however
> many cores it has.
> Hot paths that reload a store on every request:
> ||Path||Store(s) reloaded on every request||
> |SSO cookie / JWT verification
> ({{DefaultTokenAuthorityService.selectVerificationKey}})|The signing
> keystore, always. Also the gateway-level credential store when the signing
> keystore password alias (default: gateway-identity-keystore-password) is not
> defined.|
> |LDAP bind ({{KnoxLdapContextFactory.getSystemPassword}})|The topology
> credential store, when the system password alias is defined only at gateway
> level.|
> |WebSSO login ({{WebSSOResource.init}} ->
> {{TokenUtils.getSignatureAlgorithm}})|The gateway-level credential store,
> when gateway.signing.hmac.secret is not defined.|
> Throughput is therefore capped at roughly 1 / (PBKDF2 time x loads per
> request).
> h3. Proposed fix
> # Cache the loaded signing keystore and credential stores in memory, keyed by
> path and type. On every access, check the cached copy against a cheap file
> stamp (last-modified time, size, file key) and reload only when the file has
> changed. External updates, such as {{knoxcli create-alias}} or re-mounted
> secrets, are still picked up. Keep a digest of the store password with each
> entry, so a wrong password still fails as it does today. Re-check once any
> entry that was loaded within the mtime granularity window of its file's last
> change.
> # Replace the global monitor with a {{ReentrantReadWriteLock}}. Reads share
> the lock. Writes (add/remove alias, create store, add self-signed cert) take
> it exclusively. Loads of the same file run one at a time. Within one JVM this
> is required anyway: concurrent {{FileChannel.lock()}} calls on the same file
> throw {{OverlappingFileLockException}}. The credential-cache hit path becomes
> lock-free.
> # Writers ({{addCredentialsForCluster}}, {{removeCredentialsForCluster}},
> {{addCertForGateway}}) load a private copy under the write lock and drop the
> cached entry after writing, so a shared cached instance is never modified.
> # Private-key lookups ({{getSigningKey}}, {{getKeyForGateway}}) and the
> gateway identity keystore, truststore and HTTP-client keystore keep loading a
> fresh copy, but no longer under a global lock. Sharing BCFKS instances for
> private-key access is unsafe: {{BcFKSKeyStoreSpi.engineGetKey}} fills an
> unsynchronized {{privateKeyCache}} HashMap, and once a key is cached it is
> returned whatever password is supplied.
> # Add {{gateway.keystore.file.cache.enabled}} (default: true) to
> gateway-site.xml, so the cache can be turned off without a rebuild.
> h3. Expected result
> On the hot paths, each request costs an in-memory lookup plus a stat instead
> of 1-2 PBKDF2 runs under a global lock. Login and cookie verification
> throughput grows with the CPU available, and no request thread waits on
> DefaultKeystoreService for a store that is already loaded.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)