Andrey Yarovoy created HDDS-17866:
-------------------------------------
Summary: cache verified block tokens instead of decoding and
HMAC-checking the token on every request
Key: HDDS-17866
URL: https://issues.apache.org/jira/browse/HDDS-17866
Project: Apache Ozone
Issue Type: Improvement
Components: Security
Reporter: Andrey Yarovoy
Assignee: Andrey Yarovoy
h3. Problem
In secure mode, \{{HddsDispatcher.validateToken}} runs the full verification
for every container command that needs a token (every ReadChunk and GetBlock):
{noformat}
TokenVerifier.verify(cmd, encodedToken)
Token.decodeFromUrlString Base64 decode + Token.readFields
CompositeTokenVerifier -> ShortLivedTokenVerifier.verify
OzoneBlockTokenIdentifier.readFromByteArray BlockTokenSecretProto.parseFrom
verifyTokenPassword
secretKeyClient.getSecretKey(id), expiry check
ManagedSecretKey.isValidSignature(tokenId, password)
tokenId.getBytes() re-serializes the identifier that was just parsed
sign(...) HMAC (javax.crypto.Mac.doFinal)
isExpired, service check, BlockTokenVerifier access-mode check
{noformat}
A client reading a block sends the same encoded token on every request for that
block. The client encodes the token once per request
(\{{Token.encodeToUrlString}}), but the resulting string is identical each
time. So the datanode decodes, parses and HMAC-checks byte-identical tokens
again and again.
h3. Evidence
async-profiler CPU profile of a datanode serving HBase YCSB workload C (gets),
EC rs-3-2, secure cluster:
||Frame||Share of \{{HddsDispatcher.dispatch}} CPU||
|\{{HddsDispatcher.validateToken}} |14.0%|
|\{{Token.decodeFromUrlString}} (Base64 + readFields)|3.7%|
|\{{OzoneBlockTokenIdentifier.readFromByteArray}} |2.5%|
|\{{ManagedSecretKey.isValidSignature}} (HMAC; includes \{{getBytes}}
re-serialization 1.2%)|3.6%|
That is about 4% of total datanode CPU, spent on requests that carry an already
verified token. HDDS-10054 reduced the allocation cost of this path; this
change removes most of the repeated work.
h3. Proposed fix
Add a small bounded cache of successfully verified tokens to the datanode's
token verifier:
* *Key:* the exact encoded token string from the request. A hit means a
byte-identical token (identifier, password, kind, service) has already passed
the HMAC check.
* *Value:* the parsed \{{OzoneBlockTokenIdentifier}} (or the container token
identifier).
* *On a hit*, skip decode, parse and HMAC, but still run every cheap
per-request check:
** the signing secret key is still known to \{{secretKeyClient}} and not
expired;
** the token is not expired;
** the token's service matches the command's block or container ID (including
the container check from HDDS-17440);
** the token's access modes allow the command's access mode.
* *On a miss*, run the full verification as today and insert the entry only if
it succeeds. Failures are never cached, to avoid stale negative answers like
HDDS-16957.
* *Bounds:* entry count limited by a new config (for example a few thousand
entries), and each entry expires no later than its token's expiry.
* Only the secure-mode verifiers are affected. \{{NoopTokenVerifier}} is
unchanged.
Optional, separate: \{{ManagedSecretKey.isValidSignature(TokenIdentifier,
byte[])}} re-serializes the identifier with \{{getBytes()}} although
\{{Token.getIdentifier()}} already holds the signed bytes. Signing the raw
identifier bytes is equivalent only if serialization is canonical, so this
needs care and should not be part of the cache change.
h3. Security considerations
* A cache hit requires an exact match of the full encoded token, so no token
that has not passed the HMAC check is ever accepted.
* Block tokens are not individually revocable today. Expiry and secret key
rotation are the controls, and both are checked on every hit.
* Memory is bounded by the entry limit. An attacker can only cause misses,
which cost the same as today.
h3. Testing
* Unit tests for the cached verifier:
** a hit still rejects an expired token;
** a hit still rejects a token whose secret key was removed or expired;
** a hit still rejects a service mismatch or a wrong access mode;
** a different password with the same identifier misses and fails;
** verification failures are not cached;
** the size bound holds.
* Re-profile the datanode under YCSB C. \{{validateToken}} should drop from
about 14% of dispatch CPU to close to the cost of one hash lookup.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]