[
https://issues.apache.org/jira/browse/HDDS-17716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy updated HDDS-17716:
----------------------------------
Description:
*Description*
Every datanode read acquires an `XceiverClientSpi` through
`XceiverClientManager.acquireClient`. Callers include:
- `ChunkInputStream.acquireClient`;
- `BlockInputStream.acquireClient`;
- `StreamBlockInputStream.acquireGrpcClientForReadData`, called on
*{*}every{*}* positioned read;
- `releaseClient(..., invalidate=true)`.
`acquireClient` builds the cache key with `getPipelineCacheKey`. In secure mode
that appends the user:
{code:java}
if (isSecurityEnabled()) {
key.append('|').append(UserGroupInformation.getCurrentUser().getShortUserName());
}
{code}
`UserGroupInformation.getCurrentUser()` calls `AccessController.getContext()` →
`Subject.getSubject`, which walks the thread stack in the JVM
(`JVM_GetStackAccessControlContext`). HBase calls Ozone from deep RPC-handler
stacks, so the walk is long. The key is also built inside
`synchronized (clientCache)`, a monitor shared by every reader in the process,
so the walk extends that critical section.
Evidence (HBase YCSB on EC, RegionServer async-profiler JFR, 6×60s per arm)
| |stream read ON|stream read OFF|
|—|—|—|
|`getPipelineCacheKey` → `UGI.getCurrentUser`, % RS CPU|2.5%|3.4%|
|share of the EC read path CPU|~26% (18% pread acquire + 8.5% initialize)|~25%|
|wall: `acquireClient`, % busy handler time|0.9%|1.3%|
`JVM_GetStackAccessControlContext` is among the top self frames (1.1–1.4% of
all RS CPU). At this load the monitor is not contended; at higher reader
concurrency the stack walk inside `synchronized (clientCache)` becomes a convoy.
*Proposed fix*
Resolve the user *{*}once per input stream{*}*, not once per request, and pass
it to the factory:
1. Capture `UserGroupInformation.getCurrentUser().getShortUserName()` (secure
mode only) when the stream is created
(`KeyInputStream` / `BlockInputStreamFactoryImpl`). The stream is opened in the
caller's security context, which is the context that should own its datanode
clients.
2. Add `XceiverClientFactory.acquireClientForReadData(Pipeline, boolean
allowShortCircuit, String user)` (default method delegates to the existing
one), and the matching release. `getPipelineCacheKey` uses the supplied user
when non-null.
3. Compute the key *{*}before{*}* entering `synchronized (clientCache)`; only
the cache get/refcount needs the lock.
Write paths (`BlockOutputStream`) already acquire their client once per block,
so they need no change.
Rejected alternatives
- *{*}Cache the user per `XceiverClientManager`.{*}* Wrong: the key carries
the user precisely so different users (`doAs`) never share a client.
- *{*}ThreadLocal of the last user.{*}* Wrong under `doAs` on pooled threads.
Compatibility / side effects
- *{*}Semantics change:{*}* the datanode client is chosen by the user that
*{*}opened{*}* the stream, not the user active at each `read` call. For a
stream shared across `doAs` contexts this differs. Today such sharing silently
mixes users too; tokens are per request and checked on the datanode.
- No wire change. New default method on `XceiverClientFactory`.
Tests
- Unit: `TestXceiverClientManager` — the same pipeline with two users gives
two clients; the per-stream user is honored; the key is computed without
holding the cache lock (verify with a lock-held assertion hook).
- Unit (Mockito): `getCurrentUser` is invoked once per stream, not once per
read (static mock of `UserGroupInformation`).
- Integration: secure mini-cluster read tests (`TestSecureOzoneRpcClient`, EC
read tests).
- Benchmark: `BlockInputStream`/`ECBlockInputStream` pread loop (4 KB–64 KB
reads, 1/8/32 threads), CPU per read and throughput A/B, n=3 per arm.
was:
*Description*
Every datanode read acquires an `XceiverClientSpi` through
`XceiverClientManager.acquireClient`. Callers include:
- `ChunkInputStream.acquireClient`;
- `BlockInputStream.acquireClient`;
- `StreamBlockInputStream.acquireGrpcClientForReadData`, called on **every**
positioned read;
- `releaseClient(..., invalidate=true)`.
`acquireClient` builds the cache key with `getPipelineCacheKey`. In secure mode
that appends the user:
{code:java}
if (isSecurityEnabled()) {
key.append('|').append(UserGroupInformation.getCurrentUser().getShortUserName());
}
{code}
`UserGroupInformation.getCurrentUser()` calls `AccessController.getContext()` →
`Subject.getSubject`, which walks the thread stack in the JVM
(`JVM_GetStackAccessControlContext`). HBase calls Ozone from deep RPC-handler
stacks, so the walk is long. The key is also built inside
`synchronized (clientCache)`, a monitor shared by every reader in the process,
so the walk extends that critical section.
### Evidence (HBase YCSB on EC, RegionServer async-profiler JFR, 6×60s per arm)
| | stream read ON | stream read OFF |
|---|---|---|
| `getPipelineCacheKey` → `UGI.getCurrentUser`, % RS CPU | 2.5% | 3.4% |
| share of the EC read path CPU | ~26% (18% pread acquire + 8.5% initialize) |
~25% |
| wall: `acquireClient`, % busy handler time | 0.9% | 1.3% |
`JVM_GetStackAccessControlContext` is among the top self frames (1.1–1.4% of
all RS CPU). At this load the monitor is not contended; at higher reader
concurrency the stack walk inside `synchronized (clientCache)` becomes a convoy.
*Proposed fix*
Resolve the user **once per input stream**, not once per request, and pass it
to the factory:
1. Capture `UserGroupInformation.getCurrentUser().getShortUserName()` (secure
mode only) when the stream is created
(`KeyInputStream` / `BlockInputStreamFactoryImpl`). The stream is opened in the
caller's security context, which is the context that should own its datanode
clients.
2. Add `XceiverClientFactory.acquireClientForReadData(Pipeline, boolean
allowShortCircuit, String user)` (default method delegates to the existing
one), and the matching release. `getPipelineCacheKey` uses the supplied user
when non-null.
3. Compute the key **before** entering `synchronized (clientCache)`; only the
cache get/refcount needs the lock.
Write paths (`BlockOutputStream`) already acquire their client once per block,
so they need no change.
### Rejected alternatives
- **Cache the user per `XceiverClientManager`.** Wrong: the key carries the
user precisely so different users (`doAs`) never share a client.
- **ThreadLocal of the last user.** Wrong under `doAs` on pooled threads.
## Compatibility / side effects
- **Semantics change:** the datanode client is chosen by the user that
**opened** the stream, not the user active at each `read` call. For a stream
shared across `doAs` contexts this differs. Today such sharing silently mixes
users too; tokens are per request and checked on the datanode.
- No wire change. New default method on `XceiverClientFactory`.
## Tests
- Unit: `TestXceiverClientManager` — the same pipeline with two users gives two
clients; the per-stream user is honored; the key is computed without holding
the cache lock (verify with a lock-held assertion hook).
- Unit (Mockito): `getCurrentUser` is invoked once per stream, not once per
read (static mock of `UserGroupInformation`).
- Integration: secure mini-cluster read tests (`TestSecureOzoneRpcClient`, EC
read tests).
- Benchmark: `BlockInputStream`/`ECBlockInputStream` pread loop (4 KB–64 KB
reads, 1/8/32 threads), CPU per read and throughput A/B, n=3 per arm.
> Avoid UserGroupInformation.getCurrentUser() per read request in
> XceiverClientManager.getPipelineCacheKey
> --------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17716
> URL: https://issues.apache.org/jira/browse/HDDS-17716
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Client
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> *Description*
> Every datanode read acquires an `XceiverClientSpi` through
> `XceiverClientManager.acquireClient`. Callers include:
> - `ChunkInputStream.acquireClient`;
> - `BlockInputStream.acquireClient`;
> - `StreamBlockInputStream.acquireGrpcClientForReadData`, called on
> *{*}every{*}* positioned read;
> - `releaseClient(..., invalidate=true)`.
> `acquireClient` builds the cache key with `getPipelineCacheKey`. In secure
> mode that appends the user:
> {code:java}
> if (isSecurityEnabled()) {
> key.append('|').append(UserGroupInformation.getCurrentUser().getShortUserName());
> }
> {code}
> `UserGroupInformation.getCurrentUser()` calls `AccessController.getContext()`
> → `Subject.getSubject`, which walks the thread stack in the JVM
> (`JVM_GetStackAccessControlContext`). HBase calls Ozone from deep RPC-handler
> stacks, so the walk is long. The key is also built inside
> `synchronized (clientCache)`, a monitor shared by every reader in the
> process, so the walk extends that critical section.
> Evidence (HBase YCSB on EC, RegionServer async-profiler JFR, 6×60s per arm)
> | |stream read ON|stream read OFF|
> |—|—|—|
> |`getPipelineCacheKey` → `UGI.getCurrentUser`, % RS CPU|2.5%|3.4%|
> |share of the EC read path CPU|~26% (18% pread acquire + 8.5%
> initialize)|~25%|
> |wall: `acquireClient`, % busy handler time|0.9%|1.3%|
> `JVM_GetStackAccessControlContext` is among the top self frames (1.1–1.4% of
> all RS CPU). At this load the monitor is not contended; at higher reader
> concurrency the stack walk inside `synchronized (clientCache)` becomes a
> convoy.
> *Proposed fix*
> Resolve the user *{*}once per input stream{*}*, not once per request, and
> pass it to the factory:
> 1. Capture `UserGroupInformation.getCurrentUser().getShortUserName()` (secure
> mode only) when the stream is created
> (`KeyInputStream` / `BlockInputStreamFactoryImpl`). The stream is opened in
> the caller's security context, which is the context that should own its
> datanode clients.
> 2. Add `XceiverClientFactory.acquireClientForReadData(Pipeline, boolean
> allowShortCircuit, String user)` (default method delegates to the existing
> one), and the matching release. `getPipelineCacheKey` uses the supplied user
> when non-null.
> 3. Compute the key *{*}before{*}* entering `synchronized (clientCache)`; only
> the cache get/refcount needs the lock.
> Write paths (`BlockOutputStream`) already acquire their client once per
> block, so they need no change.
> Rejected alternatives
> - *{*}Cache the user per `XceiverClientManager`.{*}* Wrong: the key carries
> the user precisely so different users (`doAs`) never share a client.
> - *{*}ThreadLocal of the last user.{*}* Wrong under `doAs` on pooled threads.
> Compatibility / side effects
> - *{*}Semantics change:{*}* the datanode client is chosen by the user that
> *{*}opened{*}* the stream, not the user active at each `read` call. For a
> stream shared across `doAs` contexts this differs. Today such sharing
> silently mixes users too; tokens are per request and checked on the datanode.
> - No wire change. New default method on `XceiverClientFactory`.
> Tests
> - Unit: `TestXceiverClientManager` — the same pipeline with two users gives
> two clients; the per-stream user is honored; the key is computed without
> holding the cache lock (verify with a lock-held assertion hook).
> - Unit (Mockito): `getCurrentUser` is invoked once per stream, not once per
> read (static mock of `UserGroupInformation`).
> - Integration: secure mini-cluster read tests (`TestSecureOzoneRpcClient`,
> EC read tests).
> - Benchmark: `BlockInputStream`/`ECBlockInputStream` pread loop (4 KB–64 KB
> reads, 1/8/32 threads), CPU per read and throughput A/B, n=3 per arm.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]