[ 
https://issues.apache.org/jira/browse/HDDS-17716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Yarovoy updated HDDS-17716:
----------------------------------
    Description: 
*Description*

Every datanode read acquires an `XceiverClientSpi` through 
`XceiverClientManager.acquireClient`. Callers include:
 - `ChunkInputStream.acquireClient`;
 - `BlockInputStream.acquireClient`;
 - `StreamBlockInputStream.acquireGrpcClientForReadData`, called on 
*{*}every{*}* positioned read;
 - `releaseClient(..., invalidate=true)`.

`acquireClient` builds the cache key with `getPipelineCacheKey`. In secure mode 
that appends the user:
{code:java}
if (isSecurityEnabled()) {
key.append('|').append(UserGroupInformation.getCurrentUser().getShortUserName());
}
{code}
`UserGroupInformation.getCurrentUser()` calls `AccessController.getContext()` → 
`Subject.getSubject`, which walks the thread stack in the JVM
(`JVM_GetStackAccessControlContext`). HBase calls Ozone from deep RPC-handler 
stacks, so the walk is long. The key is also built inside
`synchronized (clientCache)`, a monitor shared by every reader in the process, 
so the walk extends that critical section.

Evidence (HBase YCSB on EC, RegionServer async-profiler JFR, 6×60s per arm)
| |stream read ON|stream read OFF|
|—|—|—|
|`getPipelineCacheKey` → `UGI.getCurrentUser`, % RS CPU|2.5%|3.4%|
|share of the EC read path CPU|~26% (18% pread acquire + 8.5% initialize)|~25%|
|wall: `acquireClient`, % busy handler time|0.9%|1.3%|

`JVM_GetStackAccessControlContext` is among the top self frames (1.1–1.4% of 
all RS CPU). At this load the monitor is not contended; at higher reader 
concurrency the stack walk inside `synchronized (clientCache)` becomes a convoy.

*Proposed fix*

Resolve the user *{*}once per input stream{*}*, not once per request, and pass 
it to the factory:

1. Capture `UserGroupInformation.getCurrentUser().getShortUserName()` (secure 
mode only) when the stream is created
(`KeyInputStream` / `BlockInputStreamFactoryImpl`). The stream is opened in the 
caller's security context, which is the context that should own its datanode 
clients.
2. Add `XceiverClientFactory.acquireClientForReadData(Pipeline, boolean 
allowShortCircuit, String user)` (default method delegates to the existing 
one), and the matching release. `getPipelineCacheKey` uses the supplied user 
when non-null.
3. Compute the key *{*}before{*}* entering `synchronized (clientCache)`; only 
the cache get/refcount needs the lock.

Write paths (`BlockOutputStream`) already acquire their client once per block, 
so they need no change.

Rejected alternatives
 - *{*}Cache the user per `XceiverClientManager`.{*}* Wrong: the key carries 
the user precisely so different users (`doAs`) never share a client.
 - *{*}ThreadLocal of the last user.{*}* Wrong under `doAs` on pooled threads.

Compatibility / side effects
 - *{*}Semantics change:{*}* the datanode client is chosen by the user that 
*{*}opened{*}* the stream, not the user active at each `read` call. For a 
stream shared across `doAs` contexts this differs. Today such sharing silently 
mixes users too; tokens are per request and checked on the datanode.
 - No wire change. New default method on `XceiverClientFactory`.

Tests
 - Unit: `TestXceiverClientManager` — the same pipeline with two users gives 
two clients; the per-stream user is honored; the key is computed without 
holding the cache lock (verify with a lock-held assertion hook).
 - Unit (Mockito): `getCurrentUser` is invoked once per stream, not once per 
read (static mock of `UserGroupInformation`).
 - Integration: secure mini-cluster read tests (`TestSecureOzoneRpcClient`, EC 
read tests).
 - Benchmark: `BlockInputStream`/`ECBlockInputStream` pread loop (4 KB–64 KB 
reads, 1/8/32 threads), CPU per read and throughput A/B, n=3 per arm.

  was:
*Description*

Every datanode read acquires an `XceiverClientSpi` through 
`XceiverClientManager.acquireClient`. Callers include:
- `ChunkInputStream.acquireClient`;
- `BlockInputStream.acquireClient`;
- `StreamBlockInputStream.acquireGrpcClientForReadData`, called on **every** 
positioned read;
- `releaseClient(..., invalidate=true)`.

`acquireClient` builds the cache key with `getPipelineCacheKey`. In secure mode 
that appends the user:

{code:java}
if (isSecurityEnabled()) {
key.append('|').append(UserGroupInformation.getCurrentUser().getShortUserName());
}
{code}

`UserGroupInformation.getCurrentUser()` calls `AccessController.getContext()` → 
`Subject.getSubject`, which walks the thread stack in the JVM
(`JVM_GetStackAccessControlContext`). HBase calls Ozone from deep RPC-handler 
stacks, so the walk is long. The key is also built inside
`synchronized (clientCache)`, a monitor shared by every reader in the process, 
so the walk extends that critical section.

### Evidence (HBase YCSB on EC, RegionServer async-profiler JFR, 6×60s per arm)

| | stream read ON | stream read OFF |
|---|---|---|
| `getPipelineCacheKey` → `UGI.getCurrentUser`, % RS CPU | 2.5% | 3.4% |
| share of the EC read path CPU | ~26% (18% pread acquire + 8.5% initialize) | 
~25% |
| wall: `acquireClient`, % busy handler time | 0.9% | 1.3% |

`JVM_GetStackAccessControlContext` is among the top self frames (1.1–1.4% of 
all RS CPU). At this load the monitor is not contended; at higher reader 
concurrency the stack walk inside `synchronized (clientCache)` becomes a convoy.

*Proposed fix*

Resolve the user **once per input stream**, not once per request, and pass it 
to the factory:

1. Capture `UserGroupInformation.getCurrentUser().getShortUserName()` (secure 
mode only) when the stream is created
(`KeyInputStream` / `BlockInputStreamFactoryImpl`). The stream is opened in the 
caller's security context, which is the context that should own its datanode 
clients.
2. Add `XceiverClientFactory.acquireClientForReadData(Pipeline, boolean 
allowShortCircuit, String user)` (default method delegates to the existing 
one), and the matching release. `getPipelineCacheKey` uses the supplied user 
when non-null.
3. Compute the key **before** entering `synchronized (clientCache)`; only the 
cache get/refcount needs the lock.

Write paths (`BlockOutputStream`) already acquire their client once per block, 
so they need no change.

### Rejected alternatives

- **Cache the user per `XceiverClientManager`.** Wrong: the key carries the 
user precisely so different users (`doAs`) never share a client.
- **ThreadLocal of the last user.** Wrong under `doAs` on pooled threads.

## Compatibility / side effects

- **Semantics change:** the datanode client is chosen by the user that 
**opened** the stream, not the user active at each `read` call. For a stream 
shared across `doAs` contexts this differs. Today such sharing silently mixes 
users too; tokens are per request and checked on the datanode.
- No wire change. New default method on `XceiverClientFactory`.

## Tests

- Unit: `TestXceiverClientManager` — the same pipeline with two users gives two 
clients; the per-stream user is honored; the key is computed without holding 
the cache lock (verify with a lock-held assertion hook).
- Unit (Mockito): `getCurrentUser` is invoked once per stream, not once per 
read (static mock of `UserGroupInformation`).
- Integration: secure mini-cluster read tests (`TestSecureOzoneRpcClient`, EC 
read tests).
- Benchmark: `BlockInputStream`/`ECBlockInputStream` pread loop (4 KB–64 KB 
reads, 1/8/32 threads), CPU per read and throughput A/B, n=3 per arm.


> Avoid UserGroupInformation.getCurrentUser() per read request in 
> XceiverClientManager.getPipelineCacheKey
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-17716
>                 URL: https://issues.apache.org/jira/browse/HDDS-17716
>             Project: Apache Ozone
>          Issue Type: Improvement
>          Components: Ozone Client
>            Reporter: Andrey Yarovoy
>            Assignee: Andrey Yarovoy
>            Priority: Major
>
> *Description*
> Every datanode read acquires an `XceiverClientSpi` through 
> `XceiverClientManager.acquireClient`. Callers include:
>  - `ChunkInputStream.acquireClient`;
>  - `BlockInputStream.acquireClient`;
>  - `StreamBlockInputStream.acquireGrpcClientForReadData`, called on 
> *{*}every{*}* positioned read;
>  - `releaseClient(..., invalidate=true)`.
> `acquireClient` builds the cache key with `getPipelineCacheKey`. In secure 
> mode that appends the user:
> {code:java}
> if (isSecurityEnabled()) {
> key.append('|').append(UserGroupInformation.getCurrentUser().getShortUserName());
> }
> {code}
> `UserGroupInformation.getCurrentUser()` calls `AccessController.getContext()` 
> → `Subject.getSubject`, which walks the thread stack in the JVM
> (`JVM_GetStackAccessControlContext`). HBase calls Ozone from deep RPC-handler 
> stacks, so the walk is long. The key is also built inside
> `synchronized (clientCache)`, a monitor shared by every reader in the 
> process, so the walk extends that critical section.
> Evidence (HBase YCSB on EC, RegionServer async-profiler JFR, 6×60s per arm)
> | |stream read ON|stream read OFF|
> |—|—|—|
> |`getPipelineCacheKey` → `UGI.getCurrentUser`, % RS CPU|2.5%|3.4%|
> |share of the EC read path CPU|~26% (18% pread acquire + 8.5% 
> initialize)|~25%|
> |wall: `acquireClient`, % busy handler time|0.9%|1.3%|
> `JVM_GetStackAccessControlContext` is among the top self frames (1.1–1.4% of 
> all RS CPU). At this load the monitor is not contended; at higher reader 
> concurrency the stack walk inside `synchronized (clientCache)` becomes a 
> convoy.
> *Proposed fix*
> Resolve the user *{*}once per input stream{*}*, not once per request, and 
> pass it to the factory:
> 1. Capture `UserGroupInformation.getCurrentUser().getShortUserName()` (secure 
> mode only) when the stream is created
> (`KeyInputStream` / `BlockInputStreamFactoryImpl`). The stream is opened in 
> the caller's security context, which is the context that should own its 
> datanode clients.
> 2. Add `XceiverClientFactory.acquireClientForReadData(Pipeline, boolean 
> allowShortCircuit, String user)` (default method delegates to the existing 
> one), and the matching release. `getPipelineCacheKey` uses the supplied user 
> when non-null.
> 3. Compute the key *{*}before{*}* entering `synchronized (clientCache)`; only 
> the cache get/refcount needs the lock.
> Write paths (`BlockOutputStream`) already acquire their client once per 
> block, so they need no change.
> Rejected alternatives
>  - *{*}Cache the user per `XceiverClientManager`.{*}* Wrong: the key carries 
> the user precisely so different users (`doAs`) never share a client.
>  - *{*}ThreadLocal of the last user.{*}* Wrong under `doAs` on pooled threads.
> Compatibility / side effects
>  - *{*}Semantics change:{*}* the datanode client is chosen by the user that 
> *{*}opened{*}* the stream, not the user active at each `read` call. For a 
> stream shared across `doAs` contexts this differs. Today such sharing 
> silently mixes users too; tokens are per request and checked on the datanode.
>  - No wire change. New default method on `XceiverClientFactory`.
> Tests
>  - Unit: `TestXceiverClientManager` — the same pipeline with two users gives 
> two clients; the per-stream user is honored; the key is computed without 
> holding the cache lock (verify with a lock-held assertion hook).
>  - Unit (Mockito): `getCurrentUser` is invoked once per stream, not once per 
> read (static mock of `UserGroupInformation`).
>  - Integration: secure mini-cluster read tests (`TestSecureOzoneRpcClient`, 
> EC read tests).
>  - Benchmark: `BlockInputStream`/`ECBlockInputStream` pread loop (4 KB–64 KB 
> reads, 1/8/32 threads), CPU per read and throughput A/B, n=3 per arm.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to