[
https://issues.apache.org/jira/browse/HDDS-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy reassigned HDDS-17051:
-------------------------------------
Assignee: Andrey Yarovoy
> EC reconstruction read loses the S3 authentication and read-consistency hint
> when a reader thread refreshes block locations from OM
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17051
> URL: https://issues.apache.org/jira/browse/HDDS-17051
> Project: Apache Ozone
> Issue Type: Bug
> Components: EC Client
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> *Problem*
> When a datanode holding an EC data replica fails, `ECBlockInputStreamProxy`
> switches to
> `ECBlockReconstructedStripeInputStream`. That stream reads the remaining
> indexes on the client's
> `ecReconstructExecutor` pool (`loadDataBuffersFromStream` submits
> `readIntoBuffer` for each selected index). Each
> index's `BlockInputStream` is opened and read on the pool thread.
> If that read gets a `StorageContainerException` or a connectivity error,
> `BlockInputStream` refreshes the block
> locations through its `refreshFunction`. In `RpcClient` that function is
> `getKeyInfo(...)`, an OM call, so it runs on
> the pool thread.
> The OM client keeps the request context in thread-locals of
> `OzoneManagerProtocolClientSideTranslatorPB`. The S3
> Gateway sets them on the request thread:
> - `threadLocalS3Auth`: the S3 credentials the request is authorized as
> - `threadLocalReadConsistencyHint`: from the `x-ozone-read-consistency`
> header
> `ecReconstructExecutor` is a plain `ThreadPoolExecutor`, so neither value is
> set on its threads. As a result:
> - *{*}Secure S3 Gateway:{*}* `OzoneClientCache` always sets
> `ozone.s3.auth.check=true`, so with `ozone.security.enabled`
> `submitRequest` throws `IllegalArgumentException("S3 Auth expected to be set
> but is null ...")`.
> - The retry lambda in `RpcClient.getInputStreamWithRetryFunction` catches
> only `IOException`, so the exception
> escapes the refresh. The task fails, and the index is marked failed.
> - The refresh that would have found the new location is lost. A degraded
> read that should succeed can then fail
> with too few good locations, and that happens exactly when the cluster is
> already degraded.
> - *{*}Without the S3 auth check{*}* (non-secure): the refresh is sent
> without S3 authentication. OM then sees the
> gateway's own identity instead of the S3 user's.
> - *{*}In all cases:{*}* the refresh does not carry the request's
> read-consistency hint. It uses the client's default
> consistency instead.
> *Proposed change*
> Build `ecReconstructExecutor` in `RpcClient` with the
> request-context-propagating executor from HDDS-16867
> (`createRequestContextPropagatingExecutor(...)` with `requestContext()`),
> instead of `createThreadPoolExecutor(...)`.
> For each task the wrapper:
> - copies the submitter's S3 auth and read-consistency hint onto the worker
> - runs the task
> - restores the worker's previous values, or removes them if it had none,
> including when the task throws
> so pool threads never keep a request's credentials after the task ends.
> The pool is created lazily, after `ozoneManagerClient`, which is already the
> case for the HDDS-16867 pool. No
> configuration, protocol or compatibility change.
> Depends on HDDS-16867.
> *Testing*
> - *{*}Unit test, in `TestOzoneECClient`{*}* (both read modes, using its
> existing mock OM transport and datanode
> storage):
> - set S3 auth and a read-consistency hint on the reading thread;
> - fail one data datanode so the read fails over to reconstruction;
> - make one GetBlock on a remaining index return `CONTAINER_NOT_FOUND`
> (`MockDatanodeStorage.failNextGetBlocks` from HDDS-16867), so the refresh
> happens on an
> `ec-reconstruct-reader-` thread;
> - assert that the data is correct, that the failed GetBlock ran on a
> reconstruct-pool thread, and that every OM
> request carries both the S3 authentication and the hint.
> It should fail without the change.
> - Existing EC reconstruction tests
> (`TestECBlockReconstructedStripeInputStream`, `TestOzoneECClient`) and
> `TestRpcClient`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
