Andrey Yarovoy created HDDS-17051:
-------------------------------------
Summary: EC reconstruction read loses the S3 authentication and
read-consistency hint when a reader thread refreshes block locations from OM
Key: HDDS-17051
URL: https://issues.apache.org/jira/browse/HDDS-17051
Project: Apache Ozone
Issue Type: Bug
Components: EC Client
Reporter: Andrey Yarovoy
## Problem
When a datanode holding an EC data replica fails, `ECBlockInputStreamProxy`
switches to
`ECBlockReconstructedStripeInputStream`. That stream reads the remaining
indexes on the client's
`ecReconstructExecutor` pool (`loadDataBuffersFromStream` submits
`readIntoBuffer` for each selected index). Each
index's `BlockInputStream` is opened and read on the pool thread.
If that read gets a `StorageContainerException` or a connectivity error,
`BlockInputStream` refreshes the block
locations through its `refreshFunction`. In `RpcClient` that function is
`getKeyInfo(...)`, an OM call, so it runs on
the pool thread.
The OM client keeps the request context in thread-locals of
`OzoneManagerProtocolClientSideTranslatorPB`. The S3
Gateway sets them on the request thread:
- `threadLocalS3Auth`: the S3 credentials the request is authorized as
- `threadLocalReadConsistencyHint`: from the `x-ozone-read-consistency` header
`ecReconstructExecutor` is a plain `ThreadPoolExecutor`, so neither value is
set on its threads. As a result:
- **Secure S3 Gateway:** `OzoneClientCache` always sets
`ozone.s3.auth.check=true`, so with `ozone.security.enabled`
`submitRequest` throws `IllegalArgumentException("S3 Auth expected to be set
but is null ...")`.
- The retry lambda in `RpcClient.getInputStreamWithRetryFunction` catches only
`IOException`, so the exception
escapes the refresh. The task fails, and the index is marked failed.
- The refresh that would have found the new location is lost. A degraded read
that should succeed can then fail
with too few good locations, and that happens exactly when the cluster is
already degraded.
- **Without the S3 auth check** (non-secure): the refresh is sent without S3
authentication. OM then sees the
gateway's own identity instead of the S3 user's.
- **In all cases:** the refresh does not carry the request's read-consistency
hint. It uses the client's default
consistency instead.
## Proposed change
Build `ecReconstructExecutor` in `RpcClient` with the
request-context-propagating executor from HDDS-16867
(`createRequestContextPropagatingExecutor(...)` with `requestContext()`),
instead of `createThreadPoolExecutor(...)`.
For each task the wrapper:
- copies the submitter's S3 auth and read-consistency hint onto the worker
- runs the task
- restores the worker's previous values, or removes them if it had none,
including when the task throws
so pool threads never keep a request's credentials after the task ends.
The pool is created lazily, after `ozoneManagerClient`, which is already the
case for the HDDS-16867 pool. No
configuration, protocol or compatibility change.
Depends on HDDS-16867.
## Testing
- **Unit test, in `TestOzoneECClient`** (both read modes, using its existing
mock OM transport and datanode
storage):
- set S3 auth and a read-consistency hint on the reading thread;
- fail one data datanode so the read fails over to reconstruction;
- make one GetBlock on a remaining index return `CONTAINER_NOT_FOUND`
(`MockDatanodeStorage.failNextGetBlocks` from HDDS-16867), so the refresh
happens on an
`ec-reconstruct-reader-` thread;
- assert that the data is correct, that the failed GetBlock ran on a
reconstruct-pool thread, and that every OM
request carries both the S3 authentication and the hint.
It should fail without the change.
- Existing EC reconstruction tests
(`TestECBlockReconstructedStripeInputStream`, `TestOzoneECClient`) and
`TestRpcClient`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
