symious opened a new pull request, #11239:
URL: https://github.com/apache/ozone/pull/11239
## What changes were proposed in this pull request?
`ContainerProtocolCalls#getBlock` and `#readChunk` fail over between
datanodes
via `tryEachDatanode`. For each datanode, they call
`XceiverClientSpi#sendCommand(request, validators)`. In `XceiverClientGrpc`
this
goes to `sendCommandWithRetry`, which again tries every datanode of the
pipeline. As a result:
- every datanode is contacted N times, so a single call can take
`N × N × ozone.client.read.timeout` before failing, and a read can take
`read.max.retries × N × N × read.timeout`;
- the request built for datanode A (`datanodeUuid`, EC replica index) may be
sent to datanode B.
This PR makes `tryEachDatanode` the only failover layer for these two
commands:
- `XceiverClientSpi`
- Add `sendCommand(request, validators, datanode)`, which sends the command
to the given datanode only. The default implementation falls back to
`sendCommand(request, validators)`, so Ratis and short-circuit clients
are
unchanged.
- Add `getDatanodesInOrder(blockID, cmdType)`. The default implementation
returns the datanodes in `Pipeline#getClosestNode` order, which is the
current behavior of `tryEachDatanode`.
- `XceiverClientGrpc`
- Extract the single-datanode part of `sendCommandWithRetry` into
`sendCommandToDatanode`: send, run validators, update the GetBlock
datanode cache. Both the existing retry loop and the new
`sendCommand(request, validators, datanode)` use it.
- `getDatanodesInOrder` reuses `sortDatanodes`, so the datanode order
(leader / cached GetBlock datanode first, non-IN_SERVICE last,
topology-aware or shuffled) is the same as before.
- `ContainerProtocolCalls`
- `tryEachDatanode` iterates over `getDatanodesInOrder`.
- The per-datanode GetBlock/ReadChunk calls use the new single-datanode
`sendCommand`.
Other commands (`listBlock`, `readSmallFile`, ...) still use the inner
failover
and are not affected.
After this change, the worst case for a read is
`read.max.retries × N × read.timeout`.
Note: `getBlockFromDatanode` (used by `ozone debug replicas chunk-info`) now
really queries only the given datanode. Before this change it could silently
return the result from another datanode.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16419
## How was this patch tested?
- `TestXceiverClientGrpc`
- `testGetBlockRetryAlNodes` / `testReadChunkRetryAllNodes` now assert that
each datanode is contacted exactly once (previously N² calls), and that
the request's `datanodeUuid` matches the datanode it is sent to.
- New `testReadChunkRetryNextNodeOnShortRead`: a short read from the first
datanode is retried on the next datanode.
- Existing ordering tests (leader, topology, IN_SERVICE first, connection
reuse after GetBlock) pass unchanged.
- `TestChunkInputStream`, `TestBlockInputStream` pass.
- Manually reproduced on a test cluster: with read.timeout=5s and 2
unresponsive datanodes, the read failed after 60s before this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
