Janus Chow created HDDS-16419:
---------------------------------
Summary: Datanode failover for GetBlock/ReadChunk is nested,
causing up to N×N attempts per read
Key: HDDS-16419
URL: https://issues.apache.org/jira/browse/HDDS-16419
Project: Apache Ozone
Issue Type: Bug
Reporter: Janus Chow
Assignee: Janus Chow
When a datanode is slow or unreachable, a single GetBlock or ReadChunk call can
take N × N × ozone.client.read.timeout before failing, where N is the number of
datanodes in the pipeline.
There are two nested failover loops:
1. ContainerProtocolCalls#tryEachDatanode (added by HDDS-8024 / HDDS-8090)
iterates over the datanodes and, for each one, calls
XceiverClientSpi#sendCommand(request, validators).
2. XceiverClientGrpc#sendCommandWithRetry, which sendCommand delegates to,
ignores the datanode chosen by the outer loop and again tries every
datanode of the pipeline.
So each outer attempt is a full pipeline failover, and every datanode is
contacted N times. The outer loop sets datanodeUuid (and the replica index for
EC) for the datanode it picked, but the inner loop may send that request to a
different datanode.
Together with the read retries in BlockInputStream, the worst case for one
read becomes:
ozone.client.read.max.retries × N × N × ozone.client.read.timeout
For example, with 2 datanodes, read.timeout=5s and max.retries=3, a read to
unresponsive datanodes fails only after 60s. The client logs show
"Failed to execute command ... on the pipeline" every 10s, and
"will try another datanode" after every second one. With the defaults
(3 datanodes, 30s timeout, 3 retries) the worst case is 810s.
This is a problem for services such as S3 Gateway. The S3 client usually
times out and retries long before this, but the gateway thread stays blocked,
so threads pile up when datanodes are slow.
Neither loop can simply be removed:
- the outer loop handles errors detected after a successful response
(e.g. short read, readLen != chunk length) and stops early on
BLOCK_TOKEN_VERIFICATION_FAILED;
- the inner loop provides failover for other commands (listBlock,
readSmallFile, ...) which do not pick a datanode themselves.
Proposal: let the outer loop send each attempt only to the datanode it picked,
so GetBlock/ReadChunk contact each datanode at most once, in the same order as
today.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
