Andrey Yarovoy created HDDS-16871:
-------------------------------------
Summary: EC: Share failed datanodes across the blocks of a key read
Key: HDDS-16871
URL: https://issues.apache.org/jira/browse/HDDS-16871
Project: Apache Ozone
Issue Type: Improvement
Components: EC Client
Reporter: Andrey Yarovoy
h2. Problem
{\{KeyInputStream}} creates one \{{ECBlockInputStreamProxy}} per block, and each
keeps its own \{{failedLocations}}. Nothing learned about a failing datanode
while
reading one block is used for the next block of the same key. When several
blocks of a key have an index on the same unreachable datanode, each block:
* starts with the healthy reader, because at creation the location still looks
available;
* waits for the connection attempts and retries to that datanode to fail;
* fails over to the reconstruction reader (see sub-task 3 for the cost of the
failover itself);
* logs its own "Failing over to reconstruction read" warning.
h2. HDFS behavior
{\{DFSInputStream}} keeps the dead nodes it found for the life of the stream and
skips them when choosing datanodes for later blocks; optionally a client-wide
{\{DeadNodeDetector}} shares them across streams. "Could not obtain block"
warnings are logged once per node (\{{warnedNodes}}).
h2. Proposal
* Keep a failed-datanode set in \{{KeyInputStream}} for the life of the key read
and pass it to each EC block stream through the block stream factory.
* \{{ECBlockInputStreamProxy.setReaderType}} counts locations on datanodes in
that
set as unavailable, so a later block starts in reconstruction mode (or the
reconstruction reader does not select those indexes) without first trying the
datanode.
* Entries expire, so a datanode that comes back is used again; model the expiry
on the write side's \{{ozone.client.exclude.nodes.expiry.time}}.
* Only connection-level failures mark a datanode. A checksum error is a problem
of one replica, not of the node, and stays per block.
* Log the failover warning once per datanode per key read; keep later ones at
debug.
A client-wide dead-node detector, shared across streams, is out of scope.
h2. Compatibility
Client only. No wire or data change. A new expiry setting, if added, gets a
default.
h2. Testing
* Unit, with a key of several blocks sharing a failing datanode: the first block
tries the datanode and fails over; later blocks never call it and start in
reconstruction mode.
* After the expiry, the datanode is tried again.
* A checksum failure on one block does not mark the datanode for other blocks.
* The warning is logged once.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
