[
https://issues.apache.org/jira/browse/HDDS-16867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy reassigned HDDS-16867:
-------------------------------------
Assignee: Andrey Yarovoy
> EC: ECBlockInputStream should read data cells from datanodes in parallel
> ------------------------------------------------------------------------
>
> Key: HDDS-16867
> URL: https://issues.apache.org/jira/browse/HDDS-16867
> Project: Apache Ozone
> Issue Type: Improvement
> Components: EC Client
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> h2. Problem
> On the healthy EC read path, \{{ECBlockInputStream}} reads one cell at a time,
> serially. Each iteration of \{{readWithStrategy}} picks the data index for the
> current position, opens its internal STANDALONE block stream, does one
> synchronous read bounded by the cell, and only then moves to the next index:
> {code:java}
> // ECBlockInputStream.readWithStrategy
> while (strategy.getTargetLength() > 0 && remaining() > 0) {
> int currentIndex = currentStreamIndex();
> BlockExtendedInputStream stream = getOrOpenStream(currentIndex);
> int read = readFromStream(stream, strategy); // one datanode round trip
> totalRead += read;
> position += read;
> }
> {code}
> A read that spans \{{N}} cells therefore costs \{{N}} serial datanode round
> trips, although the cells are on \{{N}} different datanodes and do not depend
> on
> each other. The reconstruction path is already parallel (HDDS-5952,
> {\{ECBlockReconstructedStripeInputStream.loadDataBuffersFromStream}}); the
> healthy path, which serves most reads, is not.
> Through the filesystem, multi-cell reads come from callers that pass large
> buffers. Note that for encrypted (TDE) keys only \{{read(ByteBuffer)}} passes
> the
> caller's buffer through: Hadoop's \{{CryptoInputStream.read(byte[], int,
> int)}}
> reads through its internal buffer (\{{hadoop.security.crypto.buffer.size}},
> 8 KB by default), so byte-array reads on TDE keys never span more than one
> cell.
> h2. HDFS behavior
> {\{StripeReader.readStripe}} submits one read per needed data chunk of the
> stripe to an \{{ExecutorCompletionService}} on the client-wide striped-reads
> pool
> (\{{dfs.client.read.striped.threadpool.size}}) and waits for them together.
> h2. Proposal
> When a \{{read}} call spans more than one cell, read the cells of different
> data
> indexes concurrently and deliver the bytes in order.
> * Split the target range into per-index segments. Segments on the same index
> (from consecutive stripes) stay sequential on that index's stream, so the
> fan-out is at most the number of data units.
> * Keep a single-cell read on the existing synchronous path, with no executor
> hand-off.
> * Open the needed per-index streams under the stream monitor before fanning
> out
> (\{{getOrOpenStream}} mutates \{{blockStreams[]}} and is not thread-safe).
> Each
> task reads its segments into its own \{{duplicate()}} slice of the caller
> buffer; the call joins all tasks before returning, so
> {\{ECBlockInputStream.read}} keeps its \{{synchronized}} contract.
> * On return, advance the position by the contiguous prefix that was read
> successfully. A short read or failure on one index must not be hidden by
> success on a later index.
> * Keep the spare-location retry (\{{shouldRetryFailedRead}}) per index. When
> spares cannot cover a failure, throw \{{BadDataLocationException}} carrying
> every failed index of the call, so the proxy fails over once.
> * Executor: either reuse the client's EC reconstruct executor
> (\{{ozone.client.ec.reconstruct.stripe.read.pool.limit}}) or add a separate
> bounded pool for healthy reads; to be decided in review. Either way, run the
> task in the caller thread when the pool is saturated, as both the
> reconstruction pool and HDFS do today.
> * Configuration flag to turn the parallel path off.
> Out of scope: reading ahead to the end of the stripe into an internal buffer
> (HDFS \{{curStripeBuf}}). It changes how much is read for small reads and is a
> separate decision.
> h2. Compatibility
> Client only. No wire or data change. A new configuration flag turns the
> parallel path off; its default is to be decided in review.
> h2. Testing
> In \{{TestECBlockInputStream}} with \{{ECStreamTestUtil}}:
> * a multi-cell read has reads in flight on several indexes at once
> (latch-based
> test streams that block until all expected indexes have started);
> * a single-cell read never uses the executor;
> * the bytes returned match the written data for in-cell, cross-cell,
> cross-stripe and partial-last-stripe ranges, for \{{rs-3-2}}, \{{rs-6-3}} and
> {\{rs-10-4}};
> * an error on one index: position advances by the contiguous prefix only, the
> spare-retry path still works, and the exception reports all failed indexes;
> * EOF and short reads behave as in the serial implementation;
> * with the flag off, behavior is identical to today.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
