Andrey Yarovoy created HDDS-16869:
-------------------------------------
Summary: EC: Degraded reads should read and decode only the
requested range
Key: HDDS-16869
URL: https://issues.apache.org/jira/browse/HDDS-16869
Project: Apache Ozone
Issue Type: Improvement
Components: EC Client
Reporter: Andrey Yarovoy
h2. Problem
In reconstruction mode, \{{ECBlockReconstructedInputStream}} works in whole
stripes. \{{ECBlockReconstructedStripeInputStream}} only accepts stripe-aligned
positions (\{{seek}} throws if \{{pos % stripeSize != 0}}) and reads and decodes
the full stripe, so any read, however small, costs data-unit-count full cells
from the datanodes plus a decode of the whole stripe. This includes reads on
indexes that are healthy, which could be served directly. Small reads at
scattered offsets, such as columnar-file footer and index reads, are the worst
case.
h2. HDFS behavior
{\{StripedBlockUtil.divideByteRangeIntoStripes}} splits the requested range into
{\{AlignedStripe}}s that cover only the requested cell offsets. The reader reads
those spans from healthy data indexes as plain data and, for a missing index,
reads the same span from enough other indexes and decodes only that span. The
stateful reader reads from the current position to the end of the stripe, not
from the stripe start; pread reads exactly the requested range.
h2. Proposal
* For a requested range, compute per-index spans (offset and length within the
cell, per stripe).
* Read spans on available data indexes directly, without decoding.
* For spans on missing indexes, take the union span in cell-offset space, read
that same span from data-unit-count available indexes (data or parity) and
decode only that span. The Reed-Solomon decoder works on equal-length inputs,
so any equal-offset span can be decoded.
* Keep reading full stripes when the request covers whole stripes (large
sequential reads), where the two approaches read the same amount.
* Datanode reads of a span still respect checksum boundaries, which the
per-index block stream already handles.
h2. Compatibility
Client only. No wire, configuration or data change.
h2. Testing
* Counting stream factory: a small unaligned read in reconstruction mode reads
only the requested span from healthy indexes, and only the decode span from
the inputs needed for a missing index.
* Correctness matrix: in-cell, cross-cell, cross-stripe and partial-last-stripe
ranges, with 1 to parity-count missing indexes, for \{{rs-3-2}}, \{{rs-6-3}} and
{\{rs-10-4}}, comparing bytes with the written data.
* Large sequential reads read the same number of bytes as today.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
