Chu Cheng Li created HDDS-16563:
-----------------------------------
Summary: Streaming ReadBlock can return no data at the end of a
block, surfacing to the client as a false EOF
Key: HDDS-16563
URL: https://issues.apache.org/jira/browse/HDDS-16563
Project: Apache Ozone
Issue Type: Bug
Reporter: Chu Cheng Li
Assignee: Chu Cheng Li
*Problem*
With {{{}ozone.client.stream.readblock.enable=true{}}}, a read of the last
bytes of a block can come back with zero data. The block input stream reports
end of stream, and {{KeyInputStream}} turns the short read into:
{code:java}
java.io.IOException: Inconsistent read for blockID=conID: 96 locID:
117883640217602265 bcsId: 35223
replicaIndex: null length=147058825 position=147054729 numBytesToRead=4096
numBytesRead=-1
at
org.apache.hadoop.ozone.client.io.KeyInputStream.checkPartBytesRead(KeyInputStream.java:189)
{code}
The read is the last 4096 bytes of the block: 147054729 + 4096 = 147058825 =
the block length the client got from OM. Reading the same key with streaming
reads disabled works.
Nothing is logged on the datanode for this block, and the client sees a clean
end of stream rather than an error, so the failure is invisible on both sides.
*Impact*
HBase on Ozone: a regionserver reading an HFile trailer gets
{code:java}
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile
Trailer from file
ofs://.../f/0a2fdc103067494f87f4174fa4c02acd
{code}
which fails bulk load (the tool retried 152 times and gave up) and would
equally fail region open, i.e. the data is unreadable for as long as the
condition lasts. A healthy file is reported as
corrupt.
*Steps to reproduce (as observed)*
# 3 datanodes, FSO bucket, RATIS/THREE.
# Copy a large directory into the bucket: {{ozone fs -put -t 16 <55 GB local
dir> ofs://...}}
(151 HFiles of ~355 MB each; this created blocks of varying size, the affected
one 147058825 bytes, not the 256 MB default).
# Read the tail of each file. 150 of 151 files read fine; one failed
consistently.
Reading that one file with the same command and three client configurations:
||read path||result||
|default (non-streaming)|reads fine|
|{{ozone.client.stream.readblock.enable=true}}|CorruptHFileException,
"Inconsistent read", numBytesRead=-1|
|Ratis data stream read (HDDS-9904)|CorruptHFileException|
Both streaming paths fail; the non-streaming path does not. Copying the same
local file to a new Ozone path made it readable through every path, so the file
content is fine.
*Two observations that should guide the investigation*
# *It is not permanent.* About 30 minutes later the original file read fine
through the streaming path. In between, all open containers in the cluster were
closed
({{{}ozone admin container close{}}}). The leading hypothesis is therefore that
this affects blocks in *open* containers, where the length the datanode serves
lags the length OM reports, and that it heals when the container closes. This
was not verified directly; the container state of container 96 at failure time
was not captured.
# *The non-streaming path masks it.* {{BlockInputStream}} fails over to
another replica on error, while the streaming reader is pinned to one datanode
and only excludes it when the outcome is treated as a failure
({{{}failedStreamingDatanodes{}}},
StreamBlockInputStream#getStreamingReader). A stream that ends with no data is
not a failure, so there is no failover and the read fails deterministically
against that replica.
*Candidate mechanisms*
# {{KeyValueHandler.readBlockImpl}} computes a read length of 0 for a range at
the end of the block and sends no responses; {{readBlock}} returns null
(success) and the client sees a normal end of stream. Worth checking
{{ReadBlockComputation.computeAdjustedLength}} for a range whose end is the
last byte of the block and whose start is inside the final, partial checksum
interval.
# The datanode's {{blockData.getSize()}} is smaller than the client's offset,
so the out of range branch added in HDDS-16207 fires. That branch reports on
the stream and deliberately does not log (KeyValueHandler.java, "An out of
range offset is a client fault"), which matches the absence of datanode
logging, and the client converting it into EOF matches the "failure mistaken
for a normal end of stream" behaviour described in HDDS-16241.
*Suggested fix*
* The datanode should never answer a ReadBlock for an in-range offset with
zero bytes and success. If the requested range cannot be served (block shorter
than the client believes), fail the request explicitly and log it at WARN with
block id, requested offset/length and the datanode's own block size.
* The client should not treat "stream ended before the requested range was
delivered" as EOF; it should fail, and fail over to another replica the way the
non-streaming path does.
* If the trigger really is a block in an open container, HDDS-15886 may share
the root cause; that issue is framed around hsync, while here a plain closed
key written by "ozone fs -put" is affected.
*Environment*
Ozone master @ aed7ceac7c (built from the HDDS-9904 branch merged with master;
the gRPC streaming read path involved is unmodified master code), Ratis 3.3.1,
3 datanodes on one host, HBase 2.6.6 on ofs, JDK 17/21.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
