Jose Luis López created HDFS-17986:
--------------------------------------
Summary: Replica with a corrupt meta header is not reported when a
short-circuit read skips checksum verification
Key: HDFS-17986
URL: https://issues.apache.org/jira/browse/HDFS-17986
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode, hdfs-client
Affects Versions: 3.4.3, 3.5.0, 3.6.0
Reporter: Jose Luis López
Assignee: Jose Luis López
h3. Problem
If a client reads with short-circuit reads enabled and checksum verification
disabled (\{{setVerifyChecksum(false)}}), a replica with a corrupt meta file
header is never reported to the NameNode:
# The short-circuit client parses the meta header in
\{{ShortCircuitReplica.<init>}}, gets a \{{CorruptMetaHeaderException}}, logs
"error creating ShortCircuitReplica" in \{{BlockReaderFactory}}, and falls back
to a remote read.
# The remote read does not ask for checksums, so the DataNode never opens the
meta file and never calls \{{handleBadBlock}} (HDFS-14706 only covers reads
with checksums).
# The read succeeds and nobody reports the replica. It is only found by the
block scanner (up to 3 weeks by default) or by a later read with checksums.
The failure is not cached, so every read of the block repeats the failed
short-circuit attempt and falls back to TCP.
h3. Who is affected
HBase with its default \{{hbase.regionserver.checksum.verify=true}} and
short-circuit reads. HBase reads the last block of each HFile with HDFS
checksums when it opens the file, so corruption there is reported. Corruption
in any other block of a multi-block HFile is not. HBASE-21607 is a production
occurrence of the same fallback.
Impact:
* The NameNode counts a replica as healthy when nothing that verifies checksums
(including re-replication) can use it.
* The client loses short-circuit reads for that block.
* If the data is also corrupt, HBase's own checksum usually catches it and
retries with HDFS checksums, which triggers the DataNode report. HBASE-21607
shows this is not guaranteed: a failing disk zeroed the blocks, zeroed HFile
blocks look like HBase's NULL checksum type, garbage was returned to the
client, and the replica was never reported.
h3. Reproduction
HDFS 3.5.0 (1 DN, short-circuit enabled) with HBase 2.6.7 standalone, default
settings:
# Load 1M rows (\{{hbase pe ... sequentialWrite}}), then flush and
major_compact. This gives one 256 MB HFile in 2 blocks.
# Stop HBase, overwrite byte 2 (the checksum type) of the first block's
\{{.meta}} file with 0x7f, and start HBase.
# \{{count 'TestTable'}} returns all 1M rows. The RegionServer logs
CorruptMetaHeaderException for the block on every scan, and \{{hdfs fsck}}
stays HEALTHY.
# Control: one \{{hdfs dfs -cat}} of the file (checksums on). The DataNode
reports the block within seconds and fsck shows CORRUPT.
Unit test \{{TestShortCircuitCorruptMetaHeader}} (needs native domain sockets)
covers invalid and truncated headers with replication 1/3/5/7. On trunk, all
cases pass with checksums on and all fail with checksums off.
h3. Possible fix
Have the DataNode check the meta header in \{{requestShortCircuitFdsForRead}}
(FINALIZED, non-transient replicas only) and call \{{handleBadBlock}} if it is
corrupt. Through the block scanner's suspect queue (HDFS-8224), that also
verifies the data. It should still return the descriptors: failing the request
makes the client disable short-circuit reads for the whole DataNode (see
HDFS-12528).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]