QI, Jiale created HDFS-17992:
--------------------------------
Summary: COMPOSITE_CRC file checksum fails or is wrong when a file
has an empty block
Key: HDFS-17992
URL: https://issues.apache.org/jira/browse/HDFS-17992
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client
Affects Versions: 3.2.0
Reporter: QI, Jiale
With dfs.checksum.combine.mode=COMPOSITE_CRC,
FileChecksumHelper#makeCompositeCrcResult
assumes that every located block contributes exactly one 4-byte CRC to
blockChecksumBuf.
That is not true for an empty block: the DataNode composes zero chunk CRCs and
returns an
empty block checksum (BlockChecksumHelper#computeCompositeCrc and
#reassembleNonStripedCompositeCrc both call CrcComposer#digest() with nothing
composed),
so the client appends 0 bytes for that block. As a result:
1. A zero-length file that still has an empty block (or block group) fails with
java.lang.ArrayIndexOutOfBoundsException: readInt out of bounds:
buf.length=0, offset=0
(an IOException with the same message on older branches), thrown from
CrcUtil.readInt(crcComposer.digest(), 0) because nothing was composed.
DistributedFileSystem#getFileChecksum(Path) passes Long.MAX_VALUE as the
length, so
such a file does not take the "empty file" shortcut in compute().
With debug logging enabled it fails even earlier in
populateBlockChecksumBuf, because
CrcUtil.toSingleCrcString rejects the empty byte array.
2. An empty block in the middle of a file shifts the offsets used to read the
following
block CRCs, so a wrong checksum is returned silently.
3. A trailing empty block reads 4 bytes past the valid data of
blockChecksumBuf; it only
happens to work because those bytes are zero.
MD5MD5CRC is not affected: every block, empty or not, yields a 16-byte MD5.
We hit (1) in production while migrating erasure-coded data across clusters
with DistCp:
leftover temporary files of failed Hive tasks are 0 bytes but have one empty
block group.
fsck reports them HEALTHY:
<file> 0 bytes, erasure-coded: policy=RS-6-3-1024k, 1 block(s): OK
0. <block group> len=0 Live_repl=9 ...
$ hdfs dfs -Ddfs.checksum.combine.mode=COMPOSITE_CRC -checksum <file>
checksum: readInt out of bounds: buf.length=0, offset=0
$ hdfs dfs -checksum <file>
<file> MD5-of-0MD5-of-512CRC32C ...
DistCp requests the source checksum before copying whenever the checksum type
is preserved
(-pc), so every retry fails and the whole job fails. Such files are typically
left behind
when a writer dies after allocating a block and lease recovery settles on a
length of 0.
Proposed fix: in makeCompositeCrcResult, walk the block checksums with a cursor
and skip
blocks that contribute no bytes; if no bytes are composed at all, return the
same checksum
as a file without blocks. Also make the debug string tolerate an empty block
checksum.
This keeps COMPOSITE_CRC independent of block layout: an empty file gets the
same checksum
whether or not it has empty blocks, and empty blocks never change the result of
a
non-empty file.
Related: HDFS-17006 (checksum type of empty files under COMPOSITE_CRC). With
this change
both kinds of empty files go through the same helper, so a fix there applies to
both.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]