QI, Jiale created HDFS-17992:
--------------------------------

             Summary: COMPOSITE_CRC file checksum fails or is wrong when a file 
has an empty block
                 Key: HDFS-17992
                 URL: https://issues.apache.org/jira/browse/HDFS-17992
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: hdfs-client
    Affects Versions: 3.2.0
            Reporter: QI, Jiale


With dfs.checksum.combine.mode=COMPOSITE_CRC, 
FileChecksumHelper#makeCompositeCrcResult
assumes that every located block contributes exactly one 4-byte CRC to 
blockChecksumBuf.
That is not true for an empty block: the DataNode composes zero chunk CRCs and 
returns an
empty block checksum (BlockChecksumHelper#computeCompositeCrc and
#reassembleNonStripedCompositeCrc both call CrcComposer#digest() with nothing 
composed),
so the client appends 0 bytes for that block. As a result:

1. A zero-length file that still has an empty block (or block group) fails with
     java.lang.ArrayIndexOutOfBoundsException: readInt out of bounds: 
buf.length=0, offset=0
   (an IOException with the same message on older branches), thrown from
   CrcUtil.readInt(crcComposer.digest(), 0) because nothing was composed.
   DistributedFileSystem#getFileChecksum(Path) passes Long.MAX_VALUE as the 
length, so
   such a file does not take the "empty file" shortcut in compute().
   With debug logging enabled it fails even earlier in 
populateBlockChecksumBuf, because
   CrcUtil.toSingleCrcString rejects the empty byte array.
2. An empty block in the middle of a file shifts the offsets used to read the 
following
   block CRCs, so a wrong checksum is returned silently.
3. A trailing empty block reads 4 bytes past the valid data of 
blockChecksumBuf; it only
   happens to work because those bytes are zero.

MD5MD5CRC is not affected: every block, empty or not, yields a 16-byte MD5.

We hit (1) in production while migrating erasure-coded data across clusters 
with DistCp:
leftover temporary files of failed Hive tasks are 0 bytes but have one empty 
block group.
fsck reports them HEALTHY:

  <file> 0 bytes, erasure-coded: policy=RS-6-3-1024k, 1 block(s):  OK
  0. <block group> len=0 Live_repl=9 ...

  $ hdfs dfs -Ddfs.checksum.combine.mode=COMPOSITE_CRC -checksum <file>
  checksum: readInt out of bounds: buf.length=0, offset=0
  $ hdfs dfs -checksum <file>
  <file> MD5-of-0MD5-of-512CRC32C ...

DistCp requests the source checksum before copying whenever the checksum type 
is preserved
(-pc), so every retry fails and the whole job fails. Such files are typically 
left behind
when a writer dies after allocating a block and lease recovery settles on a 
length of 0.

Proposed fix: in makeCompositeCrcResult, walk the block checksums with a cursor 
and skip
blocks that contribute no bytes; if no bytes are composed at all, return the 
same checksum
as a file without blocks. Also make the debug string tolerate an empty block 
checksum.
This keeps COMPOSITE_CRC independent of block layout: an empty file gets the 
same checksum
whether or not it has empty blocks, and empty blocks never change the result of 
a
non-empty file.

Related: HDFS-17006 (checksum type of empty files under COMPOSITE_CRC). With 
this change
both kinds of empty files go through the same helper, so a fix there applies to 
both.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to