Dennis Huo created HDFS-13056:
---------------------------------
Summary: Expose file-level composite CRCs in HDFS which are
comparable across different instances/layouts
Key: HDFS-13056
URL: https://issues.apache.org/jira/browse/HDFS-13056
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, distcp, erasure-coding, federation, hdfs
Affects Versions: 3.0.0
Reporter: Dennis Huo
FileChecksum was first introduced in
[https://issues-test.apache.org/jira/browse/HADOOP-3981] and ever since then
has remained defined as MD5-of-MD5-of-CRC, where per-512-byte chunk CRCs are
already stored as part of datanode metadata, and the MD5 approach is used to
compute an aggregate value in a distributed manner, with individual datanodes
computing the MD5-of-CRCs per-block in parallel, and the HDFS client computing
the second-level MD5.
A shortcoming of this approach which is often brought up is the fact that this
FileChecksum is sensitive to the internal block-size and chunk-size
configuration, and thus different HDFS files with different block/chunk
settings cannot be compared. More commonly, one might have different HDFS
clusters which use different block sizes, in which case any data migration
won't be able to use the FileChecksum for distcp's rsync functionality or for
verifying end-to-end data integrity (on top of low-level data integrity checks
applied at data transfer time).
This was also revisited in https://issues.apache.org/jira/browse/HDFS-8430
during the addition of checksum support for striped erasure-coded files; while
there was some discussion of using CRC composability, it still ultimately
settled on hierarchical MD5 approach, which also adds the problem that
checksums of basic replicated files are not comparable to striped files.
This feature proposes to add a "COMPOSITE-CRC" FileChecksum type which uses CRC
composition to remain completely chunk/block agnostic, and allows comparison
between striped vs replicated files, between different HDFS instances, and
possible even between HDFS and other external storage systems. This feature can
also be added in-place to be compatible with existing block metadata, and
doesn't need to change the normal path of chunk verification, so is minimally
invasive. This also means even large preexisting HDFS deployments could adopt
this feature to retroactively sync data. A detailed design document can be
found here:
https://storage.googleapis.com/dennishuo/hdfs-file-composite-crc32-v1.pdf
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
