[
https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347851#comment-16347851
]
Dennis Huo commented on HDFS-13056:
-----------------------------------
Uploaded initial end-to-end working draft against trunk which supports
CRC32/CRC32C and partial file prefixes of arbitrary bytes-per-crc or blocksize
and across replicated vs striped encodings as well.
Still a TODO to support the striped-reconstruction path, and adding stripe
support made everything a lot messier so some refactoring is in order. Also,
unittests still pending, but manual testing in a real setup works:
{code:java}
$ hdfs dfs -cp gs://hadoop-cloud-dev-dhuo/random-crctest.dat
hdfs:///tmp/random-crctest-default1.dat
$ hdfs dfs -cp gs://hadoop-cloud-dev-dhuo/random-crctest.dat
hdfs:///tmp/random-crctest-default2.dat
$ hdfs dfs -Ddfs.bytes-per-checksum=1024 -cp
gs://hadoop-cloud-dev-dhuo/random-crctest.dat
hdfs:///tmp/random-crctest-bpc1024.dat
$ hdfs dfs -Ddfs.blocksize=67108864 -cp
gs://hadoop-cloud-dev-dhuo/random-crctest.dat
hdfs:///tmp/random-crctest-blocksize64mb.dat
$ hdfs dfs -cp gs://hadoop-cloud-dev-dhuo/random-crctest-unaligned.dat
hdfs:///tmp/random-crctest-unaligned1.dat
$ hdfs dfs -Ddfs.bytes-per-checksum=1024 -cp
gs://hadoop-cloud-dev-dhuo/random-crctest-unaligned.dat
hdfs:///tmp/random-crctest-unaligned2.dat
$ hdfs dfs -Ddfs.checksum.type=CRC32 -cp
gs://hadoop-cloud-dev-dhuo/random-crctest.dat
hdfs:///tmp/random-crctest-gzipcrc32-1.dat
$ hdfs dfs -Ddfs.checksum.type=CRC32 -Ddfs.bytes-per-checksum=1024 -cp
gs://hadoop-cloud-dev-dhuo/random-crctest.dat
hdfs:///tmp/random-crctest-gzipcrc32-2.dat
$ hdfs dfs -mkdir hdfs:///tmpec
$ hdfs ec -enablePolicy -policy XOR-2-1-1024k
$ hdfs ec -setPolicy -path hdfs:///tmpec -policy XOR-2-1-1024k
$ hdfs dfs -cp gs://hadoop-cloud-dev-dhuo/random-crctest.dat
hdfs:///tmpec/random-crctest-default1.dat
$ hdfs dfs -cp gs://hadoop-cloud-dev-dhuo/random-crctest.dat
hdfs:///tmpec/random-crctest-default2.dat
$ hdfs dfs -Ddfs.bytes-per-checksum=1024 -cp
gs://hadoop-cloud-dev-dhuo/random-crctest.dat
hdfs:///tmpec/random-crctest-bpc1024.dat
$ hdfs dfs -Ddfs.blocksize=67108864 -cp
gs://hadoop-cloud-dev-dhuo/random-crctest.dat
hdfs:///tmpec/random-crctest-blocksize64mb.dat
$ hdfs dfs -cp gs://hadoop-cloud-dev-dhuo/random-crctest-unaligned.dat
hdfs:///tmpec/random-crctest-unaligned1.dat
$ hdfs dfs -Ddfs.bytes-per-checksum=1024 -cp
gs://hadoop-cloud-dev-dhuo/random-crctest-unaligned.dat
hdfs:///tmpec/random-crctest-unaligned2.dat
$ hdfs dfs -Ddfs.checksum.type=CRC32 -cp
gs://hadoop-cloud-dev-dhuo/random-crctest.dat
hdfs:///tmpec/random-crctest-gzipcrc32-1.dat
$ hdfs dfs -Ddfs.checksum.type=CRC32 -Ddfs.bytes-per-checksum=1024 -cp
gs://hadoop-cloud-dev-dhuo/random-crctest.dat
hdfs:///tmpec/random-crctest-gzipcrc32-2.dat
$ hdfs dfs -checksum hdfs:///tmp/random-crctest*.dat
hdfs:///tmp/random-crctest-blocksize64mb.dat MD5-of-131072MD5-of-512CRC32C
0000020000000000000200008baa940ef6ed21fb4bd6224ce917d127
hdfs:///tmp/random-crctest-bpc1024.dat MD5-of-131072MD5-of-1024CRC32C
000004000000000000020000930b0d7ad333786a839b044ed8d18d2d
hdfs:///tmp/random-crctest-default1.dat MD5-of-262144MD5-of-512CRC32C
000002000000000000040000c0baeeacbc4b5a3c8af5152944fe2d79
hdfs:///tmp/random-crctest-default2.dat MD5-of-262144MD5-of-512CRC32C
000002000000000000040000c0baeeacbc4b5a3c8af5152944fe2d79
hdfs:///tmp/random-crctest-gzipcrc32-1.dat MD5-of-262144MD5-of-512CRC32
00000200000000000004000049d52fdd25aa08559e20536acc34d51d
hdfs:///tmp/random-crctest-gzipcrc32-2.dat MD5-of-131072MD5-of-1024CRC32
0000040000000000000200001d5468ea4093ddb3741790b8dc3b9a57
hdfs:///tmp/random-crctest-unaligned1.dat MD5-of-262144MD5-of-512CRC32C
0000020000000000000400000da665dadca0df00456206f234d5f8b0
hdfs:///tmp/random-crctest-unaligned2.dat MD5-of-131072MD5-of-1024CRC32C
00000400000000000002000027c2198f48224a0ddb92c4dc4addd28b
$ hdfs dfs -checksum hdfs:///tmpec/random-crctest*.dat
18/02/01 01:15:54 INFO gcs.GoogleHadoopFileSystemBase: GHFS version:
1.6.2-hadoop2
hdfs:///tmpec/random-crctest-blocksize64mb.dat MD5-of-131072MD5-of-512CRC32C
0000020000000000000200005b54faaa368ed81b25984a746c767d39
hdfs:///tmpec/random-crctest-bpc1024.dat MD5-of-131072MD5-of-1024CRC32C
00000400000000000002000089a128b1e1995256bdb34fb95720dafc
hdfs:///tmpec/random-crctest-default1.dat MD5-of-262144MD5-of-512CRC32C
00000200000000000004000007ee18e8f4909647adf085ec0f464d1a
hdfs:///tmpec/random-crctest-default2.dat MD5-of-262144MD5-of-512CRC32C
00000200000000000004000007ee18e8f4909647adf085ec0f464d1a
hdfs:///tmpec/random-crctest-gzipcrc32-1.dat MD5-of-262144MD5-of-512CRC32
000002000000000000040000d79ad1fa00fad2f0adb18f49f2e90bb3
hdfs:///tmpec/random-crctest-gzipcrc32-2.dat MD5-of-131072MD5-of-1024CRC32
000004000000000000020000126ac7bc467c59942734bd8ebf690440
hdfs:///tmpec/random-crctest-unaligned1.dat MD5-of-262144MD5-of-512CRC32C
0000020000000000000400004b95df26144cba3d1a0ab87cea048b66
hdfs:///tmpec/random-crctest-unaligned2.dat MD5-of-131072MD5-of-1024CRC32C
000004000000000000020000c8b50f1216f55608975624f6a34542bc
$ hdfs dfs -Ddfs.checksum.combine.mode=COMPOSITE_CRC -checksum
hdfs:///tmp/random-crctest*.dat
18/02/01 01:15:57 INFO gcs.GoogleHadoopFileSystemBase: GHFS version:
1.6.2-hadoop2
hdfs:///tmp/random-crctest-blocksize64mb.dat COMPOSITE-CRC32C
4db86e2b00000000000000000000000000000000000000000000000000000000
hdfs:///tmp/random-crctest-bpc1024.dat COMPOSITE-CRC32C
4db86e2b00000000000000000000000000000000000000000000000000000000
hdfs:///tmp/random-crctest-default1.dat COMPOSITE-CRC32C
4db86e2b00000000000000000000000000000000000000000000000000000000
hdfs:///tmp/random-crctest-default2.dat COMPOSITE-CRC32C
4db86e2b00000000000000000000000000000000000000000000000000000000
hdfs:///tmp/random-crctest-gzipcrc32-1.dat COMPOSITE-CRC32
721d687e00000000000000000000000000000000000000000000000000000000
hdfs:///tmp/random-crctest-gzipcrc32-2.dat COMPOSITE-CRC32
721d687e00000000000000000000000000000000000000000000000000000000
hdfs:///tmp/random-crctest-unaligned1.dat COMPOSITE-CRC32C
c3842f6100000000000000000000000000000000000000000000000000000000
hdfs:///tmp/random-crctest-unaligned2.dat COMPOSITE-CRC32C
c3842f6100000000000000000000000000000000000000000000000000000000
$ hdfs dfs -Ddfs.checksum.combine.mode=COMPOSITE_CRC -checksum
hdfs:///tmpec/random-crctest*.dat
18/02/01 01:16:00 INFO gcs.GoogleHadoopFileSystemBase: GHFS version:
1.6.2-hadoop2
hdfs:///tmpec/random-crctest-blocksize64mb.dat COMPOSITE-CRC32C
4db86e2b00000000000000000000000000000000000000000000000000000000
hdfs:///tmpec/random-crctest-bpc1024.dat COMPOSITE-CRC32C
4db86e2b00000000000000000000000000000000000000000000000000000000
hdfs:///tmpec/random-crctest-default1.dat COMPOSITE-CRC32C
4db86e2b00000000000000000000000000000000000000000000000000000000
hdfs:///tmpec/random-crctest-default2.dat COMPOSITE-CRC32C
4db86e2b00000000000000000000000000000000000000000000000000000000
hdfs:///tmpec/random-crctest-gzipcrc32-1.dat COMPOSITE-CRC32
721d687e00000000000000000000000000000000000000000000000000000000
hdfs:///tmpec/random-crctest-gzipcrc32-2.dat COMPOSITE-CRC32
721d687e00000000000000000000000000000000000000000000000000000000
hdfs:///tmpec/random-crctest-unaligned1.dat COMPOSITE-CRC32C
c3842f6100000000000000000000000000000000000000000000000000000000
hdfs:///tmpec/random-crctest-unaligned2.dat COMPOSITE-CRC32C
c3842f6100000000000000000000000000000000000000000000000000000000
{code}
> Expose file-level composite CRCs in HDFS which are comparable across
> different instances/layouts
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-13056
> URL: https://issues.apache.org/jira/browse/HDFS-13056
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, distcp, erasure-coding, federation, hdfs
> Affects Versions: 3.0.0
> Reporter: Dennis Huo
> Priority: Major
> Attachments: HDFS-13056-branch-2.8.001.patch,
> HDFS-13056-branch-2.8.poc1.patch, HDFS-13056.001.patch,
> Reference_only_zhen_PPOC_hadoop2.6.X.diff, hdfs-file-composite-crc32-v1.pdf,
> hdfs-file-composite-crc32-v2.pdf
>
>
> FileChecksum was first introduced in
> [https://issues-test.apache.org/jira/browse/HADOOP-3981] and ever since then
> has remained defined as MD5-of-MD5-of-CRC, where per-512-byte chunk CRCs are
> already stored as part of datanode metadata, and the MD5 approach is used to
> compute an aggregate value in a distributed manner, with individual datanodes
> computing the MD5-of-CRCs per-block in parallel, and the HDFS client
> computing the second-level MD5.
>
> A shortcoming of this approach which is often brought up is the fact that
> this FileChecksum is sensitive to the internal block-size and chunk-size
> configuration, and thus different HDFS files with different block/chunk
> settings cannot be compared. More commonly, one might have different HDFS
> clusters which use different block sizes, in which case any data migration
> won't be able to use the FileChecksum for distcp's rsync functionality or for
> verifying end-to-end data integrity (on top of low-level data integrity
> checks applied at data transfer time).
>
> This was also revisited in https://issues.apache.org/jira/browse/HDFS-8430
> during the addition of checksum support for striped erasure-coded files;
> while there was some discussion of using CRC composability, it still
> ultimately settled on hierarchical MD5 approach, which also adds the problem
> that checksums of basic replicated files are not comparable to striped files.
>
> This feature proposes to add a "COMPOSITE-CRC" FileChecksum type which uses
> CRC composition to remain completely chunk/block agnostic, and allows
> comparison between striped vs replicated files, between different HDFS
> instances, and possible even between HDFS and other external storage systems.
> This feature can also be added in-place to be compatible with existing block
> metadata, and doesn't need to change the normal path of chunk verification,
> so is minimally invasive. This also means even large preexisting HDFS
> deployments could adopt this feature to retroactively sync data. A detailed
> design document can be found here:
> https://storage.googleapis.com/dennishuo/hdfs-file-composite-crc32-v1.pdf
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
