[
https://issues.apache.org/jira/browse/HDFS-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-4403:
------------------------------
Attachment: hdfs-4403.txt
Attached patch takes the following approach:
- Remove the CRC32 default from the protobuf.
- If the OP_BLOCK_CHECKSUM response doesn't include the checksum type, detect
this:
-- Re-connect to the same DN, and issue OP_READ_BLOCK for the first byte of the
block.
-- Extract the checksum type from the response
Given that this is a cross-version compatibility fix, it isn't really possible
to make a reasonable functional test for it. However, the existing tests should
keep this from regressing the non-cross-version case.
I tested manually as follows:
- Made a local branch off branch-2, and reverted HADOOP-9070, which was causing
an IPC incompatibility (see discussion there for more details).
- Cherry-picked HADOOP-9209 (shell command to see checksums)
- Started a cluster from the Hadoop 2.0.0 release tarball
Without this patch:
{code}
$ ./bin/hadoop fs -checksum '/*'
/issue MD5-of-0MD5-of-512CRC32
000002000000000000000000b234aa05a75fed38536bda657b20bfcf
/issue-crc32-old MD5-of-0MD5-of-512CRC32
000002000000000000000000593b23e67a7477aab90e42e41478b321
{code}
It shows both of these files as having CRC32 type checksums, which is not
correct.
With the patch applied, I get the correct result:
{code}
$ ./bin/hadoop fs -checksum '/*'
/issue MD5-of-0MD5-of-512CRC32C
000002000000000000000000b234aa05a75fed38536bda657b20bfcf
/issue-crc32-old MD5-of-0MD5-of-512CRC32
000002000000000000000000593b23e67a7477aab90e42e41478b321
{code}
(note that it identifies CRC32C for the first of the two files, which is
correct)
> DFSClient can infer checksum type when not provided by reading first byte
> -------------------------------------------------------------------------
>
> Key: HDFS-4403
> URL: https://issues.apache.org/jira/browse/HDFS-4403
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
> Affects Versions: 2.0.2-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: hdfs-4403.txt
>
>
> HDFS-3177 added the checksum type to OpBlockChecksumResponseProto, but the
> new protobuf field is optional, with a default of CRC32. This means that this
> API, when used against an older cluster (like earlier 0.23 releases) will
> falsely return CRC32 even if that cluster has written files with CRC32C. This
> can cause issues for distcp, for example.
> Instead of defaulting the protobuf field to CRC32, we can leave it with no
> default, and if the OpBlockChecksumResponseProto has no checksum type set,
> the client can send OP_READ_BLOCK to read the first byte of the block, then
> grab the checksum type out of that response (which has always been present)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
