[
https://issues.apache.org/jira/browse/HDFS-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554467#comment-13554467
]
Todd Lipcon commented on HDFS-4403:
-----------------------------------
bq. Is it possible that the server does not set the crcType and expects the
client to infer the type. The checksum happens to be something other than
CHECKSUM_CRC32. The old client treats it as CHECKSUM_CRC32 and runs into issues?
That's the bug this is trying to fix. If you currently run a newer 0.23-branch
client against an earlier 0.23, or a 2.x branch (after making it IPC
compatible) against 2.0.0, it will incorrectly report CHECKSUM_CRC32 back to
the caller, even if the file is actually CHECKSUM_CRC32C. This results in
distcp giving back errors about the copy failing due to checksum mismatch, etc.
Any _new_ server always sets the checksum type explicitly, regardless of what
type it is. It's only the old (pre-HDFS-3177) servers that wouldn't set one.
> DFSClient can infer checksum type when not provided by reading first byte
> -------------------------------------------------------------------------
>
> Key: HDFS-4403
> URL: https://issues.apache.org/jira/browse/HDFS-4403
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
> Affects Versions: 2.0.2-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: hdfs-4403.txt
>
>
> HDFS-3177 added the checksum type to OpBlockChecksumResponseProto, but the
> new protobuf field is optional, with a default of CRC32. This means that this
> API, when used against an older cluster (like earlier 0.23 releases) will
> falsely return CRC32 even if that cluster has written files with CRC32C. This
> can cause issues for distcp, for example.
> Instead of defaulting the protobuf field to CRC32, we can leave it with no
> default, and if the OpBlockChecksumResponseProto has no checksum type set,
> the client can send OP_READ_BLOCK to read the first byte of the block, then
> grab the checksum type out of that response (which has always been present)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
