[
https://issues.apache.org/jira/browse/HADOOP-8060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236946#comment-13236946
]
Kihwal Lee commented on HADOOP-8060:
------------------------------------
bq. What about making the checksum type part of the FileSystem cache key
The checksum type is a dfs config item. We can't do that in FileSystem, which
is in common. But FileSystem already has things like setVerfyChecksum() and
getFileChecksum(). So we could make the checksum type a Filesystem-level
config.
To address the issue of dynamically configurable properties, we could introduce
a file system config digest method, which is kind of like hashCode(). The
tricky part will be to get the hdfs part of the formula added to Configuration
when, say, HdfsConfiguration.init() is called. Or maybe having each file system
implement a digest method is better.
For this jira, I will just add conf as a part of the key. The equality check
will be just a shallow comparison.
> Add a capability to use of consistent checksums for append and copy
> -------------------------------------------------------------------
>
> Key: HADOOP-8060
> URL: https://issues.apache.org/jira/browse/HADOOP-8060
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, util
> Affects Versions: 0.23.0, 0.23.1, 0.24.0
> Reporter: Kihwal Lee
> Assignee: Kihwal Lee
> Fix For: 0.23.2, 0.24.0
>
>
> After the improved CRC32C checksum feature became default, some of use cases
> involving data movement are no longer supported. For example, when running
> DistCp to copy from a file stored with the CRC32 checksum to a new cluster
> with the CRC32C set to default checksum, the final data integrity check fails
> because of mismatch in checksums.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
