[
https://issues.apache.org/jira/browse/HBASE-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461321#comment-13461321
]
LiuLei commented on HBASE-6868:
-------------------------------
Hi all,
1. Local Read
When we set dfs.client.read.shortcircuit=true and
dfs.client.read.shortcircuit.skip.checksum=false, and verifyChecksum parameter
is ture in BlockReaderLocal constructor, the DFSClient read meta file and
verify checksum.
When we set dfs.client.read.shortcircuit=true and
dfs.client.read.shortcircuit.skip.checksum=true, the DFSClient don't read meta
file and don't verify checksum.
2. Remote Read
When we call DistributedFileSystem.setVerifyChecksum(false), the DFSClient
don't verify checksum.
When we call DistributedFileSystem.setVerifyChecksum(true), the DFSClient
verify checksum.
the verifyChecksum property default value is true.
I think there is another problem in local read, BlockReaderLocal class use
"static Map<Integer, LocalDatanodeInfo> localDatanodeInfoMap" property to store
local block file path and local meta file path. When I stop HDFS cluster or I
kill the local DataNode and delete file use "./hadoop dfs -rm path" command ,
the RegionServer still can read the data form local file. I think that may lead
to RegionServer read wrong data.
> Skip checksum is broke; are we double-checksumming by default?
> --------------------------------------------------------------
>
> Key: HBASE-6868
> URL: https://issues.apache.org/jira/browse/HBASE-6868
> Project: HBase
> Issue Type: Bug
> Components: HFile, wal
> Affects Versions: 0.94.0, 0.94.1
> Reporter: LiuLei
> Priority: Blocker
> Fix For: 0.94.3, 0.96.0
>
>
> The HFile contains checksums for decrease the iops, so when Hbase read HFile
> , that dont't need to read the checksum from meta file of HDFS. But HLog
> file of Hbase don't contain the checksum, so when HBase read the HLog, that
> must read checksum from meta file of HDFS. We could add setSkipChecksum per
> file to hdfs or we could write checksums into WAL if this skip checksum
> facility is enabled
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
