[
https://issues.apache.org/jira/browse/HBASE-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461312#comment-13461312
]
binlijin commented on HBASE-6868:
---------------------------------
@Lars Hofhansl
Sorry about that,
{code}
case (2) dfs.client.read.shortcircuit = true,
dfs.client.read.shortcircuit.skip.checksum=false, short circuit read turned on.
If the block is local, DFSClient will read file data direct (HRegionServer is
a DFSClient).
HFile : DFSClient will read block file and meta file. DFSClient will checksum
the data, HRegionServer(HFile) will checksum the HFile data. This is the
double-checksumming.
HLog : DFSClient will read block file and meta file. DFSClient will checksum
the data, HRegionServer will not checksum HLog data.
(2a) the block is not local.
HFile : DataNode will read block file and meta file. DFSClient will not
checksum the data, HRegionServer(HFile) will checksum the HFile data.
HLog : DataNode will read block file and meta file. DFSClient will checksum the
data, HRegionServer will not checksum HLog data.
(3a) the block is not local.
HFile : DataNode will read block file and meta file. DFSClient will not
checksum the data, HRegionServer(HFile) will checksum the HFile data.
HLog : DataNode will read block file and meta file. DFSClient will checksum the
data, HRegionServer will not checksum HLog data.
(4) dfs.client.read.shortcircuit = false,
dfs.client.read.shortcircuit.skip.checksum=true
The same as case(1)
{code}
> Skip checksum is broke; are we double-checksumming by default?
> --------------------------------------------------------------
>
> Key: HBASE-6868
> URL: https://issues.apache.org/jira/browse/HBASE-6868
> Project: HBase
> Issue Type: Bug
> Components: HFile, wal
> Affects Versions: 0.94.0, 0.94.1
> Reporter: LiuLei
> Priority: Blocker
> Fix For: 0.94.3, 0.96.0
>
>
> The HFile contains checksums for decrease the iops, so when Hbase read HFile
> , that dont't need to read the checksum from meta file of HDFS. But HLog
> file of Hbase don't contain the checksum, so when HBase read the HLog, that
> must read checksum from meta file of HDFS. We could add setSkipChecksum per
> file to hdfs or we could write checksums into WAL if this skip checksum
> facility is enabled
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
