[
https://issues.apache.org/jira/browse/HBASE-11625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15008450#comment-15008450
]
Pankaj Kumar commented on HBASE-11625:
--------------------------------------
As per the hbase-default.xml
{noformat}
<property>
<name>hbase.regionserver.checksum.verify</name>
<value>true</value>
<description>
If set to true (the default), HBase verifies the checksums for hfile
blocks. HBase writes checksums inline with the data when it writes out
hfiles. HDFS (as of this writing) writes checksums to a separate file
than the data file necessitating extra seeks. Setting this flag saves
some on i/o. Checksum verification by HDFS will be internally disabled
on hfile streams when this flag is set. If the hbase-checksum
verification
fails, we will switch back to using HDFS checksums (so do not disable
HDFS
checksums! And besides this feature applies to hfiles only, not to
WALs).
If this parameter is set to false, then hbase will not verify any
checksums,
instead it will depend on checksum verification being done in the HDFS
client.
</description>
</property>
{noformat}
Here as per the description,
*If the hbase-checksum verification fails, we will switch back to using HDFS
checksums (so do not disable HDFS checksums!*
*And besides this feature applies to hfiles only, not to WALs).*
But we have implemented it wrongly in HFileSystem (Please correct me if I'm
wrong),
{code}
if (useHBaseChecksum && !(fs instanceof LocalFileSystem)) {
conf = new Configuration(conf);
conf.setBoolean("dfs.client.read.shortcircuit.skip.checksum", true);
this.noChecksumFs = newInstanceFileSystem(conf);
this.noChecksumFs.setVerifyChecksum(false);
} else {
this.noChecksumFs = fs;
}
{code}
Here we should not disable HDFS checksum,
{code}
this.noChecksumFs.setVerifyChecksum(false);
{code}
It should be,
{code}
this.noChecksumFs.setVerifyChecksum(true);
{code}
That is why switch didn't happen to HDFS checksum.
ping [~apurtell] [~ram_krish] [~syuanjiang] [~ndimiduk] [~enis], please
provide your opinion.
If solution is fine then I will attach the patch file.
> Reading datablock throws "Invalid HFile block magic" and can not switch to
> hdfs checksum
> -----------------------------------------------------------------------------------------
>
> Key: HBASE-11625
> URL: https://issues.apache.org/jira/browse/HBASE-11625
> Project: HBase
> Issue Type: Bug
> Components: HFile
> Affects Versions: 0.94.21, 0.98.4, 0.98.5
> Reporter: qian wang
> Attachments: 2711de1fdf73419d9f8afc6a8b86ce64.gz
>
>
> when using hbase checksum,call readBlockDataInternal() in hfileblock.java, it
> could happen file corruption but it only can switch to hdfs checksum
> inputstream till validateBlockChecksum(). If the datablock's header corrupted
> when b = new HFileBlock(),it throws the exception "Invalid HFile block magic"
> and the rpc call fail
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
