[
https://issues.apache.org/jira/browse/HBASE-20674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16500852#comment-16500852
]
Mike Drob commented on HBASE-20674:
-----------------------------------
Will rename to localFS, is fine. Everything should be this.fs in HFileSystem,
will clean that too.
One thing I noticed now was the check in
https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/io/FSDataInputStreamWrapper.java#L153-L154
We do an identity comparison to check if the file system is the same as the
nochecksum file system, and if it is we don't do hbase checksumming. This check
doesn't work. FS and noSumFS are the same object previously when
{{(useHBaseChecksum is false OR fs instanceof LocalFileSystem) AND
hbase.fs.wrapper is unset}}. I think that was a bug, the last one shouldn't be
a condition for it, because both of them should have been getting wrapped in
the same manner. I imagine this was mostly used for testing and never came up
on prod systems?
I don't fully understand why we end up skipping HBase checksums on LocalFS, but
it strongly appears to have been the case of what the code actually did.
I haven't done any testing beyond unit tests, I'm not even sure that the unit
tests we are running are sufficient. I didn't add anything new to the suite.
The mechanics of check sum verification from the dfs client are very very
opaque.
> clean up short circuit read logic and docs
> ------------------------------------------
>
> Key: HBASE-20674
> URL: https://issues.apache.org/jira/browse/HBASE-20674
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 2.0.0
> Reporter: Mike Drob
> Assignee: Mike Drob
> Priority: Major
> Attachments: HBASE-20674.patch, HBASE-20674.v2.patch
>
>
> Mailing list discussion at
> https://lists.apache.org/thread.html/f6f73df0ceae29f762f9b9088e3ffd0bf8f109d3dd692df100bf4fd6@%3Cdev.hbase.apache.org%3E
> There are several inconsistencies between how our docs claim we do things and
> how we actually do things.
> There are two docs sections that attempt to address how SCR should work.
> dfs.client.read.shortcircuit.skip.checksum is advised to set to true, but our
> code in separate places ignores it and then later sets it to true anyway.
> CommonFSUtils and FSUtils duplicate code related to SCR setup.
> There is a workaround in HFileSystem for a bug that's been fixed in all
> versions of hadoop that we support. (HADOOP-9307)
> We suggest setting dfs.client.read.shortcircuit.buffer.size to a value that
> is very close to what we'd set it to anyway, without clearly explaining why
> this is important.
> There are other properties that we claim are important, but we don't offer
> any suggestions or explanations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
