[
https://issues.apache.org/jira/browse/HBASE-14283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906704#comment-14906704
]
Ben Lau commented on HBASE-14283:
---------------------------------
Hi guys, I have posted a new patch on review board. See
https://reviews.apache.org/r/38720/. The patch adds support for HFileV4 and
uses it to fix/optimize reverse scan. The patch is designed to be
rolling-restartable in 2 phases, as discussed in an above comment on Sep 14.
As currently posted, I think the patch has to go into HBase 2.0 since it
changes HConstants which is marked @Stable. It turns out that assumptions
about the header size and contents are hardcoded in several places, primarily
HConstants.java.
Let me know what you guys think of the patch. More eyes would be better
because the changes are somewhat farther reaching than they sounded initially
and the HFile format has a long history that I’m not as familiar with as most
people around here.
I also need to reach out to Facebook later for feedback since this change seems
like it will affect their external Memcached block cache. I have marked TODO
in the patch in several places where I will need to talk with Facebook.
Also, since we are adding a new HFileV4, now would be a good time to fix
anything else that is broken or add additional metadata for future
optimizations that we missed out on in HFileV3. Someone could add more
metadata after the patch stands on its own as a complete fix for reverse scans.
> Reverse scan doesn’t work with HFile inline index/bloom blocks
> --------------------------------------------------------------
>
> Key: HBASE-14283
> URL: https://issues.apache.org/jira/browse/HBASE-14283
> Project: HBase
> Issue Type: Bug
> Reporter: Ben Lau
> Assignee: Ben Lau
> Attachments: HBASE-14283-v2.patch, HBASE-14283.patch,
> hfile-seek-before.patch
>
>
> Reverse scans do not work if an HFile contains inline bloom blocks or leaf
> level index blocks. The reason is because the seekBefore() call calculates
> the previous data block’s size by assuming data blocks are contiguous which
> is not the case in HFile V2 and beyond.
> Attached is a first cut patch (targeting
> bcef28eefaf192b0ad48c8011f98b8e944340da5 on trunk) which includes:
> (1) a unit test which exposes the bug and demonstrates failures for both
> inline bloom blocks and inline index blocks
> (2) a proposed fix for inline index blocks that does not require a new HFile
> version change, but is only performant for 1 and 2-level indexes and not 3+.
> 3+ requires an HFile format update for optimal performance.
> This patch does not fix the bloom filter blocks bug. But the fix should be
> similar to the case of inline index blocks. The reason I haven’t made the
> change yet is I want to confirm that you guys would be fine with me revising
> the HFile.Reader interface.
> Specifically, these 2 functions (getGeneralBloomFilterMetadata and
> getDeleteBloomFilterMetadata) need to return the BloomFilter. Right now the
> HFileReader class doesn’t have a reference to the bloom filters (and hence
> their indices) and only constructs the IO streams and hence has no way to
> know where the bloom blocks are in the HFile. It seems that the HFile.Reader
> bloom method comments state that they “know nothing about how that metadata
> is structured” but I do not know if that is a requirement of the abstraction
> (why?) or just an incidental current property.
> We would like to do 3 things with community approval:
> (1) Update the HFile.Reader interface and implementation to contain and
> return BloomFilters directly rather than unstructured IO streams
> (2) Merge the fixes for index blocks and bloom blocks into open source
> (3) Create a new Jira ticket for open source HBase to add a ‘prevBlockSize’
> field in the block header in the next HFile version, so that seekBefore()
> calls can not only be correct but performant in all cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
