[
https://issues.apache.org/jira/browse/HBASE-14283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14956541#comment-14956541
]
Ben Lau commented on HBASE-14283:
---------------------------------
[~lhofhansl] and [~anoop.hbase], I probably should’ve called it out explicitly
but yes technically, it would affect forward scans (among other things), but I
don’t think in any noticeable way. The patched code is called by
HalfStoreFileReader.getLastKey().
This getLastKey() method is needed to figure out the split point of a region
which is eg used in splitting a region. Extra IO op there but given that
region splits don’t happen frequently nor at a very high rate when they do I
think it is fine. The getLastKey() is also used by StoreFile.Reader for
passesKeyRangeFilter() check to determine whether a store is applicable to a
scan, both forward/reverse. It affects the initial scan creation as well as
later next() RPC calls I think. Although this is too bad, it doesn’t really
matter I think in practice, because the block of the last key will get cached
in the BlockCache the first time we need to know the last key for that
halfstore. So repeated calls later in the region server will not incur any
overhead. Let me know if this addresses your concerns or not.
Incidentally I think that caching is one of the primary reasons why there seems
to be a decent # of people using reverse scan but almost no one reporting this
bug— because caching often hides it, either completely or nondeterministically
(requests succeeding on later retries). We had some problems initially
reproducing this bug reliably because if we ran forward scans in a table
concurrently with the reverse scans, it would cause the blocks to become cached
and so certain key ranges that previously would’ve caused reverse scan to fail
suddenly started working just fine.
Re: Attaching the same patch, let me know if I'm doing this wrong but it sounds
like all I should have to do is just upload the same patch file for master but
with a different name and the QA tests will pick it up. I will do that.
> Reverse scan doesn’t work with HFile inline index/bloom blocks
> --------------------------------------------------------------
>
> Key: HBASE-14283
> URL: https://issues.apache.org/jira/browse/HBASE-14283
> Project: HBase
> Issue Type: Bug
> Reporter: Ben Lau
> Assignee: Ben Lau
> Attachments: HBASE-14283-0.98.patch, HBASE-14283-branch-1.0.patch,
> HBASE-14283-branch-1.1.patch, HBASE-14283-branch-1.2.patch,
> HBASE-14283-branch-1.patch, HBASE-14283-master.patch, HBASE-14283-v2.patch,
> HBASE-14283.patch, hfile-seek-before.patch
>
>
> Reverse scans do not work if an HFile contains inline bloom blocks or leaf
> level index blocks. The reason is because the seekBefore() call calculates
> the previous data block’s size by assuming data blocks are contiguous which
> is not the case in HFile V2 and beyond.
> Attached is a first cut patch (targeting
> bcef28eefaf192b0ad48c8011f98b8e944340da5 on trunk) which includes:
> (1) a unit test which exposes the bug and demonstrates failures for both
> inline bloom blocks and inline index blocks
> (2) a proposed fix for inline index blocks that does not require a new HFile
> version change, but is only performant for 1 and 2-level indexes and not 3+.
> 3+ requires an HFile format update for optimal performance.
> This patch does not fix the bloom filter blocks bug. But the fix should be
> similar to the case of inline index blocks. The reason I haven’t made the
> change yet is I want to confirm that you guys would be fine with me revising
> the HFile.Reader interface.
> Specifically, these 2 functions (getGeneralBloomFilterMetadata and
> getDeleteBloomFilterMetadata) need to return the BloomFilter. Right now the
> HFileReader class doesn’t have a reference to the bloom filters (and hence
> their indices) and only constructs the IO streams and hence has no way to
> know where the bloom blocks are in the HFile. It seems that the HFile.Reader
> bloom method comments state that they “know nothing about how that metadata
> is structured” but I do not know if that is a requirement of the abstraction
> (why?) or just an incidental current property.
> We would like to do 3 things with community approval:
> (1) Update the HFile.Reader interface and implementation to contain and
> return BloomFilters directly rather than unstructured IO streams
> (2) Merge the fixes for index blocks and bloom blocks into open source
> (3) Create a new Jira ticket for open source HBase to add a ‘prevBlockSize’
> field in the block header in the next HFile version, so that seekBefore()
> calls can not only be correct but performant in all cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
