[
https://issues.apache.org/jira/browse/HBASE-13374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14391753#comment-14391753
]
Jonathan Lawlor commented on HBASE-13374:
-----------------------------------------
bq. If lastResult is from previous region and I am in new region, how can I get
a row from before?
Prior to this change, this else if block would set the start key to
lastResult.getRow() and the first result returned from the server would be
skipped. Rather than skipping the first Result returned from the server, it
would be better if we could set the start key such that the first Result
returned from the server would be the Result that follows lastResult. In this
case, since we are performing a reversed scan, the start key that should be
used is the key of the closest row that could occur before lastResult.getRow.
bq. Where was the int overflow? In the int i count?
Integer overflow was occurring on the following line
{code}
cacheNum++;
{code}
Looks like this increment to caching was performed as part of the
skipRowOfFirstResult hack (since the first row is skipped, increment caching by
one)
bq. How did you find the issues? Especially int overflow one?
For issue #1 (integer overflow), I tracked it down after noticing that not all
rows were being retrieved with the following scan configuration
(caching=Int.MAX, maxResultSize=1, small=true). What ends up happening is that
{{cacheNum++}} overflows and we end up sending a negative caching value to the
server (this is equivalent to telling this server that no rows should be
retrieved). The result is that all RPC's after the overflow will return empty
results and the client will think that all regions have been exhausted.
For issue #2, I tracked it down after noticing that there were still rows
missing if I used the following scan configuration (caching=100,
maxResultSize=1, small=true). In this case, what happens is that a single row
does not fit into the defined max result size (i.e. we reach the size limit
after retrieving only a single row). Thus, we will receive only one row back
from the first RPC. Then, in the next RPC, the only row returned will be that
same row. This is because the small scanner expects that by increasing the
caching limit it will be able to skip this row... it doesn't account for the
fact that the size limit may be reached before the caching limit. Thus, since
only one row is returned, and that row is skipped, the scanner interprets this
as meaning that the region is exhausted and will skip all of the remaining rows
in that region.
> Small scanners (with particular configurations) do not return all rows
> ----------------------------------------------------------------------
>
> Key: HBASE-13374
> URL: https://issues.apache.org/jira/browse/HBASE-13374
> Project: HBase
> Issue Type: Bug
> Affects Versions: 1.0.0, 2.0.0, 1.1.0, 0.98.13
> Reporter: Jonathan Lawlor
> Assignee: Jonathan Lawlor
> Priority: Blocker
> Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.13
>
> Attachments: HBASE-13374-v1.patch,
> small-scanner-data-loss-tests-0.98.patch,
> small-scanner-data-loss-tests-branch-1.0+.patch
>
>
> I recently ran into a couple data loss issues with small scans. Similar to
> HBASE-13262, these issues only appear when scans are configured in such a way
> that the max result size limit is reached before the caching limit is
> reached. As far as I can tell, this issue affects branches 0.98+
> I should note that after investigation it looks like the root cause of these
> issues is not the same as HBASE-13262. Rather, these issue are caused by
> errors in the small scanner logic (I will explain in more depth below).
> Furthermore, I do know that the solution from HBASE-13262 has not made its
> way into small scanners (it is being addressed in HBASE-13335). As a result I
> made sure to test these issues with the patch from HBASE-13335 applied and I
> saw that they were still present.
> The following two issues have been observed (both lead to data loss):
> 1. When a small scan is configured with a caching value of Integer.MAX_VALUE,
> and a maxResultSize limit that is reached before the region is exhausted,
> integer overflow will occur. This eventually leads to a preemptive skip of
> the regions.
> 2. When a small scan is configured with a maxResultSize that is smaller than
> the size of a single row, the small scanner will jump between regions
> preemptively. This issue seems to be because small scanners assume that,
> unless a region is exhausted, at least 2 rows will be returned from the
> server. This assumption isn't clearly state in the small scanners but is
> implied through the use of {{skipRowOfFirstResult}}.
> Again, I would like to stress that the root cause of these issues is *NOT*
> related to the cause of HBASE-13262. These issues occur because of
> inappropriate assumption made in the small scanner logic. The inappropriate
> assumptions are:
> 1. Integer overflow will not occur when incrementing caching
> 2. At least 2 rows will be returned from the server unless the region has
> been exhausted
> I am attaching a patch that contains tests to display these issues. If these
> issues should be split into separate JIRAs please let me know.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
