[
https://issues.apache.org/jira/browse/PHOENIX-5528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16952411#comment-16952411
]
Kadir OZDEMIR commented on PHOENIX-5528:
----------------------------------------
[~vincentpoon], another great find. There are two issues here to take care:
# When GlobalIndexChecker detects an unverified index row, it rebuilds the
row first, then it closes the current scanner, and finally opens a new region
scanner on the index table region to make sure that the row that has just been
rebuilt is visible to the scanner. When it does this, it reuses the original
scan object. Phoenix sets the time range on the scan to [0,
HConstants.LATEST_TIMESTAMP], which means retrieve the latest and greatest.
Every time a new scanner opens, there is a possibility of picking up new
unverified rows. The solution to this problem is to update the scan time range
to replace the max timestamp value HConstants.LATEST_TIMESTAMP with the current
wall clock time of the server when the scanner is opened by GlobalIndexChecker
in the very first time, so that even when the new region scanner is opened, the
newly updated rows outside the time range of the scan will not be visible to
the scan.
# The above solution will reduce the likelihood of your bug happening but will
not prevent it. GlobalIndexChecker should also maintain a set of data row keys
for which index rows are built and the corresponding index rows are included in
the scan result. This will allow it to detect that multiple verified rows are
scanned for the same data row. If so, only the first one should be returned
> Race condition in index verification causes multiple index rows to be
> returned for single data table row
> --------------------------------------------------------------------------------------------------------
>
> Key: PHOENIX-5528
> URL: https://issues.apache.org/jira/browse/PHOENIX-5528
> Project: Phoenix
> Issue Type: Bug
> Reporter: Vincent Poon
> Assignee: Kadir OZDEMIR
> Priority: Major
> Attachments: PHOENIX-5528.master.001.patch
>
>
> Warning: This is an artificially generated scenario that likely has a very
> low probability of happening in practice. But a race condition nevertheless.
> Unfortunately I don't have a test case, but was able to produce this by
> debugging a local regionserver and adding breakpoints at the right places to
> produce the ordering here.
> The core problem is that when we do an update to the data table, we produce
> two unverified index rows at first. When we scan both of these index rows
> and attempt to verify via rebuilding the data table row, we cannot guarantee
> that both verifications happen before the data table update, or both happen
> after the data table update.
> I use multiple index regions here to demonstrate, but I believe it could
> happen within a single region as well.
> Steps:
> 1) Create a test table with "pk" and "indexed_val" columns, and a global
> index on "indexed_val".
> 2) upsert into test values ('test_pk', 'test_val');
> 3) Split the index table on 'test_pk':
> hbase shell: split 'test_index', 'test_pk'.
> This creates two regions, call them regionA and regionB (which holds the
> existing index row)
> 3) start an update: upsert into test values ('test_pk', 'new_val');
> The first thing the indexing code does is create two unverified index
> rows: one is a new version of the existing index row, and the other is for
> the new indexed value.
> We pause the thread after this is done, before the row locks and data
> table write happens.
> 4) select indexed_val from test;
> This scans both the index regions in parallel. Each scan picks up a
> unverified row in its region. We pause in GlobalIndexChecker.
> Let the regionB scan proceed. It will attempt to rebuild the data table
> row. The data table still has 'test_val' as the indexed value. The rebuild
> succeeds.
> scan on regionA still paused.
> 5) The original update proceeds to update the data table indexed value to
> 'new_val'.
> 6) The scan on regionA proceeds, and attempted to rebuild the data table row.
> The rebuild succeeds with 'new_val' as the indexed value.
> 7) Both 'test_val' and 'new_val' are returned to the client, because both
> rebuilds succeeded.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
