[jira] [Created] (HBASE-16232) ITBLL fails on branch-1.3, now loosing actual keys

[Mikhail Antonov (JIRA)]

Mikhail Antonov created HBASE-16232:
---------------------------------------

             Summary: ITBLL fails on branch-1.3, now loosing actual keys
                 Key: HBASE-16232
                 URL: https://issues.apache.org/jira/browse/HBASE-16232
             Project: HBase
          Issue Type: Bug
          Components: dataloss, integration tests
    Affects Versions: 1.3.0
            Reporter: Mikhail Antonov
            Priority: Blocker


So I'm running ITBLL off branch-1.3 on recent commit (after [~stack]'s fix for 
fake keys showing up in the scans) with increased number of regions per 
regionserver and seeing the following.

{quote} 
$Verify​$Counts 
REFERENCED      0       4,999,999,994   4,999,999,994
UNDEFINED       0       3       3
UNREFERENCED    0       3       3
{quote}

So we're loosing some keys. This time those aren't fake:

{quote}
undef   
\x89\x10\xE0\xBBx\xF1\xC4\xBAY`\xC4\xD77\x87\x84\x0F    0       1       1
\x89\x11\x0F\xBA@\x0D8^\xAE \xB1\xCAh\xEB&\xE3  0       1       1
\x89\x16waxv;\xB1\xE3Z\xE6"|\xFC\xBE\x9A        0       1       1
unref   
\x15\x1F*f\x92i6\x86\x1D\x8E\xB7\xE1\xC1=\x96\xEF       0       1       1
\xF4G\xC6E\xD6\xF1\xAB\xB7\xDB\xC0\x94\xF2\xE7mN\xEC    0       1       1
U\x0F'\x88\x106\x19\x1C\x87Y"\xF3\xE6\xC1\xC8\x15
{quote}

Re-running verify step with CM off still shows this issue. Search tool reports:
{quote}
Total
\x89\x11\x0F\xBA@\x0D8^\xAE \xB1\xCAh\xEB&\xE3  5       0       5
\x89\x16waxv;\xB1\xE3Z\xE6"|\xFC\xBE\x9A        4       0       4
CELL_WITH_MISSING_ROW   15      0       15
{quote}

Will post more as I dig into.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)