[jira] [Updated] (HBASE-16074) ITBLL fails, reports lost big or tine families
[
https://issues.apache.org/jira/browse/HBASE-16074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-16074:
--
Attachment: HBASE-16074.branch-1.3.003.patch
Retry. Test passes locally. Retry.
Ran the ITBLL on my little cluster against tip of 1.3 and it fails with:
{code}
\xB7\xFF\xE6r=1
\xBA\x93\xA0\xE0\xF6\x5C\x8D\xA8>\xB82\x8F01\xC2S\x00=1
\xBC\xE0\x925\xD6H\x09\x0D\x0D\xF4Y\x8BA\x9F\xDA\x84\x00=1
\xBF\xFF\xFF\xFF\xFF\xFF\xFF\xF4=1
\xC8\xDA\xB2\xF9g\x00\xFET\x90@\xE9\xB25\xFD\xA2~\x00=1
\xCA\xAA\xAA\xAA\xAA\xAA\xAA\x9E=1
]\xADn\xE7#\xF3\xDB\xB3k\xAB\xF0k\x7F-\x1AA\x00=1
_\xFF\xFF\xFF\xFF\xFF\xFF\xFA=1
c\x85\xA4\x93HN8\xE7\x90\x8D\xA6\xA5\x8A\x15\xFF]\x00=1
eV1\xE3=1
h\xDB\x94\xEB\xA0\x82\xD4\x17\xF9\x1C\xE6o\xC9/\xE8$\x00=1
j\xAA\xAA\xAA\xAA\xAA\xAA\xA4=1
l\xCB\x83\xEC\x97\x86\xE1\x90\x7F\xA21J\x99\xF7Ji\x00=1
nH\x1C5\xD4\x16\xD9\xAE<\xE1E\xAF\x99\xBC\x1A\x8D\x00=1
uUUN=1
y\x14;9\x9E'\xE6\xB1E\xEE&\xE3\x9C`\x0E\x0D\x00=1
~W\x07t\xD2\x0B\x96\xF4\xD9P%h\xEA(\xBA\xC4\x00=1
16/07/03 22:39:39 ERROR test.IntegrationTestBigLinkedList$Verify: Found nodes
which lost big or tiny families, count=8669
{code}
> ITBLL fails, reports lost big or tine families
> --
>
> Key: HBASE-16074
> URL: https://issues.apache.org/jira/browse/HBASE-16074
> Project: HBase
> Issue Type: Bug
> Components: integration tests
>Affects Versions: 1.3.0, 0.98.20
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
>Priority: Blocker
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.21
>
> Attachments: 16074.test.branch-1.3.patch, 16074.test.patch,
> HBASE-16074.branch-1.3.001.patch, HBASE-16074.branch-1.3.002.patch,
> HBASE-16074.branch-1.3.003.patch, HBASE-16074.branch-1.3.003.patch,
> changes_to_stress_ITBLL.patch, changes_to_stress_ITBLL__a_bit_relaxed_.patch,
> itbll log with failure, itbll log with success
>
>
> Underlying MR jobs succeed but I'm seeing the following in the logs (mid-size
> distributed test cluster):
> ERROR test.IntegrationTestBigLinkedList$Verify: Found nodes which lost big or
> tiny families, count=164
> I do not know exactly yet whether it's a bug, a test issue or env setup
> issue, but need figure it out. Opening this to raise awareness and see if
> someone saw that recently.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16074) ITBLL fails, reports lost big or tine families
[ https://issues.apache.org/jira/browse/HBASE-16074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16074: -- Attachment: HBASE-16074.branch-1.3.003.patch > ITBLL fails, reports lost big or tine families > -- > > Key: HBASE-16074 > URL: https://issues.apache.org/jira/browse/HBASE-16074 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 1.3.0, 0.98.20 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Blocker > Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.21 > > Attachments: 16074.test.branch-1.3.patch, 16074.test.patch, > HBASE-16074.branch-1.3.001.patch, HBASE-16074.branch-1.3.002.patch, > HBASE-16074.branch-1.3.003.patch, changes_to_stress_ITBLL.patch, > changes_to_stress_ITBLL__a_bit_relaxed_.patch, itbll log with failure, itbll > log with success > > > Underlying MR jobs succeed but I'm seeing the following in the logs (mid-size > distributed test cluster): > ERROR test.IntegrationTestBigLinkedList$Verify: Found nodes which lost big or > tiny families, count=164 > I do not know exactly yet whether it's a bug, a test issue or env setup > issue, but need figure it out. Opening this to raise awareness and see if > someone saw that recently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16074) ITBLL fails, reports lost big or tine families
[ https://issues.apache.org/jira/browse/HBASE-16074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16074: -- Attachment: HBASE-16074.branch-1.3.002.patch > ITBLL fails, reports lost big or tine families > -- > > Key: HBASE-16074 > URL: https://issues.apache.org/jira/browse/HBASE-16074 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 1.3.0, 0.98.20 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Blocker > Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.21 > > Attachments: 16074.test.branch-1.3.patch, 16074.test.patch, > HBASE-16074.branch-1.3.001.patch, HBASE-16074.branch-1.3.002.patch, > changes_to_stress_ITBLL.patch, changes_to_stress_ITBLL__a_bit_relaxed_.patch, > itbll log with failure, itbll log with success > > > Underlying MR jobs succeed but I'm seeing the following in the logs (mid-size > distributed test cluster): > ERROR test.IntegrationTestBigLinkedList$Verify: Found nodes which lost big or > tiny families, count=164 > I do not know exactly yet whether it's a bug, a test issue or env setup > issue, but need figure it out. Opening this to raise awareness and see if > someone saw that recently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16074) ITBLL fails, reports lost big or tine families
[ https://issues.apache.org/jira/browse/HBASE-16074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-16074: --- Fix Version/s: 0.98.21 1.4.0 2.0.0 > ITBLL fails, reports lost big or tine families > -- > > Key: HBASE-16074 > URL: https://issues.apache.org/jira/browse/HBASE-16074 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 1.3.0, 0.98.20 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Blocker > Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.21 > > Attachments: 16074.test.branch-1.3.patch, 16074.test.patch, > HBASE-16074.branch-1.3.001.patch, changes_to_stress_ITBLL.patch, > changes_to_stress_ITBLL__a_bit_relaxed_.patch, itbll log with failure, itbll > log with success > > > Underlying MR jobs succeed but I'm seeing the following in the logs (mid-size > distributed test cluster): > ERROR test.IntegrationTestBigLinkedList$Verify: Found nodes which lost big or > tiny families, count=164 > I do not know exactly yet whether it's a bug, a test issue or env setup > issue, but need figure it out. Opening this to raise awareness and see if > someone saw that recently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16074) ITBLL fails, reports lost big or tine families
[ https://issues.apache.org/jira/browse/HBASE-16074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-16074: --- Affects Version/s: 0.98.20 > ITBLL fails, reports lost big or tine families > -- > > Key: HBASE-16074 > URL: https://issues.apache.org/jira/browse/HBASE-16074 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 1.3.0, 0.98.20 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Blocker > Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.21 > > Attachments: 16074.test.branch-1.3.patch, 16074.test.patch, > HBASE-16074.branch-1.3.001.patch, changes_to_stress_ITBLL.patch, > changes_to_stress_ITBLL__a_bit_relaxed_.patch, itbll log with failure, itbll > log with success > > > Underlying MR jobs succeed but I'm seeing the following in the logs (mid-size > distributed test cluster): > ERROR test.IntegrationTestBigLinkedList$Verify: Found nodes which lost big or > tiny families, count=164 > I do not know exactly yet whether it's a bug, a test issue or env setup > issue, but need figure it out. Opening this to raise awareness and see if > someone saw that recently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16074) ITBLL fails, reports lost big or tine families
[ https://issues.apache.org/jira/browse/HBASE-16074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16074: -- Attachment: HBASE-16074.branch-1.3.001.patch > ITBLL fails, reports lost big or tine families > -- > > Key: HBASE-16074 > URL: https://issues.apache.org/jira/browse/HBASE-16074 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Blocker > Fix For: 1.3.0 > > Attachments: 16074.test.branch-1.3.patch, 16074.test.patch, > HBASE-16074.branch-1.3.001.patch, changes_to_stress_ITBLL.patch, > changes_to_stress_ITBLL__a_bit_relaxed_.patch, itbll log with failure, itbll > log with success > > > Underlying MR jobs succeed but I'm seeing the following in the logs (mid-size > distributed test cluster): > ERROR test.IntegrationTestBigLinkedList$Verify: Found nodes which lost big or > tiny families, count=164 > I do not know exactly yet whether it's a bug, a test issue or env setup > issue, but need figure it out. Opening this to raise awareness and see if > someone saw that recently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16074) ITBLL fails, reports lost big or tine families
[ https://issues.apache.org/jira/browse/HBASE-16074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16074: -- Attachment: 16074.test.branch-1.3.patch > ITBLL fails, reports lost big or tine families > -- > > Key: HBASE-16074 > URL: https://issues.apache.org/jira/browse/HBASE-16074 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Blocker > Fix For: 1.3.0 > > Attachments: 16074.test.branch-1.3.patch, 16074.test.patch, > changes_to_stress_ITBLL.patch, changes_to_stress_ITBLL__a_bit_relaxed_.patch, > itbll log with failure, itbll log with success > > > Underlying MR jobs succeed but I'm seeing the following in the logs (mid-size > distributed test cluster): > ERROR test.IntegrationTestBigLinkedList$Verify: Found nodes which lost big or > tiny families, count=164 > I do not know exactly yet whether it's a bug, a test issue or env setup > issue, but need figure it out. Opening this to raise awareness and see if > someone saw that recently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16074) ITBLL fails, reports lost big or tine families
[ https://issues.apache.org/jira/browse/HBASE-16074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16074: -- Attachment: 16074.test.patch Something to test with. Not sure if it fixes things but looking at the diff... two possible issues. 1. In StoreFile#Writer, the boolean which says if we've been passed a TimeRangeTracker is not volatile. Perhaps this an issue (lots of change here because needed one Writer for compaction-time and another at Flush time...) 2. LOG if the min is -1... that'd mess us up. Shouldn't happen... This is something to try Mikail... if it fixes things I'll dig again. Will keep looking at the diff... > ITBLL fails, reports lost big or tine families > -- > > Key: HBASE-16074 > URL: https://issues.apache.org/jira/browse/HBASE-16074 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Blocker > Fix For: 1.3.0 > > Attachments: 16074.test.patch, changes_to_stress_ITBLL.patch, > changes_to_stress_ITBLL__a_bit_relaxed_.patch, itbll log with failure, itbll > log with success > > > Underlying MR jobs succeed but I'm seeing the following in the logs (mid-size > distributed test cluster): > ERROR test.IntegrationTestBigLinkedList$Verify: Found nodes which lost big or > tiny families, count=164 > I do not know exactly yet whether it's a bug, a test issue or env setup > issue, but need figure it out. Opening this to raise awareness and see if > someone saw that recently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16074) ITBLL fails, reports lost big or tine families
[ https://issues.apache.org/jira/browse/HBASE-16074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16074: -- Status: Patch Available (was: Open) > ITBLL fails, reports lost big or tine families > -- > > Key: HBASE-16074 > URL: https://issues.apache.org/jira/browse/HBASE-16074 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Blocker > Fix For: 1.3.0 > > Attachments: 16074.test.patch, changes_to_stress_ITBLL.patch, > changes_to_stress_ITBLL__a_bit_relaxed_.patch, itbll log with failure, itbll > log with success > > > Underlying MR jobs succeed but I'm seeing the following in the logs (mid-size > distributed test cluster): > ERROR test.IntegrationTestBigLinkedList$Verify: Found nodes which lost big or > tiny families, count=164 > I do not know exactly yet whether it's a bug, a test issue or env setup > issue, but need figure it out. Opening this to raise awareness and see if > someone saw that recently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16074) ITBLL fails, reports lost big or tine families
[ https://issues.apache.org/jira/browse/HBASE-16074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-16074: Attachment: itbll log with success and clean run > ITBLL fails, reports lost big or tine families > -- > > Key: HBASE-16074 > URL: https://issues.apache.org/jira/browse/HBASE-16074 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Blocker > Fix For: 1.3.0 > > Attachments: changes_to_stress_ITBLL.patch, > changes_to_stress_ITBLL__a_bit_relaxed_.patch, itbll log with failure, itbll > log with success > > > Underlying MR jobs succeed but I'm seeing the following in the logs (mid-size > distributed test cluster): > ERROR test.IntegrationTestBigLinkedList$Verify: Found nodes which lost big or > tiny families, count=164 > I do not know exactly yet whether it's a bug, a test issue or env setup > issue, but need figure it out. Opening this to raise awareness and see if > someone saw that recently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16074) ITBLL fails, reports lost big or tine families
[ https://issues.apache.org/jira/browse/HBASE-16074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-16074: Attachment: itbll log with failure [~stack] here's log of run which failed (I should say, sometimes I take dozen of attempts to repro) in case it's helfpul. Grep by "verify" and this shows the error I'm seeing. > ITBLL fails, reports lost big or tine families > -- > > Key: HBASE-16074 > URL: https://issues.apache.org/jira/browse/HBASE-16074 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Blocker > Fix For: 1.3.0 > > Attachments: changes_to_stress_ITBLL.patch, > changes_to_stress_ITBLL__a_bit_relaxed_.patch, itbll log with failure > > > Underlying MR jobs succeed but I'm seeing the following in the logs (mid-size > distributed test cluster): > ERROR test.IntegrationTestBigLinkedList$Verify: Found nodes which lost big or > tiny families, count=164 > I do not know exactly yet whether it's a bug, a test issue or env setup > issue, but need figure it out. Opening this to raise awareness and see if > someone saw that recently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16074) ITBLL fails, reports lost big or tine families
[ https://issues.apache.org/jira/browse/HBASE-16074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-16074: Attachment: changes_to_stress_ITBLL__a_bit_relaxed_.patch > ITBLL fails, reports lost big or tine families > -- > > Key: HBASE-16074 > URL: https://issues.apache.org/jira/browse/HBASE-16074 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Blocker > Fix For: 1.3.0 > > Attachments: changes_to_stress_ITBLL.patch, > changes_to_stress_ITBLL__a_bit_relaxed_.patch > > > Underlying MR jobs succeed but I'm seeing the following in the logs (mid-size > distributed test cluster): > ERROR test.IntegrationTestBigLinkedList$Verify: Found nodes which lost big or > tiny families, count=164 > I do not know exactly yet whether it's a bug, a test issue or env setup > issue, but need figure it out. Opening this to raise awareness and see if > someone saw that recently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16074) ITBLL fails, reports lost big or tine families
[ https://issues.apache.org/jira/browse/HBASE-16074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-16074: Attachment: changes_to_stress_ITBLL.patch I was able to get the errors running ITBLL in minicluster from the IDE with the following ad-hoc patch, FYI. > ITBLL fails, reports lost big or tine families > -- > > Key: HBASE-16074 > URL: https://issues.apache.org/jira/browse/HBASE-16074 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Blocker > Fix For: 1.3.0 > > Attachments: changes_to_stress_ITBLL.patch > > > Underlying MR jobs succeed but I'm seeing the following in the logs (mid-size > distributed test cluster): > ERROR test.IntegrationTestBigLinkedList$Verify: Found nodes which lost big or > tiny families, count=164 > I do not know exactly yet whether it's a bug, a test issue or env setup > issue, but need figure it out. Opening this to raise awareness and see if > someone saw that recently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16074) ITBLL fails, reports lost big or tine families
[ https://issues.apache.org/jira/browse/HBASE-16074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-16074: Description: Underlying MR jobs succeed but I'm seeing the following in the logs (mid-size distributed test cluster): ERROR test.IntegrationTestBigLinkedList$Verify: Found nodes which lost big or tiny families, count=164 I do not know exactly yet whether it's a bug, a test issue or env setup issue, but need figure it out. Opening this to raise awareness and see if someone saw that recently. was: Underlying MR jobs succeed but I'm seeing the following in the logs (mid-size distributed test cluster): ERROR test.IntegrationTestBigLinkedList$Verify: Found nodes which lost big or tiny families, count=164 I do know know exactly yet whether it's a bug, a test issue or env setup issue, but need figure it out. Opening this to raise awareness and see if someone saw that recently. > ITBLL fails, reports lost big or tine families > -- > > Key: HBASE-16074 > URL: https://issues.apache.org/jira/browse/HBASE-16074 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Blocker > Fix For: 1.3.0 > > > Underlying MR jobs succeed but I'm seeing the following in the logs (mid-size > distributed test cluster): > ERROR test.IntegrationTestBigLinkedList$Verify: Found nodes which lost big or > tiny families, count=164 > I do not know exactly yet whether it's a bug, a test issue or env setup > issue, but need figure it out. Opening this to raise awareness and see if > someone saw that recently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16074) ITBLL fails, reports lost big or tine families
[ https://issues.apache.org/jira/browse/HBASE-16074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-16074: Description: Underlying MR jobs succeed but I'm seeing the following in the logs (mid-size distributed test cluster): ERROR test.IntegrationTestBigLinkedList$Verify: Found nodes which lost big or tiny families, count=164 I do know know exactly yet whether it's a bug, a test issue or env setup issue, but need figure it out. Opening this to raise awareness and see if someone saw that recently. was: Underlying MR jobs succeed but I'm seeing the following in the logs: ERROR test.IntegrationTestBigLinkedList$Verify: Found nodes which lost big or tiny families, count=164 I do know know exactly yet whether it's a bug, a test issue or env setup issue, but need figure it out. Opening this to raise awareness and see if someone saw that recently. > ITBLL fails, reports lost big or tine families > -- > > Key: HBASE-16074 > URL: https://issues.apache.org/jira/browse/HBASE-16074 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Blocker > Fix For: 1.3.0 > > > Underlying MR jobs succeed but I'm seeing the following in the logs (mid-size > distributed test cluster): > ERROR test.IntegrationTestBigLinkedList$Verify: Found nodes which lost big or > tiny families, count=164 > I do know know exactly yet whether it's a bug, a test issue or env setup > issue, but need figure it out. Opening this to raise awareness and see if > someone saw that recently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
