[jira] [Commented] (HBASE-23771) [Flakey Tests] Test TestSplitTransactionOnCluster Again
[ https://issues.apache.org/jira/browse/HBASE-23771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026462#comment-17026462 ] Michael Stack commented on HBASE-23771: --- Pushed attached patch on branch-2. It cycles waiting on clearance of references, waiting on any outstanding compactions first and then scheduling new ones until references are gone and we can move on. Leaving open while watch what happens over next 24 hours. Need better tooling/apis for watching over compactions. Compaction logs are a little baffling and should report explicitly for example why they choose not to major compact a store file (Will file an issue). > [Flakey Tests] Test TestSplitTransactionOnCluster Again > --- > > Key: HBASE-23771 > URL: https://issues.apache.org/jira/browse/HBASE-23771 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-23771-Flakey-Tests-Test-TestSplitTransactionOn.patch > > > Parent fix had the test failures in GCE go from 35% to 4%. Let me see if can > clear the remaining fails. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23771) [Flakey Tests] Test TestSplitTransactionOnCluster Again
[ https://issues.apache.org/jira/browse/HBASE-23771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-23771: -- Attachment: 0001-HBASE-23771-Flakey-Tests-Test-TestSplitTransactionOn.patch > [Flakey Tests] Test TestSplitTransactionOnCluster Again > --- > > Key: HBASE-23771 > URL: https://issues.apache.org/jira/browse/HBASE-23771 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-23771-Flakey-Tests-Test-TestSplitTransactionOn.patch > > > Parent fix had the test failures in GCE go from 35% to 4%. Let me see if can > clear the remaining fails. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23742) Mob Data, threshold crossed data is considering as normal data.
[ https://issues.apache.org/jira/browse/HBASE-23742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Y. SREENIVASULU REDDY updated HBASE-23742: -- Priority: Minor (was: Major) > Mob Data, threshold crossed data is considering as normal data. > --- > > Key: HBASE-23742 > URL: https://issues.apache.org/jira/browse/HBASE-23742 > Project: HBase > Issue Type: Sub-task > Components: MTTR >Affects Versions: 3.0.0, 2.3.0 >Reporter: Y. SREENIVASULU REDDY >Priority: Minor > > Steps to reproduce this issue. > 1. create a table with 1 region, and mob enabled, keep threshold value to 5. > 2. Load data into the table, keep the value size should be more than 5. > 3. flush the table. > 4. observe the mobdir and data dir, hfiles should be there. > 5. load data again with different data set, keep the value size is greater > than 5. > 6. Kill -9 RS where table region is online > 7. Start RS > check the mob dir and data dir, both should have 2 hfiles each. > But data dir only have 2 hfiles, that means mob threshold crossed data is > considered as normal data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23770) [Flakey Tests] TestRegionReplicasWithRestartScenarios#testWhenRestart
[ https://issues.apache.org/jira/browse/HBASE-23770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026459#comment-17026459 ] Michael Stack commented on HBASE-23770: --- Leaving open to see how it does over next 24hours. > [Flakey Tests] TestRegionReplicasWithRestartScenarios#testWhenRestart > - > > Key: HBASE-23770 > URL: https://issues.apache.org/jira/browse/HBASE-23770 > Project: HBase > Issue Type: Bug > Components: flakies >Reporter: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: > 0001-HBASE-23770-Flakey-Tests-TestRegionReplicasWithResta.patch, Screen Shot > 2020-01-29 at 9.45.58 PM.png > > > Fails about 35% of the time in the GCE build. Let me attach a picture from > current flakies dashboard for branch-2. > The test starts a cluster of three RS w/ 3 region replicas. It then stops a > server, starts a new one, and then expects that the remaining three nodes do > not have instances where two region replicas have landed on a single server. > It fails sporadically (reproducible locally) because when the SCP runs its > assign, sometimes timing has it so Master knows of two servers only. Making > the new start before the old one is stopped (instead of other way around) > seems to fix the test -- there'll be three servers up when SCP runs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23770) [Flakey Tests] TestRegionReplicasWithRestartScenarios#testWhenRestart
[ https://issues.apache.org/jira/browse/HBASE-23770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026457#comment-17026457 ] Michael Stack commented on HBASE-23770: --- Test was recently added by: HBASE-23078 BaseLoadBalancer should consider region replicas when randomAssignment and roundRobinAssignment (#663) Guanghao Zhang 9/29/19, 2:36 AM ... to 2.2+ but 2.2 does not have failures, only branch-2 (2.3.0) -- not even Master. > [Flakey Tests] TestRegionReplicasWithRestartScenarios#testWhenRestart > - > > Key: HBASE-23770 > URL: https://issues.apache.org/jira/browse/HBASE-23770 > Project: HBase > Issue Type: Bug > Components: flakies >Reporter: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: > 0001-HBASE-23770-Flakey-Tests-TestRegionReplicasWithResta.patch, Screen Shot > 2020-01-29 at 9.45.58 PM.png > > > Fails about 35% of the time in the GCE build. Let me attach a picture from > current flakies dashboard for branch-2. > The test starts a cluster of three RS w/ 3 region replicas. It then stops a > server, starts a new one, and then expects that the remaining three nodes do > not have instances where two region replicas have landed on a single server. > It fails sporadically (reproducible locally) because when the SCP runs its > assign, sometimes timing has it so Master knows of two servers only. Making > the new start before the old one is stopped (instead of other way around) > seems to fix the test -- there'll be three servers up when SCP runs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23770) [Flakey Tests] TestRegionReplicasWithRestartScenarios#testWhenRestart
[ https://issues.apache.org/jira/browse/HBASE-23770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-23770: -- Fix Version/s: 2.3.0 3.0.0 > [Flakey Tests] TestRegionReplicasWithRestartScenarios#testWhenRestart > - > > Key: HBASE-23770 > URL: https://issues.apache.org/jira/browse/HBASE-23770 > Project: HBase > Issue Type: Bug > Components: flakies >Reporter: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: > 0001-HBASE-23770-Flakey-Tests-TestRegionReplicasWithResta.patch, Screen Shot > 2020-01-29 at 9.45.58 PM.png > > > Fails about 35% of the time in the GCE build. Let me attach a picture from > current flakies dashboard for branch-2. > The test starts a cluster of three RS w/ 3 region replicas. It then stops a > server, starts a new one, and then expects that the remaining three nodes do > not have instances where two region replicas have landed on a single server. > It fails sporadically (reproducible locally) because when the SCP runs its > assign, sometimes timing has it so Master knows of two servers only. Making > the new start before the old one is stopped (instead of other way around) > seems to fix the test -- there'll be three servers up when SCP runs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23771) [Flakey Tests] Test TestSplitTransactionOnCluster Again
[ https://issues.apache.org/jira/browse/HBASE-23771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-23771: -- Description: Parent fix had the test failures in GCE go from 35% to 4%. Let me see if can clear the remaining fails. > [Flakey Tests] Test TestSplitTransactionOnCluster Again > --- > > Key: HBASE-23771 > URL: https://issues.apache.org/jira/browse/HBASE-23771 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Major > > Parent fix had the test failures in GCE go from 35% to 4%. Let me see if can > clear the remaining fails. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23771) [Flakey Tests] Test TestSplitTransactionOnCluster Again
[ https://issues.apache.org/jira/browse/HBASE-23771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-23771: -- Summary: [Flakey Tests] Test TestSplitTransactionOnCluster Again (was: [Flakey Tests] Test) > [Flakey Tests] Test TestSplitTransactionOnCluster Again > --- > > Key: HBASE-23771 > URL: https://issues.apache.org/jira/browse/HBASE-23771 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23733) [Flakey Tests] TestSplitTransactionOnCluster
[ https://issues.apache.org/jira/browse/HBASE-23733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026451#comment-17026451 ] Michael Stack commented on HBASE-23733: --- Hmm... Much better but still an odd failure I see one up on the GCE builds ... !Screen Shot 2020-01-29 at 10.16.54 PM.png! ... and in a little bunch of local builds, its come up once or twice. Let me open new subissue to try stuff. Its the same root issue of not being able to clear References. Let me be more aggressive. > [Flakey Tests] TestSplitTransactionOnCluster > > > Key: HBASE-23733 > URL: https://issues.apache.org/jira/browse/HBASE-23733 > Project: HBase > Issue Type: Bug > Components: flakies >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.4 > > Attachments: > 0001-HBASE-23733-Flakey-Tests-TestSplitTransactionOnClust.patch, Screen Shot > 2020-01-29 at 10.16.54 PM.png > > > TestSplitTransactionOnCluster fails 80% of time in the GCE run up on > https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2/lastSuccessfulBuild/artifact/dashboard.html > We want to run a compaction to clear up references but logic doesn't handle > case where compaction already running. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23771) [Flakey Tests] Test
Michael Stack created HBASE-23771: - Summary: [Flakey Tests] Test Key: HBASE-23771 URL: https://issues.apache.org/jira/browse/HBASE-23771 Project: HBase Issue Type: Sub-task Reporter: Michael Stack -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23733) [Flakey Tests] TestSplitTransactionOnCluster
[ https://issues.apache.org/jira/browse/HBASE-23733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-23733: -- Attachment: Screen Shot 2020-01-29 at 10.16.54 PM.png > [Flakey Tests] TestSplitTransactionOnCluster > > > Key: HBASE-23733 > URL: https://issues.apache.org/jira/browse/HBASE-23733 > Project: HBase > Issue Type: Bug > Components: flakies >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.4 > > Attachments: > 0001-HBASE-23733-Flakey-Tests-TestSplitTransactionOnClust.patch, Screen Shot > 2020-01-29 at 10.16.54 PM.png > > > TestSplitTransactionOnCluster fails 80% of time in the GCE run up on > https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2/lastSuccessfulBuild/artifact/dashboard.html > We want to run a compaction to clear up references but logic doesn't handle > case where compaction already running. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23770) [Flakey Tests] TestRegionReplicasWithRestartScenarios#testWhenRestart
[ https://issues.apache.org/jira/browse/HBASE-23770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026450#comment-17026450 ] Michael Stack commented on HBASE-23770: --- Here is screen shot of current failure rate on the GCE builds. !Screen Shot 2020-01-29 at 9.45.58 PM.png! > [Flakey Tests] TestRegionReplicasWithRestartScenarios#testWhenRestart > - > > Key: HBASE-23770 > URL: https://issues.apache.org/jira/browse/HBASE-23770 > Project: HBase > Issue Type: Bug > Components: flakies >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-23770-Flakey-Tests-TestRegionReplicasWithResta.patch, Screen Shot > 2020-01-29 at 9.45.58 PM.png > > > Fails about 35% of the time in the GCE build. Let me attach a picture from > current flakies dashboard for branch-2. > The test starts a cluster of three RS w/ 3 region replicas. It then stops a > server, starts a new one, and then expects that the remaining three nodes do > not have instances where two region replicas have landed on a single server. > It fails sporadically (reproducible locally) because when the SCP runs its > assign, sometimes timing has it so Master knows of two servers only. Making > the new start before the old one is stopped (instead of other way around) > seems to fix the test -- there'll be three servers up when SCP runs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23770) [Flakey Tests] TestRegionReplicasWithRestartScenarios#testWhenRestart
[ https://issues.apache.org/jira/browse/HBASE-23770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-23770: -- Attachment: Screen Shot 2020-01-29 at 9.45.58 PM.png > [Flakey Tests] TestRegionReplicasWithRestartScenarios#testWhenRestart > - > > Key: HBASE-23770 > URL: https://issues.apache.org/jira/browse/HBASE-23770 > Project: HBase > Issue Type: Bug > Components: flakies >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-23770-Flakey-Tests-TestRegionReplicasWithResta.patch, Screen Shot > 2020-01-29 at 9.45.58 PM.png > > > Fails about 35% of the time in the GCE build. Let me attach a picture from > current flakies dashboard for branch-2. > The test starts a cluster of three RS w/ 3 region replicas. It then stops a > server, starts a new one, and then expects that the remaining three nodes do > not have instances where two region replicas have landed on a single server. > It fails sporadically (reproducible locally) because when the SCP runs its > assign, sometimes timing has it so Master knows of two servers only. Making > the new start before the old one is stopped (instead of other way around) > seems to fix the test -- there'll be three servers up when SCP runs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23770) [Flakey Tests] TestRegionReplicasWithRestartScenarios#testWhenRestart
[ https://issues.apache.org/jira/browse/HBASE-23770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026449#comment-17026449 ] Michael Stack commented on HBASE-23770: --- What I pushed on branch-2. Lets see how it does. > [Flakey Tests] TestRegionReplicasWithRestartScenarios#testWhenRestart > - > > Key: HBASE-23770 > URL: https://issues.apache.org/jira/browse/HBASE-23770 > Project: HBase > Issue Type: Bug > Components: flakies >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-23770-Flakey-Tests-TestRegionReplicasWithResta.patch, Screen Shot > 2020-01-29 at 9.45.58 PM.png > > > Fails about 35% of the time in the GCE build. Let me attach a picture from > current flakies dashboard for branch-2. > The test starts a cluster of three RS w/ 3 region replicas. It then stops a > server, starts a new one, and then expects that the remaining three nodes do > not have instances where two region replicas have landed on a single server. > It fails sporadically (reproducible locally) because when the SCP runs its > assign, sometimes timing has it so Master knows of two servers only. Making > the new start before the old one is stopped (instead of other way around) > seems to fix the test -- there'll be three servers up when SCP runs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23770) [Flakey Tests] TestRegionReplicasWithRestartScenarios#testWhenRestart
[ https://issues.apache.org/jira/browse/HBASE-23770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-23770: -- Attachment: 0001-HBASE-23770-Flakey-Tests-TestRegionReplicasWithResta.patch > [Flakey Tests] TestRegionReplicasWithRestartScenarios#testWhenRestart > - > > Key: HBASE-23770 > URL: https://issues.apache.org/jira/browse/HBASE-23770 > Project: HBase > Issue Type: Bug > Components: flakies >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-23770-Flakey-Tests-TestRegionReplicasWithResta.patch > > > Fails about 35% of the time in the GCE build. Let me attach a picture from > current flakies dashboard for branch-2. > The test starts a cluster of three RS w/ 3 region replicas. It then stops a > server, starts a new one, and then expects that the remaining three nodes do > not have instances where two region replicas have landed on a single server. > It fails sporadically (reproducible locally) because when the SCP runs its > assign, sometimes timing has it so Master knows of two servers only. Making > the new start before the old one is stopped (instead of other way around) > seems to fix the test -- there'll be three servers up when SCP runs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-22670) JDK 11 and CellComparator
[ https://issues.apache.org/jira/browse/HBASE-22670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-22670: -- Parent: (was: HBASE-22972) Issue Type: Bug (was: Sub-task) > JDK 11 and CellComparator > - > > Key: HBASE-22670 > URL: https://issues.apache.org/jira/browse/HBASE-22670 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > Labels: jdk11 > > This could act as a parent JIRA for analysing JDK 11 and the Comparator impls > that we have. > Latest JDK has support for SIMD and AVX512, which means it has support for > vectorizations. > See JDK11's ArraysSupport#mismatch() and vectorizedMismatch(). > We also have BufferMismatch#mismatch() which is not publicly exposed but it > uses the ArraysSupport#vectorizedMismatch(). > Internally vectorizedMismatch() does something similar to what > UnsafeComparator#compareToUnsafe does. Will add about the details of the > study in further comments. > The JDK also exposes new annotations like @HotSpotIntrinsicCandidate and > @ForceInline tags that helps in inlining the intrinsic calls. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22670) JDK 11 and CellComparator
[ https://issues.apache.org/jira/browse/HBASE-22670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026447#comment-17026447 ] Michael Stack commented on HBASE-22670: --- Thanks [~ram_krish] > JDK 11 and CellComparator > - > > Key: HBASE-22670 > URL: https://issues.apache.org/jira/browse/HBASE-22670 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > Labels: jdk11 > > This could act as a parent JIRA for analysing JDK 11 and the Comparator impls > that we have. > Latest JDK has support for SIMD and AVX512, which means it has support for > vectorizations. > See JDK11's ArraysSupport#mismatch() and vectorizedMismatch(). > We also have BufferMismatch#mismatch() which is not publicly exposed but it > uses the ArraysSupport#vectorizedMismatch(). > Internally vectorizedMismatch() does something similar to what > UnsafeComparator#compareToUnsafe does. Will add about the details of the > study in further comments. > The JDK also exposes new annotations like @HotSpotIntrinsicCandidate and > @ForceInline tags that helps in inlining the intrinsic calls. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23770) [Flakey Tests] TestRegionReplicasWithRestartScenarios#testWhenRestart
Michael Stack created HBASE-23770: - Summary: [Flakey Tests] TestRegionReplicasWithRestartScenarios#testWhenRestart Key: HBASE-23770 URL: https://issues.apache.org/jira/browse/HBASE-23770 Project: HBase Issue Type: Bug Components: flakies Reporter: Michael Stack Fails about 35% of the time in the GCE build. Let me attach a picture from current flakies dashboard for branch-2. The test starts a cluster of three RS w/ 3 region replicas. It then stops a server, starts a new one, and then expects that the remaining three nodes do not have instances where two region replicas have landed on a single server. It fails sporadically (reproducible locally) because when the SCP runs its assign, sometimes timing has it so Master knows of two servers only. Making the new start before the old one is stopped (instead of other way around) seems to fix the test -- there'll be three servers up when SCP runs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22670) JDK 11 and CellComparator
[ https://issues.apache.org/jira/browse/HBASE-22670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026440#comment-17026440 ] ramkrishna.s.vasudevan commented on HBASE-22670: Yes we can. I may not be able to do those perf tests again but it is better we can make it a standalone issue and then close out the parent. > JDK 11 and CellComparator > - > > Key: HBASE-22670 > URL: https://issues.apache.org/jira/browse/HBASE-22670 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > Labels: jdk11 > > This could act as a parent JIRA for analysing JDK 11 and the Comparator impls > that we have. > Latest JDK has support for SIMD and AVX512, which means it has support for > vectorizations. > See JDK11's ArraysSupport#mismatch() and vectorizedMismatch(). > We also have BufferMismatch#mismatch() which is not publicly exposed but it > uses the ArraysSupport#vectorizedMismatch(). > Internally vectorizedMismatch() does something similar to what > UnsafeComparator#compareToUnsafe does. Will add about the details of the > study in further comments. > The JDK also exposes new annotations like @HotSpotIntrinsicCandidate and > @ForceInline tags that helps in inlining the intrinsic calls. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22671) ByteBufferUtils#findCommonPrefix() can be safely changed to ArraysSupport#mismatch()
[
https://issues.apache.org/jira/browse/HBASE-22671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026438#comment-17026438
]
ramkrishna.s.vasudevan commented on HBASE-22671:
I don have the patch with me currently but I can try to make one.
> ByteBufferUtils#findCommonPrefix() can be safely changed to
> ArraysSupport#mismatch()
>
>
> Key: HBASE-22671
> URL: https://issues.apache.org/jira/browse/HBASE-22671
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
>
> Microbenchmarks reveal that finding the common prefix for encoders can safely
> be replaced with ArraysSupport#mismatch().
> the microbenchmark just compares Cells that are backed with array and BB.
> For a 27 bit common row prefix the existing BBUtils#findCommonPrefix
> {code}
> BenchmarkMode CntScoreError Units
> PrefixComparator.arrayBBCompare avgt 10 869.897 ± 9.429 ns/op
> PrefixComparator.arrayCompareavgt 10 302.074 ± 13.448 ns/op
> PrefixComparator.bbArrayCompare avgt 10 869.369 ± 5.368 ns/op
> PrefixComparator.bbCompare avgt 10 409.479 ± 1.587 ns/op
> {code}
> the same with ArraysSupport#mismatch() change gives this
> {code}
> BenchmarkMode CntScore Error Units
> PrefixComparator.arrayBBCompare avgt 10 311.946 ± 1.902 ns/op
> PrefixComparator.arrayCompareavgt 10 157.010 ± 4.482 ns/op
> PrefixComparator.bbArrayCompare avgt 10 311.568 ± 1.348 ns/op
> PrefixComparator.bbCompare avgt 10 92.540 ± 0.501 ns/op
> {code}
> How ever note that this comes in flushes/compaction and not during the read
> path.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23349) Low refCount preventing archival of compacted away files
[
https://issues.apache.org/jira/browse/HBASE-23349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026433#comment-17026433
]
ramkrishna.s.vasudevan commented on HBASE-23349:
I will list out my points here
-> Generally the scanners created at the Region level due to incoming user
requests have a lease mechanism. It is basically to release the resources. So
any long running scan there will be a point of end for the scan or may be even
if the user
is not responsive or not closing the scan we have the lease mechanism. In any
such case the scanners gets closed and the resources get released.
-> The case where it cannot happen is when CPs create their own scanners. Then
there are chances that if the CP scanner fails or does not release the
resources we may hold up the underlying resources and even recover lease will
not work
-> One way to solve this is to have a lease mechanism for CP scans also so that
we don end up in scans being alive for a longer time.
-> Coming to the benefit of the ref count based mechanism, it solves the sync
block issues which was happening for every next call. But not only that we have
two other benefits
For the first benefit, pls refer to this user mailing list
http://apache-hbase.679495.n3.nabble.com/Extremely-long-flush-times-td4104190.html
http://mail-archives.apache.org/mod_mbox/hbase-dev/201208.mbox/%3c6548f17059905b48b2a6f28ce3692baa0ce29...@oaexch4server.oa.oclc.org%3E
(see the later part of it).
We are helping the readers to carry on without the impact of flushes and the
reverse way where flushes are not blocked due to readers. Here the scans are
heavier where either you have filters applied, more deletes/versions to skip
through.
In such cases having a non sync way of readers always helps.
The other benefit is that, the current readers need not reset itself, load
the new files(after compaction which may not be in cache), reseek to the last
fetched row and then again proceed with the scan. Obviously it means that
the next scan that comes will have to anyway read from the filesystem and
then load to the cache but atleast the ongoing scans are not impacted.
-> Finally I would like to mention that Phoenix like cases where there could be
a query that reads large amount of data and has filtering applied along with
heavy writes, it may be obvious that we may face the issues
as the users have faced in the mailing list.
I am fine if every body agrees to revert the patch and put back the sync way of
readers (or any other better soln). Just saying because it should be giving a
view that am in favour of the exisiting behaviour and against any changes to it.
> Low refCount preventing archival of compacted away files
>
>
> Key: HBASE-23349
> URL: https://issues.apache.org/jira/browse/HBASE-23349
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.3.0, 1.6.0
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 1.6.0
>
>
> We have observed that refCount on compacted away store files as low as 1 is
> prevent archival.
> {code:java}
> regionserver.HStore - Can't archive compacted file
> hdfs://{{root-dir}}/hbase/data/default/t1/12a9e1112e0371955b3db8d3ebb2d298/cf1/73b72f5ddfce4a34a9e01afe7b83c1f9
> because of either isCompactedAway=true or file has reference,
> isReferencedInReads=true, refCount=1, skipping for now.
> {code}
> We should come up with core code (run as part of discharger thread)
> gracefully resolve reader lock issue by resetting ongoing scanners to start
> pointing to new store files instead of compacted away store files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-20950) Helper method to configure secure DFS cluster for tests
[
https://issues.apache.org/jira/browse/HBASE-20950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026415#comment-17026415
]
Hudson commented on HBASE-20950:
Results for branch branch-2.1
[build #1788 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1788/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1788//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1788//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1788//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Helper method to configure secure DFS cluster for tests
> ---
>
> Key: HBASE-20950
> URL: https://issues.apache.org/jira/browse/HBASE-20950
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Affects Versions: 3.0.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-20950.master.001.patch,
> HBASE-20950.master.002.patch, HBASE-20950.master.003.patch
>
>
> There is quite some boilerplate code for configuring a secure HDFS cluster
> for tests. The code is repeated in a number of test files within HBase code
> base. Convert the boilerplate code into a helper method to avoid duplication
> and lower maintenance effort.
> SecureTestCluster#setHdfsSecuredConfiguration
> TestSecureExport#setUpClusterKdc
> TestThriftSpnegoHttpServer#addSecurityConfigurations
> TestSaslFanOutOneBlockAsyncDFSOutput#setHdfsSecuredConfiguration
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23749) TestHFileWriterV3 should have tests for all data block encodings
[
https://issues.apache.org/jira/browse/HBASE-23749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026414#comment-17026414
]
Hudson commented on HBASE-23749:
Results for branch branch-2.1
[build #1788 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1788/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1788//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1788//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1788//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> TestHFileWriterV3 should have tests for all data block encodings
>
>
> Key: HBASE-23749
> URL: https://issues.apache.org/jira/browse/HBASE-23749
> Project: HBase
> Issue Type: Test
>Affects Versions: 3.0.0, 2.3.0, 1.6.0
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 1.6.0, 2.2.3, 2.1.9
>
>
> TestHFileWriterV3 tests include writing and reading HFiles for default data
> blocks encoding. However, since this test has efficient way of testing by
> looping through encoded/un-encoded keys and values and keeping offsets
> updated etc, it would be really good to have such test for each encoded data
> blocks. For that, we should explore using respective seekers e.g.
> RowIndexV1Seeker for ROW_INDEX_V1 encoded data blocks etc.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23760) (2.1) Helper method to configure secure DFS cluster for tests
[
https://issues.apache.org/jira/browse/HBASE-23760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026416#comment-17026416
]
Hudson commented on HBASE-23760:
Results for branch branch-2.1
[build #1788 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1788/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1788//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1788//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1788//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> (2.1) Helper method to configure secure DFS cluster for tests
> -
>
> Key: HBASE-23760
> URL: https://issues.apache.org/jira/browse/HBASE-23760
> Project: HBase
> Issue Type: Task
> Components: security, test
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 1.6.0, 2.1.9
>
> Attachments: HBASE-23760.001.branch-1.patch,
> HBASE-23760.001.branch-2.1.patch
>
>
> Let's get [~weichiu]'s HBASE-20950 onto branch-2.1. It will make the backport
> for HBASE-17115 that much easier.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-17115) HMaster/HRegion Info Server does not honour admin.acl
[
https://issues.apache.org/jira/browse/HBASE-17115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026413#comment-17026413
]
Hudson commented on HBASE-17115:
Results for branch branch-2.1
[build #1788 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1788/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1788//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1788//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1788//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> HMaster/HRegion Info Server does not honour admin.acl
> -
>
> Key: HBASE-17115
> URL: https://issues.apache.org/jira/browse/HBASE-17115
> Project: HBase
> Issue Type: Bug
>Reporter: Mohammad Arshad
>Assignee: Josh Elser
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.2.3, 2.1.9
>
>
> Currently there is no way to enable protected URLs like /jmx, /conf only
> for admins. This is applicable for both Master and RegionServer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] bharathv commented on issue #1106: HBASE-23764: Switch to IP address for ZK ensemble
bharathv commented on issue #1106: HBASE-23764: Switch to IP address for ZK ensemble URL: https://github.com/apache/hbase/pull/1106#issuecomment-580066626 Much easier to commit and then revert later if something is broken IMO. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23754) [DOC] HBase Web UI runs on different port
[
https://issues.apache.org/jira/browse/HBASE-23754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026405#comment-17026405
]
Sean Busbey commented on HBASE-23754:
-
No inconvenience at all. Thanks for chasing things down!
> [DOC] HBase Web UI runs on different port
> -
>
> Key: HBASE-23754
> URL: https://issues.apache.org/jira/browse/HBASE-23754
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Yiğit Oner
>Priority: Trivial
>
> Just started to work through the ref guide and started up the stand-alone
> setup. The guide mentions to look up [http://localhost:16010/] for the HBase
> Web UI, but there is no process listening to that port.
>
> After checking netstat and logs I've discovered, that the UI runs on port
> 43105
>
> {code}
> 2020-01-28 22:17:43,658 INFO [main] http.HttpServer: Jetty bound to port 43105
> 2020-01-28 22:17:43,658 INFO [main] mortbay.log: jetty-6.1.26
> 2020-01-28 22:17:43,697 INFO [main] mortbay.log: Started
> SelectChannelConnector@0.0.0.0:4310
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23749) TestHFileWriterV3 should have tests for all data block encodings
[
https://issues.apache.org/jira/browse/HBASE-23749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026403#comment-17026403
]
Hudson commented on HBASE-23749:
Results for branch branch-2
[build #2439 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2439/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2439//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2439//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2439//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> TestHFileWriterV3 should have tests for all data block encodings
>
>
> Key: HBASE-23749
> URL: https://issues.apache.org/jira/browse/HBASE-23749
> Project: HBase
> Issue Type: Test
>Affects Versions: 3.0.0, 2.3.0, 1.6.0
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 1.6.0, 2.2.3, 2.1.9
>
>
> TestHFileWriterV3 tests include writing and reading HFiles for default data
> blocks encoding. However, since this test has efficient way of testing by
> looping through encoded/un-encoded keys and values and keeping offsets
> updated etc, it would be really good to have such test for each encoded data
> blocks. For that, we should explore using respective seekers e.g.
> RowIndexV1Seeker for ROW_INDEX_V1 encoded data blocks etc.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23769) TestSecureRESTServer fails due to hostname mismatch
[
https://issues.apache.org/jira/browse/HBASE-23769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026401#comment-17026401
]
Josh Elser commented on HBASE-23769:
Set on -Dsun.security.krb5.debug=true (and maybe
-Dsun.security.spnego.debug=true can't tell if this is spnego yet) and you
should get a clear error message about what the client was asking for from the
kdc. I've seen this before with VMs where localhost resolution is wonky as a
"feature" so that things resolve from the host OS. Something like localhost
actually resolves to something other than 127.0.0.1 but then the IP that was
resolved has a reverse lookup into the fqdn.
In short, double check /etc/hosts and make sure localhost is sane.
> TestSecureRESTServer fails due to hostname mismatch
> ---
>
> Key: HBASE-23769
> URL: https://issues.apache.org/jira/browse/HBASE-23769
> Project: HBase
> Issue Type: Test
> Components: REST, test
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Nick Dimiduk
>Priority: Minor
>
> Above test fails reliably in a simpleVagrant/VirtualBox Ubuntu vm.
> {noformat}
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.005 s <<<
> FAILURE! - in org.apache.hadoop.hbase.rest.TestSecureRESTServer
> org.apache.hadoop.hbase.rest.TestSecureRESTServer Time elapsed: 0.001 s <<<
> ERROR!
> java.io.IOException: Failed on local exception: java.io.IOException: Couldn't
> setup connection for hbase/localh...@example.com to
> localhost/127.0.0.1:45719; Host Details : local host is: "bionic/10.0.2.15";
> destination host is: "localhost":45719;
> at
> org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
> Caused by: java.io.IOException: Couldn't setup connection for
> hbase/localh...@example.com to localhost/127.0.0.1:45719
> at
> org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
> Caused by: javax.security.sasl.SaslException: GSS initiate failed
> at
> org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
> Caused by: org.ietf.jgss.GSSException: No valid credentials provided
> (Mechanism level: Message stream modified (41) - Message stream modified)
> at
> org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
> Caused by: sun.security.krb5.KrbException: Message stream modified (41) -
> Message stream modified
> at
> org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
> Caused by: sun.security.krb5.Asn1Exception: Identifier doesn't match expected
> value (906)
> at
> org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] busbey commented on issue #1106: HBASE-23764: Switch to IP address for ZK ensemble
busbey commented on issue #1106: HBASE-23764: Switch to IP address for ZK ensemble URL: https://github.com/apache/hbase/pull/1106#issuecomment-580054569 Or if you want the full nightlies push a branch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] busbey commented on issue #1106: HBASE-23764: Switch to IP address for ZK ensemble
busbey commented on issue #1106: HBASE-23764: Switch to IP address for ZK ensemble URL: https://github.com/apache/hbase/pull/1106#issuecomment-580054428 If you want a full suite of unit tests just touch the root pom. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-18095) Provide an option for clients to find the server hosting META that does not involve the ZooKeeper client
[
https://issues.apache.org/jira/browse/HBASE-18095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026398#comment-17026398
]
Hudson commented on HBASE-18095:
Results for branch HBASE-18095/client-locate-meta-no-zookeeper
[build #55 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/55/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/55//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/55//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/55//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/55//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Provide an option for clients to find the server hosting META that does not
> involve the ZooKeeper client
>
>
> Key: HBASE-18095
> URL: https://issues.apache.org/jira/browse/HBASE-18095
> Project: HBase
> Issue Type: New Feature
> Components: Client
>Reporter: Andrew Kyle Purtell
>Assignee: Bharath Vissapragada
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 1.6.0
>
> Attachments: HBASE-18095.master-v1.patch, HBASE-18095.master-v2.patch
>
>
> Clients are required to connect to ZooKeeper to find the location of the
> regionserver hosting the meta table region. Site configuration provides the
> client a list of ZK quorum peers and the client uses an embedded ZK client to
> query meta location. Timeouts and retry behavior of this embedded ZK client
> are managed orthogonally to HBase layer settings and in some cases the ZK
> cannot manage what in theory the HBase client can, i.e. fail fast upon outage
> or network partition.
> We should consider new configuration settings that provide a list of
> well-known master and backup master locations, and with this information the
> client can contact any of the master processes directly. Any master in either
> active or passive state will track meta location and respond to requests for
> it with its cached last known location. If this location is stale, the client
> can ask again with a flag set that requests the master refresh its location
> cache and return the up-to-date location. Every client interaction with the
> cluster thus uses only HBase RPC as transport, with appropriate settings
> applied to the connection. The configuration toggle that enables this
> alternative meta location lookup should be false by default.
> This removes the requirement that HBase clients embed the ZK client and
> contact the ZK service directly at the beginning of the connection lifecycle.
> This has several benefits. ZK service need not be exposed to clients, and
> their potential abuse, yet no benefit ZK provides the HBase server cluster is
> compromised. Normalizing HBase client and ZK client timeout settings and
> retry behavior - in some cases, impossible, i.e. for fail-fast - is no longer
> necessary.
> And, from [~ghelmling]: There is an additional complication here for
> token-based authentication. When a delegation token is used for SASL
> authentication, the client uses the cluster ID obtained from Zookeeper to
> select the token identifier to use. So there would also need to be some
> Zookeeper-less, unauthenticated way to obtain the cluster ID as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22172) Suppress Java 11 reflective access warnings
[
https://issues.apache.org/jira/browse/HBASE-22172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026345#comment-17026345
]
Nick Dimiduk commented on HBASE-22172:
--
Can we close this as won't fix then? The idea is to fix accesses according to
the new APIs, not sweep warnings under the rug.
> Suppress Java 11 reflective access warnings
> ---
>
> Key: HBASE-22172
> URL: https://issues.apache.org/jira/browse/HBASE-22172
> Project: HBase
> Issue Type: Sub-task
> Components: java, scripts
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
> Labels: jdk11
> Attachments: hbase-22172.master.001.patch
>
>
> While running a Java 8 compiled hbase on Java 11 system, I found the
> following warnings being thrown. I think we can add the "--add-opens" flag to
> HBASE_OPTS (if the jdk version is 11) to suppress this warning.
> {code:java}
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.hadoop.hbase.util.UnsafeAvailChecker
> (file:/Users/jatsakthi/test/HBASE_TEST_AREA/hbase-3.0.0-SNAPSHOT/lib/hbase-common-3.0.0-SNAPSHOT.jar)
> to method java.nio.Bits.unaligned()
> WARNING: Please consider reporting this to the maintainers of
> org.apache.hadoop.hbase.util.UnsafeAvailChecker
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23578) [UI] Master UI shows long stack traces when table is broken
[ https://issues.apache.org/jira/browse/HBASE-23578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026342#comment-17026342 ] Xu Cang commented on HBASE-23578: - LGTM, please fix a minor conflict. [~Shuhei Yamasaki] > [UI] Master UI shows long stack traces when table is broken > --- > > Key: HBASE-23578 > URL: https://issues.apache.org/jira/browse/HBASE-23578 > Project: HBase > Issue Type: Improvement > Components: master, UI >Reporter: Shuhei Yamasaki >Priority: Minor > Attachments: stackCompact1_short.png, table_jsp.png > > > The table.jsp in Master UI shows long stack traces when table is broken. > (shown as table_jsp.png) > This messages are hard to read and web page is very wide because stack traces > displayed in a single line. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] xcangCRM commented on issue #951: HBASE-23578 [UI] Master UI shows long stack traces when table is broken
xcangCRM commented on issue #951: HBASE-23578 [UI] Master UI shows long stack traces when table is broken URL: https://github.com/apache/hbase/pull/951#issuecomment-580030351 This LGTM. Please fix the conflict. thank you @YamasakiSS This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23769) TestSecureRESTServer fails due to hostname mismatch
[
https://issues.apache.org/jira/browse/HBASE-23769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026312#comment-17026312
]
Nick Dimiduk commented on HBASE-23769:
--
Is there some assumption about hostnames that this environment is missing
[~elserj]?
> TestSecureRESTServer fails due to hostname mismatch
> ---
>
> Key: HBASE-23769
> URL: https://issues.apache.org/jira/browse/HBASE-23769
> Project: HBase
> Issue Type: Test
> Components: REST, test
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Nick Dimiduk
>Priority: Minor
>
> Above test fails reliably in a simpleVagrant/VirtualBox Ubuntu vm.
> {noformat}
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.005 s <<<
> FAILURE! - in org.apache.hadoop.hbase.rest.TestSecureRESTServer
> org.apache.hadoop.hbase.rest.TestSecureRESTServer Time elapsed: 0.001 s <<<
> ERROR!
> java.io.IOException: Failed on local exception: java.io.IOException: Couldn't
> setup connection for hbase/localh...@example.com to
> localhost/127.0.0.1:45719; Host Details : local host is: "bionic/10.0.2.15";
> destination host is: "localhost":45719;
> at
> org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
> Caused by: java.io.IOException: Couldn't setup connection for
> hbase/localh...@example.com to localhost/127.0.0.1:45719
> at
> org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
> Caused by: javax.security.sasl.SaslException: GSS initiate failed
> at
> org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
> Caused by: org.ietf.jgss.GSSException: No valid credentials provided
> (Mechanism level: Message stream modified (41) - Message stream modified)
> at
> org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
> Caused by: sun.security.krb5.KrbException: Message stream modified (41) -
> Message stream modified
> at
> org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
> Caused by: sun.security.krb5.Asn1Exception: Identifier doesn't match expected
> value (906)
> at
> org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22587) Add a REST server to the nightly test that uses a cluster for simple operations
[ https://issues.apache.org/jira/browse/HBASE-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026306#comment-17026306 ] Sakthi commented on HBASE-22587: Yeah you are right [~ndimiduk] . I was thinking from an integration testing point of view. > Add a REST server to the nightly test that uses a cluster for simple > operations > --- > > Key: HBASE-22587 > URL: https://issues.apache.org/jira/browse/HBASE-22587 > Project: HBase > Issue Type: Sub-task > Components: REST, test >Reporter: Sakthi >Priority: Major > Labels: jdk11 > > Let's add rest server to the nightly test that uses a cluster for simple > operations to catch any possible rest server issues related to dependencies, > cnfe, npe etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23769) TestSecureRESTServer fails due to hostname mismatch
[
https://issues.apache.org/jira/browse/HBASE-23769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-23769:
-
Affects Version/s: 2.3.0
> TestSecureRESTServer fails due to hostname mismatch
> ---
>
> Key: HBASE-23769
> URL: https://issues.apache.org/jira/browse/HBASE-23769
> Project: HBase
> Issue Type: Test
> Components: REST, test
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Nick Dimiduk
>Priority: Minor
>
> Above test fails reliably in a simpleVagrant/VirtualBox Ubuntu vm.
> {noformat}
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.005 s <<<
> FAILURE! - in org.apache.hadoop.hbase.rest.TestSecureRESTServer
> org.apache.hadoop.hbase.rest.TestSecureRESTServer Time elapsed: 0.001 s <<<
> ERROR!
> java.io.IOException: Failed on local exception: java.io.IOException: Couldn't
> setup connection for hbase/localh...@example.com to
> localhost/127.0.0.1:45719; Host Details : local host is: "bionic/10.0.2.15";
> destination host is: "localhost":45719;
> at
> org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
> Caused by: java.io.IOException: Couldn't setup connection for
> hbase/localh...@example.com to localhost/127.0.0.1:45719
> at
> org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
> Caused by: javax.security.sasl.SaslException: GSS initiate failed
> at
> org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
> Caused by: org.ietf.jgss.GSSException: No valid credentials provided
> (Mechanism level: Message stream modified (41) - Message stream modified)
> at
> org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
> Caused by: sun.security.krb5.KrbException: Message stream modified (41) -
> Message stream modified
> at
> org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
> Caused by: sun.security.krb5.Asn1Exception: Identifier doesn't match expected
> value (906)
> at
> org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-23769) TestSecureRESTServer fails due to hostname mismatch
Nick Dimiduk created HBASE-23769:
Summary: TestSecureRESTServer fails due to hostname mismatch
Key: HBASE-23769
URL: https://issues.apache.org/jira/browse/HBASE-23769
Project: HBase
Issue Type: Test
Components: REST, test
Affects Versions: 3.0.0
Reporter: Nick Dimiduk
Above test fails reliably in a simpleVagrant/VirtualBox Ubuntu vm.
{noformat}
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.005 s <<<
FAILURE! - in org.apache.hadoop.hbase.rest.TestSecureRESTServer
org.apache.hadoop.hbase.rest.TestSecureRESTServer Time elapsed: 0.001 s <<<
ERROR!
java.io.IOException: Failed on local exception: java.io.IOException: Couldn't
setup connection for hbase/localh...@example.com to localhost/127.0.0.1:45719;
Host Details : local host is: "bionic/10.0.2.15"; destination host is:
"localhost":45719;
at
org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
Caused by: java.io.IOException: Couldn't setup connection for
hbase/localh...@example.com to localhost/127.0.0.1:45719
at
org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
Caused by: javax.security.sasl.SaslException: GSS initiate failed
at
org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism
level: Message stream modified (41) - Message stream modified)
at
org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
Caused by: sun.security.krb5.KrbException: Message stream modified (41) -
Message stream modified
at
org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
Caused by: sun.security.krb5.Asn1Exception: Identifier doesn't match expected
value (906)
at
org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-23768) Backport to 1.x
Josh Elser created HBASE-23768: -- Summary: Backport to 1.x Key: HBASE-23768 URL: https://issues.apache.org/jira/browse/HBASE-23768 Project: HBase Issue Type: Sub-task Reporter: Josh Elser Assignee: Josh Elser Fix For: 1.6.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] joshelser commented on issue #936: HBASE-17115 Define UI admins via an ACL
joshelser commented on issue #936: HBASE-17115 Define UI admins via an ACL URL: https://github.com/apache/hbase/pull/936#issuecomment-57773 Submitted to all 2.x branches and master, after getting HBASE-20950 also backported to branch-2.1. 1.x needs some help with precommit before this change can land there. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] joshelser closed pull request #936: HBASE-17115 Define UI admins via an ACL
joshelser closed pull request #936: HBASE-17115 Define UI admins via an ACL URL: https://github.com/apache/hbase/pull/936 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (HBASE-17115) HMaster/HRegion Info Server does not honour admin.acl
[ https://issues.apache.org/jira/browse/HBASE-17115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser resolved HBASE-17115. Hadoop Flags: Reviewed Release Note: Implements authorization for the HBase Web UI by limiting access to certain endpoints which could be used to extract sensitive information from HBase. Access to these restricted endpoints can be limited to a group of administrators, identified either by a list of users (hbase.security.authentication.spnego.admin.users) or by a list of groups (hbase.security.authentication.spnego.admin.groups). By default, neither of these values are set which will preserve backwards compatibility (allowing all authenticated users to access all endpoints). Further, users who have sensitive information in the HBase service configuration can set hbase.security.authentication.ui.config.protected to true which will treat the configuration endpoint as a protected, admin-only resource. By default, all authenticated users may access the configuration endpoint. Resolution: Fixed PreCommit on 1.x looks like it's busted. Resolving this for now and will revisit a 1.x backport when I can figure out what's going on with precommit. > HMaster/HRegion Info Server does not honour admin.acl > - > > Key: HBASE-17115 > URL: https://issues.apache.org/jira/browse/HBASE-17115 > Project: HBase > Issue Type: Bug >Reporter: Mohammad Arshad >Assignee: Josh Elser >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.3, 2.1.9 > > > Currently there is no way to enable protected URLs like /jmx, /conf only > for admins. This is applicable for both Master and RegionServer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-17115) HMaster/HRegion Info Server does not honour admin.acl
[ https://issues.apache.org/jira/browse/HBASE-17115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026275#comment-17026275 ] Josh Elser commented on HBASE-17115: Alright, 2.x and master have this. Let me see if it comes back to branch-1 easily. > HMaster/HRegion Info Server does not honour admin.acl > - > > Key: HBASE-17115 > URL: https://issues.apache.org/jira/browse/HBASE-17115 > Project: HBase > Issue Type: Bug >Reporter: Mohammad Arshad >Assignee: Josh Elser >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.3, 2.1.9 > > > Currently there is no way to enable protected URLs like /jmx, /conf only > for admins. This is applicable for both Master and RegionServer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23767) Add JDK11 to the Jenkinsfile(s) for matrix builds
Nick Dimiduk created HBASE-23767: Summary: Add JDK11 to the Jenkinsfile(s) for matrix builds Key: HBASE-23767 URL: https://issues.apache.org/jira/browse/HBASE-23767 Project: HBase Issue Type: Sub-task Components: build Reporter: Nick Dimiduk We already test against multiple JDK versions in a handful of places. Let's get JDK11 added to the mix. Applies to pre-commit and nightly... anywhere else? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23684) NPE HFilesOutputSink
[
https://issues.apache.org/jira/browse/HBASE-23684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-23684:
-
Fix Version/s: 2.3.0
3.0.0
> NPE HFilesOutputSink
>
>
> Key: HBASE-23684
> URL: https://issues.apache.org/jira/browse/HBASE-23684
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.3.0
>Reporter: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
>
> Enabling the new split to hfiles feature, HBASE-23286, running branch-2 tip,
> I see this out on RegionServers:
> {code}
> 2020-01-13 17:37:08,204 INFO org.apache.hadoop.hbase.wal.OutputSink: 3 split
> writer threads finished
> 2020-01-13 17:37:08,233 INFO org.apache.hadoop.hbase.wal.WALSplitter:
> Processed 1007 edits across 0 regions cost 284 ms; edits skipped=76;
> WAL=hdfs://nameservice1/hbase/genie/WALs/hbasedn101.example.org,16020,1578934806382-splitting/hbasedn101.example.org%2C16020%2C1578934806382.1578937008832,
> size=128.5 M, length=134708720, corrupted=false, progress failed=true
> 2020-01-13 17:37:08,234 WARN
> org.apache.hadoop.hbase.regionserver.SplitLogWorker: log splitting of
> WALs/hbasedn101.example.org,16020,1578934806382-splitting/hbasedn101.example.org%2C16020%2C1578934806382.1578937008832
> failed, returning error
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.writeRemainingEntryBuffers(BoundedRecoveredHFilesOutputSink.java:173)
> at
> org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.close(BoundedRecoveredHFilesOutputSink.java:140)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:339)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:181)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.splitLog(SplitLogWorker.java:105)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.lambda$new$0(SplitLogWorker.java:84)
> at
> org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:70)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.configContextForNonMetaWriter(BoundedRecoveredHFilesOutputSink.java:225)
> at
> org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.createRecoveredHFileWriter(BoundedRecoveredHFilesOutputSink.java:213)
> at
> org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.append(BoundedRecoveredHFilesOutputSink.java:117)
> at
> org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.lambda$writeRemainingEntryBuffers$3(BoundedRecoveredHFilesOutputSink.java:155)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> {code}
> It is a bit odd because log says there were zero regions. Not sure what that
> was about.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22587) Add a REST server to the nightly test that uses a cluster for simple operations
[
https://issues.apache.org/jira/browse/HBASE-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026258#comment-17026258
]
Nick Dimiduk commented on HBASE-22587:
--
Existing tests using {{HBaseRESTTestingUtility}} don't accomplish what you're
seeking here, [~sakthi]? We need some kind of integration test that works by
shelling out to {{bin/hbase rest start}} ?
> Add a REST server to the nightly test that uses a cluster for simple
> operations
> ---
>
> Key: HBASE-22587
> URL: https://issues.apache.org/jira/browse/HBASE-22587
> Project: HBase
> Issue Type: Sub-task
> Components: REST, test
>Reporter: Sakthi
>Priority: Major
> Labels: jdk11
>
> Let's add rest server to the nightly test that uses a cluster for simple
> operations to catch any possible rest server issues related to dependencies,
> cnfe, npe etc.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] bharathv commented on issue #1106: HBASE-23764: Switch to IP address for ZK ensemble
bharathv commented on issue #1106: HBASE-23764: Switch to IP address for ZK ensemble URL: https://github.com/apache/hbase/pull/1106#issuecomment-579965921 > Should we replace all instances of localhost w/ 127.x? Based on my experiments, I think the bug is only in the ZKClient implementation. Hence I just changed the ZK ensemble config (and that works for me locally). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-22972) [JDK11] Support JDK11 LTS in HBase
[ https://issues.apache.org/jira/browse/HBASE-22972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026229#comment-17026229 ] Nick Dimiduk commented on HBASE-22972: -- I'm told HADOOP-15338 is targeting 3.3. Is waiting for, upgrading to a Hadoop 3.3 release really a requirement for our purposes? > [JDK11] Support JDK11 LTS in HBase > -- > > Key: HBASE-22972 > URL: https://issues.apache.org/jira/browse/HBASE-22972 > Project: HBase > Issue Type: Umbrella >Reporter: Duo Zhang >Priority: Blocker > Labels: jdk11 > Fix For: 3.0.0, 2.3.0 > > > This is an umbrella issue for tracking all the problems for JDK11 in HBase. > And we also rely on hadoop. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack commented on issue #1106: HBASE-23764: Switch to IP address for ZK ensemble
saintstack commented on issue #1106: HBASE-23764: Switch to IP address for ZK ensemble URL: https://github.com/apache/hbase/pull/1106#issuecomment-579956560 Should we replace all instances of localhost w/ 127.x? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-15898) Document G1GC Recommendations
[ https://issues.apache.org/jira/browse/HBASE-15898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-15898: -- Parent: (was: HBASE-22972) Issue Type: Task (was: Sub-task) > Document G1GC Recommendations > - > > Key: HBASE-15898 > URL: https://issues.apache.org/jira/browse/HBASE-15898 > Project: HBase > Issue Type: Task > Components: documentation, java >Affects Versions: 2.0.0 >Reporter: Misty Linville >Assignee: Misty Linville >Priority: Major > Labels: jdk11 > Attachments: HBASE-15898-001.PATCH, HBASE-15898.patch > > > Document G1GC recommendations for HBase -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-15898) Document G1GC Recommendations
[ https://issues.apache.org/jira/browse/HBASE-15898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026201#comment-17026201 ] Nick Dimiduk commented on HBASE-15898: -- I agree that this isn't a prerequisite for JDK11 support. > Document G1GC Recommendations > - > > Key: HBASE-15898 > URL: https://issues.apache.org/jira/browse/HBASE-15898 > Project: HBase > Issue Type: Sub-task > Components: documentation, java >Affects Versions: 2.0.0 >Reporter: Misty Linville >Assignee: Misty Linville >Priority: Major > Labels: jdk11 > Attachments: HBASE-15898-001.PATCH, HBASE-15898.patch > > > Document G1GC recommendations for HBase -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23755) [OpenTracing] Declare HTrace is unusable in the user doc
[ https://issues.apache.org/jira/browse/HBASE-23755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026186#comment-17026186 ] Wei-Chiu Chuang commented on HBASE-23755: - Thanks. Didn't realize the docs are patched against master. Don't think there's any public API change. Probably the only user facing stuff is the configuration hbase.trace.spanreceiver.classes and the hbase shell (there's a trace command) I'm thinking to update the doc to deprecate Htrace in the master branch now. The doc for opentracing will be in the feature branch, which will get added to doc when the branch is merged into trunk. > [OpenTracing] Declare HTrace is unusable in the user doc > > > Key: HBASE-23755 > URL: https://issues.apache.org/jira/browse/HBASE-23755 > Project: HBase > Issue Type: Sub-task >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > The trace doesn't work at all in HBase 2.0 and above after HBASE-18601 (the > trace doesn't get picked up at the server side). We should make a note in the > user doc stating it is > # unusable > # deprecated in HBase 2.x because HTrace is in Attic. > # removed from HBase 3.0 and above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack commented on issue #1106: HBASE-23764: Switch to IP address for ZK ensemble
saintstack commented on issue #1106: HBASE-23764: Switch to IP address for ZK ensemble URL: https://github.com/apache/hbase/pull/1106#issuecomment-579942058 @ndimiduk no. need to make non-changes in each of the modules you want to spin. Could also just push it and see This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-15898) Document G1GC Recommendations
[
https://issues.apache.org/jira/browse/HBASE-15898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026181#comment-17026181
]
HBase QA commented on HBASE-15898:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
5s{color} | {color:blue} The patch file was not named according to hbase's
naming conventions. Please see
https://yetus.apache.org/documentation/in-progress/precommit-patchnames for
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 4m
45s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 4m
55s{color} | {color:blue} patch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 base:
https://builds.apache.org/job/PreCommit-HBASE-Build/1114/artifact/patchprocess/Dockerfile
|
| JIRA Issue | HBASE-15898 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12970104/HBASE-15898-001.PATCH
|
| Optional Tests | dupname asflicense refguide |
| uname | Linux 58532041ed39 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 66d198d35f |
| refguide |
https://builds.apache.org/job/PreCommit-HBASE-Build/1114/artifact/patchprocess/branch-site/book.html
|
| refguide |
https://builds.apache.org/job/PreCommit-HBASE-Build/1114/artifact/patchprocess/patch-site/book.html
|
| Max. process+thread count | 95 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/1114/console |
| versions | git=2.11.0 maven=2018-06-17T18:33:14Z) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
> Document G1GC Recommendations
> -
>
> Key: HBASE-15898
> URL: https://issues.apache.org/jira/browse/HBASE-15898
> Project: HBase
> Issue Type: Sub-task
> Components: documentation, java
>Affects Versions: 2.0.0
>Reporter: Misty Linville
>Assignee: Misty Linville
>Priority: Major
> Labels: jdk11
> Attachments: HBASE-15898-001.PATCH, HBASE-15898.patch
>
>
> Document G1GC recommendations for HBase
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22587) Add a REST server to the nightly test that uses a cluster for simple operations
[ https://issues.apache.org/jira/browse/HBASE-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026173#comment-17026173 ] Michael Stack commented on HBASE-22587: --- Oh. I should have read the description. Thanks [~sakthi] > Add a REST server to the nightly test that uses a cluster for simple > operations > --- > > Key: HBASE-22587 > URL: https://issues.apache.org/jira/browse/HBASE-22587 > Project: HBase > Issue Type: Sub-task > Components: REST, test >Reporter: Sakthi >Priority: Major > Labels: jdk11 > > Let's add rest server to the nightly test that uses a cluster for simple > operations to catch any possible rest server issues related to dependencies, > cnfe, npe etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23045) currentPath may be stitched in a loop in replication source code.
[ https://issues.apache.org/jira/browse/HBASE-23045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani updated HBASE-23045: - Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to branch-1's. Thanks for the contribution [~gk_coder] > currentPath may be stitched in a loop in replication source code. > -- > > Key: HBASE-23045 > URL: https://issues.apache.org/jira/browse/HBASE-23045 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.2.6.1 >Reporter: kangkang.guo >Assignee: kangkang.guo >Priority: Critical > Fix For: 1.6.0, 1.2.6.1 > > Attachments: HBASE-23045.branch-1.000.patch, > HBASE-23045.branch-1.000.patch, HBASE-23045.branch-1.2.0001.patch > > > When the openReader encounters a FileNotFoundException, we may go to all > possible directories to find the current hlog. When found, the path may be > wrong, and it is looped together. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22587) Add a REST server to the nightly test that uses a cluster for simple operations
[ https://issues.apache.org/jira/browse/HBASE-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026165#comment-17026165 ] Sakthi commented on HBASE-22587: All the tasks listed here issues that are related to getting our rest server work fine with jdk 11 I guess. But the parent issue hasn't yet been started/completed (No one has taken up the task yet). The motto was for us to continuously test our rest server by doing some simple cluster operations through the rest endpoint. [~stack] > Add a REST server to the nightly test that uses a cluster for simple > operations > --- > > Key: HBASE-22587 > URL: https://issues.apache.org/jira/browse/HBASE-22587 > Project: HBase > Issue Type: Sub-task > Components: REST, test >Reporter: Sakthi >Priority: Major > Labels: jdk11 > > Let's add rest server to the nightly test that uses a cluster for simple > operations to catch any possible rest server issues related to dependencies, > cnfe, npe etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-15898) Document G1GC Recommendations
[ https://issues.apache.org/jira/browse/HBASE-15898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026164#comment-17026164 ] Michael Stack commented on HBASE-15898: --- Move this out of the parent until gets some love? > Document G1GC Recommendations > - > > Key: HBASE-15898 > URL: https://issues.apache.org/jira/browse/HBASE-15898 > Project: HBase > Issue Type: Sub-task > Components: documentation, java >Affects Versions: 2.0.0 >Reporter: Misty Linville >Assignee: Misty Linville >Priority: Major > Labels: jdk11 > Attachments: HBASE-15898-001.PATCH, HBASE-15898.patch > > > Document G1GC recommendations for HBase -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23748) Include HBASE-21284 to branch-2.2
[ https://issues.apache.org/jira/browse/HBASE-23748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026151#comment-17026151 ] Sakthi commented on HBASE-23748: Thanks [~elserj]. Let me get this in. > Include HBASE-21284 to branch-2.2 > - > > Key: HBASE-23748 > URL: https://issues.apache.org/jira/browse/HBASE-23748 > Project: HBase > Issue Type: Sub-task >Reporter: Sakthi >Assignee: Sakthi >Priority: Major > Fix For: 2.2.3 > > Attachments: hbase-23748.branch-2.2.001.patch > > > HBASE-21284 was ought to be present in 2.2. But by the time the commit was > done, the branch had been cut already. Hence this Jira to track it's > inclusion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22587) Add a REST server to the nightly test that uses a cluster for simple operations
[ https://issues.apache.org/jira/browse/HBASE-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026150#comment-17026150 ] Michael Stack commented on HBASE-22587: --- Can we resolve this parent issue now [~sakthi] given all subtasks are done? > Add a REST server to the nightly test that uses a cluster for simple > operations > --- > > Key: HBASE-22587 > URL: https://issues.apache.org/jira/browse/HBASE-22587 > Project: HBase > Issue Type: Sub-task > Components: REST, test >Reporter: Sakthi >Priority: Major > Labels: jdk11 > > Let's add rest server to the nightly test that uses a cluster for simple > operations to catch any possible rest server issues related to dependencies, > cnfe, npe etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22671) ByteBufferUtils#findCommonPrefix() can be safely changed to ArraysSupport#mismatch()
[
https://issues.apache.org/jira/browse/HBASE-22671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026148#comment-17026148
]
Michael Stack commented on HBASE-22671:
---
[~ram_krish] Seems like a benefit. Patch?
> ByteBufferUtils#findCommonPrefix() can be safely changed to
> ArraysSupport#mismatch()
>
>
> Key: HBASE-22671
> URL: https://issues.apache.org/jira/browse/HBASE-22671
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
>
> Microbenchmarks reveal that finding the common prefix for encoders can safely
> be replaced with ArraysSupport#mismatch().
> the microbenchmark just compares Cells that are backed with array and BB.
> For a 27 bit common row prefix the existing BBUtils#findCommonPrefix
> {code}
> BenchmarkMode CntScoreError Units
> PrefixComparator.arrayBBCompare avgt 10 869.897 ± 9.429 ns/op
> PrefixComparator.arrayCompareavgt 10 302.074 ± 13.448 ns/op
> PrefixComparator.bbArrayCompare avgt 10 869.369 ± 5.368 ns/op
> PrefixComparator.bbCompare avgt 10 409.479 ± 1.587 ns/op
> {code}
> the same with ArraysSupport#mismatch() change gives this
> {code}
> BenchmarkMode CntScore Error Units
> PrefixComparator.arrayBBCompare avgt 10 311.946 ± 1.902 ns/op
> PrefixComparator.arrayCompareavgt 10 157.010 ± 4.482 ns/op
> PrefixComparator.bbArrayCompare avgt 10 311.568 ± 1.348 ns/op
> PrefixComparator.bbCompare avgt 10 92.540 ± 0.501 ns/op
> {code}
> How ever note that this comes in flushes/compaction and not during the read
> path.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22670) JDK 11 and CellComparator
[ https://issues.apache.org/jira/browse/HBASE-22670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026146#comment-17026146 ] Michael Stack commented on HBASE-22670: --- [~ram_krish] good stuff. Seems like its worth spending time in here. Should we make this a standalone, perf issue so we can close out the parent? > JDK 11 and CellComparator > - > > Key: HBASE-22670 > URL: https://issues.apache.org/jira/browse/HBASE-22670 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > Labels: jdk11 > > This could act as a parent JIRA for analysing JDK 11 and the Comparator impls > that we have. > Latest JDK has support for SIMD and AVX512, which means it has support for > vectorizations. > See JDK11's ArraysSupport#mismatch() and vectorizedMismatch(). > We also have BufferMismatch#mismatch() which is not publicly exposed but it > uses the ArraysSupport#vectorizedMismatch(). > Internally vectorizedMismatch() does something similar to what > UnsafeComparator#compareToUnsafe does. Will add about the details of the > study in further comments. > The JDK also exposes new annotations like @HotSpotIntrinsicCandidate and > @ForceInline tags that helps in inlining the intrinsic calls. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23749) TestHFileWriterV3 should have tests for all data block encodings
[ https://issues.apache.org/jira/browse/HBASE-23749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani resolved HBASE-23749. -- Hadoop Flags: Reviewed Resolution: Fixed > TestHFileWriterV3 should have tests for all data block encodings > > > Key: HBASE-23749 > URL: https://issues.apache.org/jira/browse/HBASE-23749 > Project: HBase > Issue Type: Test >Affects Versions: 3.0.0, 2.3.0, 1.6.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Fix For: 3.0.0, 2.3.0, 1.6.0, 2.2.3, 2.1.9 > > > TestHFileWriterV3 tests include writing and reading HFiles for default data > blocks encoding. However, since this test has efficient way of testing by > looping through encoded/un-encoded keys and values and keeping offsets > updated etc, it would be really good to have such test for each encoded data > blocks. For that, we should explore using respective seekers e.g. > RowIndexV1Seeker for ROW_INDEX_V1 encoded data blocks etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HBASE-23749) TestHFileWriterV3 should have tests for all data block encodings
[ https://issues.apache.org/jira/browse/HBASE-23749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-23749 started by Viraj Jasani. > TestHFileWriterV3 should have tests for all data block encodings > > > Key: HBASE-23749 > URL: https://issues.apache.org/jira/browse/HBASE-23749 > Project: HBase > Issue Type: Test >Affects Versions: 3.0.0, 2.3.0, 1.6.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Fix For: 3.0.0, 2.3.0, 1.6.0, 2.2.3, 2.1.9 > > > TestHFileWriterV3 tests include writing and reading HFiles for default data > blocks encoding. However, since this test has efficient way of testing by > looping through encoded/un-encoded keys and values and keeping offsets > updated etc, it would be really good to have such test for each encoded data > blocks. For that, we should explore using respective seekers e.g. > RowIndexV1Seeker for ROW_INDEX_V1 encoded data blocks etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on issue #1106: HBASE-23764: Switch to IP address for ZK ensemble
ndimiduk commented on issue #1106: HBASE-23764: Switch to IP address for ZK ensemble URL: https://github.com/apache/hbase/pull/1106#issuecomment-579916852 @busbey is there a way to kick Yetus into running the full suite vs this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23755) [OpenTracing] Declare HTrace is unusable in the user doc
[ https://issues.apache.org/jira/browse/HBASE-23755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026129#comment-17026129 ] Nick Dimiduk commented on HBASE-23755: -- Is this for the docs only? Are you planning to deprecate any APIs here? Docs patches are usually against master only. > [OpenTracing] Declare HTrace is unusable in the user doc > > > Key: HBASE-23755 > URL: https://issues.apache.org/jira/browse/HBASE-23755 > Project: HBase > Issue Type: Sub-task >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > The trace doesn't work at all in HBase 2.0 and above after HBASE-18601 (the > trace doesn't get picked up at the server side). We should make a note in the > user doc stating it is > # unusable > # deprecated in HBase 2.x because HTrace is in Attic. > # removed from HBase 3.0 and above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23765) Use 127.0.0.1 instead of localhost when setting up Kerberos-related tests
[ https://issues.apache.org/jira/browse/HBASE-23765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026123#comment-17026123 ] Bharath Vissapragada commented on HBASE-23765: -- I created another jira based on my observations (HBASE-23764). I think it has something to do with ZKClient implementation (more details in the jira). I don't know if a similar problem exists with rest of the stack (hdfs/secure hdfs etc.) > Use 127.0.0.1 instead of localhost when setting up Kerberos-related tests > - > > Key: HBASE-23765 > URL: https://issues.apache.org/jira/browse/HBASE-23765 > Project: HBase > Issue Type: Task > Components: test >Reporter: Josh Elser >Priority: Minor > > [~ndimiduk] gave an ask over on HBASE-23760 to change some of the > Hadoop-level configuration properties around secure cluster setup from > localhost to 127.0.0.1. He and [~bharathv] have been chasing some issues with > ZooKeeper and not having a resolution of localhost to 127.0.0.1. > Before I start making a change, how sure are we that this is an issue? > Assuming that it's the nightlies that we see these on, how about we make a > change to increase the krb5 and spnego debugging to see if we aren't > resolving names properly? > There might be a debug property for DNS lookups in Java too maybe? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23764) Flaky tests due to ZK client name resolution delays
[
https://issues.apache.org/jira/browse/HBASE-23764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026120#comment-17026120
]
HBase QA commented on HBASE-23764:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
50s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
23s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 5m
25s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 5m
29s{color} | {color:blue} patch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
8s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 base:
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1106/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hbase/pull/1106 |
| JIRA Issue | HBASE-23764 |
| Optional Tests | dupname asflicense javac javadoc unit refguide xml |
| uname | Linux 04c89cf0470f 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1106/out/precommit/personality/provided.sh
|
| git revision | master / 66d198d35f |
| Default Java | 1.8.0_181 |
| refguide |
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1106/1/artifact/out/branch-site/book.html
|
| refguide |
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1106/1/artifact/out/patch-site/book.html
|
| Test Results |
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1106/1/testReport/
|
| Max. process+thread count | 255 (vs. ulimit of 1) |
| modules | C: hbase-common U: hbase-common |
| Console output |
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1106/1/console |
| versions | git=2.11.0 maven=2018-06-17T18:33:14Z) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
> Flaky tests due to ZK client name resolution delays
> ---
>
> Key: HBASE-23764
> URL: https://issues.apache.org/jira/browse/HBASE-23764
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
>Reporter: Bharath Vissapragada
>Assignee: Bharath Vissapragada
>Priority: Major
>
> [~ndimiduk] and I ran into this issue (separately) and we noticed that there
> are some performance issues with name resolution in the Zookeeper
[GitHub] [hbase] Apache-HBase commented on issue #1106: HBASE-23764: Switch to IP address for ZK ensemble
Apache-HBase commented on issue #1106: HBASE-23764: Switch to IP address for ZK ensemble URL: https://github.com/apache/hbase/pull/1106#issuecomment-579907994 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 50s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 23s | master passed | | +0 :ok: | refguide | 5m 25s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | javadoc | 0m 22s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 32s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +0 :ok: | refguide | 5m 29s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | javadoc | 0m 21s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 3m 8s | hbase-common in the patch passed. | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 30m 0s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1106/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1106 | | JIRA Issue | HBASE-23764 | | Optional Tests | dupname asflicense javac javadoc unit refguide xml | | uname | Linux 04c89cf0470f 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1106/out/precommit/personality/provided.sh | | git revision | master / 66d198d35f | | Default Java | 1.8.0_181 | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1106/1/artifact/out/branch-site/book.html | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1106/1/artifact/out/patch-site/book.html | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1106/1/testReport/ | | Max. process+thread count | 255 (vs. ulimit of 1) | | modules | C: hbase-common U: hbase-common | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1106/1/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (HBASE-23705) Add CellComparator to HFileContext
[ https://issues.apache.org/jira/browse/HBASE-23705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-23705. --- Resolution: Fixed Forgot to resolve. > Add CellComparator to HFileContext > -- > > Key: HBASE-23705 > URL: https://issues.apache.org/jira/browse/HBASE-23705 > Project: HBase > Issue Type: Sub-task > Components: io >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > The HFileContext is present when reading and writing files. It is populated > at read time using HFile trailer content and file metadata. At write time, we > create it up front. > Interesting is that though CellComparator is written to the HFile trailer, > and parse of the Trailer creates an HFileInfo which builds the HFileContext > at read time, the HFileContext does not expose what CellComparator to use > decoding and seeking. Around the codebase there are various compensations > made for this lack with decoders that actually have a decoding context (with > a reference to the hfilecontext), hard-coding use of the default > CellComparator. StoreFileInfo will use default if not passed a comparator > (even though we'd just read the trailer and even though it has reference to > filecontext) and HFile does similar. What CellComparator to use in a given > context is confused. > Let me fix this situation removing ambiguity. It will also fix bugs in parent > issue where UTs are failing because wrong CellComparator is being used. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23766) Support Point-In-Time Queries
Geoffrey Jacoby created HBASE-23766: --- Summary: Support Point-In-Time Queries Key: HBASE-23766 URL: https://issues.apache.org/jira/browse/HBASE-23766 Project: HBase Issue Type: New Feature Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby HBase currently offers a snapshot feature which allows operators to capture the state of a table at a point in time in a way that can be cloned or queried in the future. It's quite useful in some circumstances, but limited because it's a heavyweight operation, and because it requires prior knowledge of the time you want to capture. Phoenix currently offers a feature called "SCN", which uses the max timestamp on Scans to provide the illusion of a "lookback" query at a point in time. It's imperfect, however, because of HBase's filtering and cleanup logic for deletes, max versions and TTLs can prevent users from seeing certain Cells they would have been able to see at a previous point in time. Even PHOENIX-5645, and the equivalent HBASE-23602, which try to control major compaction cleanup, don't cover all edge cases completely. (For example, you can't see rows whose TTL has expired now but hadn't back then. Same with max versions.) There are useful non-Phoenix applications as well, such as a change stream that shows before/after images, as DynamoDB offers. Since full support will require new configuration options added not just to major compaction, but also to the read pipeline, I'm filing this as an umbrella JIRA so we can have smaller sub-tasks, rather than trying to cram everything into HBASE-23602. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-17115) HMaster/HRegion Info Server does not honour admin.acl
[ https://issues.apache.org/jira/browse/HBASE-17115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026107#comment-17026107 ] Josh Elser commented on HBASE-17115: Accidentally pushed this to 2.1 while trying to get HBASE-20950/HBASE-23760 cleaned up. Sorry for the revert. > HMaster/HRegion Info Server does not honour admin.acl > - > > Key: HBASE-17115 > URL: https://issues.apache.org/jira/browse/HBASE-17115 > Project: HBase > Issue Type: Bug >Reporter: Mohammad Arshad >Assignee: Josh Elser >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.3, 2.1.9 > > > Currently there is no way to enable protected URLs like /jmx, /conf only > for admins. This is applicable for both Master and RegionServer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23765) Use 127.0.0.1 instead of localhost when setting up Kerberos-related tests
Josh Elser created HBASE-23765: -- Summary: Use 127.0.0.1 instead of localhost when setting up Kerberos-related tests Key: HBASE-23765 URL: https://issues.apache.org/jira/browse/HBASE-23765 Project: HBase Issue Type: Task Components: test Reporter: Josh Elser [~ndimiduk] gave an ask over on HBASE-23760 to change some of the Hadoop-level configuration properties around secure cluster setup from localhost to 127.0.0.1. He and [~bharathv] have been chasing some issues with ZooKeeper and not having a resolution of localhost to 127.0.0.1. Before I start making a change, how sure are we that this is an issue? Assuming that it's the nightlies that we see these on, how about we make a change to increase the krb5 and spnego debugging to see if we aren't resolving names properly? There might be a debug property for DNS lookups in Java too maybe? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23760) (2.1) Helper method to configure secure DFS cluster for tests
[ https://issues.apache.org/jira/browse/HBASE-23760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-23760: --- Attachment: HBASE-23760.001.branch-1.patch > (2.1) Helper method to configure secure DFS cluster for tests > - > > Key: HBASE-23760 > URL: https://issues.apache.org/jira/browse/HBASE-23760 > Project: HBase > Issue Type: Task > Components: security, test >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 2.1.9 > > Attachments: HBASE-23760.001.branch-1.patch, > HBASE-23760.001.branch-2.1.patch > > > Let's get [~weichiu]'s HBASE-20950 onto branch-2.1. It will make the backport > for HBASE-17115 that much easier. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23760) (2.1) Helper method to configure secure DFS cluster for tests
[ https://issues.apache.org/jira/browse/HBASE-23760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-23760: --- Fix Version/s: 1.6.0 > (2.1) Helper method to configure secure DFS cluster for tests > - > > Key: HBASE-23760 > URL: https://issues.apache.org/jira/browse/HBASE-23760 > Project: HBase > Issue Type: Task > Components: security, test >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 1.6.0, 2.1.9 > > Attachments: HBASE-23760.001.branch-1.patch, > HBASE-23760.001.branch-2.1.patch > > > Let's get [~weichiu]'s HBASE-20950 onto branch-2.1. It will make the backport > for HBASE-17115 that much easier. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23749) TestHFileWriterV3 should have tests for all data block encodings
[ https://issues.apache.org/jira/browse/HBASE-23749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani updated HBASE-23749: - Fix Version/s: 2.1.9 2.2.3 1.6.0 2.3.0 3.0.0 > TestHFileWriterV3 should have tests for all data block encodings > > > Key: HBASE-23749 > URL: https://issues.apache.org/jira/browse/HBASE-23749 > Project: HBase > Issue Type: Test >Affects Versions: 3.0.0, 2.3.0, 1.6.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Fix For: 3.0.0, 2.3.0, 1.6.0, 2.2.3, 2.1.9 > > > TestHFileWriterV3 tests include writing and reading HFiles for default data > blocks encoding. However, since this test has efficient way of testing by > looping through encoded/un-encoded keys and values and keeping offsets > updated etc, it would be really good to have such test for each encoded data > blocks. For that, we should explore using respective seekers e.g. > RowIndexV1Seeker for ROW_INDEX_V1 encoded data blocks etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] bharathv commented on issue #1106: HBASE-23764: Switch to IP address for ZK ensemble
bharathv commented on issue #1106: HBASE-23764: Switch to IP address for ZK ensemble URL: https://github.com/apache/hbase/pull/1106#issuecomment-579894291 Please don't merge this yet. Just pushed it for review to kick off a build and see how it works. Probably needs community approval before committing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] bharathv opened a new pull request #1106: HBASE-23764: Switch to IP address for ZK ensemble
bharathv opened a new pull request #1106: HBASE-23764: Switch to IP address for ZK ensemble URL: https://github.com/apache/hbase/pull/1106 Tests showed that using IP address is much faster for mini cluster and a lot less flakier. See the jira for more details. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (HBASE-23754) [DOC] HBase Web UI runs on different port
[
https://issues.apache.org/jira/browse/HBASE-23754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiğit Oner resolved HBASE-23754.
Resolution: Not A Problem
> [DOC] HBase Web UI runs on different port
> -
>
> Key: HBASE-23754
> URL: https://issues.apache.org/jira/browse/HBASE-23754
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Yiğit Oner
>Priority: Trivial
>
> Just started to work through the ref guide and started up the stand-alone
> setup. The guide mentions to look up [http://localhost:16010/] for the HBase
> Web UI, but there is no process listening to that port.
>
> After checking netstat and logs I've discovered, that the UI runs on port
> 43105
>
> {code}
> 2020-01-28 22:17:43,658 INFO [main] http.HttpServer: Jetty bound to port 43105
> 2020-01-28 22:17:43,658 INFO [main] mortbay.log: jetty-6.1.26
> 2020-01-28 22:17:43,697 INFO [main] mortbay.log: Started
> SelectChannelConnector@0.0.0.0:4310
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23754) [DOC] HBase Web UI runs on different port
[
https://issues.apache.org/jira/browse/HBASE-23754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026090#comment-17026090
]
Yiğit Oner commented on HBASE-23754:
[~busbey] I've redid the setup just to be sure. Now it is listening to 16010. I
guess, I got a wrong version on the first attempt
Sorry for the inconvenience
> [DOC] HBase Web UI runs on different port
> -
>
> Key: HBASE-23754
> URL: https://issues.apache.org/jira/browse/HBASE-23754
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Yiğit Oner
>Priority: Trivial
>
> Just started to work through the ref guide and started up the stand-alone
> setup. The guide mentions to look up [http://localhost:16010/] for the HBase
> Web UI, but there is no process listening to that port.
>
> After checking netstat and logs I've discovered, that the UI runs on port
> 43105
>
> {code}
> 2020-01-28 22:17:43,658 INFO [main] http.HttpServer: Jetty bound to port 43105
> 2020-01-28 22:17:43,658 INFO [main] mortbay.log: jetty-6.1.26
> 2020-01-28 22:17:43,697 INFO [main] mortbay.log: Started
> SelectChannelConnector@0.0.0.0:4310
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23764) Flaky tests due to ZK client name resolution delays
[
https://issues.apache.org/jira/browse/HBASE-23764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharath Vissapragada updated HBASE-23764:
-
Description:
[~ndimiduk] and I ran into this issue (separately) and we noticed that there
are some performance issues with name resolution in the Zookeeper client. Since
we use ZK heavily in the unit tests, this often manifests as the following
issues
1. Test time outs starting the mini cluster (Master failed to start)
2. InterruptedException (because the tests timeout)
3. Flaky tests because a subset of the cluster fails to start for whatever
reason (replication tests especially because they spawn multiple clusters).
4. ConnectionLoss to znode /hbase/xyzz.. JVM pause?
I have strong feeling that this is a possible cause for many of our flaky tests
in Jenkins. Luckily, it looks like the following workaround to switch to an IP
address instead of hostname makes it much quicker. There are some related
discussions in the ZK community (ZOOKEEPER-1666 and related jiras).
{code:java}
--- a/hbase-common/src/main/resources/hbase-default.xml
+++ b/hbase-common/src/main/resources/hbase-default.xml
@@ -72,7 +72,7 @@ possible configurations would overwhelm and obscure the
important.
hbase.zookeeper.quorum
-localhost
+127.0.0.1
Comma separated list of servers in the ZooKeeper ensemble
(This config. should have been named hbase.zookeeper.ensemble).
For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com".
{code}
Until we figure out the actual root cause and a dependency upgrade (if needed),
we should consider making this hostname to IP switch for more stable builds.
was:
[~ndimiduk] and I ran into this issue (separately) and we noticed that there
are some performance issues with name resolution in the Zookeeper client. Since
we use ZK heavily in the unit tests, this often manifests as the following
issues
1. Test time outs starting the mini cluster (Master failed to start)
2. InterruptedException (because the tests timeout)
3. Flaky tests because a subset of the cluster fails to start for whatever
reason (replication tests especially because they spawn multiple clusters).
I have strong feeling that this is a possible cause for many of our flaky tests
in Jenkins. Luckily, it looks like the following workaround to switch to an IP
address instead of hostname makes it much quicker. There are some related
discussions in the ZK community (ZOOKEEPER-1666 and related jiras).
Until we figure out the actual root cause and a dependency upgrade (if needed),
we should consider making this hostname to IP switch for more stable builds.
> Flaky tests due to ZK client name resolution delays
> ---
>
> Key: HBASE-23764
> URL: https://issues.apache.org/jira/browse/HBASE-23764
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
>Reporter: Bharath Vissapragada
>Assignee: Bharath Vissapragada
>Priority: Major
>
> [~ndimiduk] and I ran into this issue (separately) and we noticed that there
> are some performance issues with name resolution in the Zookeeper client.
> Since we use ZK heavily in the unit tests, this often manifests as the
> following issues
> 1. Test time outs starting the mini cluster (Master failed to start)
> 2. InterruptedException (because the tests timeout)
> 3. Flaky tests because a subset of the cluster fails to start for whatever
> reason (replication tests especially because they spawn multiple clusters).
> 4. ConnectionLoss to znode /hbase/xyzz.. JVM pause?
> I have strong feeling that this is a possible cause for many of our flaky
> tests in Jenkins. Luckily, it looks like the following workaround to switch
> to an IP address instead of hostname makes it much quicker. There are some
> related discussions in the ZK community (ZOOKEEPER-1666 and related jiras).
> {code:java}
> --- a/hbase-common/src/main/resources/hbase-default.xml
> +++ b/hbase-common/src/main/resources/hbase-default.xml
> @@ -72,7 +72,7 @@ possible configurations would overwhelm and obscure the
> important.
>
>
> hbase.zookeeper.quorum
> -localhost
> +127.0.0.1
> Comma separated list of servers in the ZooKeeper ensemble
> (This config. should have been named hbase.zookeeper.ensemble).
> For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com".
> {code}
> Until we figure out the actual root cause and a dependency upgrade (if
> needed), we should consider making this hostname to IP switch for more stable
> builds.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] bharathv commented on issue #1102: HBASE-23752: Fix remaining test failures from nightly runs
bharathv commented on issue #1102: HBASE-23752: Fix remaining test failures from nightly runs URL: https://github.com/apache/hbase/pull/1102#issuecomment-579882636 > After these changes, are there still more flakes that could use the same treatment? (Looking at the precommit unit results.) @apurtell I think the remaining flakes are general flakes and not bugs. After doing some investigation I think the general cause of flakiness is this : https://issues.apache.org/jira/browse/HBASE-23764 . Making that switch locally makes the tests much faster for me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (HBASE-23764) Flaky tests due to ZK client name resolution delays
Bharath Vissapragada created HBASE-23764: Summary: Flaky tests due to ZK client name resolution delays Key: HBASE-23764 URL: https://issues.apache.org/jira/browse/HBASE-23764 Project: HBase Issue Type: Bug Components: test Affects Versions: 3.0.0 Reporter: Bharath Vissapragada Assignee: Bharath Vissapragada [~ndimiduk] and I ran into this issue (separately) and we noticed that there are some performance issues with name resolution in the Zookeeper client. Since we use ZK heavily in the unit tests, this often manifests as the following issues 1. Test time outs starting the mini cluster (Master failed to start) 2. InterruptedException (because the tests timeout) 3. Flaky tests because a subset of the cluster fails to start for whatever reason (replication tests especially because they spawn multiple clusters). I have strong feeling that this is a possible cause for many of our flaky tests in Jenkins. Luckily, it looks like the following workaround to switch to an IP address instead of hostname makes it much quicker. There are some related discussions in the ZK community (ZOOKEEPER-1666 and related jiras). Until we figure out the actual root cause and a dependency upgrade (if needed), we should consider making this hostname to IP switch for more stable builds. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] apurtell commented on issue #1102: HBASE-23752: Fix remaining test failures from nightly runs
apurtell commented on issue #1102: HBASE-23752: Fix remaining test failures from nightly runs URL: https://github.com/apache/hbase/pull/1102#issuecomment-579879485 After these changes, are there still more flakes that could use the same treatment? (Looking at the precommit unit results.) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] virajjasani closed pull request #1105: HBASE-23749 : TestHFileWriterV3 for all encoders
virajjasani closed pull request #1105: HBASE-23749 : TestHFileWriterV3 for all encoders URL: https://github.com/apache/hbase/pull/1105 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23760) (2.1) Helper method to configure secure DFS cluster for tests
[ https://issues.apache.org/jira/browse/HBASE-23760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026072#comment-17026072 ] Josh Elser commented on HBASE-23760: bq. Can these use "127.0.0.1" instead of "localhost"? I think a big part of our test instability due to hostname resolution. Thanks, Nick. Let me spin out something new to track your ask, rather than change this. > (2.1) Helper method to configure secure DFS cluster for tests > - > > Key: HBASE-23760 > URL: https://issues.apache.org/jira/browse/HBASE-23760 > Project: HBase > Issue Type: Task > Components: security, test >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 2.1.9 > > Attachments: HBASE-23760.001.branch-2.1.patch > > > Let's get [~weichiu]'s HBASE-20950 onto branch-2.1. It will make the backport > for HBASE-17115 that much easier. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] bharathv commented on a change in pull request #1102: HBASE-23752: Fix remaining test failures from nightly runs
bharathv commented on a change in pull request #1102: HBASE-23752: Fix
remaining test failures from nightly runs
URL: https://github.com/apache/hbase/pull/1102#discussion_r372530454
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/security/provider/TestCustomSaslAuthenticationProvider.java
##
@@ -431,9 +431,13 @@ public static void setupCluster() throws Exception {
UTIL.getDataTestDir("keytab").toUri().getPath());
final MiniKdc kdc = UTIL.setupMiniKdc(KEYTAB_FILE);
-// Switch back to NIO for now.
+// Switch to blocking RPC impl.
CONF.set(RpcClientFactory.CUSTOM_RPC_CLIENT_IMPL_CONF_KEY,
BlockingRpcClient.class.getName());
CONF.set(RpcServerFactory.CUSTOM_RPC_SERVER_IMPL_CONF_KEY,
SimpleRpcServer.class.getName());
+// Set the connection registry to ZKConnectionRegistry since hedging is
not supported on
+// blocking rpc clients.
Review comment:
> Why is this test forced onto the blocking rpc client? Can/Should it be
parameterized over both implementations?
Not totally sure why it should only be a blocking rpc client (and a
SimpleRpcServer impl). @joshelser Any idea? I see that you are the author of
this test.
> can we not use the MasterRegistry with hedged rpcs disabled? It appears
this configuration is supported by the MasterRegistry constructor.
We can. But my comment above has nothing to do with whether hedging is
enabled/disabled. HedgingRpcChannel is a non-blocking channel implementation.
Using a non-blocking-channel wrapped over a blocking-rpc-connection defeats the
purpose.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] ramkrish86 commented on issue #1105: HBASE-23749 : TestHFileWriterV3 for all encoders
ramkrish86 commented on issue #1105: HBASE-23749 : TestHFileWriterV3 for all encoders URL: https://github.com/apache/hbase/pull/1105#issuecomment-579869190 +1. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23760) (2.1) Helper method to configure secure DFS cluster for tests
[
https://issues.apache.org/jira/browse/HBASE-23760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026058#comment-17026058
]
Nick Dimiduk commented on HBASE-23760:
--
{noformat}
+ public static void setSSLConfiguration(HBaseTestingUtility utility, Class
clazz)
+ throws Exception {
+Configuration conf = utility.getConfiguration();
+conf.set(DFSConfigKeys.DFS_HTTP_POLICY_KEY,
HttpConfig.Policy.HTTPS_ONLY.name());
+conf.set(DFSConfigKeys.DFS_NAMENODE_HTTPS_ADDRESS_KEY, "localhost:0");
+conf.set(DFSConfigKeys.DFS_DATANODE_HTTPS_ADDRESS_KEY, "localhost:0");
{noformat}
Can these use "127.0.0.1" instead of "localhost"? I think a big part of our
test instability due to hostname resolution.
Skimmed, looks like a nice cleanup/consolidation of concerns. +1
> (2.1) Helper method to configure secure DFS cluster for tests
> -
>
> Key: HBASE-23760
> URL: https://issues.apache.org/jira/browse/HBASE-23760
> Project: HBase
> Issue Type: Task
> Components: security, test
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 2.1.9
>
> Attachments: HBASE-23760.001.branch-2.1.patch
>
>
> Let's get [~weichiu]'s HBASE-20950 onto branch-2.1. It will make the backport
> for HBASE-17115 that much easier.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on a change in pull request #1102: HBASE-23752: Fix remaining test failures from nightly runs
ndimiduk commented on a change in pull request #1102: HBASE-23752: Fix
remaining test failures from nightly runs
URL: https://github.com/apache/hbase/pull/1102#discussion_r372515033
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/security/provider/TestCustomSaslAuthenticationProvider.java
##
@@ -431,9 +431,13 @@ public static void setupCluster() throws Exception {
UTIL.getDataTestDir("keytab").toUri().getPath());
final MiniKdc kdc = UTIL.setupMiniKdc(KEYTAB_FILE);
-// Switch back to NIO for now.
+// Switch to blocking RPC impl.
CONF.set(RpcClientFactory.CUSTOM_RPC_CLIENT_IMPL_CONF_KEY,
BlockingRpcClient.class.getName());
CONF.set(RpcServerFactory.CUSTOM_RPC_SERVER_IMPL_CONF_KEY,
SimpleRpcServer.class.getName());
+// Set the connection registry to ZKConnectionRegistry since hedging is
not supported on
+// blocking rpc clients.
Review comment:
1. Why is this test forced onto the blocking rpc client? Can/Should it be
parameterized over both implementations?
1. can we not use the `MasterRegistry` with hedged rpcs disabled? It appears
this configuration is supported by the `MasterRegistry` constructor.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (HBASE-18388) Fix description on region page, explaining what a region name is made of
[
https://issues.apache.org/jira/browse/HBASE-18388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026025#comment-17026025
]
Mirko Jotic commented on HBASE-18388:
-
[~liyu] if this is still relevant I'd like to submit a patch. Is there
something I need to do before I'm allowed to submit a patch? Executing
submit-patch.py is giving me 403 from Jira API.
> Fix description on region page, explaining what a region name is made of
>
>
> Key: HBASE-18388
> URL: https://issues.apache.org/jira/browse/HBASE-18388
> Project: HBase
> Issue Type: Improvement
> Components: master, regionserver, UI
>Affects Versions: 1.3.1, 2.0.0-alpha-1
>Reporter: Lars George
>Priority: Minor
> Labels: beginner
>
> In the {{RegionListTmpl.jamon}} we have this:
> {code}
> Region names are made of the containing table's name, a comma,
> the start key, a comma, and a randomly generated region id. To
> illustrate,
> the region named
> domains,apache.org,5464829424211263407 is party to the table
> domains, has an id of 5464829424211263407 and the first
> key
> in the region is apache.org.The hbase:meta 'table'
> is an internal
> system table (or a 'catalog' table in db-speak).
> The hbase:meta table keeps a list of all regions in the system. The empty
> key is used to denote
> table start and table end. A region with an empty start key is the first
> region in a table.
> If a region has both an empty start key and an empty end key, it's the
> only region in the
> table. See http://hbase.org;>HBase Home for further
> explication.
> {code}
> This is wrong and worded oddly. What needs to be fixed facts wise is:
> - Region names contain (separated by commas) the full table name (including
> the namespace), the start key, the time the region was created, and finally a
> dot with an MD5 hash of everything before the dot. For example:
> {{test,,1499410125885.1544f69aeaf787755caa11d3567a9621.}}
> - The trailing dot is to distinguish legacy region names (like those used by
> the {{hbase:meta}} table)
> - The MD5 hash is used as the directory name within the HBase storage
> directories
> - The names for the meta table use a Jenkins hash instead, also leaving out
> the trailing dot, for example {{hbase:meta,,1.1588230740}}. The time is
> always set to {{1}}.
> - The start key is printed in safe characters, escaping unprintable characters
> - The link to the HBase home page to explain more is useless and should be
> removed.
> - Also, for region replicas, the replica ID is inserted into the name, like
> so {{replicatable,,1486289678486_0001.3e8b7655299b21b3038ff8d39062467f.}},
> see the {{_0001}} part.
> As for the wording, I would just make this all flow a little better, that "is
> party of" sounds weird to me (IMHO).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23736) Remove deprecated getTimeStampOfLastAppliedOp from MetricsSink
[
https://issues.apache.org/jira/browse/HBASE-23736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jan Hentschel updated HBASE-23736:
--
Hadoop Flags: Incompatible change,Reviewed (was: Incompatible change)
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master.
> Remove deprecated getTimeStampOfLastAppliedOp from MetricsSink
> --
>
> Key: HBASE-23736
> URL: https://issues.apache.org/jira/browse/HBASE-23736
> Project: HBase
> Issue Type: Task
>Affects Versions: 3.0.0
>Reporter: Jan Hentschel
>Assignee: Jan Hentschel
>Priority: Minor
>
> {{MetricsSink}} defines the deprecated method
> {{getTimeStampOfLastAppliedOp}}, which should be removed for 3.0.0.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] HorizonNet merged pull request #1089: HBASE-23736 Removed deprecated getTimeStampOfLastAppliedOp from MetricsSink
HorizonNet merged pull request #1089: HBASE-23736 Removed deprecated getTimeStampOfLastAppliedOp from MetricsSink URL: https://github.com/apache/hbase/pull/1089 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23746) [Flakey Tests] Caused by: org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException: hflush and hsync
[
https://issues.apache.org/jira/browse/HBASE-23746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17025893#comment-17025893
]
Hudson commented on HBASE-23746:
Results for branch master
[build #1612 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1612/]: (/)
*{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1612//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1612//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1612//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> [Flakey Tests] Caused by:
> org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException:
> hflush and hsync
> -
>
> Key: HBASE-23746
> URL: https://issues.apache.org/jira/browse/HBASE-23746
> Project: HBase
> Issue Type: Bug
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: 23746.patch
>
>
> Tests that use the new RegionProcedureStoreTestBase utility fail in branch-2
> nightlies complaining...
> {code}
> Error Message
> cannot get log writer
> Stacktrace
> java.io.IOException: cannot get log writer
> Caused by:
> org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException:
> hflush and hsync
> {code}
> They are using local fs which doesn't have support for above. Only shows when
> you run the tests with hadoop3 profile. Let me do for these tests what was
> done over in HBASE-19289
> It has been going on for a while now, probably since the move to region
> procedure store (the old wal procedure store had turned off the checks for
> sync/hflush).
> {code}
> health checks / yetus jdk8 hadoop3 checks /
> org.apache.hadoop.hbase.procedure2.store.region.TestHFileProcedurePrettyPrinter.test
> health checks / yetus jdk8 hadoop3 checks /
> org.apache.hadoop.hbase.procedure2.store.region.TestRegionProcedureStore.testLoad
> health checks / yetus jdk8 hadoop3 checks /
> org.apache.hadoop.hbase.procedure2.store.region.TestRegionProcedureStore.testCleanup
> health checks / yetus jdk8 hadoop3 checks /
> org.apache.hadoop.hbase.procedure2.store.region.TestRegionProcedureStoreCompaction.test
> health checks / yetus jdk8 hadoop3 checks /
> org.apache.hadoop.hbase.procedure2.store.region.TestRegionProcedureStoreMigration.testMigrateWithUnsupportedProcedures
> health checks / yetus jdk8 hadoop3 checks /
> org.apache.hadoop.hbase.procedure2.store.region.TestRegionProcedureStoreMigration.test
> health checks / yetus jdk8 hadoop3 checks /
> org.apache.hadoop.hbase.procedure2.store.region.TestRegionProcedureStoreWALCleaner.test
> health checks / yetus jdk8 hadoop3 checks /
> org.apache.hadoop.hbase.procedure2.store.region.TestWALProcedurePrettyPrinter.test
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23738) Remove deprecated createLocalHRegion from HBaseTestingUtility
[
https://issues.apache.org/jira/browse/HBASE-23738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17025892#comment-17025892
]
Hudson commented on HBASE-23738:
Results for branch master
[build #1612 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1612/]: (/)
*{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1612//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1612//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1612//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Remove deprecated createLocalHRegion from HBaseTestingUtility
> -
>
> Key: HBASE-23738
> URL: https://issues.apache.org/jira/browse/HBASE-23738
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Jan Hentschel
>Assignee: Jan Hentschel
>Priority: Minor
>
> {{createLocalHRegion}} in {{HBaseTestingUtility}} was deprecated back in
> 2.0.0 and should be removed for 3.0.0.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23753) Update of errorprone generated failures
[
https://issues.apache.org/jira/browse/HBASE-23753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17025894#comment-17025894
]
Hudson commented on HBASE-23753:
Results for branch master
[build #1612 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1612/]: (/)
*{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1612//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1612//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1612//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Update of errorprone generated failures
> ---
>
> Key: HBASE-23753
> URL: https://issues.apache.org/jira/browse/HBASE-23753
> Project: HBase
> Issue Type: Sub-task
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
>
> Parent issue updated hbase-thirdparty which updated errorprone. The nightly
> compile started failing with 'compile'/errorprone complaints. All look good.
> Let me fix.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #1105: HBASE-23749 : TestHFileWriterV3 for all encoders
Apache-HBase commented on issue #1105: HBASE-23749 : TestHFileWriterV3 for all encoders URL: https://github.com/apache/hbase/pull/1105#issuecomment-579751802 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 14s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 7s | master passed | | +1 :green_heart: | compile | 1m 4s | master passed | | +1 :green_heart: | checkstyle | 1m 15s | master passed | | +1 :green_heart: | shadedjars | 5m 21s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | master passed | | +0 :ok: | spotbugs | 5m 4s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 5m 2s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 43s | the patch passed | | +1 :green_heart: | compile | 1m 2s | the patch passed | | +1 :green_heart: | javac | 1m 2s | the patch passed | | +1 :green_heart: | checkstyle | 1m 13s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 5m 7s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 17m 46s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | | +1 :green_heart: | findbugs | 5m 7s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 163m 32s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. | | | | 228m 28s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.quotas.TestClusterScopeQuotaThrottle | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1105/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1105 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux b270afe8afef 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1105/out/precommit/personality/provided.sh | | git revision | master / 98cff8a26a | | Default Java | 1.8.0_181 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1105/2/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1105/2/testReport/ | | Max. process+thread count | 5684 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1105/2/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23748) Include HBASE-21284 to branch-2.2
[ https://issues.apache.org/jira/browse/HBASE-23748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17025841#comment-17025841 ] Josh Elser commented on HBASE-23748: Big +1 > Include HBASE-21284 to branch-2.2 > - > > Key: HBASE-23748 > URL: https://issues.apache.org/jira/browse/HBASE-23748 > Project: HBase > Issue Type: Sub-task >Reporter: Sakthi >Assignee: Sakthi >Priority: Major > Fix For: 2.2.3 > > Attachments: hbase-23748.branch-2.2.001.patch > > > HBASE-21284 was ought to be present in 2.2. But by the time the commit was > done, the branch had been cut already. Hence this Jira to track it's > inclusion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23633) Find a way to handle the corrupt recovered hfiles
[
https://issues.apache.org/jira/browse/HBASE-23633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17025829#comment-17025829
]
Pankaj Kumar commented on HBASE-23633:
--
I also observed this problem during test, many regions *FAILED* to open due to
CorruptHFileException.
{noformat}
2020-01-29 07:07:13,911 | INFO | RS_OPEN_REGION-RS-IP:RS-PORT-2 | Validating
hfile at
hdfs://cluster/hbase/data/default/usertable01/a2f0e8b46399ce55e864d4ee7311c845/family/recovered.hfiles/290-RS-IP%2CRS-PORT%2C1580220469213.1580222580793
for inclusion in store family region
usertable01,user35466,1580220595485.a2f0e8b46399ce55e864d4ee7311c845. |
org.apache.hadoop.hbase.regionserver.HStore.assertBulkLoadHFileOk(HStore.java:730)
2020-01-29 07:07:13,930 | ERROR | RS_OPEN_REGION-RS-IP:RS-PORT-2 | Failed open
of
region=usertable01,user35466,1580220595485.a2f0e8b46399ce55e864d4ee7311c845.,
starting to roll back the global memstore size. |
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:386)
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile
Trailer from file
hdfs://cluster/hbase/data/default/usertable01/a2f0e8b46399ce55e864d4ee7311c845/family/recovered.hfiles/290-RS-IP%2CRS-PORT%2C1580220469213.1580222580793
at org.apache.hadoop.hbase.io.hfile.HFile.openReader(HFile.java:503)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:562)
at
org.apache.hadoop.hbase.regionserver.HStore.assertBulkLoadHFileOk(HStore.java:732)
at
org.apache.hadoop.hbase.regionserver.HRegion.loadRecoveredHFilesIfAny(HRegion.java:4905)
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:863)
at
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:824)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7023)
{noformat}
After digging more into the log, observed this problem occured when
"split-log-closeStream" thread was splitting WAL into hfile and Region Server
abort due to some region. So the "split-log-closeStream" thread was interrupted
and left the recovered hfile in an intermediate state.
{noformat}
2020-01-28 23:01:04,962 | WARN | RS_LOG_REPLAY_OPS-8-5-179-5:RS-PORT-0 | log
splitting of
WALs/RS-IP,RS-PORT,1580220469213-splitting/RS-IP%2CRS-PORT%2C1580220469213.1580222580793
interrupted, resigning |
org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:111)
java.io.InterruptedIOException
at
org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.writeRemainingEntryBuffers(BoundedRecoveredHFilesOutputSink.java:186)
at
org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.close(BoundedRecoveredHFilesOutputSink.java:155)
at
org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:404)
at
org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:225)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:105)
at
org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:72)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ExecutorCompletionService.take(ExecutorCompletionService.java:193)
at
org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.writeRemainingEntryBuffers(BoundedRecoveredHFilesOutputSink.java:179)
... 9 more
{noformat}
Further I checked and confirmed from the NN audit log that file was not written
completelty and RS went down,
{noformat}
2020-01-28 23:01:04,946 | INFO | IPC Server handler 125 on 25000 | BLOCK*
allocate blk_1092127264_18392260, replicas=DN-IP1:DN-PORT, DN-IP2:DN-PORT,
DN-IP3:DN-PORT for
/hbase/data/default/usertable01/a2f0e8b46399ce55e864d4ee7311c845/family/recovered.hfiles/290-RS-IP%2CRS-PORT%2C1580220469213.1580222580793
| FSDirWriteFileOp.java:856
2020-01-29 00:01:04,956 | INFO |
org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@862fb5 | Recovering
[Lease. Holder: DFSClient_NONMAPREDUCE_-1098699935_1, pending creates: 21],
