[jira] [Commented] (HBASE-16148) Hybrid Logical Clocks(placeholder for running tests)
[
https://issues.apache.org/jira/browse/HBASE-16148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400443#comment-15400443
]
Hadoop QA commented on HBASE-16148:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 37s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 36s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s
{color} | {color:green} master passed with JDK v1.7.0_25 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} master passed with JDK v1.7.0_25 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 22s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed with JDK v1.7.0_25 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 4m 5s {color}
| {color:red} hbase-server-jdk1.7.0_25 with JDK v1.7.0_25 generated 2 new + 4
unchanged - 2 fixed = 6 total (was 6) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
32m 10s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed with JDK v1.7.0_25 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 55s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 59s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 46s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m 50s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.balancer.TestRegionLocationFinder |
\\
\\
[jira] [Commented] (HBASE-16296) Reverse scan performance degrades when scanner cache size matches page filter size
[ https://issues.apache.org/jira/browse/HBASE-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400429#comment-15400429 ] ramkrishna.s.vasudevan commented on HBASE-16296: Will this be right for SkipFilter or whileMatchFilter also? I think this step of filterRowKey(current) was added mainly in cases where the batch limit or size limit is reached and again we scan for the next set of cells /rows. But as Lars says if filterAllRemaining also needs to be checked, then both for filterlist and the normal filter case we should do it or for both it should not be. So may be Lars patch makes things consistent and we can try with that? Need to see all the cases here. > Reverse scan performance degrades when scanner cache size matches page filter > size > -- > > Key: HBASE-16296 > URL: https://issues.apache.org/jira/browse/HBASE-16296 > Project: HBase > Issue Type: Bug >Reporter: James Taylor > Attachments: 16296-MAYBE.txt, generatedata-snippet.java, > repro-snippet.java > > > When a reverse scan is done, the server seems to not know it's done when the > scanner cache size matches the number of rows in a PageFilter. See > PHOENIX-3121 for how this manifests itself. We have a standalone, pure HBase > API reproducer too that I'll attach (courtesy of [~churromorales] and > [~mujtabachohan]). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14743) Add metrics around HeapMemoryManager
[ https://issues.apache.org/jira/browse/HBASE-14743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400427#comment-15400427 ] Reid Chan commented on HBASE-14743: --- Thanks for reminding. > Add metrics around HeapMemoryManager > > > Key: HBASE-14743 > URL: https://issues.apache.org/jira/browse/HBASE-14743 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Reid Chan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14743.009.patch, HBASE-14743.009.rw3.patch, > HBASE-14743.009.v2.patch, HBASE-14743.010.patch, HBASE-14743.010.v2.patch, > HBASE-14743.011.patch, Metrics snapshot 2016-6-30.png, Screen Shot 2016-06-16 > at 5.39.13 PM.png, test2_1.png, test2_2.png, test2_3.png, test2_4.png > > > it would be good to know how many invocations there have been. > How many decided to expand memstore. > How many decided to expand block cache. > How many decided to do nothing. > etc. > When that's done use those metrics to clean up the tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14743) Add metrics around HeapMemoryManager
[ https://issues.apache.org/jira/browse/HBASE-14743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-14743: -- Release Note: A Memory metrics for RegionServer. It reveals situations happened in both MemStore and BlockCache. (was: A Memory metrics for RegionServer. It reveals situations happened both in MemStore and BlockCache.) > Add metrics around HeapMemoryManager > > > Key: HBASE-14743 > URL: https://issues.apache.org/jira/browse/HBASE-14743 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Reid Chan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14743.009.patch, HBASE-14743.009.rw3.patch, > HBASE-14743.009.v2.patch, HBASE-14743.010.patch, HBASE-14743.010.v2.patch, > HBASE-14743.011.patch, Metrics snapshot 2016-6-30.png, Screen Shot 2016-06-16 > at 5.39.13 PM.png, test2_1.png, test2_2.png, test2_3.png, test2_4.png > > > it would be good to know how many invocations there have been. > How many decided to expand memstore. > How many decided to expand block cache. > How many decided to do nothing. > etc. > When that's done use those metrics to clean up the tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16256) Purpose of EnvironmentEdge, EnvironmentEdgeManager
[ https://issues.apache.org/jira/browse/HBASE-16256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400426#comment-15400426 ] Hudson commented on HBASE-16256: SUCCESS: Integrated in HBase-1.2-IT #565 (See [https://builds.apache.org/job/HBase-1.2-IT/565/]) HBASE-16256 Purpose of EnvironmentEdge, EnvironmentEdgeManager (Sai Teja (stack: rev 4541d70598b0662da4f5a90822cf6b71f6ac48b6) * hbase-common/src/main/java/org/apache/hadoop/hbase/util/EnvironmentEdgeManager.java > Purpose of EnvironmentEdge, EnvironmentEdgeManager > -- > > Key: HBASE-16256 > URL: https://issues.apache.org/jira/browse/HBASE-16256 > Project: HBase > Issue Type: Improvement > Components: documentation, regionserver >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Trivial > Labels: beginner, clarification > Fix For: 2.0.0, 1.3.0, 1.2.3, 1.1.7 > > Attachments: HBASE-16256.master.1.patch, HBASE-16256.master.2.patch, > HBASE-16256.master.3.patch > > > I am trying to figure out the purpose of EnvironmentEdge in the HBase code. > The java doc says it has something to do with environment, I feel it is > vague. It looks like there is a bigger picture for such a design. Currently > only concrete method that is available in it is currentTime(). > Can anyone summarize the rationale behind having > EnvironmentEdge/EnvironmentEdgeManager ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14743) Add metrics around HeapMemoryManager
[ https://issues.apache.org/jira/browse/HBASE-14743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-14743: -- Release Note: A Memory metrics for RegionServer. It reveals the situations happened both in MemStore and BlockCache. > Add metrics around HeapMemoryManager > > > Key: HBASE-14743 > URL: https://issues.apache.org/jira/browse/HBASE-14743 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Reid Chan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14743.009.patch, HBASE-14743.009.rw3.patch, > HBASE-14743.009.v2.patch, HBASE-14743.010.patch, HBASE-14743.010.v2.patch, > HBASE-14743.011.patch, Metrics snapshot 2016-6-30.png, Screen Shot 2016-06-16 > at 5.39.13 PM.png, test2_1.png, test2_2.png, test2_3.png, test2_4.png > > > it would be good to know how many invocations there have been. > How many decided to expand memstore. > How many decided to expand block cache. > How many decided to do nothing. > etc. > When that's done use those metrics to clean up the tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14743) Add metrics around HeapMemoryManager
[ https://issues.apache.org/jira/browse/HBASE-14743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-14743: -- Release Note: A Memory metrics for RegionServer. It reveals situations happened both in MemStore and BlockCache. (was: A Memory metrics for RegionServer. It reveals the situations happened both in MemStore and BlockCache.) > Add metrics around HeapMemoryManager > > > Key: HBASE-14743 > URL: https://issues.apache.org/jira/browse/HBASE-14743 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Reid Chan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14743.009.patch, HBASE-14743.009.rw3.patch, > HBASE-14743.009.v2.patch, HBASE-14743.010.patch, HBASE-14743.010.v2.patch, > HBASE-14743.011.patch, Metrics snapshot 2016-6-30.png, Screen Shot 2016-06-16 > at 5.39.13 PM.png, test2_1.png, test2_2.png, test2_3.png, test2_4.png > > > it would be good to know how many invocations there have been. > How many decided to expand memstore. > How many decided to expand block cache. > How many decided to do nothing. > etc. > When that's done use those metrics to clean up the tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16284) Unauthorized client can shutdown the cluster
[
https://issues.apache.org/jira/browse/HBASE-16284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400417#comment-15400417
]
Jerry He commented on HBASE-16284:
--
Another comment.
Can we just replace the existing testShutdown() and testStopMaster() in
TestAccessController with your versions, including verifyAllowed() and
verifyDenied()?
Thanks.
> Unauthorized client can shutdown the cluster
>
>
> Key: HBASE-16284
> URL: https://issues.apache.org/jira/browse/HBASE-16284
> Project: HBase
> Issue Type: Bug
>Reporter: Deokwoo Han
> Attachments: HBASE-16284-v2.patch, HBASE-16284.patch
>
>
> An unauthorized client can shutdown the cluster as {{AccessDeniedException}}
> is ignored during {{Admin.stopMaster}} and {{Admin.shutdown}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15756) Pluggable RpcServer
[ https://issues.apache.org/jira/browse/HBASE-15756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400416#comment-15400416 ] binlijin commented on HBASE-15756: -- you can change the hbase.client.ipc.pool.size to see how the result is different. > Pluggable RpcServer > --- > > Key: HBASE-15756 > URL: https://issues.apache.org/jira/browse/HBASE-15756 > Project: HBase > Issue Type: Improvement > Components: Performance, rpc >Reporter: binlijin >Assignee: binlijin >Priority: Critical > Attachments: NettyRpcServer.patch, NettyRpcServer_forperf.patch, > gets.png, idle.png, queue.png > > > Current we use a simple RpcServer, and can not configure and use other > implementation.This issue is to make the RpcServer pluggable, so we can make > other implementation for example netty rpc server. Patch will upload laterly -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14345) Consolidate printUsage in IntegrationTestLoadAndVerify
[
https://issues.apache.org/jira/browse/HBASE-14345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400415#comment-15400415
]
Reid Chan commented on HBASE-14345:
---
Yes, change it to "Options"
I commented out the "System.exit(EXIT_FAILURE);" and attached a new screenshot.
As it shows, it exits anyway and the exception trace stack will be printed
which i thought it was ugly and useless to users. That's why
"System.exit(EXIT_FAILURE);" was explicitly added in patch.
Just want to make sure if "System.exit(EXIT_FAILURE);" should be discarded. And
i will re-attach a new patch fixed as suggestions, after your confirmation.
Thanks
> Consolidate printUsage in IntegrationTestLoadAndVerify
> --
>
> Key: HBASE-14345
> URL: https://issues.apache.org/jira/browse/HBASE-14345
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Reporter: Nick Dimiduk
>Assignee: Reid Chan
>Priority: Trivial
> Labels: beginner
> Attachments: 2016-07-30-Without System_exit.png,
> HBASE-14345.002.patch, HBASE-14345.patch, itlav-2016-07-07.png, itlv.png
>
>
> Investigating the use of {{itlav}} is a little screwy. Subclasses are not
> overriding the {{printUsage()}} methods correctly, so you have to pass
> {{--help}} to get some info and no arguments to get the rest.
> {noformat}
> [hbase@ndimiduk-112rc2-7 ~]$ hbase
> org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify --help
> usage: bin/hbase org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify
>
> Options:
> -h,--help Show usage
> -m,--monkey Which chaos monkey to run
> -monkeyProps The properties file for specifying chaos monkey
> properties.
> -ncc,--noClusterCleanUp Don't clean up the cluster at the end
> [hbase@ndimiduk-112rc2-7 ~]$ hbase
> org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify
> IntegrationTestLoadAndVerify [-Doptions]
> Loads a table with row dependencies and verifies the dependency chains
> Options
> -Dloadmapper.table=Table to write/verify (default autogen)
> -Dloadmapper.backrefs=Number of backreferences per row (default
> 50)
> -Dloadmapper.num_to_write=Number of rows per mapper (default 100,000
> per mapper)
> -Dloadmapper.deleteAfter= Delete after a successful verify (default
> true)
> -Dloadmapper.numPresplits=Number of presplit regions to start with
> (default 40)
> -Dloadmapper.map.tasks= Number of map tasks for load (default 200)
> -Dverify.reduce.tasks=Number of reduce tasks for verify (default
> 35)
> -Dverify.scannercaching= Number hbase scanner caching rows to read
> (default 50)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16148) Hybrid Logical Clocks(placeholder for running tests)
[ https://issues.apache.org/jira/browse/HBASE-16148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Teja Ranuva updated HBASE-16148: Status: Patch Available (was: Open) > Hybrid Logical Clocks(placeholder for running tests) > > > Key: HBASE-16148 > URL: https://issues.apache.org/jira/browse/HBASE-16148 > Project: HBase > Issue Type: Sub-task > Components: API >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Minor > Labels: test-patch > Attachments: HBASE-16148.master.001.patch, > HBASE-16148.master.6.patch, HBASE-16148.master.test.1.patch, > HBASE-16148.master.test.2.patch, HBASE-16148.master.test.3.patch, > HBASE-16148.master.test.4.patch, HBASE-16148.master.test.5.patch > > > This JIRA is just a placeholder to test Hybrid Logical Clocks code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16148) Hybrid Logical Clocks(placeholder for running tests)
[ https://issues.apache.org/jira/browse/HBASE-16148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Teja Ranuva updated HBASE-16148: Status: Open (was: Patch Available) > Hybrid Logical Clocks(placeholder for running tests) > > > Key: HBASE-16148 > URL: https://issues.apache.org/jira/browse/HBASE-16148 > Project: HBase > Issue Type: Sub-task > Components: API >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Minor > Labels: test-patch > Attachments: HBASE-16148.master.001.patch, > HBASE-16148.master.6.patch, HBASE-16148.master.test.1.patch, > HBASE-16148.master.test.2.patch, HBASE-16148.master.test.3.patch, > HBASE-16148.master.test.4.patch, HBASE-16148.master.test.5.patch > > > This JIRA is just a placeholder to test Hybrid Logical Clocks code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16148) Hybrid Logical Clocks(placeholder for running tests)
[ https://issues.apache.org/jira/browse/HBASE-16148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Teja Ranuva updated HBASE-16148: Attachment: HBASE-16148.master.6.patch HLC patch, testing. > Hybrid Logical Clocks(placeholder for running tests) > > > Key: HBASE-16148 > URL: https://issues.apache.org/jira/browse/HBASE-16148 > Project: HBase > Issue Type: Sub-task > Components: API >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Minor > Labels: test-patch > Attachments: HBASE-16148.master.001.patch, > HBASE-16148.master.6.patch, HBASE-16148.master.test.1.patch, > HBASE-16148.master.test.2.patch, HBASE-16148.master.test.3.patch, > HBASE-16148.master.test.4.patch, HBASE-16148.master.test.5.patch > > > This JIRA is just a placeholder to test Hybrid Logical Clocks code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-15756) Pluggable RpcServer
[ https://issues.apache.org/jira/browse/HBASE-15756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400410#comment-15400410 ] binlijin edited comment on HBASE-15756 at 7/30/16 3:18 AM: --- The more client connections, the result will be more different. So i think i know you benchmark test. was (Author: aoxiang): The more client connections, the result will be more different. > Pluggable RpcServer > --- > > Key: HBASE-15756 > URL: https://issues.apache.org/jira/browse/HBASE-15756 > Project: HBase > Issue Type: Improvement > Components: Performance, rpc >Reporter: binlijin >Assignee: binlijin >Priority: Critical > Attachments: NettyRpcServer.patch, NettyRpcServer_forperf.patch, > gets.png, idle.png, queue.png > > > Current we use a simple RpcServer, and can not configure and use other > implementation.This issue is to make the RpcServer pluggable, so we can make > other implementation for example netty rpc server. Patch will upload laterly -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16209) Provide an ExponentialBackOffPolicy sleep between failed region open requests
[ https://issues.apache.org/jira/browse/HBASE-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400407#comment-15400407 ] Hudson commented on HBASE-16209: FAILURE: Integrated in HBase-Trunk_matrix #1320 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1320/]) HBASE-16209 Removed unnecessary method and method calls from (matteo.bertozzi: rev 3729320099eaaab8ef19c9772a24ae5ba868beea) * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java * hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/AssignmentManagerStatusTmpl.jamon * hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java > Provide an ExponentialBackOffPolicy sleep between failed region open requests > - > > Key: HBASE-16209 > URL: https://issues.apache.org/jira/browse/HBASE-16209 > Project: HBase > Issue Type: Bug >Reporter: Joseph >Assignee: Joseph > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16209-branch-1-addendum.patch, > HBASE-16209-branch-1.patch, HBASE-16209-v2.patch, HBASE-16209.patch > > > Related to HBASE-16138. As of now we currently have no pause between retrying > failed region open requests. And with a low maximumAttempt default, we can > quickly use up all our regionOpen retries if the server is in a bad state. I > added in a ExponentialBackOffPolicy so that we spread out the timing of our > open region retries in AssignmentManager. Review board at > https://reviews.apache.org/r/50011/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16181) Backup of hbase:backup table
[ https://issues.apache.org/jira/browse/HBASE-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400408#comment-15400408 ] Hudson commented on HBASE-16181: FAILURE: Integrated in HBase-Trunk_matrix #1320 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1320/]) HBASE-16181 Fix AssignmentManager MBean name (Reid Chan) (stack: rev 69d170063f9136c78507d1d46ce405a6102a40a4) * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsAssignmentManagerSource.java > Backup of hbase:backup table > > > Key: HBASE-16181 > URL: https://issues.apache.org/jira/browse/HBASE-16181 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16181-v1.patch, HBASE-16181-v2.patch > > > Snapshot of HBase system tables is not supported, we need either move > hbase:backup into different name space or fix snapshots. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15756) Pluggable RpcServer
[ https://issues.apache.org/jira/browse/HBASE-15756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400410#comment-15400410 ] binlijin commented on HBASE-15756: -- The more client connections, the result will be more different. > Pluggable RpcServer > --- > > Key: HBASE-15756 > URL: https://issues.apache.org/jira/browse/HBASE-15756 > Project: HBase > Issue Type: Improvement > Components: Performance, rpc >Reporter: binlijin >Assignee: binlijin >Priority: Critical > Attachments: NettyRpcServer.patch, NettyRpcServer_forperf.patch, > gets.png, idle.png, queue.png > > > Current we use a simple RpcServer, and can not configure and use other > implementation.This issue is to make the RpcServer pluggable, so we can make > other implementation for example netty rpc server. Patch will upload laterly -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15656) Fix unused protobuf warning in Admin.proto
[
https://issues.apache.org/jira/browse/HBASE-15656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400409#comment-15400409
]
Hudson commented on HBASE-15656:
FAILURE: Integrated in HBase-Trunk_matrix #1320 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1320/])
HBASE-15656 Fix unused protobuf warning in Admin.proto (Yi Liang) (stack: rev
26c042668902d297011109d4b382ebfa7007399c)
* hbase-protocol/src/main/protobuf/HBase.proto
*
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java
* hbase-protocol/src/main/protobuf/WAL.proto
* hbase-protocol/src/main/protobuf/SecureBulkLoad.proto
*
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/WALProtos.java
* hbase-protocol/src/main/protobuf/Admin.proto
*
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java
*
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/SecureBulkLoadProtos.java
> Fix unused protobuf warning in Admin.proto
> --
>
> Key: HBASE-15656
> URL: https://issues.apache.org/jira/browse/HBASE-15656
> Project: HBase
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: Elliott Clark
>Assignee: Yi Liang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-15656-V1.patch
>
>
> {code}
> Warning: Unused import: "Admin.proto" imports "Client.proto" which is not
> used.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16256) Purpose of EnvironmentEdge, EnvironmentEdgeManager
[ https://issues.apache.org/jira/browse/HBASE-16256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400406#comment-15400406 ] Hudson commented on HBASE-16256: FAILURE: Integrated in HBase-Trunk_matrix #1320 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1320/]) HBASE-16256 Purpose of EnvironmentEdge, EnvironmentEdgeManager (Sai Teja (stack: rev c29024c01780d055c38e5fe3397d93ace5466df5) * hbase-common/src/main/java/org/apache/hadoop/hbase/util/EnvironmentEdgeManager.java > Purpose of EnvironmentEdge, EnvironmentEdgeManager > -- > > Key: HBASE-16256 > URL: https://issues.apache.org/jira/browse/HBASE-16256 > Project: HBase > Issue Type: Improvement > Components: documentation, regionserver >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Trivial > Labels: beginner, clarification > Fix For: 2.0.0, 1.3.0, 1.2.3, 1.1.7 > > Attachments: HBASE-16256.master.1.patch, HBASE-16256.master.2.patch, > HBASE-16256.master.3.patch > > > I am trying to figure out the purpose of EnvironmentEdge in the HBase code. > The java doc says it has something to do with environment, I feel it is > vague. It looks like there is a bigger picture for such a design. Currently > only concrete method that is available in it is currentTime(). > Can anyone summarize the rationale behind having > EnvironmentEdge/EnvironmentEdgeManager ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16256) Purpose of EnvironmentEdge, EnvironmentEdgeManager
[ https://issues.apache.org/jira/browse/HBASE-16256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400405#comment-15400405 ] Hudson commented on HBASE-16256: FAILURE: Integrated in HBase-1.3-IT #769 (See [https://builds.apache.org/job/HBase-1.3-IT/769/]) HBASE-16256 Purpose of EnvironmentEdge, EnvironmentEdgeManager (Sai Teja (stack: rev 7b22c76f7df3e46c3b0a77e8d670e1ec4890adc6) * hbase-common/src/main/java/org/apache/hadoop/hbase/util/EnvironmentEdgeManager.java > Purpose of EnvironmentEdge, EnvironmentEdgeManager > -- > > Key: HBASE-16256 > URL: https://issues.apache.org/jira/browse/HBASE-16256 > Project: HBase > Issue Type: Improvement > Components: documentation, regionserver >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Trivial > Labels: beginner, clarification > Fix For: 2.0.0, 1.3.0, 1.2.3, 1.1.7 > > Attachments: HBASE-16256.master.1.patch, HBASE-16256.master.2.patch, > HBASE-16256.master.3.patch > > > I am trying to figure out the purpose of EnvironmentEdge in the HBase code. > The java doc says it has something to do with environment, I feel it is > vague. It looks like there is a bigger picture for such a design. Currently > only concrete method that is available in it is currentTime(). > Can anyone summarize the rationale behind having > EnvironmentEdge/EnvironmentEdgeManager ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14345) Consolidate printUsage in IntegrationTestLoadAndVerify
[
https://issues.apache.org/jira/browse/HBASE-14345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reid Chan updated HBASE-14345:
--
Attachment: 2016-07-30-Without System_exit.png
> Consolidate printUsage in IntegrationTestLoadAndVerify
> --
>
> Key: HBASE-14345
> URL: https://issues.apache.org/jira/browse/HBASE-14345
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Reporter: Nick Dimiduk
>Assignee: Reid Chan
>Priority: Trivial
> Labels: beginner
> Attachments: 2016-07-30-Without System_exit.png,
> HBASE-14345.002.patch, HBASE-14345.patch, itlav-2016-07-07.png, itlv.png
>
>
> Investigating the use of {{itlav}} is a little screwy. Subclasses are not
> overriding the {{printUsage()}} methods correctly, so you have to pass
> {{--help}} to get some info and no arguments to get the rest.
> {noformat}
> [hbase@ndimiduk-112rc2-7 ~]$ hbase
> org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify --help
> usage: bin/hbase org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify
>
> Options:
> -h,--help Show usage
> -m,--monkey Which chaos monkey to run
> -monkeyProps The properties file for specifying chaos monkey
> properties.
> -ncc,--noClusterCleanUp Don't clean up the cluster at the end
> [hbase@ndimiduk-112rc2-7 ~]$ hbase
> org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify
> IntegrationTestLoadAndVerify [-Doptions]
> Loads a table with row dependencies and verifies the dependency chains
> Options
> -Dloadmapper.table=Table to write/verify (default autogen)
> -Dloadmapper.backrefs=Number of backreferences per row (default
> 50)
> -Dloadmapper.num_to_write=Number of rows per mapper (default 100,000
> per mapper)
> -Dloadmapper.deleteAfter= Delete after a successful verify (default
> true)
> -Dloadmapper.numPresplits=Number of presplit regions to start with
> (default 40)
> -Dloadmapper.map.tasks= Number of map tasks for load (default 200)
> -Dverify.reduce.tasks=Number of reduce tasks for verify (default
> 35)
> -Dverify.scannercaching= Number hbase scanner caching rows to read
> (default 50)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15756) Pluggable RpcServer
[ https://issues.apache.org/jira/browse/HBASE-15756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400400#comment-15400400 ] binlijin commented on HBASE-15756: -- The result is much different with my's result, i am curious how you do the test, my dear sir. > Pluggable RpcServer > --- > > Key: HBASE-15756 > URL: https://issues.apache.org/jira/browse/HBASE-15756 > Project: HBase > Issue Type: Improvement > Components: Performance, rpc >Reporter: binlijin >Assignee: binlijin >Priority: Critical > Attachments: NettyRpcServer.patch, NettyRpcServer_forperf.patch, > gets.png, idle.png, queue.png > > > Current we use a simple RpcServer, and can not configure and use other > implementation.This issue is to make the RpcServer pluggable, so we can make > other implementation for example netty rpc server. Patch will upload laterly -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15756) Pluggable RpcServer
[ https://issues.apache.org/jira/browse/HBASE-15756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400398#comment-15400398 ] binlijin commented on HBASE-15756: -- Netty's NioWork read request from channel to OffHeap, current we copy to heap to decode into the Call. HBaseProtocolEncoder write a CompositeChannelBuffer result which a wrap with BufferChain's buffers. Netty's NioWork write result to channel via ((GatheringByteChannel) ch).write(buffers); So i think there is no further copy. This is the NettyRpcServer_forperf.patch with hbase branch-1. If there is wrong, please correct me. So i think Anoop Sam John's concern can be resolved. > Pluggable RpcServer > --- > > Key: HBASE-15756 > URL: https://issues.apache.org/jira/browse/HBASE-15756 > Project: HBase > Issue Type: Improvement > Components: Performance, rpc >Reporter: binlijin >Assignee: binlijin >Priority: Critical > Attachments: NettyRpcServer.patch, NettyRpcServer_forperf.patch, > gets.png, idle.png, queue.png > > > Current we use a simple RpcServer, and can not configure and use other > implementation.This issue is to make the RpcServer pluggable, so we can make > other implementation for example netty rpc server. Patch will upload laterly -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16301) Trigger flush without waiting when compaction is disabled on a table
[
https://issues.apache.org/jira/browse/HBASE-16301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400389#comment-15400389
]
huaxiang sun commented on HBASE-16301:
--
Thanks [~enis] for review. I thought about moving it inside
isTooManyStoreFiles() when I first worked on the patch. But at that time, I
thought that isTooManyStoreFiles() really means too many store files so aborted
that idea. Looking at flushOneForGlobalPressure() again, it seems that it
should be moved to isTooManyStoreFiles() to make the logic right as it really
means flushable.
One minor improvement here is that for bestFlushableRegion and bestAnyRegion,
they can be resolved in one loop and return a pair instead of two separate
loops.
https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java#L153
{code}
Region bestFlushableRegion = getBiggestMemstoreRegion(regionsBySize,
excludedRegions, true);
// Find the biggest region, total, even if it might have too many flushes.
Region bestAnyRegion = getBiggestMemstoreRegion(
regionsBySize, excludedRegions, false);
{code}
> Trigger flush without waiting when compaction is disabled on a table
>
>
> Key: HBASE-16301
> URL: https://issues.apache.org/jira/browse/HBASE-16301
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Attachments: HBASE-16301-v001.patch
>
>
> When compaction is disabled on a table, flush needs to wait
> MemStoreFlusher#blockingWaitTime (default value is 90 seconds) before it goes
> ahead to flush. It has side effect that client may be blocked due to
> RegionTooBusyException. Please see the mail sent to dev list.
> http://mail-archives.apache.org/mod_mbox/hbase-dev/201607.mbox/%3c2d66b8ca-7c6f-40ea-a861-2de5482ec...@cloudera.com%3E
> I guess that the right behavior is to do flush without waiting if compaction
> is disabled on a table. Attached a patch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15756) Pluggable RpcServer
[ https://issues.apache.org/jira/browse/HBASE-15756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400386#comment-15400386 ] binlijin commented on HBASE-15756: -- First let me try to explain Netty's Thread Model first: Netty's NioWork read request from channel, decode into Call and handoff it to HBase Handlers via Queue, when HBase Handlers done all the work, it will handoff the result to the channel's WriteBufferQueue, then Netty's NioWork batch write the result to the client. So i think the big different Between the NettyRpcServer with the current RpcServer is the write response. @Jurriaan Mous, if there is wrong, please correct me. > Pluggable RpcServer > --- > > Key: HBASE-15756 > URL: https://issues.apache.org/jira/browse/HBASE-15756 > Project: HBase > Issue Type: Improvement > Components: Performance, rpc >Reporter: binlijin >Assignee: binlijin >Priority: Critical > Attachments: NettyRpcServer.patch, NettyRpcServer_forperf.patch, > gets.png, idle.png, queue.png > > > Current we use a simple RpcServer, and can not configure and use other > implementation.This issue is to make the RpcServer pluggable, so we can make > other implementation for example netty rpc server. Patch will upload laterly -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16256) Purpose of EnvironmentEdge, EnvironmentEdgeManager
[ https://issues.apache.org/jira/browse/HBASE-16256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400380#comment-15400380 ] Hudson commented on HBASE-16256: FAILURE: Integrated in HBase-1.3 #799 (See [https://builds.apache.org/job/HBase-1.3/799/]) HBASE-16256 Purpose of EnvironmentEdge, EnvironmentEdgeManager (Sai Teja (stack: rev 7b22c76f7df3e46c3b0a77e8d670e1ec4890adc6) * hbase-common/src/main/java/org/apache/hadoop/hbase/util/EnvironmentEdgeManager.java > Purpose of EnvironmentEdge, EnvironmentEdgeManager > -- > > Key: HBASE-16256 > URL: https://issues.apache.org/jira/browse/HBASE-16256 > Project: HBase > Issue Type: Improvement > Components: documentation, regionserver >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Trivial > Labels: beginner, clarification > Fix For: 2.0.0, 1.3.0, 1.2.3, 1.1.7 > > Attachments: HBASE-16256.master.1.patch, HBASE-16256.master.2.patch, > HBASE-16256.master.3.patch > > > I am trying to figure out the purpose of EnvironmentEdge in the HBase code. > The java doc says it has something to do with environment, I feel it is > vague. It looks like there is a bigger picture for such a design. Currently > only concrete method that is available in it is currentTime(). > Can anyone summarize the rationale behind having > EnvironmentEdge/EnvironmentEdgeManager ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16213) A new HFileBlock structure for fast random get
[
https://issues.apache.org/jira/browse/HBASE-16213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400366#comment-15400366
]
binlijin commented on HBASE-16213:
--
alter 'table_put_10B_100w', {NAME => 'cf', DATA_BLOCK_ENCODING => 'NONE',
BLOCKSIZE => '65536'}
major_compact 'table_put_10B_100w'
We can alter the table with different BLOCKSIZE also.
> A new HFileBlock structure for fast random get
> --
>
> Key: HBASE-16213
> URL: https://issues.apache.org/jira/browse/HBASE-16213
> Project: HBase
> Issue Type: New Feature
> Components: Performance
>Reporter: binlijin
>Assignee: binlijin
> Attachments: HBASE-16213-master_v1.patch, HBASE-16213.patch,
> HBASE-16213_branch1_v3.patch, HBASE-16213_v2.patch, hfile-cpu.png,
> new-hfile-block.xlsx
>
>
> HFileBlock store cells sequential, current when to get a row from the block,
> it scan from the first cell until the row's cell.
> The new structure store every row's start offset with data, so it can find
> the exact row with binarySearch.
> I use EncodedSeekPerformanceTest test the performance.
> First use ycsb write 100w data, every row have only one qualifier, and
> valueLength=16B/64/256B/1k.
> Then use EncodedSeekPerformanceTest to test random read 1w or 100w row, and
> also record HFileBlock's dataSize/dataWithMetaSize in the encoding.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16213) A new HFileBlock structure for fast random get
[
https://issues.apache.org/jira/browse/HBASE-16213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400363#comment-15400363
]
binlijin commented on HBASE-16213:
--
When i do the test
（1）create "table_put_10B_100w", "cf"
（2) use ycsb write to table "table_put_10B_100w" 100w rows, and value=16B,
BLOCKSIZE => '65536'
（3）flush table 'table_put_10B_100w'
(4) alter 'table_put_10B_100w',{NAME => 'cf',DATA_BLOCK_ENCODING => 'NONE'}
major_compact 'table_put_10B_100w'
we can do the test with DATA_BLOCK_ENCODING => 'NONE'
(5) alter 'table_put_10B_100w',{NAME => 'cf',DATA_BLOCK_ENCODING =>
'ROW_INDEX_V1'} major_compact 'table_put_10B_100w'
we can do the test with DATA_BLOCK_ENCODING => ‘ROW_INDEX_V1'
> A new HFileBlock structure for fast random get
> --
>
> Key: HBASE-16213
> URL: https://issues.apache.org/jira/browse/HBASE-16213
> Project: HBase
> Issue Type: New Feature
> Components: Performance
>Reporter: binlijin
>Assignee: binlijin
> Attachments: HBASE-16213-master_v1.patch, HBASE-16213.patch,
> HBASE-16213_branch1_v3.patch, HBASE-16213_v2.patch, hfile-cpu.png,
> new-hfile-block.xlsx
>
>
> HFileBlock store cells sequential, current when to get a row from the block,
> it scan from the first cell until the row's cell.
> The new structure store every row's start offset with data, so it can find
> the exact row with binarySearch.
> I use EncodedSeekPerformanceTest test the performance.
> First use ycsb write 100w data, every row have only one qualifier, and
> valueLength=16B/64/256B/1k.
> Then use EncodedSeekPerformanceTest to test random read 1w or 100w row, and
> also record HFileBlock's dataSize/dataWithMetaSize in the encoding.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16285) Drop RPC requests if it must be considered as timeout at client
[ https://issues.apache.org/jira/browse/HBASE-16285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400320#comment-15400320 ] Sai Teja Ranuva edited comment on HBASE-16285 at 7/30/16 1:06 AM: -- It would be great if the rationale behind choosing 200ms or 50ms for DEAFAULT_DROP_TIMEOUT_REQUEST_DELAY is put in the comments for future reference. was (Author: saitejar): It would be great if the rationale behind choosing 200ms(or 50ms as suggested by [~stack]) for DEAFAULT_DROP_TIMEOUT_REQUEST_DELAY is put in the comments for future reference. > Drop RPC requests if it must be considered as timeout at client > --- > > Key: HBASE-16285 > URL: https://issues.apache.org/jira/browse/HBASE-16285 > Project: HBase > Issue Type: Improvement >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-16285-branch-1-v1.patch, HBASE-16285-v1.patch > > > After HBASE-15593, we have a timeout param in header of RPC requests. We can > use it in more scenes. > A straightforward scene is to drop requests if it has waited so long in RPC > queue and has been dropped by client. Even if we handle this request and send > the response back, it will not be used any more. And client may have sent a > retry. In an extreme case, if the server is slow, all requests may be timeout > or queue-full-exception because we should handle previous requests which have > been dropped by client and many resources at server are wasted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-9899) for idempotent operation dups, return the result instead of throwing conflict exception
[ https://issues.apache.org/jira/browse/HBASE-9899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400331#comment-15400331 ] Guanghao Zhang edited comment on HBASE-9899 at 7/30/16 12:58 AM: - There are some situation to use these non-idempotent operations (increment/append/checkAndPut/...). When use 0.94, we set not retry for these non-idempotent operations. Now we upgrade our cluster to 0.98 and found that it use nonce to solve this. But it maybe throw OperationConflictException even the increment/append success. A example (client rpc retries number set to 3) is: 1. first increment rpc request success 2. client timeout and send second rpc request, but nonce is same and save in server. The server found that it has already succeed, so return a OperationConflictException to make sure that increment operation only be applied once in server. This patch will solve this problem by read the previous result when receive a duplicate rpc request. 1. Store the mvcc to OperationContext. When first rpc request succeed, store the mvcc for this operation nonce. 2. When there are duplicate rpc request, convert to read result by the mvcc. was (Author: zghaobac): There are some situation to use these non-idempotent operations (increment/append/checkAndPut/...). When use 0.94, we set not retry for these non-idempotent operations. Now we upgrade our cluster to 0.98 and found that it use nonce to solve this. But it maybe throw OperationConflictException even the increment/append success. A example (client rpc retries number set to 3) is: 1. first increment rpc request success 2. client timeout and send second rpc request success, but nonce is same and save in server. It found it succeed, so return a OperationConflictException to make sure that increment operation only be applied once in server. This patch will solve this problem by read the previous result when receive a duplicate rpc request. 1. Store the mvcc to OperationContext. When first rpc request succeed, store the mvcc for this operation nonce. 2. When there are duplicate rpc request, convert to read result by the mvcc. > for idempotent operation dups, return the result instead of throwing conflict > exception > --- > > Key: HBASE-9899 > URL: https://issues.apache.org/jira/browse/HBASE-9899 > Project: HBase > Issue Type: Improvement >Reporter: Sergey Shelukhin >Assignee: Enis Soztutar > Attachments: HBASE-9899-v1.patch > > > After HBASE-3787, we could store mvcc in operation context, and use it to > convert the modification request into read on dups instead of throwing > OperationConflictException. > MVCC tracking will have to be aware of such MVCC numbers present. Given that > scanners are usually relatively short-lived, that would prevent low watermark > from advancing for quite a bit more time -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-9899) for idempotent operation dups, return the result instead of throwing conflict exception
[ https://issues.apache.org/jira/browse/HBASE-9899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400331#comment-15400331 ] Guanghao Zhang commented on HBASE-9899: --- There are some situation to use these non-idempotent operations (increment/append/checkAndPut/...). When use 0.94, we set not retry for these non-idempotent operations. Now we upgrade our cluster to 0.98 and found that it use nonce to solve this. But it maybe throw OperationConflictException even the increment/append success. A example (client rpc retries number set to 3) is: 1. first increment rpc request success 2. client timeout and send second rpc request success, but nonce is same and save in server. It found it succeed, so return a OperationConflictException to make sure that increment operation only be applied once in server. This patch will solve this problem by read the previous result when receive a duplicate rpc request. 1. Store the mvcc to OperationContext. When first rpc request succeed, store the mvcc for this operation nonce. 2. When there are duplicate rpc request, convert to read result by the mvcc. > for idempotent operation dups, return the result instead of throwing conflict > exception > --- > > Key: HBASE-9899 > URL: https://issues.apache.org/jira/browse/HBASE-9899 > Project: HBase > Issue Type: Improvement >Reporter: Sergey Shelukhin >Assignee: Enis Soztutar > Attachments: HBASE-9899-v1.patch > > > After HBASE-3787, we could store mvcc in operation context, and use it to > convert the modification request into read on dups instead of throwing > OperationConflictException. > MVCC tracking will have to be aware of such MVCC numbers present. Given that > scanners are usually relatively short-lived, that would prevent low watermark > from advancing for quite a bit more time -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16288) HFile intermediate block level indexes might recurse forever creating multi TB files
[
https://issues.apache.org/jira/browse/HBASE-16288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400321#comment-15400321
]
Heng Chen commented on HBASE-16288:
---
{quote}
Now we have a min number of entries in a given index block, defaults to 16, as
well as the max size. Max size is ignored while we have less than desired
number of entries.
{quote}
It make sense. patch v4 lgtm. +1
> HFile intermediate block level indexes might recurse forever creating multi
> TB files
>
>
> Key: HBASE-16288
> URL: https://issues.apache.org/jira/browse/HBASE-16288
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.6, 1.2.3
>
> Attachments: hbase-16288_v1.patch, hbase-16288_v2.patch,
> hbase-16288_v3.patch, hbase-16288_v4.patch
>
>
> Mighty [~elserj] was debugging an opentsdb cluster where some region
> directory ended up having 5TB+ files under /.tmp/
> Further debugging and analysis, we were able to reproduce the problem locally
> where we never we recursing in this code path for writing intermediate level
> indices:
> {code:title=HFileBlockIndex.java}
> if (curInlineChunk != null) {
> while (rootChunk.getRootSize() > maxChunkSize) {
> rootChunk = writeIntermediateLevel(out, rootChunk);
> numLevels += 1;
> }
> }
> {code}
> The problem happens if we end up with a very large rowKey (larger than
> "hfile.index.block.max.size" being the first key in the block, then moving
> all the way to the root-level index building. We will keep writing and
> building the next level of intermediate level indices with a single
> very-large key. This can happen in flush / compaction / region recovery
> causing cluster inoperability due to ever-growing files.
> Seems the issue was also reported earlier, with a temporary workaround:
> https://github.com/OpenTSDB/opentsdb/issues/490
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16285) Drop RPC requests if it must be considered as timeout at client
[ https://issues.apache.org/jira/browse/HBASE-16285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400320#comment-15400320 ] Sai Teja Ranuva commented on HBASE-16285: - It would be great if the rationale behind choosing 200ms(or 50ms as suggested by [~stack]) for DEAFAULT_DROP_TIMEOUT_REQUEST_DELAY is put in the comments for future reference. > Drop RPC requests if it must be considered as timeout at client > --- > > Key: HBASE-16285 > URL: https://issues.apache.org/jira/browse/HBASE-16285 > Project: HBase > Issue Type: Improvement >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-16285-branch-1-v1.patch, HBASE-16285-v1.patch > > > After HBASE-15593, we have a timeout param in header of RPC requests. We can > use it in more scenes. > A straightforward scene is to drop requests if it has waited so long in RPC > queue and has been dropped by client. Even if we handle this request and send > the response back, it will not be used any more. And client may have sent a > retry. In an extreme case, if the server is slow, all requests may be timeout > or queue-full-exception because we should handle previous requests which have > been dropped by client and many resources at server are wasted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16234) Expect and handle nulls when assigning replicas
[
https://issues.apache.org/jira/browse/HBASE-16234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400316#comment-15400316
]
Heng Chen commented on HBASE-16234:
---
v4 lgtm.
{quote}
Could do with less repetition (break out a method?)
{quote}
Agreed. But i think we should do it in another issue, this one is just for
bug fix, let's keep it small and clear. wdyt? [~stack]
> Expect and handle nulls when assigning replicas
> ---
>
> Key: HBASE-16234
> URL: https://issues.apache.org/jira/browse/HBASE-16234
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Affects Versions: 2.0.0
>Reporter: Harsh J
>Assignee: Yi Liang
> Attachments: HBASE-16234-V1.patch, HBASE-16234-V2.patch,
> HBASE-16234-V3.patch, HBASE-16234-V4.patch
>
>
> Observed this on a cluster:
> {code}
> FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting
> shutdown.
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.AssignmentManager.replicaRegionsNotRecordedInMeta(AssignmentManager.java:2799)
>
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assignAllUserRegions(AssignmentManager.java:2778)
>
> at
> org.apache.hadoop.hbase.master.AssignmentManager.processDeadServersAndRegionsInTransition(AssignmentManager.java:638)
>
> at
> org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:485)
>
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:723)
>
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:169)
> at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1481)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> It looks like {{FSTableDescriptors#get(…)}} can be expected to return null in
> some cases, but {{AssignmentManager.replicaRegionsNotRecordedInMeta(…)}} does
> not currently have any handling for such a possibility.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16270) When using region replicas, get thousands of occurrences of UnexpectedStateException: Current snapshot id is -1
[
https://issues.apache.org/jira/browse/HBASE-16270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400284#comment-15400284
]
Matteo Bertozzi commented on HBASE-16270:
-
when do we have snapshotId -1? always if the replica is not a primary?
can we add some if !RegionReplicaUtil.isDefaultReplica() to avoid to get in
that code path instead of just doing a check on -1?
> When using region replicas, get thousands of occurrences of
> UnexpectedStateException: Current snapshot id is -1
> ---
>
> Key: HBASE-16270
> URL: https://issues.apache.org/jira/browse/HBASE-16270
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.1.2
>Reporter: Robert Yokota
> Attachments: HBASE-16270-branch-1.2.patch, HBASE-16270-master.patch
>
>
> We have an HBase (1.1.2) production cluster with 58 region servers and a
> staging cluster with 6 region servers.
> For both clusters, we enabled region replicas with the following settings:
> hbase.regionserver.storefile.refresh.period = 0
> hbase.region.replica.replication.enabled = true
> hbase.region.replica.replication.memstore.enabled = true
> hbase.master.hfilecleaner.ttl = 360
> hbase.master.loadbalancer.class =
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer
> hbase.meta.replica.count = 3
> hbase.regionserver.meta.storefile.refresh.period = 3
> hbase.region.replica.wait.for.primary.flush = true
> hbase.region.replica.storefile.refresh.memstore.multiplier = 4
> We then altered our HBase tables to have REGION_REPLICATION => 2
> Both clusters got into a state where a few region servers were spewing the
> following error below in the HBase logs. In one instance the error occurred
> over 70K times. At this time, these region servers would see 10x write
> traffic (the hadoop.HBase.RegionServer.Server.writeRequestCount metric) and
> in some instances would crash.
> At the same time, we would see secondary regions move and then go into the
> "reads are disabled" state for extended periods.
> It appears there is a race condition where the DefaultMemStore::clearSnapshot
> method might be called more than once in succession. The first call would set
> snapshotId to -1, but the second call would throw an exception. It seems the
> second call should just return if the snapshotId is already -1, rather than
> throwing an exception.
> Thu Jul 21 08:38:50 UTC 2016,
> RpcRetryingCaller{globalStartTime=1469090201543, pause=100, retries=35},
> org.apache.hadoop.hbase.regionserver.UnexpectedStateException:
> org.apache.hadoop.hbase.regionserver.UnexpectedStateException: Current
> snapshot id is -1,passed 1469085004304
> at
> org.apache.hadoop.hbase.regionserver.DefaultMemStore.clearSnapshot(DefaultMemStore.java:187)
> at
> org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1054)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$500(HStore.java:128)
> at
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.replayFlush(HStore.java:2270)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayFlushInStores(HRegion.java:4487)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayWALFlushCommitMarker(HRegion.java:4388)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayWALFlushMarker(HRegion.java:4191)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doReplayBatchOp(RSRpcServices.java:776)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.replay(RSRpcServices.java:1655)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22255)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16196) Update jruby to a newer version.
[ https://issues.apache.org/jira/browse/HBASE-16196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400279#comment-15400279 ] stack commented on HBASE-16196: --- We know why the hadoop failures? I tried the patch. Still spews this at start: stack@ve0524:~$ ./hbase/bin/hbase --config ~/conf_hbase shell SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/stack/hbase-1.3.0-SNAPSHOT/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/stack/hadoop-2.7.3-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] HBase Shell; enter 'help' for list of supported commands. Type "exit" to leave the HBase Shell Version 1.3.0-SNAPSHOT, ra76cede216a1bb7a68c3c96aeeccfcf98f0d8441, Fri Jul 29 16:56:14 PDT 2016 ... Would be sweet to fix but not directly related. It seems to take longer to start up. Let me get some numbers. You see that? Probably loading more stuff... given its 25M! vs 13M. ls -la ~/.m2/repository/org/jruby/jruby-complete/9.1.2.0/jruby-complete-9.1.2.0.jar -rw-rw-r-- 1 stack stack 23458977 Jul 29 16:57 /home/stack/.m2/repository/org/jruby/jruby-complete/9.1.2.0/jruby-complete-9.1.2.0.jar -rw-rw-r-- 1 stack stack 13832273 Mar 22 11:04 /home/stack/.m2/repository/org/jruby/jruby-complete/1.6.8/jruby-complete-1.6.8.jar I poked around. It seems to work. Nice. > Update jruby to a newer version. > > > Key: HBASE-16196 > URL: https://issues.apache.org/jira/browse/HBASE-16196 > Project: HBase > Issue Type: Bug > Components: dependencies, shell >Reporter: Elliott Clark >Assignee: Matt Mullins >Priority: Critical > Fix For: 2.0.0, 1.4.0 > > Attachments: 0001-Update-to-JRuby-9.1.2.0-and-JLine-2.12.patch, > hbase-16196.branch-1.patch, hbase-16196.v2.branch-1.patch > > > Ruby 1.8.7 is no longer maintained. > The TTY library in the old jruby is bad. The newer one is less bad. > Since this is only a dependency on the hbase-shell module and not on > hbase-client or hbase-server this should be a pretty simple thing that > doesn't have any backwards compat issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16270) When using region replicas, get thousands of occurrences of UnexpectedStateException: Current snapshot id is -1
[
https://issues.apache.org/jira/browse/HBASE-16270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400280#comment-15400280
]
Matteo Bertozzi edited comment on HBASE-16270 at 7/30/16 12:07 AM:
---
nevermind... did not understood patch.
the -1 check is on the internal snapshotId not the passed one
was (Author: mbertozzi):
nevermind... did not understood code
> When using region replicas, get thousands of occurrences of
> UnexpectedStateException: Current snapshot id is -1
> ---
>
> Key: HBASE-16270
> URL: https://issues.apache.org/jira/browse/HBASE-16270
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.1.2
>Reporter: Robert Yokota
> Attachments: HBASE-16270-branch-1.2.patch, HBASE-16270-master.patch
>
>
> We have an HBase (1.1.2) production cluster with 58 region servers and a
> staging cluster with 6 region servers.
> For both clusters, we enabled region replicas with the following settings:
> hbase.regionserver.storefile.refresh.period = 0
> hbase.region.replica.replication.enabled = true
> hbase.region.replica.replication.memstore.enabled = true
> hbase.master.hfilecleaner.ttl = 360
> hbase.master.loadbalancer.class =
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer
> hbase.meta.replica.count = 3
> hbase.regionserver.meta.storefile.refresh.period = 3
> hbase.region.replica.wait.for.primary.flush = true
> hbase.region.replica.storefile.refresh.memstore.multiplier = 4
> We then altered our HBase tables to have REGION_REPLICATION => 2
> Both clusters got into a state where a few region servers were spewing the
> following error below in the HBase logs. In one instance the error occurred
> over 70K times. At this time, these region servers would see 10x write
> traffic (the hadoop.HBase.RegionServer.Server.writeRequestCount metric) and
> in some instances would crash.
> At the same time, we would see secondary regions move and then go into the
> "reads are disabled" state for extended periods.
> It appears there is a race condition where the DefaultMemStore::clearSnapshot
> method might be called more than once in succession. The first call would set
> snapshotId to -1, but the second call would throw an exception. It seems the
> second call should just return if the snapshotId is already -1, rather than
> throwing an exception.
> Thu Jul 21 08:38:50 UTC 2016,
> RpcRetryingCaller{globalStartTime=1469090201543, pause=100, retries=35},
> org.apache.hadoop.hbase.regionserver.UnexpectedStateException:
> org.apache.hadoop.hbase.regionserver.UnexpectedStateException: Current
> snapshot id is -1,passed 1469085004304
> at
> org.apache.hadoop.hbase.regionserver.DefaultMemStore.clearSnapshot(DefaultMemStore.java:187)
> at
> org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1054)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$500(HStore.java:128)
> at
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.replayFlush(HStore.java:2270)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayFlushInStores(HRegion.java:4487)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayWALFlushCommitMarker(HRegion.java:4388)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayWALFlushMarker(HRegion.java:4191)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doReplayBatchOp(RSRpcServices.java:776)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.replay(RSRpcServices.java:1655)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22255)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16270) When using region replicas, get thousands of occurrences of UnexpectedStateException: Current snapshot id is -1
[
https://issues.apache.org/jira/browse/HBASE-16270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400280#comment-15400280
]
Matteo Bertozzi commented on HBASE-16270:
-
nevermind... did not understood code
> When using region replicas, get thousands of occurrences of
> UnexpectedStateException: Current snapshot id is -1
> ---
>
> Key: HBASE-16270
> URL: https://issues.apache.org/jira/browse/HBASE-16270
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.1.2
>Reporter: Robert Yokota
> Attachments: HBASE-16270-branch-1.2.patch, HBASE-16270-master.patch
>
>
> We have an HBase (1.1.2) production cluster with 58 region servers and a
> staging cluster with 6 region servers.
> For both clusters, we enabled region replicas with the following settings:
> hbase.regionserver.storefile.refresh.period = 0
> hbase.region.replica.replication.enabled = true
> hbase.region.replica.replication.memstore.enabled = true
> hbase.master.hfilecleaner.ttl = 360
> hbase.master.loadbalancer.class =
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer
> hbase.meta.replica.count = 3
> hbase.regionserver.meta.storefile.refresh.period = 3
> hbase.region.replica.wait.for.primary.flush = true
> hbase.region.replica.storefile.refresh.memstore.multiplier = 4
> We then altered our HBase tables to have REGION_REPLICATION => 2
> Both clusters got into a state where a few region servers were spewing the
> following error below in the HBase logs. In one instance the error occurred
> over 70K times. At this time, these region servers would see 10x write
> traffic (the hadoop.HBase.RegionServer.Server.writeRequestCount metric) and
> in some instances would crash.
> At the same time, we would see secondary regions move and then go into the
> "reads are disabled" state for extended periods.
> It appears there is a race condition where the DefaultMemStore::clearSnapshot
> method might be called more than once in succession. The first call would set
> snapshotId to -1, but the second call would throw an exception. It seems the
> second call should just return if the snapshotId is already -1, rather than
> throwing an exception.
> Thu Jul 21 08:38:50 UTC 2016,
> RpcRetryingCaller{globalStartTime=1469090201543, pause=100, retries=35},
> org.apache.hadoop.hbase.regionserver.UnexpectedStateException:
> org.apache.hadoop.hbase.regionserver.UnexpectedStateException: Current
> snapshot id is -1,passed 1469085004304
> at
> org.apache.hadoop.hbase.regionserver.DefaultMemStore.clearSnapshot(DefaultMemStore.java:187)
> at
> org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1054)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$500(HStore.java:128)
> at
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.replayFlush(HStore.java:2270)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayFlushInStores(HRegion.java:4487)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayWALFlushCommitMarker(HRegion.java:4388)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayWALFlushMarker(HRegion.java:4191)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doReplayBatchOp(RSRpcServices.java:776)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.replay(RSRpcServices.java:1655)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22255)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16270) When using region replicas, get thousands of occurrences of UnexpectedStateException: Current snapshot id is -1
[
https://issues.apache.org/jira/browse/HBASE-16270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400266#comment-15400266
]
Matteo Bertozzi commented on HBASE-16270:
-
In 1.1, 1.2, 1.3, branch-1, master I see that before calling clearSnapshot() we
check snapshotId > 0. since HBASE-11569
so this patch should not be needed in the latest code base.
{code}
private boolean updateStorefiles(final List sfs, final long
snapshotId)
...
if (snapshotId > 0) {
this.memstore.clearSnapshot(snapshotId);
}
{code}
https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java#L1042
can you reproduce the issue with the latest >= 1.1? if so, can you write a
unit-test?
> When using region replicas, get thousands of occurrences of
> UnexpectedStateException: Current snapshot id is -1
> ---
>
> Key: HBASE-16270
> URL: https://issues.apache.org/jira/browse/HBASE-16270
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.1.2
>Reporter: Robert Yokota
> Attachments: HBASE-16270-branch-1.2.patch, HBASE-16270-master.patch
>
>
> We have an HBase (1.1.2) production cluster with 58 region servers and a
> staging cluster with 6 region servers.
> For both clusters, we enabled region replicas with the following settings:
> hbase.regionserver.storefile.refresh.period = 0
> hbase.region.replica.replication.enabled = true
> hbase.region.replica.replication.memstore.enabled = true
> hbase.master.hfilecleaner.ttl = 360
> hbase.master.loadbalancer.class =
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer
> hbase.meta.replica.count = 3
> hbase.regionserver.meta.storefile.refresh.period = 3
> hbase.region.replica.wait.for.primary.flush = true
> hbase.region.replica.storefile.refresh.memstore.multiplier = 4
> We then altered our HBase tables to have REGION_REPLICATION => 2
> Both clusters got into a state where a few region servers were spewing the
> following error below in the HBase logs. In one instance the error occurred
> over 70K times. At this time, these region servers would see 10x write
> traffic (the hadoop.HBase.RegionServer.Server.writeRequestCount metric) and
> in some instances would crash.
> At the same time, we would see secondary regions move and then go into the
> "reads are disabled" state for extended periods.
> It appears there is a race condition where the DefaultMemStore::clearSnapshot
> method might be called more than once in succession. The first call would set
> snapshotId to -1, but the second call would throw an exception. It seems the
> second call should just return if the snapshotId is already -1, rather than
> throwing an exception.
> Thu Jul 21 08:38:50 UTC 2016,
> RpcRetryingCaller{globalStartTime=1469090201543, pause=100, retries=35},
> org.apache.hadoop.hbase.regionserver.UnexpectedStateException:
> org.apache.hadoop.hbase.regionserver.UnexpectedStateException: Current
> snapshot id is -1,passed 1469085004304
> at
> org.apache.hadoop.hbase.regionserver.DefaultMemStore.clearSnapshot(DefaultMemStore.java:187)
> at
> org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1054)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$500(HStore.java:128)
> at
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.replayFlush(HStore.java:2270)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayFlushInStores(HRegion.java:4487)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayWALFlushCommitMarker(HRegion.java:4388)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayWALFlushMarker(HRegion.java:4191)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doReplayBatchOp(RSRpcServices.java:776)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.replay(RSRpcServices.java:1655)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22255)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14743) Add metrics around HeapMemoryManager
[ https://issues.apache.org/jira/browse/HBASE-14743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400264#comment-15400264 ] stack commented on HBASE-14743: --- A release note would be here. You 'Edit' this issue. You'll then get a release note text box. Write a curt note on what your change introduced. Your audience is user/operator, not developer. Thanks [~reidchan] > Add metrics around HeapMemoryManager > > > Key: HBASE-14743 > URL: https://issues.apache.org/jira/browse/HBASE-14743 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Reid Chan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14743.009.patch, HBASE-14743.009.rw3.patch, > HBASE-14743.009.v2.patch, HBASE-14743.010.patch, HBASE-14743.010.v2.patch, > HBASE-14743.011.patch, Metrics snapshot 2016-6-30.png, Screen Shot 2016-06-16 > at 5.39.13 PM.png, test2_1.png, test2_2.png, test2_3.png, test2_4.png > > > it would be good to know how many invocations there have been. > How many decided to expand memstore. > How many decided to expand block cache. > How many decided to do nothing. > etc. > When that's done use those metrics to clean up the tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16209) Provide an ExponentialBackOffPolicy sleep between failed region open requests
[ https://issues.apache.org/jira/browse/HBASE-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400261#comment-15400261 ] Joseph commented on HBASE-16209: Oh ok cool, thanks! > Provide an ExponentialBackOffPolicy sleep between failed region open requests > - > > Key: HBASE-16209 > URL: https://issues.apache.org/jira/browse/HBASE-16209 > Project: HBase > Issue Type: Bug >Reporter: Joseph >Assignee: Joseph > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16209-branch-1-addendum.patch, > HBASE-16209-branch-1.patch, HBASE-16209-v2.patch, HBASE-16209.patch > > > Related to HBASE-16138. As of now we currently have no pause between retrying > failed region open requests. And with a low maximumAttempt default, we can > quickly use up all our regionOpen retries if the server is in a bad state. I > added in a ExponentialBackOffPolicy so that we spread out the timing of our > open region retries in AssignmentManager. Review board at > https://reviews.apache.org/r/50011/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16209) Provide an ExponentialBackOffPolicy sleep between failed region open requests
[ https://issues.apache.org/jira/browse/HBASE-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph resolved HBASE-16209. Resolution: Fixed > Provide an ExponentialBackOffPolicy sleep between failed region open requests > - > > Key: HBASE-16209 > URL: https://issues.apache.org/jira/browse/HBASE-16209 > Project: HBase > Issue Type: Bug >Reporter: Joseph >Assignee: Joseph > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16209-branch-1-addendum.patch, > HBASE-16209-branch-1.patch, HBASE-16209-v2.patch, HBASE-16209.patch > > > Related to HBASE-16138. As of now we currently have no pause between retrying > failed region open requests. And with a low maximumAttempt default, we can > quickly use up all our regionOpen retries if the server is in a bad state. I > added in a ExponentialBackOffPolicy so that we spread out the timing of our > open region retries in AssignmentManager. Review board at > https://reviews.apache.org/r/50011/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16296) Reverse scan performance degrades when scanner cache size matches page filter size
[
https://issues.apache.org/jira/browse/HBASE-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400164#comment-15400164
]
Ted Yu edited comment on HBASE-16296 at 7/29/16 11:51 PM:
--
I wonder if the following change would help :
{code}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
index 89e723e..fa3836f 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -5951,6 +5951,10 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver, Regi
// Technically, if we hit limits before on this row, we don't need
this call.
if (filterRowKey(current)) {
incrementCountOfRowsFilteredMetric(scannerContext);
+if (isFilterDoneInternal()) {
+return
scannerContext.setScannerState(NextState.NO_MORE_VALUES).hasMoreValues();
+}
+
// Typically the count of rows scanned is incremented inside
#populateResult. However,
// here we are filtering a row based purely on its row key,
preventing us from calling
// #populateResult. Thus, perform the necessary increment here to
rows scanned metric
{code}
The added check would skip calling nextRow() if filter is done.
was (Author: yuzhih...@gmail.com):
I wonder if the following change would help :
{code}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
index 89e723e..fa3836f 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -5951,6 +5951,10 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver, Regi
// Technically, if we hit limits before on this row, we don't need
this call.
if (filterRowKey(current)) {
incrementCountOfRowsFilteredMetric(scannerContext);
+if (isFilterDoneInternal()) {
+return
scannerContext.setScannerState(NextState.NO_MORE_VALUES).hasMoreValues();
+}
+
// Typically the count of rows scanned is incremented inside
#populateResult. However,
// here we are filtering a row based purely on its row key,
preventing us from calling
// #populateResult. Thus, perform the necessary increment here to
rows scanned metric
{code}
> Reverse scan performance degrades when scanner cache size matches page filter
> size
> --
>
> Key: HBASE-16296
> URL: https://issues.apache.org/jira/browse/HBASE-16296
> Project: HBase
> Issue Type: Bug
>Reporter: James Taylor
> Attachments: 16296-MAYBE.txt, generatedata-snippet.java,
> repro-snippet.java
>
>
> When a reverse scan is done, the server seems to not know it's done when the
> scanner cache size matches the number of rows in a PageFilter. See
> PHOENIX-3121 for how this manifests itself. We have a standalone, pure HBase
> API reproducer too that I'll attach (courtesy of [~churromorales] and
> [~mujtabachohan]).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16285) Drop RPC requests if it must be considered as timeout at client
[
https://issues.apache.org/jira/browse/HBASE-16285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400257#comment-15400257
]
stack commented on HBASE-16285:
---
On EnvironmentEdge, see the patch just committed at HBASE-16256. I'd say, it
ok to not use EE.
bq. But maybe we need not configurable, we can make it a const. 200ms may be
too large, perhaps 50ms is OK for most scenes?
How about no delay padding? (I agree w/ [~anoop.hbase] that it would be good if
we could save on new configs).
On the patch...
bq. 96if (System.currentTimeMillis() >= call.deadline) {
Is it possible that further up in the code, we just did a
System.currentTimeMillis (trying to save on our doing too many of them).
This patch is excellent.
> Drop RPC requests if it must be considered as timeout at client
> ---
>
> Key: HBASE-16285
> URL: https://issues.apache.org/jira/browse/HBASE-16285
> Project: HBase
> Issue Type: Improvement
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-16285-branch-1-v1.patch, HBASE-16285-v1.patch
>
>
> After HBASE-15593, we have a timeout param in header of RPC requests. We can
> use it in more scenes.
> A straightforward scene is to drop requests if it has waited so long in RPC
> queue and has been dropped by client. Even if we handle this request and send
> the response back, it will not be used any more. And client may have sent a
> retry. In an extreme case, if the server is slow, all requests may be timeout
> or queue-full-exception because we should handle previous requests which have
> been dropped by client and many resources at server are wasted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16209) Provide an ExponentialBackOffPolicy sleep between failed region open requests
[ https://issues.apache.org/jira/browse/HBASE-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400251#comment-15400251 ] Matteo Bertozzi commented on HBASE-16209: - i took care of both on commit > Provide an ExponentialBackOffPolicy sleep between failed region open requests > - > > Key: HBASE-16209 > URL: https://issues.apache.org/jira/browse/HBASE-16209 > Project: HBase > Issue Type: Bug >Reporter: Joseph >Assignee: Joseph > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16209-branch-1-addendum.patch, > HBASE-16209-branch-1.patch, HBASE-16209-v2.patch, HBASE-16209.patch > > > Related to HBASE-16138. As of now we currently have no pause between retrying > failed region open requests. And with a low maximumAttempt default, we can > quickly use up all our regionOpen retries if the server is in a bad state. I > added in a ExponentialBackOffPolicy so that we spread out the timing of our > open region retries in AssignmentManager. Review board at > https://reviews.apache.org/r/50011/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16186) Fix AssignmentManager MBean name
[ https://issues.apache.org/jira/browse/HBASE-16186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16186: -- Resolution: Fixed Hadoop Flags: Incompatible change,Reviewed Release Note: The AssignmentManager MBean was named AssignmentManger (note misspelling). This patch fixed the misspelling. Status: Resolved (was: Patch Available) Marking as an incompatible change. Pushed to master only. Thanks for the patch [~reidchan] > Fix AssignmentManager MBean name > > > Key: HBASE-16186 > URL: https://issues.apache.org/jira/browse/HBASE-16186 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 1.2.1 >Reporter: Lars George >Assignee: Reid Chan > Labels: beginner > Fix For: 2.0.0 > > Attachments: HBASE-16181.patch > > > The MBean has a spelling error, listed as "AssignmentManger" (note the > missing "a"). This is a publicly available name that tools might already use > to filter metrics etc. We should change this across major versions only? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16270) When using region replicas, get thousands of occurrences of UnexpectedStateException: Current snapshot id is -1
[
https://issues.apache.org/jira/browse/HBASE-16270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400245#comment-15400245
]
stack commented on HBASE-16270:
---
[~mbertozzi] This looks good to me. To you?
> When using region replicas, get thousands of occurrences of
> UnexpectedStateException: Current snapshot id is -1
> ---
>
> Key: HBASE-16270
> URL: https://issues.apache.org/jira/browse/HBASE-16270
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.1.2
>Reporter: Robert Yokota
> Attachments: HBASE-16270-branch-1.2.patch, HBASE-16270-master.patch
>
>
> We have an HBase (1.1.2) production cluster with 58 region servers and a
> staging cluster with 6 region servers.
> For both clusters, we enabled region replicas with the following settings:
> hbase.regionserver.storefile.refresh.period = 0
> hbase.region.replica.replication.enabled = true
> hbase.region.replica.replication.memstore.enabled = true
> hbase.master.hfilecleaner.ttl = 360
> hbase.master.loadbalancer.class =
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer
> hbase.meta.replica.count = 3
> hbase.regionserver.meta.storefile.refresh.period = 3
> hbase.region.replica.wait.for.primary.flush = true
> hbase.region.replica.storefile.refresh.memstore.multiplier = 4
> We then altered our HBase tables to have REGION_REPLICATION => 2
> Both clusters got into a state where a few region servers were spewing the
> following error below in the HBase logs. In one instance the error occurred
> over 70K times. At this time, these region servers would see 10x write
> traffic (the hadoop.HBase.RegionServer.Server.writeRequestCount metric) and
> in some instances would crash.
> At the same time, we would see secondary regions move and then go into the
> "reads are disabled" state for extended periods.
> It appears there is a race condition where the DefaultMemStore::clearSnapshot
> method might be called more than once in succession. The first call would set
> snapshotId to -1, but the second call would throw an exception. It seems the
> second call should just return if the snapshotId is already -1, rather than
> throwing an exception.
> Thu Jul 21 08:38:50 UTC 2016,
> RpcRetryingCaller{globalStartTime=1469090201543, pause=100, retries=35},
> org.apache.hadoop.hbase.regionserver.UnexpectedStateException:
> org.apache.hadoop.hbase.regionserver.UnexpectedStateException: Current
> snapshot id is -1,passed 1469085004304
> at
> org.apache.hadoop.hbase.regionserver.DefaultMemStore.clearSnapshot(DefaultMemStore.java:187)
> at
> org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1054)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$500(HStore.java:128)
> at
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.replayFlush(HStore.java:2270)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayFlushInStores(HRegion.java:4487)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayWALFlushCommitMarker(HRegion.java:4388)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayWALFlushMarker(HRegion.java:4191)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doReplayBatchOp(RSRpcServices.java:776)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.replay(RSRpcServices.java:1655)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22255)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16256) Purpose of EnvironmentEdge, EnvironmentEdgeManager
[ https://issues.apache.org/jira/browse/HBASE-16256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16256: -- Resolution: Fixed Assignee: Sai Teja Ranuva Hadoop Flags: Reviewed Fix Version/s: 1.1.7 1.2.3 1.3.0 Status: Resolved (was: Patch Available) Pushed to branch-1.1+ Thanks for the nice doc [~saitejar] > Purpose of EnvironmentEdge, EnvironmentEdgeManager > -- > > Key: HBASE-16256 > URL: https://issues.apache.org/jira/browse/HBASE-16256 > Project: HBase > Issue Type: Improvement > Components: documentation, regionserver >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Trivial > Labels: beginner, clarification > Fix For: 2.0.0, 1.3.0, 1.2.3, 1.1.7 > > Attachments: HBASE-16256.master.1.patch, HBASE-16256.master.2.patch, > HBASE-16256.master.3.patch > > > I am trying to figure out the purpose of EnvironmentEdge in the HBase code. > The java doc says it has something to do with environment, I feel it is > vague. It looks like there is a bigger picture for such a design. Currently > only concrete method that is available in it is currentTime(). > Can anyone summarize the rationale behind having > EnvironmentEdge/EnvironmentEdgeManager ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-16209) Provide an ExponentialBackOffPolicy sleep between failed region open requests
[ https://issues.apache.org/jira/browse/HBASE-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph reopened HBASE-16209: > Provide an ExponentialBackOffPolicy sleep between failed region open requests > - > > Key: HBASE-16209 > URL: https://issues.apache.org/jira/browse/HBASE-16209 > Project: HBase > Issue Type: Bug >Reporter: Joseph >Assignee: Joseph > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16209-branch-1-addendum.patch, > HBASE-16209-branch-1.patch, HBASE-16209-v2.patch, HBASE-16209.patch > > > Related to HBASE-16138. As of now we currently have no pause between retrying > failed region open requests. And with a low maximumAttempt default, we can > quickly use up all our regionOpen retries if the server is in a bad state. I > added in a ExponentialBackOffPolicy so that we spread out the timing of our > open region retries in AssignmentManager. Review board at > https://reviews.apache.org/r/50011/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16209) Provide an ExponentialBackOffPolicy sleep between failed region open requests
[ https://issues.apache.org/jira/browse/HBASE-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400241#comment-15400241 ] Joseph commented on HBASE-16209: Oh sorry, I hadn't attached my addendum for the Master branch. Also should I submit another patch for the whitespace issue on branch-1? > Provide an ExponentialBackOffPolicy sleep between failed region open requests > - > > Key: HBASE-16209 > URL: https://issues.apache.org/jira/browse/HBASE-16209 > Project: HBase > Issue Type: Bug >Reporter: Joseph >Assignee: Joseph > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16209-branch-1-addendum.patch, > HBASE-16209-branch-1.patch, HBASE-16209-v2.patch, HBASE-16209.patch > > > Related to HBASE-16138. As of now we currently have no pause between retrying > failed region open requests. And with a low maximumAttempt default, we can > quickly use up all our regionOpen retries if the server is in a bad state. I > added in a ExponentialBackOffPolicy so that we spread out the timing of our > open region retries in AssignmentManager. Review board at > https://reviews.apache.org/r/50011/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16234) Expect and handle nulls when assigning replicas
[
https://issues.apache.org/jira/browse/HBASE-16234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400235#comment-15400235
]
Matteo Bertozzi commented on HBASE-16234:
-
agreed with stack, we can extract that logic in a method something like
getRegionReplication(TableName) or getNumReplicas(TableName) or some better
name. also "server" is already a MasterService, so there is no need to cast
(patch is half and half).
> Expect and handle nulls when assigning replicas
> ---
>
> Key: HBASE-16234
> URL: https://issues.apache.org/jira/browse/HBASE-16234
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Affects Versions: 2.0.0
>Reporter: Harsh J
>Assignee: Yi Liang
> Attachments: HBASE-16234-V1.patch, HBASE-16234-V2.patch,
> HBASE-16234-V3.patch, HBASE-16234-V4.patch
>
>
> Observed this on a cluster:
> {code}
> FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting
> shutdown.
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.AssignmentManager.replicaRegionsNotRecordedInMeta(AssignmentManager.java:2799)
>
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assignAllUserRegions(AssignmentManager.java:2778)
>
> at
> org.apache.hadoop.hbase.master.AssignmentManager.processDeadServersAndRegionsInTransition(AssignmentManager.java:638)
>
> at
> org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:485)
>
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:723)
>
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:169)
> at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1481)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> It looks like {{FSTableDescriptors#get(…)}} can be expected to return null in
> some cases, but {{AssignmentManager.replicaRegionsNotRecordedInMeta(…)}} does
> not currently have any handling for such a possibility.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16275) Change ServerManager#onlineServers from ConcurrentHashMap to ConcurrentSkipListMap
[ https://issues.apache.org/jira/browse/HBASE-16275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400226#comment-15400226 ] stack commented on HBASE-16275: --- Nice numbers [~huaxiang] > Change ServerManager#onlineServers from ConcurrentHashMap to > ConcurrentSkipListMap > -- > > Key: HBASE-16275 > URL: https://issues.apache.org/jira/browse/HBASE-16275 > Project: HBase > Issue Type: Improvement >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16275-v001.patch, HBASE-16275-v002.patch, > TestServerManagerPerf.java > > > In Class ServerManager, onlineServers is declared as ConcurrentHashMap. In > findServerWithSameHostnamePortWithLock(), it has to do a loop to find if > there is a ServerName with same host:port pair. If replaced with > ConcurrentSkipListMap, findServerWithSameHostnamePortWithLock() can be > replaced with a O(logN) implementation. > I run some performance comparison(test the function only), it seems that > there is no difference if there are 1000 servers. With more servers, > ConcurrentSkipListMap implementation is going to win big. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16234) Expect and handle nulls when assigning replicas
[
https://issues.apache.org/jira/browse/HBASE-16234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400222#comment-15400222
]
stack commented on HBASE-16234:
---
Patch LGTM. Could do with less repetition (break out a method?). @matteo or
[~enis] might have input but they can review post commit if have to. Thanks
[~carp84]
> Expect and handle nulls when assigning replicas
> ---
>
> Key: HBASE-16234
> URL: https://issues.apache.org/jira/browse/HBASE-16234
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Affects Versions: 2.0.0
>Reporter: Harsh J
>Assignee: Yi Liang
> Attachments: HBASE-16234-V1.patch, HBASE-16234-V2.patch,
> HBASE-16234-V3.patch, HBASE-16234-V4.patch
>
>
> Observed this on a cluster:
> {code}
> FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting
> shutdown.
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.AssignmentManager.replicaRegionsNotRecordedInMeta(AssignmentManager.java:2799)
>
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assignAllUserRegions(AssignmentManager.java:2778)
>
> at
> org.apache.hadoop.hbase.master.AssignmentManager.processDeadServersAndRegionsInTransition(AssignmentManager.java:638)
>
> at
> org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:485)
>
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:723)
>
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:169)
> at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1481)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> It looks like {{FSTableDescriptors#get(…)}} can be expected to return null in
> some cases, but {{AssignmentManager.replicaRegionsNotRecordedInMeta(…)}} does
> not currently have any handling for such a possibility.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15882) Upgrade to yetus precommit 0.3.0
[ https://issues.apache.org/jira/browse/HBASE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400215#comment-15400215 ] stack commented on HBASE-15882: --- I see we are still on yetus 0.2.1. @busbey is out driving north to south for next few days but will be back next week > Upgrade to yetus precommit 0.3.0 > > > Key: HBASE-15882 > URL: https://issues.apache.org/jira/browse/HBASE-15882 > Project: HBase > Issue Type: Improvement > Components: build >Reporter: Sean Busbey >Assignee: Jurriaan Mous >Priority: Critical > Attachments: HBASE-15882.patch > > > Now that Yetus 0.3.0 is out, we should update our precommit builds so we can > use docker again. > Most of the changes to our personality should be covered in the updated hbase > example that ships with 0.3.0. we'll have to forward port the flakey test > work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16209) Provide an ExponentialBackOffPolicy sleep between failed region open requests
[
https://issues.apache.org/jira/browse/HBASE-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400207#comment-15400207
]
Hadoop QA commented on HBASE-16209:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
1s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} branch-1 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} branch-1 passed with JDK v1.7.0_25 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} branch-1 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} branch-1 passed with JDK v1.7.0_25 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_25 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m 20s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 11m 40s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 12m 59s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 14m 19s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.7.0_25 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 139m 22s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 168m 10s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.TestAssignmentManagerOnCluster |
| |
hadoop.hbase.client.replication.TestReplicationAdminWithTwoDifferentZKClusters |
| | hadoop.hbase.client.TestFromClientSide |
| | hadoop.hbase.util.TestHBaseFsck |
| | hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster |
| | hadoop.hbase.TestPartialResultsFromClientSide |
| |
[jira] [Commented] (HBASE-16148) Hybrid Logical Clocks(placeholder for running tests)
[
https://issues.apache.org/jira/browse/HBASE-16148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400204#comment-15400204
]
Hadoop QA commented on HBASE-16148:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m 59s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 9s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} master passed with JDK v1.7.0_25 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s
{color} | {color:green} master passed with JDK v1.7.0_25 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.7.0_25 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 1s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed with JDK v1.7.0_25 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 56s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 91m 11s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 154m 19s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821078/HBASE-16148.master.test.5.patch
|
| JIRA Issue | HBASE-16148 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool |
[jira] [Commented] (HBASE-16209) Provide an ExponentialBackOffPolicy sleep between failed region open requests
[ https://issues.apache.org/jira/browse/HBASE-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400198#comment-15400198 ] Matteo Bertozzi commented on HBASE-16209: - committed to branch-1 and master > Provide an ExponentialBackOffPolicy sleep between failed region open requests > - > > Key: HBASE-16209 > URL: https://issues.apache.org/jira/browse/HBASE-16209 > Project: HBase > Issue Type: Bug >Reporter: Joseph >Assignee: Joseph > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16209-branch-1-addendum.patch, > HBASE-16209-branch-1.patch, HBASE-16209-v2.patch, HBASE-16209.patch > > > Related to HBASE-16138. As of now we currently have no pause between retrying > failed region open requests. And with a low maximumAttempt default, we can > quickly use up all our regionOpen retries if the server is in a bad state. I > added in a ExponentialBackOffPolicy so that we spread out the timing of our > open region retries in AssignmentManager. Review board at > https://reviews.apache.org/r/50011/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16209) Provide an ExponentialBackOffPolicy sleep between failed region open requests
[ https://issues.apache.org/jira/browse/HBASE-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16209: Resolution: Fixed Status: Resolved (was: Patch Available) > Provide an ExponentialBackOffPolicy sleep between failed region open requests > - > > Key: HBASE-16209 > URL: https://issues.apache.org/jira/browse/HBASE-16209 > Project: HBase > Issue Type: Bug >Reporter: Joseph >Assignee: Joseph > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16209-branch-1-addendum.patch, > HBASE-16209-branch-1.patch, HBASE-16209-v2.patch, HBASE-16209.patch > > > Related to HBASE-16138. As of now we currently have no pause between retrying > failed region open requests. And with a low maximumAttempt default, we can > quickly use up all our regionOpen retries if the server is in a bad state. I > added in a ExponentialBackOffPolicy so that we spread out the timing of our > open region retries in AssignmentManager. Review board at > https://reviews.apache.org/r/50011/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16148) Hybrid Logical Clocks(placeholder for running tests)
[ https://issues.apache.org/jira/browse/HBASE-16148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Teja Ranuva updated HBASE-16148: Status: Patch Available (was: Open) > Hybrid Logical Clocks(placeholder for running tests) > > > Key: HBASE-16148 > URL: https://issues.apache.org/jira/browse/HBASE-16148 > Project: HBase > Issue Type: Sub-task > Components: API >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Minor > Labels: test-patch > Attachments: HBASE-16148.master.001.patch, > HBASE-16148.master.test.1.patch, HBASE-16148.master.test.2.patch, > HBASE-16148.master.test.3.patch, HBASE-16148.master.test.4.patch, > HBASE-16148.master.test.5.patch > > > This JIRA is just a placeholder to test Hybrid Logical Clocks code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16148) Hybrid Logical Clocks(placeholder for running tests)
[ https://issues.apache.org/jira/browse/HBASE-16148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Teja Ranuva updated HBASE-16148: Status: Open (was: Patch Available) > Hybrid Logical Clocks(placeholder for running tests) > > > Key: HBASE-16148 > URL: https://issues.apache.org/jira/browse/HBASE-16148 > Project: HBase > Issue Type: Sub-task > Components: API >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Minor > Labels: test-patch > Attachments: HBASE-16148.master.001.patch, > HBASE-16148.master.test.1.patch, HBASE-16148.master.test.2.patch, > HBASE-16148.master.test.3.patch, HBASE-16148.master.test.4.patch, > HBASE-16148.master.test.5.patch > > > This JIRA is just a placeholder to test Hybrid Logical Clocks code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16209) Provide an ExponentialBackOffPolicy sleep between failed region open requests
[ https://issues.apache.org/jira/browse/HBASE-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400188#comment-15400188 ] Hudson commented on HBASE-16209: FAILURE: Integrated in HBase-1.4 #314 (See [https://builds.apache.org/job/HBase-1.4/314/]) Back port HBASE-16209 Provide an exponential back off policy in (eclark: rev 7c97acf6e345023f043964d023816d5b3329dde9) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ClosedRegionHandler.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java * hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/AssignmentManagerStatusTmpl.jamon * hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java > Provide an ExponentialBackOffPolicy sleep between failed region open requests > - > > Key: HBASE-16209 > URL: https://issues.apache.org/jira/browse/HBASE-16209 > Project: HBase > Issue Type: Bug >Reporter: Joseph >Assignee: Joseph > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16209-branch-1-addendum.patch, > HBASE-16209-branch-1.patch, HBASE-16209-v2.patch, HBASE-16209.patch > > > Related to HBASE-16138. As of now we currently have no pause between retrying > failed region open requests. And with a low maximumAttempt default, we can > quickly use up all our regionOpen retries if the server is in a bad state. I > added in a ExponentialBackOffPolicy so that we spread out the timing of our > open region retries in AssignmentManager. Review board at > https://reviews.apache.org/r/50011/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16301) Trigger flush without waiting when compaction is disabled on a table
[
https://issues.apache.org/jira/browse/HBASE-16301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400175#comment-15400175
]
Enis Soztutar commented on HBASE-16301:
---
Can you move this check to be inside isTooManyStoreFiles()
{code}
region.getTableDesc().isCompactionEnabled()
{code}
Remove all commented-out code please:
{code}
+//TEST_UTIL.startMiniCluster(1);
{code}
> Trigger flush without waiting when compaction is disabled on a table
>
>
> Key: HBASE-16301
> URL: https://issues.apache.org/jira/browse/HBASE-16301
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Attachments: HBASE-16301-v001.patch
>
>
> When compaction is disabled on a table, flush needs to wait
> MemStoreFlusher#blockingWaitTime (default value is 90 seconds) before it goes
> ahead to flush. It has side effect that client may be blocked due to
> RegionTooBusyException. Please see the mail sent to dev list.
> http://mail-archives.apache.org/mod_mbox/hbase-dev/201607.mbox/%3c2d66b8ca-7c6f-40ea-a861-2de5482ec...@cloudera.com%3E
> I guess that the right behavior is to do flush without waiting if compaction
> is disabled on a table. Attached a patch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16298) ESAPI.properties missing in hbase-server.jar
[
https://issues.apache.org/jira/browse/HBASE-16298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-16298:
--
Status: Patch Available (was: Open)
> ESAPI.properties missing in hbase-server.jar
>
>
> Key: HBASE-16298
> URL: https://issues.apache.org/jira/browse/HBASE-16298
> Project: HBase
> Issue Type: Bug
> Components: Interface
>Affects Versions: 1.2.2, 1.1.5
> Environment: Debian 8.2
> Linux 3.16.0-4-amd64
> OpenJDK 8u91-b14-3ubuntu1~16.04.1
>Reporter: Sylvain Veyrié
> Attachments: HBASE-16298.patch
>
>
> For policy/compliance reasons, we removed the tests jars from lib/ directory
> on HBase Master. Everything was working fine from 1.0 to 1.1.3.
> When I upgraded from 1.1.3 to 1.1.5, the /master-status page started to
> return an error 500: {{java.lang.IllegalArgumentException: Failed to load
> ESAPI.properties as a classloader resource.}}
> After some search, I found out that ESAPI has been added following
> HBASE-15122 which also added the ESAPI.properties files into
> src/main/resources.
> However, it seems an exclusion has been put on packaging: the file is absent
> from hbase-server-1.1.5.jar, but present in hbase-server-1.1.5-tests.jar,
> which is in the lib/ directory in the tar.gz distribution.
> Our workaround is to deploy back hbase-server-1.1.5-tests.jar in lib/.
> However, it does not seem right to require tests jar to have HBase master
> propertly work.
> Even if it is the current HBase policy to keep those jars, I think the
> hbase-server.jar should contain ESAPI.properties.
> The same thing applies for 1.2 branch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16296) Reverse scan performance degrades when scanner cache size matches page filter size
[
https://issues.apache.org/jira/browse/HBASE-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400164#comment-15400164
]
Ted Yu commented on HBASE-16296:
I wonder if the following change would help :
{code}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
index 89e723e..fa3836f 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -5951,6 +5951,10 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver, Regi
// Technically, if we hit limits before on this row, we don't need
this call.
if (filterRowKey(current)) {
incrementCountOfRowsFilteredMetric(scannerContext);
+if (isFilterDoneInternal()) {
+return
scannerContext.setScannerState(NextState.NO_MORE_VALUES).hasMoreValues();
+}
+
// Typically the count of rows scanned is incremented inside
#populateResult. However,
// here we are filtering a row based purely on its row key,
preventing us from calling
// #populateResult. Thus, perform the necessary increment here to
rows scanned metric
{code}
> Reverse scan performance degrades when scanner cache size matches page filter
> size
> --
>
> Key: HBASE-16296
> URL: https://issues.apache.org/jira/browse/HBASE-16296
> Project: HBase
> Issue Type: Bug
>Reporter: James Taylor
> Attachments: 16296-MAYBE.txt, generatedata-snippet.java,
> repro-snippet.java
>
>
> When a reverse scan is done, the server seems to not know it's done when the
> scanner cache size matches the number of rows in a PageFilter. See
> PHOENIX-3121 for how this manifests itself. We have a standalone, pure HBase
> API reproducer too that I'll attach (courtesy of [~churromorales] and
> [~mujtabachohan]).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-9899) for idempotent operation dups, return the result instead of throwing conflict exception
[ https://issues.apache.org/jira/browse/HBASE-9899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400110#comment-15400110 ] stack commented on HBASE-9899: -- Do you have a commit comment to go w/ your patch explaining what it does [~zghaobac]? I skimmed the patch. It looks great. Nice to see an old TODO being implemented. Were you experiencing this issue? > for idempotent operation dups, return the result instead of throwing conflict > exception > --- > > Key: HBASE-9899 > URL: https://issues.apache.org/jira/browse/HBASE-9899 > Project: HBase > Issue Type: Improvement >Reporter: Sergey Shelukhin >Assignee: Enis Soztutar > Attachments: HBASE-9899-v1.patch > > > After HBASE-3787, we could store mvcc in operation context, and use it to > convert the modification request into read on dups instead of throwing > OperationConflictException. > MVCC tracking will have to be aware of such MVCC numbers present. Given that > scanners are usually relatively short-lived, that would prevent low watermark > from advancing for quite a bit more time -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16209) Provide an ExponentialBackOffPolicy sleep between failed region open requests
[ https://issues.apache.org/jira/browse/HBASE-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400088#comment-15400088 ] Matteo Bertozzi commented on HBASE-16209: - +1 on the addendum > Provide an ExponentialBackOffPolicy sleep between failed region open requests > - > > Key: HBASE-16209 > URL: https://issues.apache.org/jira/browse/HBASE-16209 > Project: HBase > Issue Type: Bug >Reporter: Joseph >Assignee: Joseph > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16209-branch-1-addendum.patch, > HBASE-16209-branch-1.patch, HBASE-16209-v2.patch, HBASE-16209.patch > > > Related to HBASE-16138. As of now we currently have no pause between retrying > failed region open requests. And with a low maximumAttempt default, we can > quickly use up all our regionOpen retries if the server is in a bad state. I > added in a ExponentialBackOffPolicy so that we spread out the timing of our > open region retries in AssignmentManager. Review board at > https://reviews.apache.org/r/50011/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14345) Consolidate printUsage in IntegrationTestLoadAndVerify
[
https://issues.apache.org/jira/browse/HBASE-14345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400080#comment-15400080
]
Enis Soztutar commented on HBASE-14345:
---
Is this a typo?
{code}
+System.err.println("Doptions");
{code}
We should not call System.exit() if it is not main() method. {{Tool}} instances
can be invoked programmatically as well in which case you don't want to call
exit.
{code}
+ System.exit(EXIT_FAILURE);
{code}
> Consolidate printUsage in IntegrationTestLoadAndVerify
> --
>
> Key: HBASE-14345
> URL: https://issues.apache.org/jira/browse/HBASE-14345
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Reporter: Nick Dimiduk
>Assignee: Reid Chan
>Priority: Trivial
> Labels: beginner
> Attachments: HBASE-14345.002.patch, HBASE-14345.patch,
> itlav-2016-07-07.png, itlv.png
>
>
> Investigating the use of {{itlav}} is a little screwy. Subclasses are not
> overriding the {{printUsage()}} methods correctly, so you have to pass
> {{--help}} to get some info and no arguments to get the rest.
> {noformat}
> [hbase@ndimiduk-112rc2-7 ~]$ hbase
> org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify --help
> usage: bin/hbase org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify
>
> Options:
> -h,--help Show usage
> -m,--monkey Which chaos monkey to run
> -monkeyProps The properties file for specifying chaos monkey
> properties.
> -ncc,--noClusterCleanUp Don't clean up the cluster at the end
> [hbase@ndimiduk-112rc2-7 ~]$ hbase
> org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify
> IntegrationTestLoadAndVerify [-Doptions]
> Loads a table with row dependencies and verifies the dependency chains
> Options
> -Dloadmapper.table=Table to write/verify (default autogen)
> -Dloadmapper.backrefs=Number of backreferences per row (default
> 50)
> -Dloadmapper.num_to_write=Number of rows per mapper (default 100,000
> per mapper)
> -Dloadmapper.deleteAfter= Delete after a successful verify (default
> true)
> -Dloadmapper.numPresplits=Number of presplit regions to start with
> (default 40)
> -Dloadmapper.map.tasks= Number of map tasks for load (default 200)
> -Dverify.reduce.tasks=Number of reduce tasks for verify (default
> 35)
> -Dverify.scannercaching= Number hbase scanner caching rows to read
> (default 50)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15656) Fix unused protobuf warning in Admin.proto
[
https://issues.apache.org/jira/browse/HBASE-15656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-15656:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Pushed to master. Thanks for the patch [~easyliangjob]
> Fix unused protobuf warning in Admin.proto
> --
>
> Key: HBASE-15656
> URL: https://issues.apache.org/jira/browse/HBASE-15656
> Project: HBase
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: Elliott Clark
>Assignee: Yi Liang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-15656-V1.patch
>
>
> {code}
> Warning: Unused import: "Admin.proto" imports "Client.proto" which is not
> used.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16296) Reverse scan performance degrades when scanner cache size matches page filter size
[ https://issues.apache.org/jira/browse/HBASE-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400063#comment-15400063 ] Ted Yu commented on HBASE-16296: The FilterList involves not only PageFilter but also WhileMatchFilter. The false return value of filterRow() is from WhileMatchFilter. > Reverse scan performance degrades when scanner cache size matches page filter > size > -- > > Key: HBASE-16296 > URL: https://issues.apache.org/jira/browse/HBASE-16296 > Project: HBase > Issue Type: Bug >Reporter: James Taylor > Attachments: 16296-MAYBE.txt, generatedata-snippet.java, > repro-snippet.java > > > When a reverse scan is done, the server seems to not know it's done when the > scanner cache size matches the number of rows in a PageFilter. See > PHOENIX-3121 for how this manifests itself. We have a standalone, pure HBase > API reproducer too that I'll attach (courtesy of [~churromorales] and > [~mujtabachohan]). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16288) HFile intermediate block level indexes might recurse forever creating multi TB files
[
https://issues.apache.org/jira/browse/HBASE-16288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-16288:
--
Attachment: hbase-16288_v4.patch
v4.
Slight change to the algorithm. Now we have a min number of entries in a given
index block, defaults to 16, as well as the max size. Max size is ignored while
we have less than desired number of entries. This is useful since the index is
supposed to be B-Tree like index, and it does not make sense to have index
levels with 1-2 entries per level. We don't want to end up with 50-level
indices which will make seeking very inefficient.
> HFile intermediate block level indexes might recurse forever creating multi
> TB files
>
>
> Key: HBASE-16288
> URL: https://issues.apache.org/jira/browse/HBASE-16288
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.6, 1.2.3
>
> Attachments: hbase-16288_v1.patch, hbase-16288_v2.patch,
> hbase-16288_v3.patch, hbase-16288_v4.patch
>
>
> Mighty [~elserj] was debugging an opentsdb cluster where some region
> directory ended up having 5TB+ files under /.tmp/
> Further debugging and analysis, we were able to reproduce the problem locally
> where we never we recursing in this code path for writing intermediate level
> indices:
> {code:title=HFileBlockIndex.java}
> if (curInlineChunk != null) {
> while (rootChunk.getRootSize() > maxChunkSize) {
> rootChunk = writeIntermediateLevel(out, rootChunk);
> numLevels += 1;
> }
> }
> {code}
> The problem happens if we end up with a very large rowKey (larger than
> "hfile.index.block.max.size" being the first key in the block, then moving
> all the way to the root-level index building. We will keep writing and
> building the next level of intermediate level indices with a single
> very-large key. This can happen in flush / compaction / region recovery
> causing cluster inoperability due to ever-growing files.
> Seems the issue was also reported earlier, with a temporary workaround:
> https://github.com/OpenTSDB/opentsdb/issues/490
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15656) Fix unused protobuf warning in Admin.proto
[
https://issues.apache.org/jira/browse/HBASE-15656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1548#comment-1548
]
Hadoop QA commented on HBASE-15656:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 39s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
6s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 19s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821067/HBASE-15656-V1.patch |
| JIRA Issue | HBASE-15656 |
| Optional Tests | asflicense cc unit hbaseprotoc |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / fe44383 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/2829/testReport/ |
| modules | C: hbase-protocol U: hbase-protocol |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/2829/console |
| Powered by | Apache Yetus 0.2.1 http://yetus.apache.org |
This message was automatically generated.
> Fix unused protobuf warning in Admin.proto
> --
>
> Key: HBASE-15656
> URL: https://issues.apache.org/jira/browse/HBASE-15656
> Project: HBase
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: Elliott Clark
>Assignee: Yi Liang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-15656-V1.patch
>
>
> {code}
> Warning: Unused import: "Admin.proto" imports "Client.proto" which is not
> used.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16306) Add specific imports to avoid namespace clash in defaultSource.scala
[ https://issues.apache.org/jira/browse/HBASE-16306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16306: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks for the patch [~saitejar] > Add specific imports to avoid namespace clash in defaultSource.scala > > > Key: HBASE-16306 > URL: https://issues.apache.org/jira/browse/HBASE-16306 > Project: HBase > Issue Type: Bug >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16306.master.1.patch > > > I am working on adding Hybrid Logical clocks to HBase. As part of it, I wish > to add TimestampType file in hbase common. Spark has some types defined in - > org.apache.spark.sql.types, all of which are being imported. It also has a > TimestampType defined. As currently in this file, all of the hbase common is > imported, creating a namespace clash. I have changed to only import specific > hbase common classes that are required in this file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16148) Hybrid Logical Clocks(placeholder for running tests)
[ https://issues.apache.org/jira/browse/HBASE-16148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399966#comment-15399966 ] Sai Teja Ranuva edited comment on HBASE-16148 at 7/29/16 8:37 PM: -- remove setting of timestamp in meta table accessor to mutate meta table. was (Author: saitejar): remove setting of timestamp by master to mutate meta table. > Hybrid Logical Clocks(placeholder for running tests) > > > Key: HBASE-16148 > URL: https://issues.apache.org/jira/browse/HBASE-16148 > Project: HBase > Issue Type: Sub-task > Components: API >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Minor > Labels: test-patch > Attachments: HBASE-16148.master.001.patch, > HBASE-16148.master.test.1.patch, HBASE-16148.master.test.2.patch, > HBASE-16148.master.test.3.patch, HBASE-16148.master.test.4.patch, > HBASE-16148.master.test.5.patch > > > This JIRA is just a placeholder to test Hybrid Logical Clocks code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16148) Hybrid Logical Clocks(placeholder for running tests)
[ https://issues.apache.org/jira/browse/HBASE-16148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Teja Ranuva updated HBASE-16148: Attachment: HBASE-16148.master.test.5.patch remove setting of timestamp by master to mutate meta table. > Hybrid Logical Clocks(placeholder for running tests) > > > Key: HBASE-16148 > URL: https://issues.apache.org/jira/browse/HBASE-16148 > Project: HBase > Issue Type: Sub-task > Components: API >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Minor > Labels: test-patch > Attachments: HBASE-16148.master.001.patch, > HBASE-16148.master.test.1.patch, HBASE-16148.master.test.2.patch, > HBASE-16148.master.test.3.patch, HBASE-16148.master.test.4.patch, > HBASE-16148.master.test.5.patch > > > This JIRA is just a placeholder to test Hybrid Logical Clocks code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15656) Fix unused protobuf warning in Admin.proto
[
https://issues.apache.org/jira/browse/HBASE-15656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liang updated HBASE-15656:
-
Affects Version/s: 2.0.0
> Fix unused protobuf warning in Admin.proto
> --
>
> Key: HBASE-15656
> URL: https://issues.apache.org/jira/browse/HBASE-15656
> Project: HBase
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: Elliott Clark
>Assignee: Yi Liang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-15656-V1.patch
>
>
> {code}
> Warning: Unused import: "Admin.proto" imports "Client.proto" which is not
> used.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15656) Fix unused protobuf warning in Admin.proto
[
https://issues.apache.org/jira/browse/HBASE-15656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liang updated HBASE-15656:
-
Fix Version/s: 2.0.0
Status: Patch Available (was: Open)
> Fix unused protobuf warning in Admin.proto
> --
>
> Key: HBASE-15656
> URL: https://issues.apache.org/jira/browse/HBASE-15656
> Project: HBase
> Issue Type: Task
>Reporter: Elliott Clark
>Assignee: Yi Liang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-15656-V1.patch
>
>
> {code}
> Warning: Unused import: "Admin.proto" imports "Client.proto" which is not
> used.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16209) Provide an ExponentialBackOffPolicy sleep between failed region open requests
[ https://issues.apache.org/jira/browse/HBASE-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph updated HBASE-16209: --- Attachment: HBASE-16209-branch-1-addendum.patch > Provide an ExponentialBackOffPolicy sleep between failed region open requests > - > > Key: HBASE-16209 > URL: https://issues.apache.org/jira/browse/HBASE-16209 > Project: HBase > Issue Type: Bug >Reporter: Joseph >Assignee: Joseph > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16209-branch-1-addendum.patch, > HBASE-16209-branch-1.patch, HBASE-16209-v2.patch, HBASE-16209.patch > > > Related to HBASE-16138. As of now we currently have no pause between retrying > failed region open requests. And with a low maximumAttempt default, we can > quickly use up all our regionOpen retries if the server is in a bad state. I > added in a ExponentialBackOffPolicy so that we spread out the timing of our > open region retries in AssignmentManager. Review board at > https://reviews.apache.org/r/50011/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16209) Provide an ExponentialBackOffPolicy sleep between failed region open requests
[ https://issues.apache.org/jira/browse/HBASE-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph updated HBASE-16209: --- Status: Patch Available (was: Reopened) > Provide an ExponentialBackOffPolicy sleep between failed region open requests > - > > Key: HBASE-16209 > URL: https://issues.apache.org/jira/browse/HBASE-16209 > Project: HBase > Issue Type: Bug >Reporter: Joseph >Assignee: Joseph > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16209-branch-1-addendum.patch, > HBASE-16209-branch-1.patch, HBASE-16209-v2.patch, HBASE-16209.patch > > > Related to HBASE-16138. As of now we currently have no pause between retrying > failed region open requests. And with a low maximumAttempt default, we can > quickly use up all our regionOpen retries if the server is in a bad state. I > added in a ExponentialBackOffPolicy so that we spread out the timing of our > open region retries in AssignmentManager. Review board at > https://reviews.apache.org/r/50011/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16306) Add specific imports to avoid namespace clash in defaultSource.scala
[
https://issues.apache.org/jira/browse/HBASE-16306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399930#comment-15399930
]
Hadoop QA commented on HBASE-16306:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s
{color} | {color:green} master passed with JDK v1.7.0_25 {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed with JDK v1.7.0_25 {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 42s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 9s
{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 45s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821058/HBASE-16306.master.1.patch
|
| JIRA Issue | HBASE-16306 |
| Optional Tests | asflicense scalac scaladoc unit compile |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / fe44383 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/2826/testReport/ |
| modules | C: hbase-spark U: hbase-spark |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/2826/console |
| Powered by | Apache Yetus 0.2.1 http://yetus.apache.org |
This message was automatically generated.
> Add specific imports to avoid namespace clash in defaultSource.scala
>
>
> Key: HBASE-16306
> URL: https://issues.apache.org/jira/browse/HBASE-16306
> Project: HBase
> Issue Type: Bug
>Reporter: Sai Teja Ranuva
>Assignee: Sai Teja Ranuva
>Priority: Minor
> Attachments: HBASE-16306.master.1.patch
>
>
> I am working on adding Hybrid Logical clocks to HBase. As part of it, I wish
> to add TimestampType file in hbase common. Spark has some types defined in -
> org.apache.spark.sql.types, all of which are being imported. It also has a
> TimestampType defined. As currently in this file, all of the hbase common is
> imported, creating a namespace clash. I have changed to only import specific
> hbase common classes that are required in this file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15656) Fix unused protobuf warning in Admin.proto
[
https://issues.apache.org/jira/browse/HBASE-15656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liang updated HBASE-15656:
-
Attachment: HBASE-15656-V1.patch
> Fix unused protobuf warning in Admin.proto
> --
>
> Key: HBASE-15656
> URL: https://issues.apache.org/jira/browse/HBASE-15656
> Project: HBase
> Issue Type: Task
>Reporter: Elliott Clark
>Assignee: Yi Liang
>Priority: Minor
> Attachments: HBASE-15656-V1.patch
>
>
> {code}
> Warning: Unused import: "Admin.proto" imports "Client.proto" which is not
> used.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14345) Consolidate printUsage in IntegrationTestLoadAndVerify
[
https://issues.apache.org/jira/browse/HBASE-14345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399927#comment-15399927
]
Appy commented on HBASE-14345:
--
+1
lets wait for a day or two to see if anyone things differently.
> Consolidate printUsage in IntegrationTestLoadAndVerify
> --
>
> Key: HBASE-14345
> URL: https://issues.apache.org/jira/browse/HBASE-14345
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Reporter: Nick Dimiduk
>Assignee: Reid Chan
>Priority: Trivial
> Labels: beginner
> Attachments: HBASE-14345.002.patch, HBASE-14345.patch,
> itlav-2016-07-07.png, itlv.png
>
>
> Investigating the use of {{itlav}} is a little screwy. Subclasses are not
> overriding the {{printUsage()}} methods correctly, so you have to pass
> {{--help}} to get some info and no arguments to get the rest.
> {noformat}
> [hbase@ndimiduk-112rc2-7 ~]$ hbase
> org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify --help
> usage: bin/hbase org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify
>
> Options:
> -h,--help Show usage
> -m,--monkey Which chaos monkey to run
> -monkeyProps The properties file for specifying chaos monkey
> properties.
> -ncc,--noClusterCleanUp Don't clean up the cluster at the end
> [hbase@ndimiduk-112rc2-7 ~]$ hbase
> org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify
> IntegrationTestLoadAndVerify [-Doptions]
> Loads a table with row dependencies and verifies the dependency chains
> Options
> -Dloadmapper.table=Table to write/verify (default autogen)
> -Dloadmapper.backrefs=Number of backreferences per row (default
> 50)
> -Dloadmapper.num_to_write=Number of rows per mapper (default 100,000
> per mapper)
> -Dloadmapper.deleteAfter= Delete after a successful verify (default
> true)
> -Dloadmapper.numPresplits=Number of presplit regions to start with
> (default 40)
> -Dloadmapper.map.tasks= Number of map tasks for load (default 200)
> -Dverify.reduce.tasks=Number of reduce tasks for verify (default
> 35)
> -Dverify.scannercaching= Number hbase scanner caching rows to read
> (default 50)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14345) Consolidate printUsage in IntegrationTestLoadAndVerify
[
https://issues.apache.org/jira/browse/HBASE-14345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399924#comment-15399924
]
Appy commented on HBASE-14345:
--
-D args are used up by GenericOptionsParser and then removed from args list
(see ToolRunner#run()). So they never make to processOptions(). A simple fix
what Reid has posted should suffice.
> Consolidate printUsage in IntegrationTestLoadAndVerify
> --
>
> Key: HBASE-14345
> URL: https://issues.apache.org/jira/browse/HBASE-14345
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Reporter: Nick Dimiduk
>Assignee: Reid Chan
>Priority: Trivial
> Labels: beginner
> Attachments: HBASE-14345.002.patch, HBASE-14345.patch,
> itlav-2016-07-07.png, itlv.png
>
>
> Investigating the use of {{itlav}} is a little screwy. Subclasses are not
> overriding the {{printUsage()}} methods correctly, so you have to pass
> {{--help}} to get some info and no arguments to get the rest.
> {noformat}
> [hbase@ndimiduk-112rc2-7 ~]$ hbase
> org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify --help
> usage: bin/hbase org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify
>
> Options:
> -h,--help Show usage
> -m,--monkey Which chaos monkey to run
> -monkeyProps The properties file for specifying chaos monkey
> properties.
> -ncc,--noClusterCleanUp Don't clean up the cluster at the end
> [hbase@ndimiduk-112rc2-7 ~]$ hbase
> org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify
> IntegrationTestLoadAndVerify [-Doptions]
> Loads a table with row dependencies and verifies the dependency chains
> Options
> -Dloadmapper.table=Table to write/verify (default autogen)
> -Dloadmapper.backrefs=Number of backreferences per row (default
> 50)
> -Dloadmapper.num_to_write=Number of rows per mapper (default 100,000
> per mapper)
> -Dloadmapper.deleteAfter= Delete after a successful verify (default
> true)
> -Dloadmapper.numPresplits=Number of presplit regions to start with
> (default 40)
> -Dloadmapper.map.tasks= Number of map tasks for load (default 200)
> -Dverify.reduce.tasks=Number of reduce tasks for verify (default
> 35)
> -Dverify.scannercaching= Number hbase scanner caching rows to read
> (default 50)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15656) Fix unused protobuf warning in Admin.proto
[
https://issues.apache.org/jira/browse/HBASE-15656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399923#comment-15399923
]
Yi Liang commented on HBASE-15656:
--
{code}
yis-macbook-pro:hbase-protocol yliang$ protoc -Isrc/main/protobuf
--java_out=src/main/java src/main/protobuf/Admin.proto
[libprotobuf WARNING google/protobuf/descriptor.cc:5411] Warning: Unused
import: "Admin.proto" imports "Client.proto" which is not used.
yis-macbook-pro:hbase-protocol yliang$ protoc -Isrc/main/protobuf
--java_out=src/main/java src/main/protobuf/HBase.proto
[libprotobuf WARNING google/protobuf/descriptor.cc:5411] Warning: Unused
import: "HBase.proto" imports "Cell.proto" which is not used.
yis-macbook-pro:hbase-protocol yliang$ protoc -Isrc/main/protobuf
--java_out=src/main/java src/main/protobuf/SecureBulkLoad.proto
[libprotobuf WARNING google/protobuf/descriptor.cc:5411] Warning: Unused
import: "SecureBulkLoad.proto" imports "HBase.proto" which is not used.
yis-macbook-pro:hbase-protocol yliang$ protoc -Isrc/main/protobuf
--java_out=src/main/java src/main/protobuf/WAL.proto
[libprotobuf WARNING google/protobuf/descriptor.cc:5411] Warning: Unused
import: "WAL.proto" imports "Client.proto" which is not used.
{code}
I have made changes for this 4 proto file
> Fix unused protobuf warning in Admin.proto
> --
>
> Key: HBASE-15656
> URL: https://issues.apache.org/jira/browse/HBASE-15656
> Project: HBase
> Issue Type: Task
>Reporter: Elliott Clark
>Priority: Minor
>
> {code}
> Warning: Unused import: "Admin.proto" imports "Client.proto" which is not
> used.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-15656) Fix unused protobuf warning in Admin.proto
[
https://issues.apache.org/jira/browse/HBASE-15656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liang reassigned HBASE-15656:
Assignee: Yi Liang
> Fix unused protobuf warning in Admin.proto
> --
>
> Key: HBASE-15656
> URL: https://issues.apache.org/jira/browse/HBASE-15656
> Project: HBase
> Issue Type: Task
>Reporter: Elliott Clark
>Assignee: Yi Liang
>Priority: Minor
>
> {code}
> Warning: Unused import: "Admin.proto" imports "Client.proto" which is not
> used.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16209) Provide an ExponentialBackOffPolicy sleep between failed region open requests
[ https://issues.apache.org/jira/browse/HBASE-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399912#comment-15399912 ] Hudson commented on HBASE-16209: FAILURE: Integrated in HBase-Trunk_matrix #1318 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1318/]) HBASE-16209 provide an ExponentialBackOffPolicy sleep between failed (eclark: rev fe44383619d0d11396382f2bfa493ad93c8b0e5f) * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java * hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/AssignmentManagerStatusTmpl.jamon * hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java > Provide an ExponentialBackOffPolicy sleep between failed region open requests > - > > Key: HBASE-16209 > URL: https://issues.apache.org/jira/browse/HBASE-16209 > Project: HBase > Issue Type: Bug >Reporter: Joseph >Assignee: Joseph > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16209-branch-1.patch, HBASE-16209-v2.patch, > HBASE-16209.patch > > > Related to HBASE-16138. As of now we currently have no pause between retrying > failed region open requests. And with a low maximumAttempt default, we can > quickly use up all our regionOpen retries if the server is in a bad state. I > added in a ExponentialBackOffPolicy so that we spread out the timing of our > open region retries in AssignmentManager. Review board at > https://reviews.apache.org/r/50011/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16306) Add specific imports to avoid namespace clash in defaultSource.scala
[ https://issues.apache.org/jira/browse/HBASE-16306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Teja Ranuva updated HBASE-16306: Description: I am working on adding Hybrid Logical clocks to HBase. As part of it, I wish to add TimestampType file in hbase common. Spark has some types defined in - org.apache.spark.sql.types, all of which are being imported. It also has a TimestampType defined. As currently in this file, all of the hbase common is imported, creating a namespace clash. I have changed to only import specific hbase common classes that are required in this file. (was: I am working on adding Hybrid Logical clocks to HBase. As part of it, I wish to add TimestampType file in hbase common. Spark has some types defined in - org.apache.spark.sql.types, all of which are being imported. It also has a TimestampType defined. As currently in this file, all of the hbase common is imported, creating a namespace clash. I have changed to imports to only specific hbase common classes that are required in this file.) > Add specific imports to avoid namespace clash in defaultSource.scala > > > Key: HBASE-16306 > URL: https://issues.apache.org/jira/browse/HBASE-16306 > Project: HBase > Issue Type: Bug >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Minor > Attachments: HBASE-16306.master.1.patch > > > I am working on adding Hybrid Logical clocks to HBase. As part of it, I wish > to add TimestampType file in hbase common. Spark has some types defined in - > org.apache.spark.sql.types, all of which are being imported. It also has a > TimestampType defined. As currently in this file, all of the hbase common is > imported, creating a namespace clash. I have changed to only import specific > hbase common classes that are required in this file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16306) Add specific imports to avoid namespace clash in defaultSource.scala
[ https://issues.apache.org/jira/browse/HBASE-16306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Teja Ranuva updated HBASE-16306: Attachment: HBASE-16306.master.1.patch > Add specific imports to avoid namespace clash in defaultSource.scala > > > Key: HBASE-16306 > URL: https://issues.apache.org/jira/browse/HBASE-16306 > Project: HBase > Issue Type: Bug >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Minor > Attachments: HBASE-16306.master.1.patch > > > I am working on adding Hybrid Logical clocks to HBase. As part of it, I wish > to add TimestampType file in hbase common. Spark has some types defined in - > org.apache.spark.sql.types, all of which are being imported. It also has a > TimestampType defined. As currently in this file, all of the hbase common is > imported, creating a namespace clash. I have changed to imports to only > specific hbase common classes that are required in this file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16306) Add specific imports to avoid namespace clash in defaultSource.scala
[ https://issues.apache.org/jira/browse/HBASE-16306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Teja Ranuva updated HBASE-16306: Status: Patch Available (was: Open) > Add specific imports to avoid namespace clash in defaultSource.scala > > > Key: HBASE-16306 > URL: https://issues.apache.org/jira/browse/HBASE-16306 > Project: HBase > Issue Type: Bug >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Minor > Attachments: HBASE-16306.master.1.patch > > > I am working on adding Hybrid Logical clocks to HBase. As part of it, I wish > to add TimestampType file in hbase common. Spark has some types defined in - > org.apache.spark.sql.types, all of which are being imported. It also has a > TimestampType defined. As currently in this file, all of the hbase common is > imported, creating a namespace clash. I have changed to imports to only > specific hbase common classes that are required in this file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16306) Add specific imports to avoid namespace clash in defaultSource.scala
Sai Teja Ranuva created HBASE-16306: --- Summary: Add specific imports to avoid namespace clash in defaultSource.scala Key: HBASE-16306 URL: https://issues.apache.org/jira/browse/HBASE-16306 Project: HBase Issue Type: Bug Reporter: Sai Teja Ranuva Assignee: Sai Teja Ranuva Priority: Minor I am working on adding Hybrid Logical clocks to HBase. As part of it, I wish to add TimestampType file in hbase common. Spark has some types defined in - org.apache.spark.sql.types, all of which are being imported. It also has a TimestampType defined. As currently in this file, all of the hbase common is imported, creating a namespace clash. I have changed to imports to only specific hbase common classes that are required in this file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16305) Multi threaded scan performance bottlenecked on Connection
[ https://issues.apache.org/jira/browse/HBASE-16305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399836#comment-15399836 ] Jerry He commented on HBASE-16305: -- Is it re-creatable via the PerformanceEvaluation tool? There is a '--oneCon=true or false' option for multi-threaded runs. > Multi threaded scan performance bottlenecked on Connection > -- > > Key: HBASE-16305 > URL: https://issues.apache.org/jira/browse/HBASE-16305 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark > > Pretty simple repro. On a single region server host ~10 or more regions. Then > for each region spawn a thread that starts a scan. Try an pull the scan as > fast as possible. > If all the scans are created through the same connection then it's slow. If > all the scans are created through different connections then the scan speed > is doubled. > Since we recommend a single connection per application this is pretty > surprising behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16300) LruBlockCache.CACHE_FIXED_OVERHEAD should calculate LruBlockCache size correctly
[
https://issues.apache.org/jira/browse/HBASE-16300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399768#comment-15399768
]
Hudson commented on HBASE-16300:
FAILURE: Integrated in HBase-1.4 #313 (See
[https://builds.apache.org/job/HBase-1.4/313/])
HBASE-16300 LruBlockCache.CACHE_FIXED_OVERHEAD should calculate (liyu: rev
08b9e6bee0a21b0af39a378faab5567beae1cdac)
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
> LruBlockCache.CACHE_FIXED_OVERHEAD should calculate LruBlockCache size
> correctly
>
>
> Key: HBASE-16300
> URL: https://issues.apache.org/jira/browse/HBASE-16300
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Sun
>Assignee: Yu Sun
> Fix For: 2.0.0, 1.3.0, 0.98.21, 1.2.3
>
> Attachments: HBASE-16300-v1.patch
>
>
> in current master {{LruBlockCache}}, CACHE_FIXED_OVERHEAD is calculated as
> this:
> {code}
> public final static long CACHE_FIXED_OVERHEAD = ClassSize.align(
> (3 * Bytes.SIZEOF_LONG) + (10 * ClassSize.REFERENCE) +
> (5 * Bytes.SIZEOF_FLOAT) + (2 * Bytes.SIZEOF_BOOLEAN)
> + ClassSize.OBJECT);
> {code}
> after some investigation. I think there are some wrong here, {{class
> LruBlockCache}}, except static varible(which is belongs to class), there are
> 4 long varibles(maxBlockSize,maxSize,blockSize and overhead), 9 reference
> varibles and 2 boolean varibles, so the above code will not calculate
> LruBlockCache instance size correctly.
> current related ut not failed mostly due to the result is 8 bytes aligned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16289) AsyncProcess stuck messages need to print region/server
[ https://issues.apache.org/jira/browse/HBASE-16289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399769#comment-15399769 ] Hudson commented on HBASE-16289: FAILURE: Integrated in HBase-1.4 #313 (See [https://builds.apache.org/job/HBase-1.4/313/]) HBASE-16289 AsyncProcess stuck messages need to print region/server (liyu: rev 96d8dcb6f13a252a1f2f259065b558f460eb8100) * hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java > AsyncProcess stuck messages need to print region/server > --- > > Key: HBASE-16289 > URL: https://issues.apache.org/jira/browse/HBASE-16289 > Project: HBase > Issue Type: Bug > Components: Operability >Reporter: stack >Assignee: Yu Li >Priority: Critical > Labels: beginner > Fix For: 2.0.0, 1.3.0, 1.1.6, 1.2.3 > > Attachments: HBASE-16289.patch, HBASE-16289.patch > > > Fix these out of AsyncProcess... > '2016-07-21 20:16:56.477 o.a.h.h.c.AsyncProcess [INFO] #13, waiting for 25 > actions to finish' > They are totally opaque, when you see one, it means a request is ten seconds > old at least. It is awkward to do but we should be able to emit region/server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16209) Provide an ExponentialBackOffPolicy sleep between failed region open requests
[ https://issues.apache.org/jira/browse/HBASE-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399752#comment-15399752 ] Joseph commented on HBASE-16209: Oh sorry, I must have overlooked that. I'll attach an addendum addressing this. > Provide an ExponentialBackOffPolicy sleep between failed region open requests > - > > Key: HBASE-16209 > URL: https://issues.apache.org/jira/browse/HBASE-16209 > Project: HBase > Issue Type: Bug >Reporter: Joseph >Assignee: Joseph > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16209-branch-1.patch, HBASE-16209-v2.patch, > HBASE-16209.patch > > > Related to HBASE-16138. As of now we currently have no pause between retrying > failed region open requests. And with a low maximumAttempt default, we can > quickly use up all our regionOpen retries if the server is in a bad state. I > added in a ExponentialBackOffPolicy so that we spread out the timing of our > open region retries in AssignmentManager. Review board at > https://reviews.apache.org/r/50011/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-16209) Provide an ExponentialBackOffPolicy sleep between failed region open requests
[ https://issues.apache.org/jira/browse/HBASE-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph reopened HBASE-16209: > Provide an ExponentialBackOffPolicy sleep between failed region open requests > - > > Key: HBASE-16209 > URL: https://issues.apache.org/jira/browse/HBASE-16209 > Project: HBase > Issue Type: Bug >Reporter: Joseph >Assignee: Joseph > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16209-branch-1.patch, HBASE-16209-v2.patch, > HBASE-16209.patch > > > Related to HBASE-16138. As of now we currently have no pause between retrying > failed region open requests. And with a low maximumAttempt default, we can > quickly use up all our regionOpen retries if the server is in a bad state. I > added in a ExponentialBackOffPolicy so that we spread out the timing of our > open region retries in AssignmentManager. Review board at > https://reviews.apache.org/r/50011/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16209) Provide an ExponentialBackOffPolicy sleep between failed region open requests
[
https://issues.apache.org/jira/browse/HBASE-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399748#comment-15399748
]
Matteo Bertozzi commented on HBASE-16209:
-
sorry, late for the review.. but why are we doing another lookup for the region
state? we are already looping on the region state... so the new "state" that we
are looking up should be the same to the "rs" we are looping on, no?
{code}
<%for RegionState rs : rit %>
+ String name = rs.getRegion().getEncodedName();
+ RegionState state = assignmentManager.getState(name);
+ AtomicInteger numOpenRetries =
failedRegionTracker.get(name);
{code}
> Provide an ExponentialBackOffPolicy sleep between failed region open requests
> -
>
> Key: HBASE-16209
> URL: https://issues.apache.org/jira/browse/HBASE-16209
> Project: HBase
> Issue Type: Bug
>Reporter: Joseph
>Assignee: Joseph
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16209-branch-1.patch, HBASE-16209-v2.patch,
> HBASE-16209.patch
>
>
> Related to HBASE-16138. As of now we currently have no pause between retrying
> failed region open requests. And with a low maximumAttempt default, we can
> quickly use up all our regionOpen retries if the server is in a bad state. I
> added in a ExponentialBackOffPolicy so that we spread out the timing of our
> open region retries in AssignmentManager. Review board at
> https://reviews.apache.org/r/50011/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16305) Multi threaded scan performance bottlenecked on Connection
Elliott Clark created HBASE-16305: - Summary: Multi threaded scan performance bottlenecked on Connection Key: HBASE-16305 URL: https://issues.apache.org/jira/browse/HBASE-16305 Project: HBase Issue Type: Bug Reporter: Elliott Clark Pretty simple repro. On a single region server host ~10 or more regions. Then for each region spawn a thread that starts a scan. Try an pull the scan as fast as possible. If all the scans are created through the same connection then it's slow. If all the scans are created through different connections then the scan speed is doubled. Since we recommend a single connection per application this is pretty surprising behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16300) LruBlockCache.CACHE_FIXED_OVERHEAD should calculate LruBlockCache size correctly
[
https://issues.apache.org/jira/browse/HBASE-16300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399713#comment-15399713
]
Hudson commented on HBASE-16300:
FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1250 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1250/])
HBASE-16300 LruBlockCache.CACHE_FIXED_OVERHEAD should calculate (liyu: rev
27c583467e53cb52667eff45afabb1ffb8547be5)
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
> LruBlockCache.CACHE_FIXED_OVERHEAD should calculate LruBlockCache size
> correctly
>
>
> Key: HBASE-16300
> URL: https://issues.apache.org/jira/browse/HBASE-16300
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Sun
>Assignee: Yu Sun
> Fix For: 2.0.0, 1.3.0, 0.98.21, 1.2.3
>
> Attachments: HBASE-16300-v1.patch
>
>
> in current master {{LruBlockCache}}, CACHE_FIXED_OVERHEAD is calculated as
> this:
> {code}
> public final static long CACHE_FIXED_OVERHEAD = ClassSize.align(
> (3 * Bytes.SIZEOF_LONG) + (10 * ClassSize.REFERENCE) +
> (5 * Bytes.SIZEOF_FLOAT) + (2 * Bytes.SIZEOF_BOOLEAN)
> + ClassSize.OBJECT);
> {code}
> after some investigation. I think there are some wrong here, {{class
> LruBlockCache}}, except static varible(which is belongs to class), there are
> 4 long varibles(maxBlockSize,maxSize,blockSize and overhead), 9 reference
> varibles and 2 boolean varibles, so the above code will not calculate
> LruBlockCache instance size correctly.
> current related ut not failed mostly due to the result is 8 bytes aligned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16300) LruBlockCache.CACHE_FIXED_OVERHEAD should calculate LruBlockCache size correctly
[
https://issues.apache.org/jira/browse/HBASE-16300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399695#comment-15399695
]
Hudson commented on HBASE-16300:
FAILURE: Integrated in HBase-0.98-matrix #378 (See
[https://builds.apache.org/job/HBase-0.98-matrix/378/])
HBASE-16300 LruBlockCache.CACHE_FIXED_OVERHEAD should calculate (liyu: rev
27c583467e53cb52667eff45afabb1ffb8547be5)
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
> LruBlockCache.CACHE_FIXED_OVERHEAD should calculate LruBlockCache size
> correctly
>
>
> Key: HBASE-16300
> URL: https://issues.apache.org/jira/browse/HBASE-16300
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Sun
>Assignee: Yu Sun
> Fix For: 2.0.0, 1.3.0, 0.98.21, 1.2.3
>
> Attachments: HBASE-16300-v1.patch
>
>
> in current master {{LruBlockCache}}, CACHE_FIXED_OVERHEAD is calculated as
> this:
> {code}
> public final static long CACHE_FIXED_OVERHEAD = ClassSize.align(
> (3 * Bytes.SIZEOF_LONG) + (10 * ClassSize.REFERENCE) +
> (5 * Bytes.SIZEOF_FLOAT) + (2 * Bytes.SIZEOF_BOOLEAN)
> + ClassSize.OBJECT);
> {code}
> after some investigation. I think there are some wrong here, {{class
> LruBlockCache}}, except static varible(which is belongs to class), there are
> 4 long varibles(maxBlockSize,maxSize,blockSize and overhead), 9 reference
> varibles and 2 boolean varibles, so the above code will not calculate
> LruBlockCache instance size correctly.
> current related ut not failed mostly due to the result is 8 bytes aligned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16301) Trigger flush without waiting when compaction is disabled on a table
[ https://issues.apache.org/jira/browse/HBASE-16301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399661#comment-15399661 ] huaxiang sun commented on HBASE-16301: -- Thanks [~tedyu], I am revisiting to see if it is possible to add some assert. > Trigger flush without waiting when compaction is disabled on a table > > > Key: HBASE-16301 > URL: https://issues.apache.org/jira/browse/HBASE-16301 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Attachments: HBASE-16301-v001.patch > > > When compaction is disabled on a table, flush needs to wait > MemStoreFlusher#blockingWaitTime (default value is 90 seconds) before it goes > ahead to flush. It has side effect that client may be blocked due to > RegionTooBusyException. Please see the mail sent to dev list. > http://mail-archives.apache.org/mod_mbox/hbase-dev/201607.mbox/%3c2d66b8ca-7c6f-40ea-a861-2de5482ec...@cloudera.com%3E > I guess that the right behavior is to do flush without waiting if compaction > is disabled on a table. Attached a patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16209) Provide an ExponentialBackOffPolicy sleep between failed region open requests
[ https://issues.apache.org/jira/browse/HBASE-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-16209: -- Resolution: Fixed Fix Version/s: 1.4.0 2.0.0 Status: Resolved (was: Patch Available) > Provide an ExponentialBackOffPolicy sleep between failed region open requests > - > > Key: HBASE-16209 > URL: https://issues.apache.org/jira/browse/HBASE-16209 > Project: HBase > Issue Type: Bug >Reporter: Joseph >Assignee: Joseph > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16209-branch-1.patch, HBASE-16209-v2.patch, > HBASE-16209.patch > > > Related to HBASE-16138. As of now we currently have no pause between retrying > failed region open requests. And with a low maximumAttempt default, we can > quickly use up all our regionOpen retries if the server is in a bad state. I > added in a ExponentialBackOffPolicy so that we spread out the timing of our > open region retries in AssignmentManager. Review board at > https://reviews.apache.org/r/50011/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
