[jira] [Updated] (HBASE-17182) Memory leak from openScanner of HBase thrift2
[ https://issues.apache.org/jira/browse/HBASE-17182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian Yi updated HBASE-17182: Assignee: Jian Yi > Memory leak from openScanner of HBase thrift2 > - > > Key: HBASE-17182 > URL: https://issues.apache.org/jira/browse/HBASE-17182 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Jian Yi >Assignee: Jian Yi > > Client call openScanner, but client (coredump or others) not closeScanner, > the scanner will not be removed from scannerMap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17262) Refactor RpcServer so as to make it extendable and/or pluggable
[ https://issues.apache.org/jira/browse/HBASE-17262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-17262: - Assignee: binlijin Fix Version/s: 2.0.0 > Refactor RpcServer so as to make it extendable and/or pluggable > --- > > Key: HBASE-17262 > URL: https://issues.apache.org/jira/browse/HBASE-17262 > Project: HBase > Issue Type: Sub-task > Components: Performance, rpc >Reporter: binlijin >Assignee: binlijin > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17263) Netty based rpc server impl
[ https://issues.apache.org/jira/browse/HBASE-17263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-17263: - Assignee: binlijin Fix Version/s: 2.0.0 > Netty based rpc server impl > - > > Key: HBASE-17263 > URL: https://issues.apache.org/jira/browse/HBASE-17263 > Project: HBase > Issue Type: Sub-task > Components: Performance, rpc >Reporter: binlijin >Assignee: binlijin > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15756) Pluggable RpcServer
[ https://issues.apache.org/jira/browse/HBASE-15756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15756: --- Fix Version/s: 2.0.0 > Pluggable RpcServer > --- > > Key: HBASE-15756 > URL: https://issues.apache.org/jira/browse/HBASE-15756 > Project: HBase > Issue Type: Improvement > Components: Performance, rpc >Reporter: binlijin >Assignee: binlijin >Priority: Critical > Fix For: 2.0.0 > > Attachments: Netty4RpcServer_forperf.patch, NettyRpcServer.patch, > NettyRpcServer_forperf.patch, PooledByteBufAllocator.patch, > PooledByteBufAllocator2.patch, gc.png, gets.png, gets.png, idle.png, > patched.vs.patched_and_cached.vs.no_patch.png, queue.png > > > Current we use a simple RpcServer, and can not configure and use other > implementation.This issue is to make the RpcServer pluggable, so we can make > other implementation for example netty rpc server. Patch will upload laterly -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17182) Memory leak from openScanner of HBase thrift2
[ https://issues.apache.org/jira/browse/HBASE-17182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724675#comment-15724675 ] Phil Yang commented on HBASE-17182: --- Any concerns not use Guava's cache? Thanks. > Memory leak from openScanner of HBase thrift2 > - > > Key: HBASE-17182 > URL: https://issues.apache.org/jira/browse/HBASE-17182 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Jian Yi > > Client call openScanner, but client (coredump or others) not closeScanner, > the scanner will not be removed from scannerMap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15756) Pluggable RpcServer
[ https://issues.apache.org/jira/browse/HBASE-15756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724630#comment-15724630 ] binlijin commented on HBASE-15756: -- Thanks, man! > Pluggable RpcServer > --- > > Key: HBASE-15756 > URL: https://issues.apache.org/jira/browse/HBASE-15756 > Project: HBase > Issue Type: Improvement > Components: Performance, rpc >Reporter: binlijin >Assignee: binlijin >Priority: Critical > Attachments: Netty4RpcServer_forperf.patch, NettyRpcServer.patch, > NettyRpcServer_forperf.patch, PooledByteBufAllocator.patch, > PooledByteBufAllocator2.patch, gc.png, gets.png, gets.png, idle.png, > patched.vs.patched_and_cached.vs.no_patch.png, queue.png > > > Current we use a simple RpcServer, and can not configure and use other > implementation.This issue is to make the RpcServer pluggable, so we can make > other implementation for example netty rpc server. Patch will upload laterly -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17263) Netty based rpc server impl
binlijin created HBASE-17263: Summary: Netty based rpc server impl Key: HBASE-17263 URL: https://issues.apache.org/jira/browse/HBASE-17263 Project: HBase Issue Type: Sub-task Reporter: binlijin -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15756) Pluggable RpcServer
[ https://issues.apache.org/jira/browse/HBASE-15756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724616#comment-15724616 ] ramkrishna.s.vasudevan commented on HBASE-15756: Yes. It can be done with offheap because from the netty buffers we tend to copy the requests to onheap buffers. Instead now you will need to just copy to the buffers from the pool. seeing the private impl of [~aoxiang] - that is what was being done. So should work fine along with the benefits that they have seen with netty by running in prod for 2 months. > Pluggable RpcServer > --- > > Key: HBASE-15756 > URL: https://issues.apache.org/jira/browse/HBASE-15756 > Project: HBase > Issue Type: Improvement > Components: Performance, rpc >Reporter: binlijin >Assignee: binlijin >Priority: Critical > Attachments: Netty4RpcServer_forperf.patch, NettyRpcServer.patch, > NettyRpcServer_forperf.patch, PooledByteBufAllocator.patch, > PooledByteBufAllocator2.patch, gc.png, gets.png, gets.png, idle.png, > patched.vs.patched_and_cached.vs.no_patch.png, queue.png > > > Current we use a simple RpcServer, and can not configure and use other > implementation.This issue is to make the RpcServer pluggable, so we can make > other implementation for example netty rpc server. Patch will upload laterly -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17262) Refactor RpcServer so as to make it extendable and/or pluggable
binlijin created HBASE-17262: Summary: Refactor RpcServer so as to make it extendable and/or pluggable Key: HBASE-17262 URL: https://issues.apache.org/jira/browse/HBASE-17262 Project: HBase Issue Type: Sub-task Reporter: binlijin -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15756) Pluggable RpcServer
[ https://issues.apache.org/jira/browse/HBASE-15756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724589#comment-15724589 ] binlijin commented on HBASE-15756: -- We have use NettyRpcServer on production for two months in Alibaba search, and this have show big improvement in our cluster. Discuss with [~anoop.hbase] NettyRpcServer can work with write pipeline off heap, so start continue this work. 1. Refactoring in the RpcServer so as to make it extendable and/or pluggable 2. The new Rpc server impl which is netty based. > Pluggable RpcServer > --- > > Key: HBASE-15756 > URL: https://issues.apache.org/jira/browse/HBASE-15756 > Project: HBase > Issue Type: Improvement > Components: Performance, rpc >Reporter: binlijin >Assignee: binlijin >Priority: Critical > Attachments: Netty4RpcServer_forperf.patch, NettyRpcServer.patch, > NettyRpcServer_forperf.patch, PooledByteBufAllocator.patch, > PooledByteBufAllocator2.patch, gc.png, gets.png, gets.png, idle.png, > patched.vs.patched_and_cached.vs.no_patch.png, queue.png > > > Current we use a simple RpcServer, and can not configure and use other > implementation.This issue is to make the RpcServer pluggable, so we can make > other implementation for example netty rpc server. Patch will upload laterly -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16981) Expand Mob Compaction Partition policy from daily to weekly, monthly and beyond
[ https://issues.apache.org/jira/browse/HBASE-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724555#comment-15724555 ] Jingcheng Du commented on HBASE-16981: -- Thanks [~huaxiang]! I will take some time to review it. Thanks. > Expand Mob Compaction Partition policy from daily to weekly, monthly and > beyond > --- > > Key: HBASE-16981 > URL: https://issues.apache.org/jira/browse/HBASE-16981 > Project: HBase > Issue Type: New Feature > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-16981.master.001.patch, > HBASE-16981.master.002.patch, > Supportingweeklyandmonthlymobcompactionpartitionpolicyinhbase.pdf > > > Today the mob region holds all mob files for all regions. With daily > partition mob compaction policy, after major mob compaction, there is still > one file per region daily. Given there is 365 days in one year, at least 365 > files per region. Since HDFS has limitation for number of files under one > folder, this is not going to scale if there are lots of regions. To reduce > mob file number, we want to introduce other partition policies such as > weekly, monthly to compact mob files within one week or month into one file. > This jira is create to track this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17249) Get/Scan's setTimeRange/setColumnFamilyTimeRange can take the TimeRange reference as the parameter instead of creating a new setColumnFamilyTimeRange instance
[
https://issues.apache.org/jira/browse/HBASE-17249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724549#comment-15724549
]
Anoop Sam John commented on HBASE-17249:
{code}
@Override public Scan setColumnFamilyTimeRange(byte[] cf, TimeRange tr) {
375 return (Scan) super.setColumnFamilyTimeRange(cf, tr);
376 }
{code}
Pls fix the formatting. I can see in another place also. Normally we will put
@Override above the public... method name line
Just noticing that we dont have setTimeRange in Query. But not related to
this patch. if ok, can fix that also here. Just like
setColumnFamilyTimeRange. But I leave it to u.
> Get/Scan's setTimeRange/setColumnFamilyTimeRange can take the TimeRange
> reference as the parameter instead of creating a new setColumnFamilyTimeRange
> instance
> --
>
> Key: HBASE-17249
> URL: https://issues.apache.org/jira/browse/HBASE-17249
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Attachments: HBASE-17249-master-001.patch,
> HBASE-17249-master-002.patch
>
>
> Going through the code, found For Get/Scan's
> setTimeRange/setColumnFamilyTimeRange, it can use TimeRange as reference
> instead of creating a new one.
> Reference:
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java#L500
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java#L506
> We can implement this in a similar way as filter:
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java#L510
> I checked it is same with branch-1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17261) Balancer makes no sense on tip of branch-1: says balanced when not
[
https://issues.apache.org/jira/browse/HBASE-17261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724490#comment-15724490
]
Guanghao Zhang commented on HBASE-17261:
bq. currently on 1.2
branch-1.2 ? But HBASE-15529 was only merged to branch-1 and master. So
branch-1.2 should not has this problem.
> Balancer makes no sense on tip of branch-1: says balanced when not
> --
>
> Key: HBASE-17261
> URL: https://issues.apache.org/jira/browse/HBASE-17261
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: Guanghao Zhang
> Attachments: HBASE-17261.patch
>
>
> Running ITBLL on tip of branch-1, I see this in log when I try to balance:
> {code}
> 2016-12-05 16:42:21,031 INFO
> [RpcServer.deafult.FPBQ.Fifo.handler=46,queue=1,port=16000]
> balancer.StochasticLoadBalancer: Skipping load balancing because balanced
> cluster; total cost is 525.2547686174673|
> , sum multiplier is 111087.0 min cost which need balance is 0.05
> {code}
> Its some old nonsense.
> Does this every time I balance. Can't even force a balance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17239) Add UnsafeByteOperations#wrap(ByteInput, int offset, int len) API
[ https://issues.apache.org/jira/browse/HBASE-17239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724482#comment-15724482 ] ramkrishna.s.vasudevan commented on HBASE-17239: Yes will do it then. Thanks Stack. > Add UnsafeByteOperations#wrap(ByteInput, int offset, int len) API > - > > Key: HBASE-17239 > URL: https://issues.apache.org/jira/browse/HBASE-17239 > Project: HBase > Issue Type: Improvement > Components: Protobufs >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17239.addendum.patch, HBASE-17239_V1.patch, > HBASE-17239_V2.patch, HBASE-17239_V3.patch > > > Instead of exposing the newInstance(ByteInput, boolean) in CIS we could just > add a UnsafeByteOperations#wrap(ByteInput, offset, len). And we could just > call that and do a #newcodedInput() over that. So internally we do return a > immutable version of the ByteInput only. This way we can avoid > CIS#newInstance(ByteInput) exposure and can keep it package private as done > in COS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17261) Balancer makes no sense on tip of branch-1: says balanced when not
[
https://issues.apache.org/jira/browse/HBASE-17261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724474#comment-15724474
]
stack commented on HBASE-17261:
---
Let me try this in a few days... currently on 1.2 testing. Thank you for
posting patch [~zghaobac]
> Balancer makes no sense on tip of branch-1: says balanced when not
> --
>
> Key: HBASE-17261
> URL: https://issues.apache.org/jira/browse/HBASE-17261
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: Guanghao Zhang
> Attachments: HBASE-17261.patch
>
>
> Running ITBLL on tip of branch-1, I see this in log when I try to balance:
> {code}
> 2016-12-05 16:42:21,031 INFO
> [RpcServer.deafult.FPBQ.Fifo.handler=46,queue=1,port=16000]
> balancer.StochasticLoadBalancer: Skipping load balancing because balanced
> cluster; total cost is 525.2547686174673|
> , sum multiplier is 111087.0 min cost which need balance is 0.05
> {code}
> Its some old nonsense.
> Does this every time I balance. Can't even force a balance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17261) Balancer makes no sense on tip of branch-1: says balanced when not
[
https://issues.apache.org/jira/browse/HBASE-17261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724470#comment-15724470
]
Guanghao Zhang commented on HBASE-17261:
{code}
private static final String REGION_REPLICA_HOST_COST_KEY =
"hbase.master.balancer.stochastic.regionReplicaHostCostKey";
private static final float DEFAULT_REGION_REPLICA_HOST_COST_KEY = 10;
{code}
The default region replica cost multiplier is too big and it has the most
weight in total cost. So when replica cost is small, it can't balance. Upload a
patch for this.
> Balancer makes no sense on tip of branch-1: says balanced when not
> --
>
> Key: HBASE-17261
> URL: https://issues.apache.org/jira/browse/HBASE-17261
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: Guanghao Zhang
> Attachments: HBASE-17261.patch
>
>
> Running ITBLL on tip of branch-1, I see this in log when I try to balance:
> {code}
> 2016-12-05 16:42:21,031 INFO
> [RpcServer.deafult.FPBQ.Fifo.handler=46,queue=1,port=16000]
> balancer.StochasticLoadBalancer: Skipping load balancing because balanced
> cluster; total cost is 525.2547686174673|
> , sum multiplier is 111087.0 min cost which need balance is 0.05
> {code}
> Its some old nonsense.
> Does this every time I balance. Can't even force a balance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17261) Balancer makes no sense on tip of branch-1: says balanced when not
[
https://issues.apache.org/jira/browse/HBASE-17261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-17261:
---
Attachment: HBASE-17261.patch
> Balancer makes no sense on tip of branch-1: says balanced when not
> --
>
> Key: HBASE-17261
> URL: https://issues.apache.org/jira/browse/HBASE-17261
> Project: HBase
> Issue Type: Bug
>Reporter: stack
> Attachments: HBASE-17261.patch
>
>
> Running ITBLL on tip of branch-1, I see this in log when I try to balance:
> {code}
> 2016-12-05 16:42:21,031 INFO
> [RpcServer.deafult.FPBQ.Fifo.handler=46,queue=1,port=16000]
> balancer.StochasticLoadBalancer: Skipping load balancing because balanced
> cluster; total cost is 525.2547686174673|
> , sum multiplier is 111087.0 min cost which need balance is 0.05
> {code}
> Its some old nonsense.
> Does this every time I balance. Can't even force a balance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-17261) Balancer makes no sense on tip of branch-1: says balanced when not
[
https://issues.apache.org/jira/browse/HBASE-17261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang reassigned HBASE-17261:
--
Assignee: Guanghao Zhang
> Balancer makes no sense on tip of branch-1: says balanced when not
> --
>
> Key: HBASE-17261
> URL: https://issues.apache.org/jira/browse/HBASE-17261
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: Guanghao Zhang
> Attachments: HBASE-17261.patch
>
>
> Running ITBLL on tip of branch-1, I see this in log when I try to balance:
> {code}
> 2016-12-05 16:42:21,031 INFO
> [RpcServer.deafult.FPBQ.Fifo.handler=46,queue=1,port=16000]
> balancer.StochasticLoadBalancer: Skipping load balancing because balanced
> cluster; total cost is 525.2547686174673|
> , sum multiplier is 111087.0 min cost which need balance is 0.05
> {code}
> Its some old nonsense.
> Does this every time I balance. Can't even force a balance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17261) Balancer makes no sense on tip of branch-1: says balanced when not
[
https://issues.apache.org/jira/browse/HBASE-17261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-17261:
---
Status: Patch Available (was: Open)
> Balancer makes no sense on tip of branch-1: says balanced when not
> --
>
> Key: HBASE-17261
> URL: https://issues.apache.org/jira/browse/HBASE-17261
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: Guanghao Zhang
> Attachments: HBASE-17261.patch
>
>
> Running ITBLL on tip of branch-1, I see this in log when I try to balance:
> {code}
> 2016-12-05 16:42:21,031 INFO
> [RpcServer.deafult.FPBQ.Fifo.handler=46,queue=1,port=16000]
> balancer.StochasticLoadBalancer: Skipping load balancing because balanced
> cluster; total cost is 525.2547686174673|
> , sum multiplier is 111087.0 min cost which need balance is 0.05
> {code}
> Its some old nonsense.
> Does this every time I balance. Can't even force a balance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17261) Balancer makes no sense on tip of branch-1: says balanced when not
[
https://issues.apache.org/jira/browse/HBASE-17261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724439#comment-15724439
]
stack commented on HBASE-17261:
---
All default configs regards balancer. Thanks.
> Balancer makes no sense on tip of branch-1: says balanced when not
> --
>
> Key: HBASE-17261
> URL: https://issues.apache.org/jira/browse/HBASE-17261
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>
> Running ITBLL on tip of branch-1, I see this in log when I try to balance:
> {code}
> 2016-12-05 16:42:21,031 INFO
> [RpcServer.deafult.FPBQ.Fifo.handler=46,queue=1,port=16000]
> balancer.StochasticLoadBalancer: Skipping load balancing because balanced
> cluster; total cost is 525.2547686174673|
> , sum multiplier is 111087.0 min cost which need balance is 0.05
> {code}
> Its some old nonsense.
> Does this every time I balance. Can't even force a balance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17239) Add UnsafeByteOperations#wrap(ByteInput, int offset, int len) API
[ https://issues.apache.org/jira/browse/HBASE-17239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724437#comment-15724437 ] stack commented on HBASE-17239: --- None yet. Do this simple one first I'd say. Should go in smooth enough. > Add UnsafeByteOperations#wrap(ByteInput, int offset, int len) API > - > > Key: HBASE-17239 > URL: https://issues.apache.org/jira/browse/HBASE-17239 > Project: HBase > Issue Type: Improvement > Components: Protobufs >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17239.addendum.patch, HBASE-17239_V1.patch, > HBASE-17239_V2.patch, HBASE-17239_V3.patch > > > Instead of exposing the newInstance(ByteInput, boolean) in CIS we could just > add a UnsafeByteOperations#wrap(ByteInput, offset, len). And we could just > call that and do a #newcodedInput() over that. So internally we do return a > immutable version of the ByteInput only. This way we can avoid > CIS#newInstance(ByteInput) exposure and can keep it package private as done > in COS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15437) Response size calculated in RPCServer for warning tooLarge responses does NOT count CellScanner payload
[
https://issues.apache.org/jira/browse/HBASE-15437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry He updated HBASE-15437:
-
Attachment: HBASE-15437-v4.patch
v4 to fix the findbugs.
> Response size calculated in RPCServer for warning tooLarge responses does NOT
> count CellScanner payload
> ---
>
> Key: HBASE-15437
> URL: https://issues.apache.org/jira/browse/HBASE-15437
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC
>Reporter: deepankar
>Assignee: Jerry He
> Attachments: HBASE-15437-v1.patch, HBASE-15437-v2.patch,
> HBASE-15437-v3.patch, HBASE-15437-v4.patch, HBASE-15437.patch
>
>
> After HBASE-13158 where we respond back to RPCs with cells in the payload ,
> the protobuf response will just have the count the cells to read from
> payload, but there are set of features where we log warn in RPCServer
> whenever the response is tooLarge, but this size now is not considering the
> sizes of the cells in the PayloadCellScanner. Code form RPCServer
> {code}
> long responseSize = result.getSerializedSize();
> // log any RPC responses that are slower than the configured warn
> // response time or larger than configured warning size
> boolean tooSlow = (processingTime > warnResponseTime &&
> warnResponseTime > -1);
> boolean tooLarge = (responseSize > warnResponseSize && warnResponseSize
> > -1);
> if (tooSlow || tooLarge) {
> // when tagging, we let TooLarge trump TooSmall to keep output simple
> // note that large responses will often also be slow.
> logResponse(new Object[]{param},
> md.getName(), md.getName() + "(" + param.getClass().getName() +
> ")",
> (tooLarge ? "TooLarge" : "TooSlow"),
> status.getClient(), startTime, processingTime, qTime,
> responseSize);
> }
> {code}
> Should this feature be not supported any more or should we add a method to
> CellScanner or a new interface which returns the serialized size (but this
> might not include the compression codecs which might be used during response
> ?) Any other Idea this could be fixed ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17239) Add UnsafeByteOperations#wrap(ByteInput, int offset, int len) API
[ https://issues.apache.org/jira/browse/HBASE-17239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724427#comment-15724427 ] ramkrishna.s.vasudevan commented on HBASE-17239: Thanks for the addendum [~saint@gmail.com]. I thought the entire patch should again be copied to src/main/patches. I can try pushing this to upstream along with other pb changes that we are planning to push. Is there any PR already active for that? > Add UnsafeByteOperations#wrap(ByteInput, int offset, int len) API > - > > Key: HBASE-17239 > URL: https://issues.apache.org/jira/browse/HBASE-17239 > Project: HBase > Issue Type: Improvement > Components: Protobufs >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17239.addendum.patch, HBASE-17239_V1.patch, > HBASE-17239_V2.patch, HBASE-17239_V3.patch > > > Instead of exposing the newInstance(ByteInput, boolean) in CIS we could just > add a UnsafeByteOperations#wrap(ByteInput, offset, len). And we could just > call that and do a #newcodedInput() over that. So internally we do return a > immutable version of the ByteInput only. This way we can avoid > CIS#newInstance(ByteInput) exposure and can keep it package private as done > in COS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16941) FavoredNodes - Split/Merge code paths
[ https://issues.apache.org/jira/browse/HBASE-16941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724413#comment-15724413 ] Thiruvel Thirumoolan commented on HBASE-16941: -- Test failure is unrelated to this patch. Latest patch is HBASE-16941.master.014.patch > FavoredNodes - Split/Merge code paths > - > > Key: HBASE-16941 > URL: https://issues.apache.org/jira/browse/HBASE-16941 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-16941.master.001.patch, > HBASE-16941.master.002.patch, HBASE-16941.master.003.patch, > HBASE-16941.master.004.patch, HBASE-16941.master.005.patch, > HBASE-16941.master.006.patch, HBASE-16941.master.007.patch, > HBASE-16941.master.008.patch, HBASE-16941.master.009.patch, > HBASE-16941.master.010.patch, HBASE-16941.master.011.patch, > HBASE-16941.master.012.patch, HBASE-16941.master.013.patch, > HBASE-16941.master.014.patch > > > This jira is to deal with the split/merge logic discussed as part of > HBASE-15532. The design document can be seen at HBASE-15531. The specific > changes are: > Split and merged regions should inherit favored node information from parent > regions. For splits also include some randomness so even if there are > subsequent splits, the regions will be more or less distributed. For split, > we include 2 FN from the parent and generate one random node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17072) CPU usage starts to climb up to 90-100% when using G1GC; purge ThreadLocal usage
[
https://issues.apache.org/jira/browse/HBASE-17072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724412#comment-15724412
]
stack commented on HBASE-17072:
---
Master should be ok [~apurtell] I think given its atomic ref (which ever thread
does next read, 'wins'...Other has to reset the stream).
> CPU usage starts to climb up to 90-100% when using G1GC; purge ThreadLocal
> usage
>
>
> Key: HBASE-17072
> URL: https://issues.apache.org/jira/browse/HBASE-17072
> Project: HBase
> Issue Type: Bug
> Components: Performance, regionserver
>Affects Versions: 1.0.0, 2.0.0, 1.2.0
>Reporter: Eiichi Sato
>Assignee: Eiichi Sato
>Priority: Critical
> Fix For: 2.0.0, 1.4.0, 0.98.24
>
> Attachments: HBASE-17072-0.98.patch, HBASE-17072.branch-1.001.patch,
> HBASE-17072.master.001.patch, HBASE-17072.master.002.patch,
> HBASE-17072.master.003.patch, HBASE-17072.master.004.patch,
> HBASE-17072.master.005.patch, HBASE-17072.master.005.patch,
> disable-block-header-cache.patch, mat-threadlocals.png, mat-threads.png,
> metrics.png, slave1.svg, slave2.svg, slave3.svg, slave4.svg
>
>
> h5. Problem
> CPU usage of a region server in our CDH 5.4.5 cluster, at some point, starts
> to gradually get higher up to nearly 90-100% when using G1GC. We've also run
> into this problem on CDH 5.7.3 and CDH 5.8.2.
> In our production cluster, it normally takes a few weeks for this to happen
> after restarting a RS. We reproduced this on our test cluster and attached
> the results. Please note that, to make it easy to reproduce, we did some
> "anti-tuning" on a table when running tests.
> In metrics.png, soon after we started running some workloads against a test
> cluster (CDH 5.8.2) at about 7 p.m. CPU usage of the two RSs started to rise.
> Flame Graphs (slave1.svg to slave4.svg) are generated from jstack dumps of
> each RS process around 10:30 a.m. the next day.
> After investigating heapdumps from another occurrence on a test cluster
> running CDH 5.7.3, we found that the ThreadLocalMap contain a lot of
> contiguous entries of {{HFileBlock$PrefetchedHeader}} probably due to primary
> clustering. This caused more loops in
> {{ThreadLocalMap#expungeStaleEntries()}}, consuming a certain amount of CPU
> time. What is worse is that the method is called from RPC metrics code,
> which means even a small amount of per-RPC time soon adds up to a huge amount
> of CPU time.
> This is very similar to the issue in HBASE-16616, but we have many
> {{HFileBlock$PrefetchedHeader}} not only {{Counter$IndexHolder}} instances.
> Here are some OQL counts from Eclipse Memory Analyzer (MAT). This shows a
> number of ThreadLocal instances in the ThreadLocalMap of a single handler
> thread.
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name =
> "org.apache.hadoop.hbase.io.hfile.HFileBlock$PrefetchedHeader"
> #=> 10980 instances
> {code}
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name = "org.apache.hadoop.hbase.util.Counter$IndexHolder"
> #=> 2052 instances
> {code}
> Although as described in HBASE-16616 this somewhat seems to be an issue in
> G1GC side regarding weakly-reachable objects, we should keep ThreadLocal
> usage minimal and avoid creating an indefinite number (in this case, a number
> of HFiles) of ThreadLocal instances.
> HBASE-16146 removes ThreadLocals from the RPC metrics code. That may solve
> the issue (I just saw the patch, never tested it at all), but the
> {{HFileBlock$PrefetchedHeader}} are still there in the ThreadLocalMap, which
> may cause issues in the future again.
> h5. Our Solution
> We simply removed the whole {{HFileBlock$PrefetchedHeader}} caching and
> fortunately we didn't notice any performance degradation for our production
> workloads.
> Because the PrefetchedHeader caching uses ThreadLocal and because RPCs are
> handled randomly in any of the handlers, small Get or small Scan RPCs do not
> benefit from the caching (See HBASE-10676 and HBASE-11402 for the details).
> Probably, we need to see how well reads are saved by the caching for large
> Scan or Get RPCs and especially for compactions if we really remove the
> caching. It's probably better if we can remove ThreadLocals without breaking
> the current caching behavior.
> FWIW, I'm attaching the patch we applied. It's for CDH 5.4.5.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17239) Add UnsafeByteOperations#wrap(ByteInput, int offset, int len) API
[ https://issues.apache.org/jira/browse/HBASE-17239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724406#comment-15724406 ] stack commented on HBASE-17239: --- Want me to try and get this pushed upstream [~ram_krish] or you want to do it? > Add UnsafeByteOperations#wrap(ByteInput, int offset, int len) API > - > > Key: HBASE-17239 > URL: https://issues.apache.org/jira/browse/HBASE-17239 > Project: HBase > Issue Type: Improvement > Components: Protobufs >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17239.addendum.patch, HBASE-17239_V1.patch, > HBASE-17239_V2.patch, HBASE-17239_V3.patch > > > Instead of exposing the newInstance(ByteInput, boolean) in CIS we could just > add a UnsafeByteOperations#wrap(ByteInput, offset, len). And we could just > call that and do a #newcodedInput() over that. So internally we do return a > immutable version of the ByteInput only. This way we can avoid > CIS#newInstance(ByteInput) exposure and can keep it package private as done > in COS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17261) Balancer makes no sense on tip of branch-1: says balanced when not
[
https://issues.apache.org/jira/browse/HBASE-17261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724403#comment-15724403
]
Guanghao Zhang commented on HBASE-17261:
bq. sum multiplier is 111087.0
Did the cluster use all default config in StochasticLoadBalancer?
bq. What you think is up?
We have been used this in our cluster. But I thought the default value should
be zero. This config can be used only for some power user. I will upload a
patch for this.
> Balancer makes no sense on tip of branch-1: says balanced when not
> --
>
> Key: HBASE-17261
> URL: https://issues.apache.org/jira/browse/HBASE-17261
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>
> Running ITBLL on tip of branch-1, I see this in log when I try to balance:
> {code}
> 2016-12-05 16:42:21,031 INFO
> [RpcServer.deafult.FPBQ.Fifo.handler=46,queue=1,port=16000]
> balancer.StochasticLoadBalancer: Skipping load balancing because balanced
> cluster; total cost is 525.2547686174673|
> , sum multiplier is 111087.0 min cost which need balance is 0.05
> {code}
> Its some old nonsense.
> Does this every time I balance. Can't even force a balance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17261) Balancer makes no sense on tip of branch-1: says balanced when not
[
https://issues.apache.org/jira/browse/HBASE-17261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724377#comment-15724377
]
stack commented on HBASE-17261:
---
[~zghaobac] Thanks for taking a look. I didn't know how to interpret the above
log message. The numbers made no sense to me. My cluster was out-of-whack and
running the balancer even w/ force was doing nothing just spewing above log
line. I haven't tried taking a look to see what is going on. What you think is
up?
> Balancer makes no sense on tip of branch-1: says balanced when not
> --
>
> Key: HBASE-17261
> URL: https://issues.apache.org/jira/browse/HBASE-17261
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>
> Running ITBLL on tip of branch-1, I see this in log when I try to balance:
> {code}
> 2016-12-05 16:42:21,031 INFO
> [RpcServer.deafult.FPBQ.Fifo.handler=46,queue=1,port=16000]
> balancer.StochasticLoadBalancer: Skipping load balancing because balanced
> cluster; total cost is 525.2547686174673|
> , sum multiplier is 111087.0 min cost which need balance is 0.05
> {code}
> Its some old nonsense.
> Does this every time I balance. Can't even force a balance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16941) FavoredNodes - Split/Merge code paths
[
https://issues.apache.org/jira/browse/HBASE-16941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724357#comment-15724357
]
Hadoop QA commented on HBASE-16941:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 28m 42s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 4s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
37m 0s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 8s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 154m 9s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
48s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 246m 45s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.security.access.TestCoprocessorWhitelistMasterObserver |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12841861/HBASE-16941.master.014.patch
|
| JIRA Issue | HBASE-16941 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux e68774069b37 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / c7b8b63 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4804/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4804/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
[jira] [Commented] (HBASE-16981) Expand Mob Compaction Partition policy from daily to weekly, monthly and beyond
[ https://issues.apache.org/jira/browse/HBASE-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724343#comment-15724343 ] huaxiang sun commented on HBASE-16981: -- Hi [~jingcheng.du] and [~anoop.hbase], could you help to review the doc and provide your comments? Thanks! > Expand Mob Compaction Partition policy from daily to weekly, monthly and > beyond > --- > > Key: HBASE-16981 > URL: https://issues.apache.org/jira/browse/HBASE-16981 > Project: HBase > Issue Type: New Feature > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-16981.master.001.patch, > HBASE-16981.master.002.patch, > Supportingweeklyandmonthlymobcompactionpartitionpolicyinhbase.pdf > > > Today the mob region holds all mob files for all regions. With daily > partition mob compaction policy, after major mob compaction, there is still > one file per region daily. Given there is 365 days in one year, at least 365 > files per region. Since HDFS has limitation for number of files under one > folder, this is not going to scale if there are lots of regions. To reduce > mob file number, we want to introduce other partition policies such as > weekly, monthly to compact mob files within one week or month into one file. > This jira is create to track this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16940) Address review of "Backup/Restore (HBASE-7912, HBASE-14030, HBASE-14123) mega patch" posted on RB
[ https://issues.apache.org/jira/browse/HBASE-16940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16940: -- Attachment: HBASE-16940.HBASE-7912.v2.patch Recent https://reviews.apache.org/r/52748 patch v2. cc: [~te...@apache.org] > Address review of "Backup/Restore (HBASE-7912, HBASE-14030, HBASE-14123) mega > patch" posted on RB > -- > > Key: HBASE-16940 > URL: https://issues.apache.org/jira/browse/HBASE-16940 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-16940-v1.patch, HBASE-16940-v2.patch, > HBASE-16940.HBASE-7912.v2.patch, HBASE-16940.addendum, HBASE-16940.addendum.2 > > > Review 52748 remaining issues. > https://reviews.apache.org/r/52748 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16700) Allow for coprocessor whitelisting
[
https://issues.apache.org/jira/browse/HBASE-16700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724220#comment-15724220
]
Hudson commented on HBASE-16700:
SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2080 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/2080/])
HBASE-16700 Allow for coprocessor whitelisting (enis: rev
c7b8b63cd1fc19e3be722ae6c71791d04ef48b9d)
* (add)
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/CoprocessorWhitelistMasterObserver.java
* (edit) src/main/asciidoc/_chapters/cp.adoc
* (add)
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestCoprocessorWhitelistMasterObserver.java
> Allow for coprocessor whitelisting
> --
>
> Key: HBASE-16700
> URL: https://issues.apache.org/jira/browse/HBASE-16700
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors
>Reporter: Clay B.
>Assignee: Clay B.
>Priority: Minor
> Labels: security
> Fix For: 2.0.0
>
> Attachments: HBASE-16700.000.patch, HBASE-16700.001.patch,
> HBASE-16700.002.patch, HBASE-16700.003.patch, HBASE-16700.004.patch,
> HBASE-16700.005.patch, HBASE-16700.006.patch, HBASE-16700.007.patch,
> HBASE-16700.008.patch
>
>
> Today one can turn off all non-system coprocessors with
> {{hbase.coprocessor.user.enabled}} however, this disables very useful things
> like Apache Phoenix's coprocessors. Some tenants of a multi-user HBase may
> also need to run bespoke coprocessors. But as an operator I would not want
> wanton coprocessor usage. Ideally, one could do one of two things:
> * Allow coprocessors defined in {{hbase-site.xml}} -- this can only be
> administratively changed in most cases
> * Allow coprocessors from table descriptors but only if the coprocessor is
> whitelisted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-17237) Override the correct compact method in HMobStore
[ https://issues.apache.org/jira/browse/HBASE-17237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du reassigned HBASE-17237: Assignee: Jingcheng Du (was: Jingcheng Du) > Override the correct compact method in HMobStore > > > Key: HBASE-17237 > URL: https://issues.apache.org/jira/browse/HBASE-17237 > Project: HBase > Issue Type: Bug > Components: mob >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Attachments: HBASE-17237.patch > > > The method Store#compact(CompactionContext compaction, ThroughputController > throughputController) is deprecated now, and HMobStore overrides it. > HMobStore needs to override compact(CompactionContext compaction, > ThroughputController throughputController, User user) instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16336) Removing peers seem to be leaving spare queues
[ https://issues.apache.org/jira/browse/HBASE-16336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724132#comment-15724132 ] Guanghao Zhang commented on HBASE-16336: HBASE-12769 try to fix this by hbck. A more automatic way is to add a replication zk node checker on master. It periodically check and delete the useless replication zk node. In our use case, we found there are dead rs znode leaved and the dead rs znode only can be transferred when other rs restarted. So the replication zk node checker should check the dead rs znode too. I know the more proper solution is HBASE-11392 and HBASE-12439. But for branch-1, we can resolve this by a replication zk node checker. Any ideas? [~enis] > Removing peers seem to be leaving spare queues > -- > > Key: HBASE-16336 > URL: https://issues.apache.org/jira/browse/HBASE-16336 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Joseph > > I have been running IntegrationTestReplication repeatedly with the backported > Replication Table changes. Every other iteration of the test fails with, but > these queues should have been deleted when we removed the peers. I believe > this may be related to HBASE-16096, HBASE-16208, or HBASE-16081. > 16/08/02 08:36:07 ERROR util.AbstractHBaseTool: Error running command-line > tool > org.apache.hadoop.hbase.replication.ReplicationException: undeleted queue for > peerId: TestPeer, replicator: > hbase4124.ash2.facebook.com,16020,1470150251042, queueId: TestPeer > at > org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.checkQueuesDeleted(ReplicationPeersZKImpl.java:544) > at > org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.addPeer(ReplicationPeersZKImpl.java:127) > at > org.apache.hadoop.hbase.client.replication.ReplicationAdmin.addPeer(ReplicationAdmin.java:200) > at > org.apache.hadoop.hbase.test.IntegrationTestReplication$VerifyReplicationLoop.setupTablesAndReplication(IntegrationTestReplication.java:239) > at > org.apache.hadoop.hbase.test.IntegrationTestReplication$VerifyReplicationLoop.run(IntegrationTestReplication.java:325) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > org.apache.hadoop.hbase.test.IntegrationTestReplication.runTestFromCommandLine(IntegrationTestReplication.java:418) > at > org.apache.hadoop.hbase.IntegrationTestBase.doWork(IntegrationTestBase.java:134) > at > org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:112) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > org.apache.hadoop.hbase.test.IntegrationTestReplication.main(IntegrationTestReplication.java:424) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17182) Memory leak from openScanner of HBase thrift2
[
https://issues.apache.org/jira/browse/HBASE-17182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724088#comment-15724088
]
Jian Yi commented on HBASE-17182:
-
class ScannerInfo {
final ResultScanner resultScanner;
private long lastAccessTime;
ScannerInfo(ResultScanner scanner) {
lastAccessTime = EnvironmentEdgeManager.currentTime();
resultScanner = scanner;
}
synchronized boolean updateAccessTime() {
lastAccessTime = EnvironmentEdgeManager.currentTime();
return true;
}
synchronized boolean timedOut(int maxIdleTime) {
long timeoutTime = lastAccessTime + maxIdleTime;
if (EnvironmentEdgeManager.currentTime() > timeoutTime) {
return true;
}
return false;
}
}
> Memory leak from openScanner of HBase thrift2
> -
>
> Key: HBASE-17182
> URL: https://issues.apache.org/jira/browse/HBASE-17182
> Project: HBase
> Issue Type: Bug
> Components: Thrift
>Reporter: Jian Yi
>
> Client call openScanner, but client (coredump or others) not closeScanner,
> the scanner will not be removed from scannerMap.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17182) Memory leak from openScanner of HBase thrift2
[ https://issues.apache.org/jira/browse/HBASE-17182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724074#comment-15724074 ] Jian Yi commented on HBASE-17182: - Add ScannerInfo to record the last access time. > Memory leak from openScanner of HBase thrift2 > - > > Key: HBASE-17182 > URL: https://issues.apache.org/jira/browse/HBASE-17182 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Jian Yi > > Client call openScanner, but client (coredump or others) not closeScanner, > the scanner will not be removed from scannerMap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16261) MultiHFileOutputFormat Enhancement
[
https://issues.apache.org/jira/browse/HBASE-16261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724050#comment-15724050
]
Hadoop QA commented on HBASE-16261:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 1s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 0s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 114m 24s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 160m 17s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.0 Server=1.12.0 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12841847/HBase-16261-V6.patch |
| JIRA Issue | HBASE-16261 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 06e8d613a7af 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / c7b8b63 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4802/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4802/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> MultiHFileOutputFormat Enhancement
>
>
> Key: HBASE-16261
> URL: https://issues.apache.org/jira/browse/HBASE-16261
> Project: HBase
> Issue Type: Sub-task
> Components: hbase, mapreduce
>Affects Versions: 2.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16261-V1.patch,
[jira] [Commented] (HBASE-17256) Rpc handler monitoring will be removed when the task queue is full
[ https://issues.apache.org/jira/browse/HBASE-17256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724047#comment-15724047 ] Hudson commented on HBASE-17256: SUCCESS: Integrated in Jenkins build HBase-1.4 #559 (See [https://builds.apache.org/job/HBase-1.4/559/]) HBASE-17256 Rpc handler monitoring will be removed when the task queue (tedyu: rev c2801a2ea87a870195557122ede08095f15b19c1) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/TaskMonitor.java > Rpc handler monitoring will be removed when the task queue is full > -- > > Key: HBASE-17256 > URL: https://issues.apache.org/jira/browse/HBASE-17256 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 0.98.23, 1.2.4 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17256-v1.patch > > > The RPC handlers monitoring will displayed in the Web UI when the cluster has > just started. As time goes on, RPC handlers disappeared . > We have the RPC handlers listed as tasks and stored in CircularFifoBuffer. > CircularFifoBuffer is a first in first out buffer with a fixed size that > replaces its oldest element if full. > When we refresh the page, completed tasks will be removed from > CircularFifoBuffer. > Not refresh the page for a long time, the oldest element (RPC handlers) will > be replaced by new tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17261) Balancer makes no sense on tip of branch-1: says balanced when not
[
https://issues.apache.org/jira/browse/HBASE-17261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724030#comment-15724030
]
Guanghao Zhang commented on HBASE-17261:
We can update the default value of
hbase.master.balancer.stochastic.minCostNeedBalance to 0.0. And keep the
default behavior sames with before HBASE-15529. Any ideas? [~stack]
> Balancer makes no sense on tip of branch-1: says balanced when not
> --
>
> Key: HBASE-17261
> URL: https://issues.apache.org/jira/browse/HBASE-17261
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>
> Running ITBLL on tip of branch-1, I see this in log when I try to balance:
> {code}
> 2016-12-05 16:42:21,031 INFO
> [RpcServer.deafult.FPBQ.Fifo.handler=46,queue=1,port=16000]
> balancer.StochasticLoadBalancer: Skipping load balancing because balanced
> cluster; total cost is 525.2547686174673|
> , sum multiplier is 111087.0 min cost which need balance is 0.05
> {code}
> Its some old nonsense.
> Does this every time I balance. Can't even force a balance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15437) Response size calculated in RPCServer for warning tooLarge responses does NOT count CellScanner payload
[
https://issues.apache.org/jira/browse/HBASE-15437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724007#comment-15724007
]
Hadoop QA commented on HBASE-15437:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
2s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
33m 54s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 34s
{color} | {color:red} hbase-server generated 2 new + 0 unchanged - 0 fixed = 2
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 104m 7s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 153m 42s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Dead store to lastBlock in
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RpcController,
ClientProtos$GetRequest) At
RSRpcServices.java:org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RpcController,
ClientProtos$GetRequest) At RSRpcServices.java:[line 2308] |
| | Dead store to lastBlock in
org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RpcController,
ClientProtos$MutateRequest) At
RSRpcServices.java:org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RpcController,
ClientProtos$MutateRequest) At RSRpcServices.java:[line 2635] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12841840/HBASE-15437-v3.patch |
| JIRA Issue | HBASE-15437 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 6713386f52b6 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / c7b8b63 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|
[jira] [Commented] (HBASE-17261) Balancer makes no sense on tip of branch-1: says balanced when not
[
https://issues.apache.org/jira/browse/HBASE-17261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724002#comment-15724002
]
Guanghao Zhang commented on HBASE-17261:
After HBASE-15529, cluster need balance when (total cost / sum multiplier) >
minCostNeedBalance. So this means the average cost is less than the defaul
minCostNeedBalance 0.05.
> Balancer makes no sense on tip of branch-1: says balanced when not
> --
>
> Key: HBASE-17261
> URL: https://issues.apache.org/jira/browse/HBASE-17261
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>
> Running ITBLL on tip of branch-1, I see this in log when I try to balance:
> {code}
> 2016-12-05 16:42:21,031 INFO
> [RpcServer.deafult.FPBQ.Fifo.handler=46,queue=1,port=16000]
> balancer.StochasticLoadBalancer: Skipping load balancing because balanced
> cluster; total cost is 525.2547686174673|
> , sum multiplier is 111087.0 min cost which need balance is 0.05
> {code}
> Its some old nonsense.
> Does this every time I balance. Can't even force a balance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog
[ https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723987#comment-15723987 ] Duo Zhang commented on HBASE-17049: --- [~ram_krish] Busy these days, haven't got enough time to setup the test environment yet... > Find out why AsyncFSWAL issues much more syncs than FSHLog > -- > > Key: HBASE-17049 > URL: https://issues.apache.org/jira/browse/HBASE-17049 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang > Fix For: 2.0.0 > > Attachments: delay-sync.patch > > > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-17257) Add column-aliasing capability to hbase-client
[ https://issues.apache.org/jira/browse/HBASE-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723952#comment-15723952 ] Daniel Vimont edited comment on HBASE-17257 at 12/6/16 1:24 AM: I very much hope that my original research is mistaken, and that a system table/region CAN be split (because arguably one of the ugliest aspects of my current design is the need for a separate namespace for the aliasMappingTable). I will get out some details of the overall design soon, and I will review my original research which seemed to indicate that system tables/regions cannot be split. was (Author: daniel_vimont): I very much hope that my original research is mistaken, and that a system table/region CAN be split (because arguably one of the ugliest aspects of my current design is the need for a separate namespace for the aliasMappingTable). I will get out some details of the overall design soon, and I will review my original research which seemed to indicate that system tables cannot be split. > Add column-aliasing capability to hbase-client > -- > > Key: HBASE-17257 > URL: https://issues.apache.org/jira/browse/HBASE-17257 > Project: HBase > Issue Type: New Feature > Components: Client >Affects Versions: 2.0.0 >Reporter: Daniel Vimont >Assignee: Daniel Vimont > Labels: features > > Column aliasing will provide the option for a 1, 2, or 4 byte alias value to > be stored in each cell of an "alias enabled" column-family, in place of the > full-length column-qualifier. Aliasing is intended to operate completely > invisibly to the end-user developer, with absolutely no "awareness" of > aliasing required to be coded into a front-end application. No new public > hbase-client interfaces are to be introduced, and only a few new public > methods should need to be added to existing interfaces, primarily to allow an > administrator to designate that a new column-family is to be alias-enabled by > setting its aliasSize attribute to 1, 2, or 4. > To facilitate such functionality, new subclasses of HTable, > BufferedMutatorImpl, and HTableMultiplexer are to be provided. The overriding > methods of these new subclasses will invoke methods of the new AliasManager > class to facilitate qualifier-to-alias conversions (for user-submitted Gets, > Scans, and Mutations) and alias-to-qualifier conversions (for Results > returned from HBase) for any Table that has one or more alias-enabled column > families. All conversion logic will be encapsulated in the new AliasManager > class, and all qualifier-to-alias mappings will be persisted in a new > aliasMappingTable in a new, reserved namespace. > An informal polling of HBase users at HBaseCon East and at the > Strata/Hadoop-World conference in Sept. 2016 showed that Column Aliasing > could be a popular enhancement to standard HBase functionality, due to the > fact that full column-qualifiers are stored in each cell, and reducing this > qualifier storage requirement down to 1, 2, or 4 bytes per cell could prove > beneficial in terms of reduced storage and bandwidth needs. Aliasing is > intended chiefly for column-families which are of the "narrow and tall" > variety (i.e., that are designed to use relatively few distinct > column-qualifiers throughout a large number of rows, throughout the lifespan > of the column-family). A column-family that is set up with an alias-size of 1 > byte can contain up to 255 unique column-qualifiers; a 2 byte alias-size > allows for up to 65,535 unique column-qualifiers; and a 4 byte alias-size > allows for up to 4,294,967,295 unique column-qualifiers. > Fuller specifications will be entered into the comments section below. Note > that it may well not be viable to add aliasing support in the new "async" > classes that appear to be currently under development. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17257) Add column-aliasing capability to hbase-client
[ https://issues.apache.org/jira/browse/HBASE-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723952#comment-15723952 ] Daniel Vimont commented on HBASE-17257: --- I very much hope that my original research is mistaken, and that a system table/region CAN be split (because arguably one of the ugliest aspects of my current design is the need for a separate namespace for the aliasMappingTable). I will get out some details of the overall design soon, and I will review my original research which seemed to indicate that system tables cannot be split. > Add column-aliasing capability to hbase-client > -- > > Key: HBASE-17257 > URL: https://issues.apache.org/jira/browse/HBASE-17257 > Project: HBase > Issue Type: New Feature > Components: Client >Affects Versions: 2.0.0 >Reporter: Daniel Vimont >Assignee: Daniel Vimont > Labels: features > > Column aliasing will provide the option for a 1, 2, or 4 byte alias value to > be stored in each cell of an "alias enabled" column-family, in place of the > full-length column-qualifier. Aliasing is intended to operate completely > invisibly to the end-user developer, with absolutely no "awareness" of > aliasing required to be coded into a front-end application. No new public > hbase-client interfaces are to be introduced, and only a few new public > methods should need to be added to existing interfaces, primarily to allow an > administrator to designate that a new column-family is to be alias-enabled by > setting its aliasSize attribute to 1, 2, or 4. > To facilitate such functionality, new subclasses of HTable, > BufferedMutatorImpl, and HTableMultiplexer are to be provided. The overriding > methods of these new subclasses will invoke methods of the new AliasManager > class to facilitate qualifier-to-alias conversions (for user-submitted Gets, > Scans, and Mutations) and alias-to-qualifier conversions (for Results > returned from HBase) for any Table that has one or more alias-enabled column > families. All conversion logic will be encapsulated in the new AliasManager > class, and all qualifier-to-alias mappings will be persisted in a new > aliasMappingTable in a new, reserved namespace. > An informal polling of HBase users at HBaseCon East and at the > Strata/Hadoop-World conference in Sept. 2016 showed that Column Aliasing > could be a popular enhancement to standard HBase functionality, due to the > fact that full column-qualifiers are stored in each cell, and reducing this > qualifier storage requirement down to 1, 2, or 4 bytes per cell could prove > beneficial in terms of reduced storage and bandwidth needs. Aliasing is > intended chiefly for column-families which are of the "narrow and tall" > variety (i.e., that are designed to use relatively few distinct > column-qualifiers throughout a large number of rows, throughout the lifespan > of the column-family). A column-family that is set up with an alias-size of 1 > byte can contain up to 255 unique column-qualifiers; a 2 byte alias-size > allows for up to 65,535 unique column-qualifiers; and a 4 byte alias-size > allows for up to 4,294,967,295 unique column-qualifiers. > Fuller specifications will be entered into the comments section below. Note > that it may well not be viable to add aliasing support in the new "async" > classes that appear to be currently under development. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17260) Procedure v2 - Add setOwner() overload taking a User instance
[
https://issues.apache.org/jira/browse/HBASE-17260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723929#comment-15723929
]
Hadoop QA commented on HBASE-17260:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
23m 27s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 27s
{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 92m 34s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 130m 49s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12841832/HBASE-17260-v0.patch |
| JIRA Issue | HBASE-17260 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 0c499257effe 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / c7b8b63 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4799/testReport/ |
| modules | C: hbase-procedure hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4799/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Procedure v2 - Add setOwner() overload taking
[jira] [Updated] (HBASE-17072) CPU usage starts to climb up to 90-100% when using G1GC; purge ThreadLocal usage
[
https://issues.apache.org/jira/browse/HBASE-17072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-17072:
---
Attachment: HBASE-17072-0.98.patch
When backporting this to 0.98 I found a multithreaded unit test in
TestHFileBlock would fail due to concurrent read/update to the cached header
ByteBuffer's position. In that test there is more than one read in progress
with the same next offset. This is what I've done differently:
{code}
PrefetchedHeader ph = prefetchedHeader.getAndSet(null); // be multithread
safe
ByteBuffer headerBuf = null;
if (ph != null) {
if (ph.offset == offset) {
headerBuf = ph.buf; // our previous read, use the cached buffer
} else {
prefetchedHeader.set(ph); // not our previous read, put back
}
}
{code}
Only the first thread through will get the buffer if the offset matches. If the
offset does not match or if another thread claimed the buffer a concurrently
executing thread will just get {{null}}.
The master patch does only:
{code}
PrefetchedHeader ph = this.prefetchedHeader.get();
return ph != null && ph.offset == offset? ph.buf: null;
{code}
More than one thread can get a reference to the buffer if the offset test
succeeds. Could be fine for master, just pointing this out. Might be relevant
for other backports. 1.1?
> CPU usage starts to climb up to 90-100% when using G1GC; purge ThreadLocal
> usage
>
>
> Key: HBASE-17072
> URL: https://issues.apache.org/jira/browse/HBASE-17072
> Project: HBase
> Issue Type: Bug
> Components: Performance, regionserver
>Affects Versions: 1.0.0, 2.0.0, 1.2.0
>Reporter: Eiichi Sato
>Assignee: Eiichi Sato
>Priority: Critical
> Fix For: 2.0.0, 1.4.0, 0.98.24
>
> Attachments: HBASE-17072-0.98.patch, HBASE-17072.branch-1.001.patch,
> HBASE-17072.master.001.patch, HBASE-17072.master.002.patch,
> HBASE-17072.master.003.patch, HBASE-17072.master.004.patch,
> HBASE-17072.master.005.patch, HBASE-17072.master.005.patch,
> disable-block-header-cache.patch, mat-threadlocals.png, mat-threads.png,
> metrics.png, slave1.svg, slave2.svg, slave3.svg, slave4.svg
>
>
> h5. Problem
> CPU usage of a region server in our CDH 5.4.5 cluster, at some point, starts
> to gradually get higher up to nearly 90-100% when using G1GC. We've also run
> into this problem on CDH 5.7.3 and CDH 5.8.2.
> In our production cluster, it normally takes a few weeks for this to happen
> after restarting a RS. We reproduced this on our test cluster and attached
> the results. Please note that, to make it easy to reproduce, we did some
> "anti-tuning" on a table when running tests.
> In metrics.png, soon after we started running some workloads against a test
> cluster (CDH 5.8.2) at about 7 p.m. CPU usage of the two RSs started to rise.
> Flame Graphs (slave1.svg to slave4.svg) are generated from jstack dumps of
> each RS process around 10:30 a.m. the next day.
> After investigating heapdumps from another occurrence on a test cluster
> running CDH 5.7.3, we found that the ThreadLocalMap contain a lot of
> contiguous entries of {{HFileBlock$PrefetchedHeader}} probably due to primary
> clustering. This caused more loops in
> {{ThreadLocalMap#expungeStaleEntries()}}, consuming a certain amount of CPU
> time. What is worse is that the method is called from RPC metrics code,
> which means even a small amount of per-RPC time soon adds up to a huge amount
> of CPU time.
> This is very similar to the issue in HBASE-16616, but we have many
> {{HFileBlock$PrefetchedHeader}} not only {{Counter$IndexHolder}} instances.
> Here are some OQL counts from Eclipse Memory Analyzer (MAT). This shows a
> number of ThreadLocal instances in the ThreadLocalMap of a single handler
> thread.
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name =
> "org.apache.hadoop.hbase.io.hfile.HFileBlock$PrefetchedHeader"
> #=> 10980 instances
> {code}
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name = "org.apache.hadoop.hbase.util.Counter$IndexHolder"
> #=> 2052 instances
> {code}
> Although as described in HBASE-16616 this somewhat seems to be an issue in
> G1GC side regarding weakly-reachable objects, we should keep ThreadLocal
> usage minimal and avoid creating an indefinite number (in this case, a number
> of HFiles) of ThreadLocal instances.
> HBASE-16146 removes ThreadLocals from the RPC metrics code. That may solve
> the issue (I just saw the patch, never tested it at all), but the
>
[jira] [Updated] (HBASE-17072) CPU usage starts to climb up to 90-100% when using G1GC; purge ThreadLocal usage
[
https://issues.apache.org/jira/browse/HBASE-17072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-17072:
---
Fix Version/s: 0.98.24
> CPU usage starts to climb up to 90-100% when using G1GC; purge ThreadLocal
> usage
>
>
> Key: HBASE-17072
> URL: https://issues.apache.org/jira/browse/HBASE-17072
> Project: HBase
> Issue Type: Bug
> Components: Performance, regionserver
>Affects Versions: 1.0.0, 2.0.0, 1.2.0
>Reporter: Eiichi Sato
>Assignee: Eiichi Sato
>Priority: Critical
> Fix For: 2.0.0, 1.4.0, 0.98.24
>
> Attachments: HBASE-17072-0.98.patch, HBASE-17072.branch-1.001.patch,
> HBASE-17072.master.001.patch, HBASE-17072.master.002.patch,
> HBASE-17072.master.003.patch, HBASE-17072.master.004.patch,
> HBASE-17072.master.005.patch, HBASE-17072.master.005.patch,
> disable-block-header-cache.patch, mat-threadlocals.png, mat-threads.png,
> metrics.png, slave1.svg, slave2.svg, slave3.svg, slave4.svg
>
>
> h5. Problem
> CPU usage of a region server in our CDH 5.4.5 cluster, at some point, starts
> to gradually get higher up to nearly 90-100% when using G1GC. We've also run
> into this problem on CDH 5.7.3 and CDH 5.8.2.
> In our production cluster, it normally takes a few weeks for this to happen
> after restarting a RS. We reproduced this on our test cluster and attached
> the results. Please note that, to make it easy to reproduce, we did some
> "anti-tuning" on a table when running tests.
> In metrics.png, soon after we started running some workloads against a test
> cluster (CDH 5.8.2) at about 7 p.m. CPU usage of the two RSs started to rise.
> Flame Graphs (slave1.svg to slave4.svg) are generated from jstack dumps of
> each RS process around 10:30 a.m. the next day.
> After investigating heapdumps from another occurrence on a test cluster
> running CDH 5.7.3, we found that the ThreadLocalMap contain a lot of
> contiguous entries of {{HFileBlock$PrefetchedHeader}} probably due to primary
> clustering. This caused more loops in
> {{ThreadLocalMap#expungeStaleEntries()}}, consuming a certain amount of CPU
> time. What is worse is that the method is called from RPC metrics code,
> which means even a small amount of per-RPC time soon adds up to a huge amount
> of CPU time.
> This is very similar to the issue in HBASE-16616, but we have many
> {{HFileBlock$PrefetchedHeader}} not only {{Counter$IndexHolder}} instances.
> Here are some OQL counts from Eclipse Memory Analyzer (MAT). This shows a
> number of ThreadLocal instances in the ThreadLocalMap of a single handler
> thread.
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name =
> "org.apache.hadoop.hbase.io.hfile.HFileBlock$PrefetchedHeader"
> #=> 10980 instances
> {code}
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name = "org.apache.hadoop.hbase.util.Counter$IndexHolder"
> #=> 2052 instances
> {code}
> Although as described in HBASE-16616 this somewhat seems to be an issue in
> G1GC side regarding weakly-reachable objects, we should keep ThreadLocal
> usage minimal and avoid creating an indefinite number (in this case, a number
> of HFiles) of ThreadLocal instances.
> HBASE-16146 removes ThreadLocals from the RPC metrics code. That may solve
> the issue (I just saw the patch, never tested it at all), but the
> {{HFileBlock$PrefetchedHeader}} are still there in the ThreadLocalMap, which
> may cause issues in the future again.
> h5. Our Solution
> We simply removed the whole {{HFileBlock$PrefetchedHeader}} caching and
> fortunately we didn't notice any performance degradation for our production
> workloads.
> Because the PrefetchedHeader caching uses ThreadLocal and because RPCs are
> handled randomly in any of the handlers, small Get or small Scan RPCs do not
> benefit from the caching (See HBASE-10676 and HBASE-11402 for the details).
> Probably, we need to see how well reads are saved by the caching for large
> Scan or Get RPCs and especially for compactions if we really remove the
> caching. It's probably better if we can remove ThreadLocals without breaking
> the current caching behavior.
> FWIW, I'm attaching the patch we applied. It's for CDH 5.4.5.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17000) [RegionServer] Compute region filesystem space use and report to Master
[
https://issues.apache.org/jira/browse/HBASE-17000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723894#comment-15723894
]
Hadoop QA commented on HBASE-17000:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 4m 26s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
47s {color} | {color:green} HBASE-16961 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} HBASE-16961 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
21s {color} | {color:green} HBASE-16961 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
3s {color} | {color:green} HBASE-16961 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 43s
{color} | {color:red} hbase-protocol-shaded in HBASE-16961 has 24 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s
{color} | {color:green} HBASE-16961 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 11 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 56s
{color} | {color:red} The patch causes 270 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 51s
{color} | {color:red} The patch causes 270 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 47s
{color} | {color:red} The patch causes 270 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 43s
{color} | {color:red} The patch causes 270 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 38s
{color} | {color:red} The patch causes 270 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} hbaseprotoc {color} | {color:red} 0m 21s
{color} | {color:red} Patch generated 1 new protoc errors in hbase-server.
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 90m 16s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 158m 27s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch
[jira] [Created] (HBASE-17261) Balancer makes no sense on tip of branch-1: says balanced when not
stack created HBASE-17261:
-
Summary: Balancer makes no sense on tip of branch-1: says balanced
when not
Key: HBASE-17261
URL: https://issues.apache.org/jira/browse/HBASE-17261
Project: HBase
Issue Type: Bug
Reporter: stack
Running ITBLL on tip of branch-1, I see this in log when I try to balance:
{code}
2016-12-05 16:42:21,031 INFO
[RpcServer.deafult.FPBQ.Fifo.handler=46,queue=1,port=16000]
balancer.StochasticLoadBalancer: Skipping load balancing because balanced
cluster; total cost is 525.2547686174673|
, sum multiplier is 111087.0 min cost which need balance is 0.05
{code}
Its some old nonsense.
Does this every time I balance. Can't even force a balance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16941) FavoredNodes - Split/Merge code paths
[ https://issues.apache.org/jira/browse/HBASE-16941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvel Thirumoolan updated HBASE-16941: - Attachment: HBASE-16941.master.014.patch > FavoredNodes - Split/Merge code paths > - > > Key: HBASE-16941 > URL: https://issues.apache.org/jira/browse/HBASE-16941 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-16941.master.001.patch, > HBASE-16941.master.002.patch, HBASE-16941.master.003.patch, > HBASE-16941.master.004.patch, HBASE-16941.master.005.patch, > HBASE-16941.master.006.patch, HBASE-16941.master.007.patch, > HBASE-16941.master.008.patch, HBASE-16941.master.009.patch, > HBASE-16941.master.010.patch, HBASE-16941.master.011.patch, > HBASE-16941.master.012.patch, HBASE-16941.master.013.patch, > HBASE-16941.master.014.patch > > > This jira is to deal with the split/merge logic discussed as part of > HBASE-15532. The design document can be seen at HBASE-15531. The specific > changes are: > Split and merged regions should inherit favored node information from parent > regions. For splits also include some randomness so even if there are > subsequent splits, the regions will be more or less distributed. For split, > we include 2 FN from the parent and generate one random node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17051) libhbase++: implement RPC client and connection management
[
https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723860#comment-15723860
]
Hadoop QA commented on HBASE-17051:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 32m 48s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 11s
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 2s
{color} | {color:green} HBASE-14850 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 2s
{color} | {color:green} HBASE-14850 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
2s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 3s
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 2s
{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
5s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
42m 55s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
4s {color} | {color:green} the patch passed {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 76m 19s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:date2016-12-05 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12841846/HBASE-17051-HBASE-14850.004.patch
|
| JIRA Issue | HBASE-17051 |
| Optional Tests | cc compile shellcheck shelldocs |
| uname | Linux 419665b220a6 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | nobuild |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | HBASE-14850 / 99ccddf |
| shellcheck | v0.4.5 |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4801/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> libhbase++: implement RPC client and connection management
> --
>
> Key: HBASE-17051
> URL: https://issues.apache.org/jira/browse/HBASE-17051
> Project: HBase
> Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HBASE-17051-HBASE-14850.000.patch,
> HBASE-17051-HBASE-14850.001.patch, HBASE-17051-HBASE-14850.002.patch,
> HBASE-17051-HBASE-14850.003.patch, HBASE-17051-HBASE-14850.004.patch
>
>
> This proposes building RPC client and connection management layer, which
> supports the equivalent functions resides in RpcClient.java and
> RpcConnection.java. Specifically, handler/pipeline concepts will be used for
> implementation, similar to NettyRpcClient and NettyRpcConnection in java side.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16941) FavoredNodes - Split/Merge code paths
[ https://issues.apache.org/jira/browse/HBASE-16941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723828#comment-15723828 ] Thiruvel Thirumoolan commented on HBASE-16941: -- Since I moved some of the classes to "favored" package, white space issues with old code also got flagged. I have fixed and uploaded a new patch (HBASE-16941.master.013.patch). > FavoredNodes - Split/Merge code paths > - > > Key: HBASE-16941 > URL: https://issues.apache.org/jira/browse/HBASE-16941 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-16941.master.001.patch, > HBASE-16941.master.002.patch, HBASE-16941.master.003.patch, > HBASE-16941.master.004.patch, HBASE-16941.master.005.patch, > HBASE-16941.master.006.patch, HBASE-16941.master.007.patch, > HBASE-16941.master.008.patch, HBASE-16941.master.009.patch, > HBASE-16941.master.010.patch, HBASE-16941.master.011.patch, > HBASE-16941.master.012.patch, HBASE-16941.master.013.patch > > > This jira is to deal with the split/merge logic discussed as part of > HBASE-15532. The design document can be seen at HBASE-15531. The specific > changes are: > Split and merged regions should inherit favored node information from parent > regions. For splits also include some randomness so even if there are > subsequent splits, the regions will be more or less distributed. For split, > we include 2 FN from the parent and generate one random node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17250) For Get and scan in one case, checkFamily can be skipped in Region#getScanner
[ https://issues.apache.org/jira/browse/HBASE-17250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723826#comment-15723826 ] Matteo Bertozzi commented on HBASE-17250: - [~tedyu] I don't think scan.setAttribute() is the right place for it. from the patch looks like the "skipCheckFamily" is specific on how we implemented the get() code. we are using that getScanner() in both scan() and get() but in get we already checked the families before calling getScanner(). maybe an alternative to the flag, is that in both cases we check the families before doing anything. since in both cases we call the coprocessors with the scan or get object, and in theory we want to make sure the families are correct. in this case we check early and getScanner() will end up without any check. but this means that coprocessors that are using directly region.getScanner() should do validation.. so maybe the skipCheckFamily flag is safe for compatibility and clarity > For Get and scan in one case, checkFamily can be skipped in Region#getScanner > - > > Key: HBASE-17250 > URL: https://issues.apache.org/jira/browse/HBASE-17250 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Attachments: HBASE-17250-master-001.patch > > > For get(), checkFamily is done in prepareGet(), so checkFamily can be skipped > in Region#getScanner(). For scan(), if there is no Family configured in scan, > the families are from table descriptor, so checkFamily in > Region#getScanner(). can be skipped in this case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16941) FavoredNodes - Split/Merge code paths
[ https://issues.apache.org/jira/browse/HBASE-16941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvel Thirumoolan updated HBASE-16941: - Attachment: HBASE-16941.master.013.patch > FavoredNodes - Split/Merge code paths > - > > Key: HBASE-16941 > URL: https://issues.apache.org/jira/browse/HBASE-16941 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-16941.master.001.patch, > HBASE-16941.master.002.patch, HBASE-16941.master.003.patch, > HBASE-16941.master.004.patch, HBASE-16941.master.005.patch, > HBASE-16941.master.006.patch, HBASE-16941.master.007.patch, > HBASE-16941.master.008.patch, HBASE-16941.master.009.patch, > HBASE-16941.master.010.patch, HBASE-16941.master.011.patch, > HBASE-16941.master.012.patch, HBASE-16941.master.013.patch > > > This jira is to deal with the split/merge logic discussed as part of > HBASE-15532. The design document can be seen at HBASE-15531. The specific > changes are: > Split and merged regions should inherit favored node information from parent > regions. For splits also include some randomness so even if there are > subsequent splits, the regions will be more or less distributed. For split, > we include 2 FN from the parent and generate one random node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17250) For Get and scan in one case, checkFamily can be skipped in Region#getScanner
[ https://issues.apache.org/jira/browse/HBASE-17250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723799#comment-15723799 ] Ted Yu commented on HBASE-17250: How about passing the boolean through scan.setAttribute() ? > For Get and scan in one case, checkFamily can be skipped in Region#getScanner > - > > Key: HBASE-17250 > URL: https://issues.apache.org/jira/browse/HBASE-17250 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Attachments: HBASE-17250-master-001.patch > > > For get(), checkFamily is done in prepareGet(), so checkFamily can be skipped > in Region#getScanner(). For scan(), if there is no Family configured in scan, > the families are from table descriptor, so checkFamily in > Region#getScanner(). can be skipped in this case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17170) HBase is also retrying DoNotRetryIOException because of class loader differences.
[
https://issues.apache.org/jira/browse/HBASE-17170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723796#comment-15723796
]
Hudson commented on HBASE-17170:
SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2079 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/2079/])
HBASE-17170 HBase is also retrying DoNotRetryIOException because of (tedyu: rev
1c8822ddff02c0b4f64b42b316900f2a970ff098)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RemoteWithExtrasException.java
> HBase is also retrying DoNotRetryIOException because of class loader
> differences.
> -
>
> Key: HBASE-17170
> URL: https://issues.apache.org/jira/browse/HBASE-17170
> Project: HBase
> Issue Type: Bug
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17170.master.001.patch,
> HBASE-17170.master.002.patch
>
>
> The class loader used by API exposed by hadoop and the context class loader
> used by RunJar(bin/hadoop jar phoenix-client.jar …. ) are different resulting
> in classes loaded from jar not visible to other current class loader used by
> API.
> {code}
> 16/04/26 21:18:00 INFO client.RpcRetryingCaller: Call exception, tries=32,
> retries=35, started=491541 ms ago, cancelled=false, msg=
> 16/04/26 21:18:21 INFO client.RpcRetryingCaller: Call exception, tries=33,
> retries=35, started=511747 ms ago, cancelled=false, msg=
> 16/04/26 21:18:41 INFO client.RpcRetryingCaller: Call exception, tries=34,
> retries=35, started=531820 ms ago, cancelled=false, msg=
> Exception in thread "main" org.apache.phoenix.exception.PhoenixIOException:
> Failed after attempts=35, exceptions:
> Tue Apr 26 21:09:49 UTC 2016,
> RpcRetryingCaller{globalStartTime=1461704989282, pause=100, retries=35},
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.NamespaceExistException):
> org.apache.hadoop.hbase.NamespaceExistException: SYSTEM
> at
> org.apache.hadoop.hbase.master.TableNamespaceManager.create(TableNamespaceManager.java:156)
> at
> org.apache.hadoop.hbase.master.TableNamespaceManager.create(TableNamespaceManager.java:131)
> at org.apache.hadoop.hbase.master.HMaster.createNamespace(HMaster.java:2553)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.createNamespace(MasterRpcServices.java:447)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58043)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:102)
> {code}
> The actual problem is stated in the comment below
> https://issues.apache.org/jira/browse/PHOENIX-3495?focusedCommentId=15677081=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15677081
> If we are not loading hbase classes from Hadoop classpath(from where hadoop
> jars are getting loaded), then the RemoteException will not get unwrapped
> because of ClassNotFoundException and the client will keep on retrying even
> if the cause of exception is DoNotRetryIOException.
> RunJar#main() context class loader.
> {code}
> ClassLoader loader = createClassLoader(file, workDir);
> Thread.currentThread().setContextClassLoader(loader);
> Class mainClass = Class.forName(mainClassName, true, loader);
> Method main = mainClass.getMethod("main", new Class[] {
> Array.newInstance(String.class, 0).getClass()
> });
> HBase classes can be loaded from jar(phoenix-client.jar):-
> hadoop --config /etc/hbase/conf/ jar
> ~/git/apache/phoenix/phoenix-client/target/phoenix-4.9.0-HBase-1.2-client.jar
> org.apache.phoenix.mapreduce.CsvBulkLoadTool --table GIGANTIC_TABLE --input
> /tmp/b.csv --zookeeper localhost:2181
> {code}
> API(using current class loader).
> {code}
> public class RpcRetryingCaller {
> public IOException unwrapRemoteException() {
> try {
> Class realClass = Class.forName(getClassName());
> return instantiateException(realClass.asSubclass(IOException.class));
> } catch(Exception e) {
> // cannot instantiate the original exception, just return this
> }
> return this;
> }
> {code}
> *Possible solution:-*
> We can create our own HBaseRemoteWithExtrasException(extension of
> RemoteWithExtrasException) so that default class loader will be the one from
> where the hbase classes are loaded and extend unwrapRemoteException() to
> throw exception if the unwrapping doesn’t take place because of CNF
> exception?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17256) Rpc handler monitoring will be removed when the task queue is full
[ https://issues.apache.org/jira/browse/HBASE-17256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723798#comment-15723798 ] Hudson commented on HBASE-17256: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2079 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2079/]) HBASE-17256 Rpc handler monitoring will be removed when the task queue (tedyu: rev 3190605801fb18ac1c0d47fdc8964a93fab0d8f2) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/TaskMonitor.java > Rpc handler monitoring will be removed when the task queue is full > -- > > Key: HBASE-17256 > URL: https://issues.apache.org/jira/browse/HBASE-17256 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 0.98.23, 1.2.4 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17256-v1.patch > > > The RPC handlers monitoring will displayed in the Web UI when the cluster has > just started. As time goes on, RPC handlers disappeared . > We have the RPC handlers listed as tasks and stored in CircularFifoBuffer. > CircularFifoBuffer is a first in first out buffer with a fixed size that > replaces its oldest element if full. > When we refresh the page, completed tasks will be removed from > CircularFifoBuffer. > Not refresh the page for a long time, the oldest element (RPC handlers) will > be replaced by new tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17221) Abstract out an interface for RpcServer.Call
[
https://issues.apache.org/jira/browse/HBASE-17221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723794#comment-15723794
]
Hudson commented on HBASE-17221:
SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2079 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/2079/])
HBASE-17221 Abstract out an interface for RpcServer.Call (jerryjch: rev
39653862a4d24b5309e972ca38554a7f81bc94fd)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallRunner.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
* (add) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcCall.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServerInterface.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RWQueueRpcExecutor.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/AdaptiveLifoCoDelCallQueue.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/TestMetaTableAccessor.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcCallContext.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestSimpleRpcScheduler.java
> Abstract out an interface for RpcServer.Call
>
>
> Key: HBASE-17221
> URL: https://issues.apache.org/jira/browse/HBASE-17221
> Project: HBase
> Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-17221-v2.patch, HBASE-17221-v3.patch,
> HBASE-17221-v4.patch, HBASE-17221.patch
>
>
> RpcServer.Call is a concrete class, but it is marked as:
> {noformat}
> @InterfaceAudience.LimitedPrivate({HBaseInterfaceAudience.COPROC,
> HBaseInterfaceAudience.PHOENIX})
> {noformat}
> Let's abstract out an interface out of it for potential consumers that want
> to pass it around.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17252) Wrong arguments for ValueAndTagRewriteCell in CellUtil
[
https://issues.apache.org/jira/browse/HBASE-17252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723795#comment-15723795
]
Hudson commented on HBASE-17252:
SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2079 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/2079/])
HBASE-17252 Wrong arguments for ValueAndTagRewriteCell in CellUtil (tedyu: rev
1fad920321f32686f39021bdbc5f64abcd35cb58)
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java
> Wrong arguments for ValueAndTagRewriteCell in CellUtil
> --
>
> Key: HBASE-17252
> URL: https://issues.apache.org/jira/browse/HBASE-17252
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 17252.v1.txt, 17252.v2.txt
>
>
> In deepClone():
> {code}
> return new ValueAndTagRewriteCell(clonedBaseCell, this.tags,
> this.value);
> {code}
> The tags parameter should be the last parameter.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17239) Add UnsafeByteOperations#wrap(ByteInput, int offset, int len) API
[ https://issues.apache.org/jira/browse/HBASE-17239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723797#comment-15723797 ] Hudson commented on HBASE-17239: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2079 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2079/]) HBASE-17239 Add UnsafeByteOperations#wrap(ByteInput, int offset, int (stack: rev 1f8d8bfa8b3fbc60705afa5b74318daff81c6fd3) * (edit) hbase-protocol-shaded/src/main/patches/HBASE-17239.patch > Add UnsafeByteOperations#wrap(ByteInput, int offset, int len) API > - > > Key: HBASE-17239 > URL: https://issues.apache.org/jira/browse/HBASE-17239 > Project: HBase > Issue Type: Improvement > Components: Protobufs >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17239.addendum.patch, HBASE-17239_V1.patch, > HBASE-17239_V2.patch, HBASE-17239_V3.patch > > > Instead of exposing the newInstance(ByteInput, boolean) in CIS we could just > add a UnsafeByteOperations#wrap(ByteInput, offset, len). And we could just > call that and do a #newcodedInput() over that. So internally we do return a > immutable version of the ByteInput only. This way we can avoid > CIS#newInstance(ByteInput) exposure and can keep it package private as done > in COS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-17250) For Get and scan in one case, checkFamily can be skipped in Region#getScanner
[ https://issues.apache.org/jira/browse/HBASE-17250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723787#comment-15723787 ] huaxiang sun edited comment on HBASE-17250 at 12/5/16 11:56 PM: Upload an initial patch. May not be a good idea to expose implementation details through getScanner() interface. But I could not find a way to better pass the flag around. There are other places such as https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L6894 which similar skipCheckFamily can be applied. If folks agree that this is the right approach to take, will upload a comprehensive patch, thanks. was (Author: huaxiang): Upload an initial patch. May not be a good idea to expose implementation details through getScanner() interface. But I could find a way to better pass the flag around. There are other places such as https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L6894 which similar skipCheckFamily can be applied. If folks agrees that this is the right approach to take, will upload a comprehensive patch, thanks. > For Get and scan in one case, checkFamily can be skipped in Region#getScanner > - > > Key: HBASE-17250 > URL: https://issues.apache.org/jira/browse/HBASE-17250 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Attachments: HBASE-17250-master-001.patch > > > For get(), checkFamily is done in prepareGet(), so checkFamily can be skipped > in Region#getScanner(). For scan(), if there is no Family configured in scan, > the families are from table descriptor, so checkFamily in > Region#getScanner(). can be skipped in this case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17250) For Get and scan in one case, checkFamily can be skipped in Region#getScanner
[ https://issues.apache.org/jira/browse/HBASE-17250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-17250: - Attachment: HBASE-17250-master-001.patch Upload an initial patch. May not be a good idea to expose implementation details through getScanner() interface. But I could find a way to better pass the flag around. There are other places such as https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L6894 which similar skipCheckFamily can be applied. If folks agrees that this is the right approach to take, will upload a comprehensive patch, thanks. > For Get and scan in one case, checkFamily can be skipped in Region#getScanner > - > > Key: HBASE-17250 > URL: https://issues.apache.org/jira/browse/HBASE-17250 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Attachments: HBASE-17250-master-001.patch > > > For get(), checkFamily is done in prepareGet(), so checkFamily can be skipped > in Region#getScanner(). For scan(), if there is no Family configured in scan, > the families are from table descriptor, so checkFamily in > Region#getScanner(). can be skipped in this case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17249) Get/Scan's setTimeRange/setColumnFamilyTimeRange can take the TimeRange reference as the parameter instead of creating a new setColumnFamilyTimeRange instance
[
https://issues.apache.org/jira/browse/HBASE-17249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723742#comment-15723742
]
Hadoop QA commented on HBASE-17249:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 52s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 55s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
6s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 27s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12841834/HBASE-17249-master-002.patch
|
| JIRA Issue | HBASE-17249 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 6c0dacb6c1ec 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / c7b8b63 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4798/testReport/ |
| modules | C: hbase-client U: hbase-client |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4798/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Get/Scan's setTimeRange/setColumnFamilyTimeRange can take the TimeRange
> reference as the parameter instead of creating a new setColumnFamilyTimeRange
> instance
> --
>
> Key:
[jira] [Commented] (HBASE-16941) FavoredNodes - Split/Merge code paths
[
https://issues.apache.org/jira/browse/HBASE-16941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723727#comment-15723727
]
Hadoop QA commented on HBASE-16941:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 16 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 52s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 46s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 96m 21s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 143m 1s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12841812/HBASE-16941.master.012.patch
|
| JIRA Issue | HBASE-16941 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 9daf4fe0bc2d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 1f8d8bf |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/4796/artifact/patchprocess/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4796/testReport/ |
| modules | C: hbase-common hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4796/console |
[jira] [Updated] (HBASE-16261) MultiHFileOutputFormat Enhancement
[ https://issues.apache.org/jira/browse/HBASE-16261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liang updated HBASE-16261: - Attachment: HBase-16261-V6.patch > MultiHFileOutputFormat Enhancement > > > Key: HBASE-16261 > URL: https://issues.apache.org/jira/browse/HBASE-16261 > Project: HBase > Issue Type: Sub-task > Components: hbase, mapreduce >Affects Versions: 2.0.0 >Reporter: Yi Liang >Assignee: Yi Liang >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16261-V1.patch, HBASE-16261-V2.patch, > HBASE-16261-V3.patch, HBASE-16261-V4.patch, HBASE-16261-V5.patch, > HBase-16261-V6.patch > > > MultiHFileOutputFormat follow HFileOutputFormat2 > (1) HFileOutputFormat2 can read one table's region split keys. and then > output multiple hfiles for one family, and each hfile map to one region. We > can add partitioner in MultiHFileOutputFormat to make it support this feature. > (2) HFileOutputFormat2 support Customized Compression algorithm for column > family and BloomFilter, also support customized DataBlockEncoding for the > output hfiles. We can also make MultiHFileOutputFormat to support these > features. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17051) libhbase++: implement RPC client and connection management
[ https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723700#comment-15723700 ] Xiaobing Zhou commented on HBASE-17051: --- Thanks [~enis] for the review. Posted v004 to address all your points except the last one. Will do that in upcoming patch. > libhbase++: implement RPC client and connection management > -- > > Key: HBASE-17051 > URL: https://issues.apache.org/jira/browse/HBASE-17051 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HBASE-17051-HBASE-14850.000.patch, > HBASE-17051-HBASE-14850.001.patch, HBASE-17051-HBASE-14850.002.patch, > HBASE-17051-HBASE-14850.003.patch, HBASE-17051-HBASE-14850.004.patch > > > This proposes building RPC client and connection management layer, which > supports the equivalent functions resides in RpcClient.java and > RpcConnection.java. Specifically, handler/pipeline concepts will be used for > implementation, similar to NettyRpcClient and NettyRpcConnection in java side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17051) libhbase++: implement RPC client and connection management
[ https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HBASE-17051: -- Attachment: HBASE-17051-HBASE-14850.004.patch > libhbase++: implement RPC client and connection management > -- > > Key: HBASE-17051 > URL: https://issues.apache.org/jira/browse/HBASE-17051 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HBASE-17051-HBASE-14850.000.patch, > HBASE-17051-HBASE-14850.001.patch, HBASE-17051-HBASE-14850.002.patch, > HBASE-17051-HBASE-14850.003.patch, HBASE-17051-HBASE-14850.004.patch > > > This proposes building RPC client and connection management layer, which > supports the equivalent functions resides in RpcClient.java and > RpcConnection.java. Specifically, handler/pipeline concepts will be used for > implementation, similar to NettyRpcClient and NettyRpcConnection in java side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-17218) [C++] Use Google Style guide and cpplint
[
https://issues.apache.org/jira/browse/HBASE-17218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar resolved HBASE-17218.
---
Resolution: Fixed
Fix Version/s: HBASE-14850
I have pushed this to the branch. [~xiaobingo], [~sudeeps] lets use
{code}
bin/format-code.sh
{code}
and
{code}
make lint
{code}
going forward for patches.
> [C++] Use Google Style guide and cpplint
>
>
> Key: HBASE-17218
> URL: https://issues.apache.org/jira/browse/HBASE-17218
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: HBASE-14850
>
> Attachments: HBASE-17218_v1.patch
>
>
> This was discussed elsewhere in other sub jiras. Let's use the Google's Style
> guide going forward (instead of LLVM one).
> There is an Eclipse formatter, a cpplint script and clang-format integration
> https://github.com/google/styleguide.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17260) Procedure v2 - Add setOwner() overload taking a User instance
[ https://issues.apache.org/jira/browse/HBASE-17260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723682#comment-15723682 ] Stephen Yuan Jiang commented on HBASE-17260: +1, LGTM > Procedure v2 - Add setOwner() overload taking a User instance > - > > Key: HBASE-17260 > URL: https://issues.apache.org/jira/browse/HBASE-17260 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17260-v0.patch > > > since we should have a User instance in most of the cases, we should just be > able to pass it, rather than converting it to getShortName() every time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17025) [Shell] Support space quota get/set via the shell
[
https://issues.apache.org/jira/browse/HBASE-17025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-17025:
---
Attachment: HBASE-17025.001.patch
.001 Parking a patch for the shell additions. {{set_quota}} has some new
additions, but {{list_quotas}} should be reusable as-is.
> [Shell] Support space quota get/set via the shell
> -
>
> Key: HBASE-17025
> URL: https://issues.apache.org/jira/browse/HBASE-17025
> Project: HBase
> Issue Type: Sub-task
> Components: shell
>Reporter: Josh Elser
> Attachments: HBASE-17025.001.patch
>
>
> Need to make sure that admins can use the shell to get/set the new space
> quotas.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15437) Response size calculated in RPCServer for warning tooLarge responses does NOT count CellScanner payload
[
https://issues.apache.org/jira/browse/HBASE-15437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry He updated HBASE-15437:
-
Assignee: Jerry He (was: deepankar)
Status: Patch Available (was: Open)
> Response size calculated in RPCServer for warning tooLarge responses does NOT
> count CellScanner payload
> ---
>
> Key: HBASE-15437
> URL: https://issues.apache.org/jira/browse/HBASE-15437
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC
>Reporter: deepankar
>Assignee: Jerry He
> Attachments: HBASE-15437-v1.patch, HBASE-15437-v2.patch,
> HBASE-15437-v3.patch, HBASE-15437.patch
>
>
> After HBASE-13158 where we respond back to RPCs with cells in the payload ,
> the protobuf response will just have the count the cells to read from
> payload, but there are set of features where we log warn in RPCServer
> whenever the response is tooLarge, but this size now is not considering the
> sizes of the cells in the PayloadCellScanner. Code form RPCServer
> {code}
> long responseSize = result.getSerializedSize();
> // log any RPC responses that are slower than the configured warn
> // response time or larger than configured warning size
> boolean tooSlow = (processingTime > warnResponseTime &&
> warnResponseTime > -1);
> boolean tooLarge = (responseSize > warnResponseSize && warnResponseSize
> > -1);
> if (tooSlow || tooLarge) {
> // when tagging, we let TooLarge trump TooSmall to keep output simple
> // note that large responses will often also be slow.
> logResponse(new Object[]{param},
> md.getName(), md.getName() + "(" + param.getClass().getName() +
> ")",
> (tooLarge ? "TooLarge" : "TooSlow"),
> status.getClient(), startTime, processingTime, qTime,
> responseSize);
> }
> {code}
> Should this feature be not supported any more or should we add a method to
> CellScanner or a new interface which returns the serialized size (but this
> might not include the compression codecs which might be used during response
> ?) Any other Idea this could be fixed ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15437) Response size calculated in RPCServer for warning tooLarge responses does NOT count CellScanner payload
[
https://issues.apache.org/jira/browse/HBASE-15437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry He updated HBASE-15437:
-
Status: Open (was: Patch Available)
> Response size calculated in RPCServer for warning tooLarge responses does NOT
> count CellScanner payload
> ---
>
> Key: HBASE-15437
> URL: https://issues.apache.org/jira/browse/HBASE-15437
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC
>Reporter: deepankar
>Assignee: deepankar
> Attachments: HBASE-15437-v1.patch, HBASE-15437-v2.patch,
> HBASE-15437-v3.patch, HBASE-15437.patch
>
>
> After HBASE-13158 where we respond back to RPCs with cells in the payload ,
> the protobuf response will just have the count the cells to read from
> payload, but there are set of features where we log warn in RPCServer
> whenever the response is tooLarge, but this size now is not considering the
> sizes of the cells in the PayloadCellScanner. Code form RPCServer
> {code}
> long responseSize = result.getSerializedSize();
> // log any RPC responses that are slower than the configured warn
> // response time or larger than configured warning size
> boolean tooSlow = (processingTime > warnResponseTime &&
> warnResponseTime > -1);
> boolean tooLarge = (responseSize > warnResponseSize && warnResponseSize
> > -1);
> if (tooSlow || tooLarge) {
> // when tagging, we let TooLarge trump TooSmall to keep output simple
> // note that large responses will often also be slow.
> logResponse(new Object[]{param},
> md.getName(), md.getName() + "(" + param.getClass().getName() +
> ")",
> (tooLarge ? "TooLarge" : "TooSlow"),
> status.getClient(), startTime, processingTime, qTime,
> responseSize);
> }
> {code}
> Should this feature be not supported any more or should we add a method to
> CellScanner or a new interface which returns the serialized size (but this
> might not include the compression codecs which might be used during response
> ?) Any other Idea this could be fixed ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15437) Response size calculated in RPCServer for warning tooLarge responses does NOT count CellScanner payload
[
https://issues.apache.org/jira/browse/HBASE-15437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry He updated HBASE-15437:
-
Attachment: HBASE-15437-v3.patch
A new patch after HBASE-17221
> Response size calculated in RPCServer for warning tooLarge responses does NOT
> count CellScanner payload
> ---
>
> Key: HBASE-15437
> URL: https://issues.apache.org/jira/browse/HBASE-15437
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC
>Reporter: deepankar
>Assignee: deepankar
> Attachments: HBASE-15437-v1.patch, HBASE-15437-v2.patch,
> HBASE-15437-v3.patch, HBASE-15437.patch
>
>
> After HBASE-13158 where we respond back to RPCs with cells in the payload ,
> the protobuf response will just have the count the cells to read from
> payload, but there are set of features where we log warn in RPCServer
> whenever the response is tooLarge, but this size now is not considering the
> sizes of the cells in the PayloadCellScanner. Code form RPCServer
> {code}
> long responseSize = result.getSerializedSize();
> // log any RPC responses that are slower than the configured warn
> // response time or larger than configured warning size
> boolean tooSlow = (processingTime > warnResponseTime &&
> warnResponseTime > -1);
> boolean tooLarge = (responseSize > warnResponseSize && warnResponseSize
> > -1);
> if (tooSlow || tooLarge) {
> // when tagging, we let TooLarge trump TooSmall to keep output simple
> // note that large responses will often also be slow.
> logResponse(new Object[]{param},
> md.getName(), md.getName() + "(" + param.getClass().getName() +
> ")",
> (tooLarge ? "TooLarge" : "TooSlow"),
> status.getClient(), startTime, processingTime, qTime,
> responseSize);
> }
> {code}
> Should this feature be not supported any more or should we add a method to
> CellScanner or a new interface which returns the serialized size (but this
> might not include the compression codecs which might be used during response
> ?) Any other Idea this could be fixed ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17249) Get/Scan's setTimeRange/setColumnFamilyTimeRange can take the TimeRange reference as the parameter instead of creating a new setColumnFamilyTimeRange instance
[ https://issues.apache.org/jira/browse/HBASE-17249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723655#comment-15723655 ] huaxiang sun commented on HBASE-17249: -- Hi [~anoop.hbase], just want to point out for cases like below, the key is not cloned and hold reference to entry.getKey(). Think that this needs to be cleaned in the future, thanks for the review. https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Get.java#L131 > Get/Scan's setTimeRange/setColumnFamilyTimeRange can take the TimeRange > reference as the parameter instead of creating a new setColumnFamilyTimeRange > instance > -- > > Key: HBASE-17249 > URL: https://issues.apache.org/jira/browse/HBASE-17249 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Attachments: HBASE-17249-master-001.patch, > HBASE-17249-master-002.patch > > > Going through the code, found For Get/Scan's > setTimeRange/setColumnFamilyTimeRange, it can use TimeRange as reference > instead of creating a new one. > Reference: > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java#L500 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java#L506 > We can implement this in a similar way as filter: > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java#L510 > I checked it is same with branch-1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17249) Get/Scan's setTimeRange/setColumnFamilyTimeRange can take the TimeRange reference as the parameter instead of creating a new setColumnFamilyTimeRange instance
[ https://issues.apache.org/jira/browse/HBASE-17249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-17249: - Attachment: HBASE-17249-master-002.patch Upload a new patch addressing Anoop's comments. > Get/Scan's setTimeRange/setColumnFamilyTimeRange can take the TimeRange > reference as the parameter instead of creating a new setColumnFamilyTimeRange > instance > -- > > Key: HBASE-17249 > URL: https://issues.apache.org/jira/browse/HBASE-17249 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Attachments: HBASE-17249-master-001.patch, > HBASE-17249-master-002.patch > > > Going through the code, found For Get/Scan's > setTimeRange/setColumnFamilyTimeRange, it can use TimeRange as reference > instead of creating a new one. > Reference: > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java#L500 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java#L506 > We can implement this in a similar way as filter: > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java#L510 > I checked it is same with branch-1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17260) Procedure v2 - Add setOwner() overload taking a User instance
[ https://issues.apache.org/jira/browse/HBASE-17260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-17260: Attachment: HBASE-17260-v0.patch > Procedure v2 - Add setOwner() overload taking a User instance > - > > Key: HBASE-17260 > URL: https://issues.apache.org/jira/browse/HBASE-17260 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17260-v0.patch > > > since we should have a User instance in most of the cases, we should just be > able to pass it, rather than converting it to getShortName() every time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17260) Procedure v2 - Add setOwner() overload taking a User instance
[ https://issues.apache.org/jira/browse/HBASE-17260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-17260: Status: Patch Available (was: Open) > Procedure v2 - Add setOwner() overload taking a User instance > - > > Key: HBASE-17260 > URL: https://issues.apache.org/jira/browse/HBASE-17260 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17260-v0.patch > > > since we should have a User instance in most of the cases, we should just be > able to pass it, rather than converting it to getShortName() every time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17260) Procedure v2 - Add setOwner() overload taking a User instance
Matteo Bertozzi created HBASE-17260: --- Summary: Procedure v2 - Add setOwner() overload taking a User instance Key: HBASE-17260 URL: https://issues.apache.org/jira/browse/HBASE-17260 Project: HBase Issue Type: Sub-task Components: proc-v2 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Trivial Fix For: 2.0.0 since we should have a User instance in most of the cases, we should just be able to pass it, rather than converting it to getShortName() every time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16700) Allow for coprocessor whitelisting
[
https://issues.apache.org/jira/browse/HBASE-16700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-16700:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.0.0
Status: Resolved (was: Patch Available)
I've committed this. Thanks Clay for the patch.
> Allow for coprocessor whitelisting
> --
>
> Key: HBASE-16700
> URL: https://issues.apache.org/jira/browse/HBASE-16700
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors
>Reporter: Clay B.
>Assignee: Clay B.
>Priority: Minor
> Labels: security
> Fix For: 2.0.0
>
> Attachments: HBASE-16700.000.patch, HBASE-16700.001.patch,
> HBASE-16700.002.patch, HBASE-16700.003.patch, HBASE-16700.004.patch,
> HBASE-16700.005.patch, HBASE-16700.006.patch, HBASE-16700.007.patch,
> HBASE-16700.008.patch
>
>
> Today one can turn off all non-system coprocessors with
> {{hbase.coprocessor.user.enabled}} however, this disables very useful things
> like Apache Phoenix's coprocessors. Some tenants of a multi-user HBase may
> also need to run bespoke coprocessors. But as an operator I would not want
> wanton coprocessor usage. Ideally, one could do one of two things:
> * Allow coprocessors defined in {{hbase-site.xml}} -- this can only be
> administratively changed in most cases
> * Allow coprocessors from table descriptors but only if the coprocessor is
> whitelisted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16700) Allow for coprocessor whitelisting
[
https://issues.apache.org/jira/browse/HBASE-16700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-16700:
--
Assignee: Clay B.
> Allow for coprocessor whitelisting
> --
>
> Key: HBASE-16700
> URL: https://issues.apache.org/jira/browse/HBASE-16700
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors
>Reporter: Clay B.
>Assignee: Clay B.
>Priority: Minor
> Labels: security
> Attachments: HBASE-16700.000.patch, HBASE-16700.001.patch,
> HBASE-16700.002.patch, HBASE-16700.003.patch, HBASE-16700.004.patch,
> HBASE-16700.005.patch, HBASE-16700.006.patch, HBASE-16700.007.patch,
> HBASE-16700.008.patch
>
>
> Today one can turn off all non-system coprocessors with
> {{hbase.coprocessor.user.enabled}} however, this disables very useful things
> like Apache Phoenix's coprocessors. Some tenants of a multi-user HBase may
> also need to run bespoke coprocessors. But as an operator I would not want
> wanton coprocessor usage. Ideally, one could do one of two things:
> * Allow coprocessors defined in {{hbase-site.xml}} -- this can only be
> administratively changed in most cases
> * Allow coprocessors from table descriptors but only if the coprocessor is
> whitelisted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17069) RegionServer writes invalid META entries for split daughters in some circumstances
[
https://issues.apache.org/jira/browse/HBASE-17069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723560#comment-15723560
]
stack commented on HBASE-17069:
---
Which IT job do you know? I was presuming ITBLL.
> RegionServer writes invalid META entries for split daughters in some
> circumstances
> --
>
> Key: HBASE-17069
> URL: https://issues.apache.org/jira/browse/HBASE-17069
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.4
>Reporter: Andrew Purtell
>Priority: Critical
> Attachments: daughter_1_d55ef81c2f8299abbddfce0445067830.log,
> daughter_2_08629d59564726da2497f70451aafcdb.log, logs.tar.gz,
> parent-393d2bfd8b1c52ce08540306659624f2.log
>
>
> I have been seeing frequent ITBLL failures testing various versions of 1.2.x.
> Over the lifetime of 1.2.x the following issues have been fixed:
> - HBASE-15315 (Remove always set super user call as high priority)
> - HBASE-16093 (Fix splits failed before creating daughter regions leave meta
> inconsistent)
> And this one is pending:
> - HBASE-17044 (Fix merge failed before creating merged region leaves meta
> inconsistent)
> I can apply all of the above to branch-1.2 and still see this failure:
> *The life of stillborn region d55ef81c2f8299abbddfce0445067830*
> *Master sees SPLITTING_NEW*
> {noformat}
> 2016-11-08 04:23:21,186 INFO [AM.ZK.Worker-pool2-t82] master.RegionStates:
> Transition null to {d55ef81c2f8299abbddfce0445067830 state=SPLITTING_NEW,
> ts=1478579001186, server=node-3.cluster,16020,1478578389506}
> {noformat}
> *The RegionServer creates it*
> {noformat}
> 2016-11-08 04:23:26,035 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for GomnU: blockCache=LruBlockCache{blockCount=34,
> currentSize=14996112, freeSize=12823716208, maxSize=12838712320,
> heapSize=14996112, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,038 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for big: blockCache=LruBlockCache{blockCount=34,
> currentSize=14996112, freeSize=12823716208, maxSize=12838712320,
> heapSize=14996112, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,442 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for meta: blockCache=LruBlockCache{blockCount=63,
> currentSize=17187656, freeSize=12821524664, maxSize=12838712320,
> heapSize=17187656, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,713 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for nwmrW: blockCache=LruBlockCache{blockCount=96,
> currentSize=19178440, freeSize=12819533880, maxSize=12838712320,
> heapSize=19178440, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,715 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for piwbr: blockCache=LruBlockCache{blockCount=96,
> currentSize=19178440, freeSize=12819533880, maxSize=12838712320,
> heapSize=19178440, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,717 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for tiny: blockCache=LruBlockCache{blockCount=96,
> currentSize=19178440, freeSize=12819533880, maxSize=12838712320,
> heapSize=19178440, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5,
[jira] [Commented] (HBASE-17170) HBase is also retrying DoNotRetryIOException because of class loader differences.
[
https://issues.apache.org/jira/browse/HBASE-17170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723515#comment-15723515
]
Hudson commented on HBASE-17170:
FAILURE: Integrated in Jenkins build HBase-1.4 #558 (See
[https://builds.apache.org/job/HBase-1.4/558/])
HBASE-17170 HBase is also retrying DoNotRetryIOException because of (tedyu: rev
600fa8de77feba07d33db24838c02fbd7d0f2161)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RemoteWithExtrasException.java
> HBase is also retrying DoNotRetryIOException because of class loader
> differences.
> -
>
> Key: HBASE-17170
> URL: https://issues.apache.org/jira/browse/HBASE-17170
> Project: HBase
> Issue Type: Bug
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17170.master.001.patch,
> HBASE-17170.master.002.patch
>
>
> The class loader used by API exposed by hadoop and the context class loader
> used by RunJar(bin/hadoop jar phoenix-client.jar …. ) are different resulting
> in classes loaded from jar not visible to other current class loader used by
> API.
> {code}
> 16/04/26 21:18:00 INFO client.RpcRetryingCaller: Call exception, tries=32,
> retries=35, started=491541 ms ago, cancelled=false, msg=
> 16/04/26 21:18:21 INFO client.RpcRetryingCaller: Call exception, tries=33,
> retries=35, started=511747 ms ago, cancelled=false, msg=
> 16/04/26 21:18:41 INFO client.RpcRetryingCaller: Call exception, tries=34,
> retries=35, started=531820 ms ago, cancelled=false, msg=
> Exception in thread "main" org.apache.phoenix.exception.PhoenixIOException:
> Failed after attempts=35, exceptions:
> Tue Apr 26 21:09:49 UTC 2016,
> RpcRetryingCaller{globalStartTime=1461704989282, pause=100, retries=35},
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.NamespaceExistException):
> org.apache.hadoop.hbase.NamespaceExistException: SYSTEM
> at
> org.apache.hadoop.hbase.master.TableNamespaceManager.create(TableNamespaceManager.java:156)
> at
> org.apache.hadoop.hbase.master.TableNamespaceManager.create(TableNamespaceManager.java:131)
> at org.apache.hadoop.hbase.master.HMaster.createNamespace(HMaster.java:2553)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.createNamespace(MasterRpcServices.java:447)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58043)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:102)
> {code}
> The actual problem is stated in the comment below
> https://issues.apache.org/jira/browse/PHOENIX-3495?focusedCommentId=15677081=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15677081
> If we are not loading hbase classes from Hadoop classpath(from where hadoop
> jars are getting loaded), then the RemoteException will not get unwrapped
> because of ClassNotFoundException and the client will keep on retrying even
> if the cause of exception is DoNotRetryIOException.
> RunJar#main() context class loader.
> {code}
> ClassLoader loader = createClassLoader(file, workDir);
> Thread.currentThread().setContextClassLoader(loader);
> Class mainClass = Class.forName(mainClassName, true, loader);
> Method main = mainClass.getMethod("main", new Class[] {
> Array.newInstance(String.class, 0).getClass()
> });
> HBase classes can be loaded from jar(phoenix-client.jar):-
> hadoop --config /etc/hbase/conf/ jar
> ~/git/apache/phoenix/phoenix-client/target/phoenix-4.9.0-HBase-1.2-client.jar
> org.apache.phoenix.mapreduce.CsvBulkLoadTool --table GIGANTIC_TABLE --input
> /tmp/b.csv --zookeeper localhost:2181
> {code}
> API(using current class loader).
> {code}
> public class RpcRetryingCaller {
> public IOException unwrapRemoteException() {
> try {
> Class realClass = Class.forName(getClassName());
> return instantiateException(realClass.asSubclass(IOException.class));
> } catch(Exception e) {
> // cannot instantiate the original exception, just return this
> }
> return this;
> }
> {code}
> *Possible solution:-*
> We can create our own HBaseRemoteWithExtrasException(extension of
> RemoteWithExtrasException) so that default class loader will be the one from
> where the hbase classes are loaded and extend unwrapRemoteException() to
> throw exception if the unwrapping doesn’t take place because of CNF
> exception?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17259) Missing functionality to remove space quota
[ https://issues.apache.org/jira/browse/HBASE-17259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-17259: --- Attachment: HBASE-17259.001.patch .001 parking a patch here for the additions. Need to get some more changes onto the feature branch before triggering QA. > Missing functionality to remove space quota > --- > > Key: HBASE-17259 > URL: https://issues.apache.org/jira/browse/HBASE-17259 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: 2.0.0 > > Attachments: HBASE-17259.001.patch > > > I'm noticing that while I have create and update APIs for quotas, I missed > the remove functionality. > Need to add public API for that and some tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723424#comment-15723424 ] Aaron Tokhy commented on HBASE-15314: - Attached the patch just now. > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Aaron Tokhy > Attachments: HBASE-15314-v2.patch, HBASE-15314-v3.patch, > HBASE-15314.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Tokhy updated HBASE-15314: Attachment: HBASE-15314-v3.patch > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Aaron Tokhy > Attachments: HBASE-15314-v2.patch, HBASE-15314-v3.patch, > HBASE-15314.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17069) RegionServer writes invalid META entries for split daughters in some circumstances
[
https://issues.apache.org/jira/browse/HBASE-17069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723415#comment-15723415
]
stack commented on HBASE-17069:
---
Ran another test over w/e that passed. 3B. Let me keep at it. Make monkey more
aggressive.
> RegionServer writes invalid META entries for split daughters in some
> circumstances
> --
>
> Key: HBASE-17069
> URL: https://issues.apache.org/jira/browse/HBASE-17069
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.4
>Reporter: Andrew Purtell
>Priority: Critical
> Attachments: daughter_1_d55ef81c2f8299abbddfce0445067830.log,
> daughter_2_08629d59564726da2497f70451aafcdb.log, logs.tar.gz,
> parent-393d2bfd8b1c52ce08540306659624f2.log
>
>
> I have been seeing frequent ITBLL failures testing various versions of 1.2.x.
> Over the lifetime of 1.2.x the following issues have been fixed:
> - HBASE-15315 (Remove always set super user call as high priority)
> - HBASE-16093 (Fix splits failed before creating daughter regions leave meta
> inconsistent)
> And this one is pending:
> - HBASE-17044 (Fix merge failed before creating merged region leaves meta
> inconsistent)
> I can apply all of the above to branch-1.2 and still see this failure:
> *The life of stillborn region d55ef81c2f8299abbddfce0445067830*
> *Master sees SPLITTING_NEW*
> {noformat}
> 2016-11-08 04:23:21,186 INFO [AM.ZK.Worker-pool2-t82] master.RegionStates:
> Transition null to {d55ef81c2f8299abbddfce0445067830 state=SPLITTING_NEW,
> ts=1478579001186, server=node-3.cluster,16020,1478578389506}
> {noformat}
> *The RegionServer creates it*
> {noformat}
> 2016-11-08 04:23:26,035 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for GomnU: blockCache=LruBlockCache{blockCount=34,
> currentSize=14996112, freeSize=12823716208, maxSize=12838712320,
> heapSize=14996112, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,038 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for big: blockCache=LruBlockCache{blockCount=34,
> currentSize=14996112, freeSize=12823716208, maxSize=12838712320,
> heapSize=14996112, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,442 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for meta: blockCache=LruBlockCache{blockCount=63,
> currentSize=17187656, freeSize=12821524664, maxSize=12838712320,
> heapSize=17187656, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,713 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for nwmrW: blockCache=LruBlockCache{blockCount=96,
> currentSize=19178440, freeSize=12819533880, maxSize=12838712320,
> heapSize=19178440, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,715 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for piwbr: blockCache=LruBlockCache{blockCount=96,
> currentSize=19178440, freeSize=12819533880, maxSize=12838712320,
> heapSize=19178440, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,717 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for tiny: blockCache=LruBlockCache{blockCount=96,
> currentSize=19178440, freeSize=12819533880, maxSize=12838712320,
> heapSize=19178440, minSize=12196776960, minFactor=0.95,
[jira] [Updated] (HBASE-16941) FavoredNodes - Split/Merge code paths
[ https://issues.apache.org/jira/browse/HBASE-16941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvel Thirumoolan updated HBASE-16941: - Attachment: HBASE-16941.master.012.patch > FavoredNodes - Split/Merge code paths > - > > Key: HBASE-16941 > URL: https://issues.apache.org/jira/browse/HBASE-16941 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-16941.master.001.patch, > HBASE-16941.master.002.patch, HBASE-16941.master.003.patch, > HBASE-16941.master.004.patch, HBASE-16941.master.005.patch, > HBASE-16941.master.006.patch, HBASE-16941.master.007.patch, > HBASE-16941.master.008.patch, HBASE-16941.master.009.patch, > HBASE-16941.master.010.patch, HBASE-16941.master.011.patch, > HBASE-16941.master.012.patch > > > This jira is to deal with the split/merge logic discussed as part of > HBASE-15532. The design document can be seen at HBASE-15531. The specific > changes are: > Split and merged regions should inherit favored node information from parent > regions. For splits also include some randomness so even if there are > subsequent splits, the regions will be more or less distributed. For split, > we include 2 FN from the parent and generate one random node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17256) Rpc handler monitoring will be removed when the task queue is full
[ https://issues.apache.org/jira/browse/HBASE-17256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17256: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.4.0 2.0.0 Status: Resolved (was: Patch Available) Thanks for the patch, Guangxu > Rpc handler monitoring will be removed when the task queue is full > -- > > Key: HBASE-17256 > URL: https://issues.apache.org/jira/browse/HBASE-17256 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 0.98.23, 1.2.4 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17256-v1.patch > > > The RPC handlers monitoring will displayed in the Web UI when the cluster has > just started. As time goes on, RPC handlers disappeared . > We have the RPC handlers listed as tasks and stored in CircularFifoBuffer. > CircularFifoBuffer is a first in first out buffer with a fixed size that > replaces its oldest element if full. > When we refresh the page, completed tasks will be removed from > CircularFifoBuffer. > Not refresh the page for a long time, the oldest element (RPC handlers) will > be replaced by new tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-17256) Rpc handler monitoring will be removed when the task queue is full
[ https://issues.apache.org/jira/browse/HBASE-17256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reassigned HBASE-17256: -- Assignee: Guangxu Cheng > Rpc handler monitoring will be removed when the task queue is full > -- > > Key: HBASE-17256 > URL: https://issues.apache.org/jira/browse/HBASE-17256 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 0.98.23, 1.2.4 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Attachments: HBASE-17256-v1.patch > > > The RPC handlers monitoring will displayed in the Web UI when the cluster has > just started. As time goes on, RPC handlers disappeared . > We have the RPC handlers listed as tasks and stored in CircularFifoBuffer. > CircularFifoBuffer is a first in first out buffer with a fixed size that > replaces its oldest element if full. > When we refresh the page, completed tasks will be removed from > CircularFifoBuffer. > Not refresh the page for a long time, the oldest element (RPC handlers) will > be replaced by new tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17239) Add UnsafeByteOperations#wrap(ByteInput, int offset, int len) API
[ https://issues.apache.org/jira/browse/HBASE-17239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17239: -- Attachment: HBASE-17239.addendum.patch I pushed the attached addendum [~ram_krish] to make "$ mvn install -Pcompile-protobuf" work again. Please check sir (Breakage spotted by our [~mbertozzi]) > Add UnsafeByteOperations#wrap(ByteInput, int offset, int len) API > - > > Key: HBASE-17239 > URL: https://issues.apache.org/jira/browse/HBASE-17239 > Project: HBase > Issue Type: Improvement > Components: Protobufs >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17239.addendum.patch, HBASE-17239_V1.patch, > HBASE-17239_V2.patch, HBASE-17239_V3.patch > > > Instead of exposing the newInstance(ByteInput, boolean) in CIS we could just > add a UnsafeByteOperations#wrap(ByteInput, offset, len). And we could just > call that and do a #newcodedInput() over that. So internally we do return a > immutable version of the ByteInput only. This way we can avoid > CIS#newInstance(ByteInput) exposure and can keep it package private as done > in COS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17154) Backup progress command usage small fix
[ https://issues.apache.org/jira/browse/HBASE-17154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17154: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.0.0 Status: Resolved (was: Patch Available) > Backup progress command usage small fix > > > Key: HBASE-17154 > URL: https://issues.apache.org/jira/browse/HBASE-17154 > Project: HBase > Issue Type: Bug >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17154-v1.patch > > > Usage should describe that backupId is optional and command w/o backup id > shows progress of a current backup session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17154) Backup progress command usage small fix
[ https://issues.apache.org/jira/browse/HBASE-17154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17154: --- Priority: Minor (was: Major) > Backup progress command usage small fix > > > Key: HBASE-17154 > URL: https://issues.apache.org/jira/browse/HBASE-17154 > Project: HBase > Issue Type: Bug >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17154-v1.patch > > > Usage should describe that backupId is optional and command w/o backup id > shows progress of a current backup session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17147) Reduce logging in BackupLogCleaner
[
https://issues.apache.org/jira/browse/HBASE-17147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17147:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.0.0
Status: Resolved (was: Patch Available)
> Reduce logging in BackupLogCleaner
> --
>
> Key: HBASE-17147
> URL: https://issues.apache.org/jira/browse/HBASE-17147
> Project: HBase
> Issue Type: Bug
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17147-v1.patch
>
>
> Every minute log cleaner logs the following:
> {quote}
> 2016-11-21 11:23:09,565 DEBUG
> [ve0524.halxg.cloudera.com,16000,1479750005565_ChoreService_1]
> impl.BackupSystemTable: Has backup sessions from hbase:backup
> 2016-11-21 11:23:09,567 DEBUG [hconnection-0x4e9c4023-shared-pool11-t85]
> ipc.RpcConnection: Use SIMPLE authentication for service ClientService,
> sasl=false
> 2016-11-21 11:23:09,568 DEBUG [hconnection-0x4e9c4023-shared-pool11-t85]
> ipc.NettyRpcConnection: Connecting to
> ve0528.halxg.cloudera.com/10.17.240.22:16020
> 2016-11-21 11:23:09,575 DEBUG [hconnection-0x4e9c4023-shared-pool11-t86]
> ipc.RpcConnection: Use SIMPLE authentication for service ClientService,
> sasl=false
> 2016-11-21 11:23:09,576 DEBUG [hconnection-0x4e9c4023-shared-pool11-t86]
> ipc.NettyRpcConnection: Connecting to
> ve0528.halxg.cloudera.com/10.17.240.22:16020
> 2016-11-21 11:23:09,579 DEBUG
> [ve0524.halxg.cloudera.com,16000,1479750005565_ChoreService_1]
> ipc.RpcConnection: Use SIMPLE authentication for service ClientService,
> sasl=false
> 2016-11-21 11:23:09,579 DEBUG
> [ve0524.halxg.cloudera.com,16000,1479750005565_ChoreService_1]
> ipc.NettyRpcConnection: Connecting to
> ve0528.halxg.cloudera.com/10.17.240.22:16020
> 2016-11-21 11:23:09,581 DEBUG
> [ve0524.halxg.cloudera.com,16000,1479750005565_ChoreService_1]
> master.BackupLogCleaner: BackupLogCleaner has no backup sessions
> 2016-11-21 11:23:09,760 DEBUG [Default-IPC-NioEventLoopGroup-1-51]
> ipc.NettyRpcDuplexHandler: shutdown connection to
> ve0528.halxg.cloudera.com/10.17.240.22:16020 because idle for a long time
> 2016-11-21 11:23:09,775 DEBUG [Default-IPC-NioEventLoopGroup-1-52]
> ipc.NettyRpcDuplexHandler: shutdown connection to
> ve0528.halxg.cloudera.com/10.17.240.22:16020 because idle for a long time
> 2016-11-21 11:23:09,778 DEBUG [Default-IPC-NioEventLoopGroup-1-53]
> ipc.NettyRpcDuplexHandler: shutdown connection to
> ve0528.halxg.cloudera.com/10.17.240.22:16020 because idle for a long time
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16314) Retry on table snapshot failure
[
https://issues.apache.org/jira/browse/HBASE-16314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722991#comment-15722991
]
Ted Yu commented on HBASE-16314:
{code}
551 LOG.warn("Snapshot attempt " + attempts + " failed for table "
+ tableName
552 + ", sleeping for " + delay + "ms", ee);
553 if (attempts < maxAttempts) {
{code}
If attempts == maxAttempts, the log message is not accurate - there would be no
sleep.
> Retry on table snapshot failure
> ---
>
> Key: HBASE-16314
> URL: https://issues.apache.org/jira/browse/HBASE-16314
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: HBASE-7912
>
> Attachments: HBASE-16314-v1.patch, HBASE-16314-v2.patch,
> HBASE-16314-v3.patch, HBASE-16314-v4.patch
>
>
> Table snapshot (during full backup) can fail with NotServingRegionException
> (splits, AM region moves). We need to add retry logic here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17199) Back-port HBASE-17151 to HBASE-7912 branch
[ https://issues.apache.org/jira/browse/HBASE-17199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17199: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Integrated with @param added for conf. > Back-port HBASE-17151 to HBASE-7912 branch > -- > > Key: HBASE-17199 > URL: https://issues.apache.org/jira/browse/HBASE-17199 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-17199-v1.patch, HBASE-17199.HBASE-7912.v2.patch > > > HBASE-17151 introduces new API to read HFile w/o instantiating block cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17221) Abstract out an interface for RpcServer.Call
[
https://issues.apache.org/jira/browse/HBASE-17221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry He updated HBASE-17221:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks for the review, [~stack]
> Abstract out an interface for RpcServer.Call
>
>
> Key: HBASE-17221
> URL: https://issues.apache.org/jira/browse/HBASE-17221
> Project: HBase
> Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-17221-v2.patch, HBASE-17221-v3.patch,
> HBASE-17221-v4.patch, HBASE-17221.patch
>
>
> RpcServer.Call is a concrete class, but it is marked as:
> {noformat}
> @InterfaceAudience.LimitedPrivate({HBaseInterfaceAudience.COPROC,
> HBaseInterfaceAudience.PHOENIX})
> {noformat}
> Let's abstract out an interface out of it for potential consumers that want
> to pass it around.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17199) Back-port HBASE-17151 to HBASE-7912 branch
[
https://issues.apache.org/jira/browse/HBASE-17199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722951#comment-15722951
]
Ted Yu commented on HBASE-17199:
lgtm
{code}
+ * Creates reader with cache configuration disabled
+ * @param fs filesystem
+ * @param path Path to file to read
{code}
Please add javadoc for conf parameter.
> Back-port HBASE-17151 to HBASE-7912 branch
> --
>
> Key: HBASE-17199
> URL: https://issues.apache.org/jira/browse/HBASE-17199
> Project: HBase
> Issue Type: Task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-17199-v1.patch, HBASE-17199.HBASE-7912.v2.patch
>
>
> HBASE-17151 introduces new API to read HFile w/o instantiating block cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17155) Unify table list output across backup / backup set commands
[
https://issues.apache.org/jira/browse/HBASE-17155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17155:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.0.0
Status: Resolved (was: Patch Available)
Thanks for the patch, Vlad.
Thanks for the review, Enis.
> Unify table list output across backup / backup set commands
> ---
>
> Key: HBASE-17155
> URL: https://issues.apache.org/jira/browse/HBASE-17155
> Project: HBase
> Issue Type: Bug
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17155-v1.patch
>
>
> Would be good to unify the output format of table list: x={ycsb,x_1} across
> all backup command line tools
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17155) Unify table list output across backup / backup set commands
[
https://issues.apache.org/jira/browse/HBASE-17155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17155:
---
Summary: Unify table list output across backup / backup set commands (was:
Unify table list output across backup/ set commands)
> Unify table list output across backup / backup set commands
> ---
>
> Key: HBASE-17155
> URL: https://issues.apache.org/jira/browse/HBASE-17155
> Project: HBase
> Issue Type: Bug
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Minor
> Attachments: HBASE-17155-v1.patch
>
>
> Would be good to unify the output format of table list: x={ycsb,x_1} across
> all backup command line tools
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-16337) Removing peers seem to be leaving spare queues
[ https://issues.apache.org/jira/browse/HBASE-16337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling resolved HBASE-16337. --- Resolution: Duplicate Closing as a dupe, thanks for pointing it out. > Removing peers seem to be leaving spare queues > -- > > Key: HBASE-16337 > URL: https://issues.apache.org/jira/browse/HBASE-16337 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Joseph > > I have been running IntegrationTestReplication repeatedly with the backported > Replication Table changes. Every other iteration of the test fails with, but > these queues should have been deleted when we removed the peers. I believe > this may be related to HBASE-16096, HBASE-16208, or HBASE-16081. > 16/08/02 08:36:07 ERROR util.AbstractHBaseTool: Error running command-line > tool > org.apache.hadoop.hbase.replication.ReplicationException: undeleted queue for > peerId: TestPeer, replicator: > hbase4124.ash2.facebook.com,16020,1470150251042, queueId: TestPeer > at > org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.checkQueuesDeleted(ReplicationPeersZKImpl.java:544) > at > org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.addPeer(ReplicationPeersZKImpl.java:127) > at > org.apache.hadoop.hbase.client.replication.ReplicationAdmin.addPeer(ReplicationAdmin.java:200) > at > org.apache.hadoop.hbase.test.IntegrationTestReplication$VerifyReplicationLoop.setupTablesAndReplication(IntegrationTestReplication.java:239) > at > org.apache.hadoop.hbase.test.IntegrationTestReplication$VerifyReplicationLoop.run(IntegrationTestReplication.java:325) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > org.apache.hadoop.hbase.test.IntegrationTestReplication.runTestFromCommandLine(IntegrationTestReplication.java:418) > at > org.apache.hadoop.hbase.IntegrationTestBase.doWork(IntegrationTestBase.java:134) > at > org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:112) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > org.apache.hadoop.hbase.test.IntegrationTestReplication.main(IntegrationTestReplication.java:424) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
