[jira] [Commented] (HBASE-16664) Timeout logic in AsyncProcess is broken
[ https://issues.apache.org/jira/browse/HBASE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567711#comment-15567711 ] Heng Chen commented on HBASE-16664: --- OK. If all your guys agree with operation timeout should be applied for batch. I will 0 vote. As for the patch, tracker in AsyncRequestFutureImpl has race condition, you could create it in constructor if callable is null. Otherwise the patch v5 is ok for me. > Timeout logic in AsyncProcess is broken > --- > > Key: HBASE-16664 > URL: https://issues.apache.org/jira/browse/HBASE-16664 > Project: HBase > Issue Type: Bug >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: 1.patch, HBASE-16664-branch-1-v1.patch, > HBASE-16664-branch-1-v1.patch, HBASE-16664-branch-1.1-v1.patch, > HBASE-16664-branch-1.2-v1.patch, HBASE-16664-branch-1.3-v1.patch, > HBASE-16664-v1.patch, HBASE-16664-v2.patch, HBASE-16664-v3.patch, > HBASE-16664-v4.patch, HBASE-16664-v5.patch, testhcm.patch > > > Have not checked the root cause, but I think timeout of all operations in > AsyncProcess is broken -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16663) JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
[
https://issues.apache.org/jira/browse/HBASE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567696#comment-15567696
]
Hudson commented on HBASE-16663:
FAILURE: Integrated in Jenkins build HBase-1.4 #462 (See
[https://builds.apache.org/job/HBase-1.4/462/])
HBASE-16663 JMX ConnectorServer stopped when unauthorized user try to
(apurtell: rev 6db4ef84796225dcf780021cbf567eaeed793f1e)
* (add)
hbase-server/src/test/java/org/apache/hadoop/hbase/TestJMXConnectorServer.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerCoprocessorHost.java
> JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
>
>
> Key: HBASE-16663
> URL: https://issues.apache.org/jira/browse/HBASE-16663
> Project: HBase
> Issue Type: Bug
> Components: metrics, security
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 0.98.24
>
> Attachments: HBASE-16663-0.98-V4.patch, HBASE-16663-0.98.patch,
> HBASE-16663-V2.patch, HBASE-16663-V3.patch, HBASE-16663-V4.patch,
> HBASE-16663-branch-1.patch, HBASE-16663.patch
>
>
> After HBASE-16284, unauthorized user will not able allowed to stop
> HM/RS/cluster, but while executing "cpHost.preStopMaster()", ConnectorServer
> will be stopped before AccessController validation.
> hbase-site.xml,
> {noformat}
>
> hbase.coprocessor.master.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
>
> hbase.coprocessor.regionserver.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
> {noformat}
> HBaseAdmin.stopMaster(),
> {noformat}
> 2016-09-20 21:12:26,796 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 21:13:55,380 WARN
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 21:14:00,495 ERROR
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> master.MasterRpcServices: Exception occurred while stopping master
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopMaster(AccessController.java:1297)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost$68.call(MasterCoprocessorHost.java:821)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1188)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.preStopMaster(MasterCoprocessorHost.java:817)
> at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2352)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.stopMaster(MasterRpcServices.java:1364)
> {noformat}
> HBaseAdmin.stopRegionServer(rs-host-port),
> {noformat}
> 2016-09-20 20:59:01,234 INFO
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 20:59:01,250 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 20:59:01,253 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> regionserver.HRegionServer: The region server did not stop
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:84)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.execOperation(RegionServerCoprocessorHost.java:256)
>
[jira] [Commented] (HBASE-16729) Define the behavior of (default) empty FilterList
[
https://issues.apache.org/jira/browse/HBASE-16729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567682#comment-15567682
]
ramkrishna.s.vasudevan commented on HBASE-16729:
Better to revisit other APIs also.
> Define the behavior of (default) empty FilterList
> -
>
> Key: HBASE-16729
> URL: https://issues.apache.org/jira/browse/HBASE-16729
> Project: HBase
> Issue Type: Wish
>Reporter: ChiaPing Tsai
>Priority: Trivial
>
> Current empty FilterList filters all data, because the
> FilterList#isFamilyEssential always returns false which causes the null cell
> retrieved by RegionScannerImpl.storeHeap.
> It seems to me that empty FilterList should do nothing, because the following
> code is common.
> {noformat}
> private static Filter makeFilter() {
> FilterList filterList = new FilterList ();
> for (some conditions) {
> // add some filters. Or nothing to add.
> }
> return filterList;
> }
> {noformat}
> If we keep the current logic which filters all data, we should add enough
> comments to explain it. Or add the FilterList#size() or FilterList#empty()
> for preventing filtering all data.
> Any comments? Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15437) Response size calculated in RPCServer for warning tooLarge responses does NOT count CellScanner payload
[
https://issues.apache.org/jira/browse/HBASE-15437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567685#comment-15567685
]
Jerry He commented on HBASE-15437:
--
That is my thinking as well, Anoop.
The last patch gets the size after encoding/compression. Or I get it wrong?
> Response size calculated in RPCServer for warning tooLarge responses does NOT
> count CellScanner payload
> ---
>
> Key: HBASE-15437
> URL: https://issues.apache.org/jira/browse/HBASE-15437
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC
>Reporter: deepankar
>Assignee: deepankar
> Attachments: HBASE-15437-v1.patch, HBASE-15437.patch
>
>
> After HBASE-13158 where we respond back to RPCs with cells in the payload ,
> the protobuf response will just have the count the cells to read from
> payload, but there are set of features where we log warn in RPCServer
> whenever the response is tooLarge, but this size now is not considering the
> sizes of the cells in the PayloadCellScanner. Code form RPCServer
> {code}
> long responseSize = result.getSerializedSize();
> // log any RPC responses that are slower than the configured warn
> // response time or larger than configured warning size
> boolean tooSlow = (processingTime > warnResponseTime &&
> warnResponseTime > -1);
> boolean tooLarge = (responseSize > warnResponseSize && warnResponseSize
> > -1);
> if (tooSlow || tooLarge) {
> // when tagging, we let TooLarge trump TooSmall to keep output simple
> // note that large responses will often also be slow.
> logResponse(new Object[]{param},
> md.getName(), md.getName() + "(" + param.getClass().getName() +
> ")",
> (tooLarge ? "TooLarge" : "TooSlow"),
> status.getClient(), startTime, processingTime, qTime,
> responseSize);
> }
> {code}
> Should this feature be not supported any more or should we add a method to
> CellScanner or a new interface which returns the serialized size (but this
> might not include the compression codecs which might be used during response
> ?) Any other Idea this could be fixed ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16729) Define the behavior of (default) empty FilterList
[
https://issues.apache.org/jira/browse/HBASE-16729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567678#comment-15567678
]
ramkrishna.s.vasudevan commented on HBASE-16729:
I think it is better we use the size of the filter list to ensure what to be
done. If there are no elements then isFamilyEssential should return true IMO.
It is equivalent to no filters.
> Define the behavior of (default) empty FilterList
> -
>
> Key: HBASE-16729
> URL: https://issues.apache.org/jira/browse/HBASE-16729
> Project: HBase
> Issue Type: Wish
>Reporter: ChiaPing Tsai
>Priority: Trivial
>
> Current empty FilterList filters all data, because the
> FilterList#isFamilyEssential always returns false which causes the null cell
> retrieved by RegionScannerImpl.storeHeap.
> It seems to me that empty FilterList should do nothing, because the following
> code is common.
> {noformat}
> private static Filter makeFilter() {
> FilterList filterList = new FilterList ();
> for (some conditions) {
> // add some filters. Or nothing to add.
> }
> return filterList;
> }
> {noformat}
> If we keep the current logic which filters all data, we should add enough
> comments to explain it. Or add the FilterList#size() or FilterList#empty()
> for preventing filtering all data.
> Any comments? Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16731) Inconsistent results from the Get/Scan if we use the empty FilterList
[
https://issues.apache.org/jira/browse/HBASE-16731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567675#comment-15567675
]
ramkrishna.s.vasudevan commented on HBASE-16731:
Nice one.
So here
{code}
if (this.filter == null || !scan.doLoadColumnFamiliesOnDemand()
|| this.filter.isFamilyEssential(entry.getKey())) {
scanners.add(scanner);
} else {
joinedScanners.add(scanner);
}
{code}
In case of scans the region's default true property for on demand family
loading works fine and hence it goes to the else case but in 'gets' it goes to
the 'if' case. Just saw in code that we have a region level property for this.
So to make things simple here is it not enough that
{code}
Scan scan = new Scan(get);
{code}
After this just set the scan with
{code}
scan.setLoadColumnFamiliesOnDemand(region.isLoadingCfsOnDemandDefault());
{code}
So you plan to override this from the client side also - by adding a new
feature to 'gets' also?
> Inconsistent results from the Get/Scan if we use the empty FilterList
> -
>
> Key: HBASE-16731
> URL: https://issues.apache.org/jira/browse/HBASE-16731
> Project: HBase
> Issue Type: Bug
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16731.v0.patch, HBASE-16731.v1.patch,
> HBASE-16731.v2.patch
>
>
> RSRpcServices#get() converts the Get to Scan without
> scan#setLoadColumnFamiliesOnDemand. It causes that the result retrieved from
> Get and Scan will be different if we use the empty filter. Scan doesn't
> return any data but Get does.
> see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729]
> Any comments? Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567669#comment-15567669 ] Duo Zhang commented on HBASE-15921: --- [~enis] Any other concerns? For curator, I think it will simplify our client code a lot as we do not need to deal with retry and recovery, and the API is more friendly. And the client only needs to access a small set of data on zk, you can see the ClusterRegistry interface in patch v6, that's all. So I think implement the client logic separately is good for both the client and server implementations as the zk implementation does not need to deal with client any more. And I know that sometimes we need the client API to do something at server side, but I think this is another story. We can discuss it later. And the client side zk related implementation is not very complicated so if we decide to remove the dependency of curator it is OK to reimplement it at that time. Thanks. > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Jurriaan Mous >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15921-v2.patch, HBASE-15921-v3.patch, > HBASE-15921-v4.patch, HBASE-15921-v5.patch, HBASE-15921-v6.patch, > HBASE-15921-v7.patch, HBASE-15921-v8.patch, HBASE-15921.demo.patch, > HBASE-15921.patch, HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16505) Save deadline in RpcCallContext according to request's timeout
[ https://issues.apache.org/jira/browse/HBASE-16505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567657#comment-15567657 ] Phil Yang commented on HBASE-16505: --- Change to a new title, it may be more clear. Thanks > Save deadline in RpcCallContext according to request's timeout > -- > > Key: HBASE-16505 > URL: https://issues.apache.org/jira/browse/HBASE-16505 > Project: HBase > Issue Type: Sub-task >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0 > > Attachments: HBASE-16505-branch-1-v1.patch, > HBASE-16505-branch-1-v2.patch, HBASE-16505-branch-1-v3.patch, > HBASE-16505-v1.patch, HBASE-16505-v10.patch, HBASE-16505-v10.patch, > HBASE-16505-v11.patch, HBASE-16505-v11.patch, HBASE-16505-v12.patch, > HBASE-16505-v13.patch, HBASE-16505-v2.patch, HBASE-16505-v3.patch, > HBASE-16505-v4.patch, HBASE-16505-v5.patch, HBASE-16505-v6.patch, > HBASE-16505-v7.patch, HBASE-16505-v8.patch, HBASE-16505-v9.patch > > > If we want to know the correct setting of timeout in read/write path, we need > add a new parameter in operation-methods of Region. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16505) Save deadline in RpcCallContext according to request's timeout
[ https://issues.apache.org/jira/browse/HBASE-16505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-16505: -- Summary: Save deadline in RpcCallContext according to request's timeout (was: Pass deadline to HRegion operations) > Save deadline in RpcCallContext according to request's timeout > -- > > Key: HBASE-16505 > URL: https://issues.apache.org/jira/browse/HBASE-16505 > Project: HBase > Issue Type: Sub-task >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0 > > Attachments: HBASE-16505-branch-1-v1.patch, > HBASE-16505-branch-1-v2.patch, HBASE-16505-branch-1-v3.patch, > HBASE-16505-v1.patch, HBASE-16505-v10.patch, HBASE-16505-v10.patch, > HBASE-16505-v11.patch, HBASE-16505-v11.patch, HBASE-16505-v12.patch, > HBASE-16505-v13.patch, HBASE-16505-v2.patch, HBASE-16505-v3.patch, > HBASE-16505-v4.patch, HBASE-16505-v5.patch, HBASE-16505-v6.patch, > HBASE-16505-v7.patch, HBASE-16505-v8.patch, HBASE-16505-v9.patch > > > If we want to know the correct setting of timeout in read/write path, we need > add a new parameter in operation-methods of Region. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[
https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567648#comment-15567648
]
Heng Chen commented on HBASE-16807:
---
Just skip the cache is OK here?
> RegionServer will fail to report new active Hmaster until
> HMaster/RegionServer failover
> ---
>
> Key: HBASE-16807
> URL: https://issues.apache.org/jira/browse/HBASE-16807
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>
> It's little weird, but it happened in the product environment that few
> RegionServer missed master znode create notification on master failover. In
> that case ZooKeeperNodeTracker will not refresh the cached data and
> MasterAddressTracker
> will always return old active HM detail to Region server on ServiceException.
> Though We create region server stub on failure but without refreshing the
> MasterAddressTracker data.
> In HRegionServer.createRegionServerStatusStub()
> {code}
> boolean refresh = false; // for the first time, use cached data
> RegionServerStatusService.BlockingInterface intf = null;
> boolean interrupted = false;
> try {
> while (keepLooping()) {
> sn = this.masterAddressTracker.getMasterAddress(refresh);
> if (sn == null) {
> if (!keepLooping()) {
> // give up with no connection.
> LOG.debug("No master found and cluster is stopped; bailing out");
> return null;
> }
> if (System.currentTimeMillis() > (previousLogTime + 1000)) {
> LOG.debug("No master found; retry");
> previousLogTime = System.currentTimeMillis();
> }
> refresh = true; // let's try pull it from ZK directly
> if (sleep(200)) {
> interrupted = true;
> }
> continue;
> }
> {code}
> Here we refresh node only when 'sn' is NULL otherwise it will use same cached
> data.
> So in above case RegionServer will never report active HMaster successfully
> until HMaster failover or RegionServer restart.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16663) JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
[
https://issues.apache.org/jira/browse/HBASE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567638#comment-15567638
]
Hudson commented on HBASE-16663:
SUCCESS: Integrated in Jenkins build HBase-1.2-JDK8 #40 (See
[https://builds.apache.org/job/HBase-1.2-JDK8/40/])
HBASE-16663 JMX ConnectorServer stopped when unauthorized user try to
(apurtell: rev 6123106496819bd538a0a7744c54f57795289b8e)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerCoprocessorHost.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
* (add)
hbase-server/src/test/java/org/apache/hadoop/hbase/TestJMXConnectorServer.java
> JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
>
>
> Key: HBASE-16663
> URL: https://issues.apache.org/jira/browse/HBASE-16663
> Project: HBase
> Issue Type: Bug
> Components: metrics, security
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 0.98.24
>
> Attachments: HBASE-16663-0.98-V4.patch, HBASE-16663-0.98.patch,
> HBASE-16663-V2.patch, HBASE-16663-V3.patch, HBASE-16663-V4.patch,
> HBASE-16663-branch-1.patch, HBASE-16663.patch
>
>
> After HBASE-16284, unauthorized user will not able allowed to stop
> HM/RS/cluster, but while executing "cpHost.preStopMaster()", ConnectorServer
> will be stopped before AccessController validation.
> hbase-site.xml,
> {noformat}
>
> hbase.coprocessor.master.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
>
> hbase.coprocessor.regionserver.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
> {noformat}
> HBaseAdmin.stopMaster(),
> {noformat}
> 2016-09-20 21:12:26,796 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 21:13:55,380 WARN
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 21:14:00,495 ERROR
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> master.MasterRpcServices: Exception occurred while stopping master
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopMaster(AccessController.java:1297)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost$68.call(MasterCoprocessorHost.java:821)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1188)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.preStopMaster(MasterCoprocessorHost.java:817)
> at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2352)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.stopMaster(MasterRpcServices.java:1364)
> {noformat}
> HBaseAdmin.stopRegionServer(rs-host-port),
> {noformat}
> 2016-09-20 20:59:01,234 INFO
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 20:59:01,250 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 20:59:01,253 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> regionserver.HRegionServer: The region server did not stop
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:84)
> at
>
[jira] [Commented] (HBASE-16783) Use ByteBufferPool for the header and message during Rpc response
[ https://issues.apache.org/jira/browse/HBASE-16783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567633#comment-15567633 ] ramkrishna.s.vasudevan commented on HBASE-16783: Yes. I have done some work to do the point 2 already. Patch will come later. I think the default size is 64K and not 64M? Correct me if am wrong. Definitly if it was 64M I would not have done this change. bq.All using the pool is not making sense IMO May be yes. That may be we can handle if we go with the 2nd way? > Use ByteBufferPool for the header and message during Rpc response > - > > Key: HBASE-16783 > URL: https://issues.apache.org/jira/browse/HBASE-16783 > Project: HBase > Issue Type: Improvement >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Attachments: HBASE-16783.patch, HBASE-16783_1.patch > > > With ByteBufferPool in place we could avoid the byte[] creation in > RpcServer#createHeaderAndMessageBytes and try using the Buffer from the pool > rather than creating byte[] every time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16664) Timeout logic in AsyncProcess is broken
[ https://issues.apache.org/jira/browse/HBASE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567630#comment-15567630 ] stack commented on HBASE-16664: --- Operation timeout has always been a murky story full of surprises seemingly going out of its way to keep the hapless operator baffled. The descriptions in Table on how the operation timeout works (or will work) promises to clean up this mess. Operation timeout should apply whether a fat batch or a small Get. Above it is suggested that if you need a short timeout for small Gets but a long one for big batches of MultiGets or Puts, then you could use two Tables differently configured. That'd work It might frustrate or surprise the operator some but it is 'understandable'. For MultiPuts, could we encourage folks to use the BufferedMutator if doing large Puts? We already do I suppose. Could we add an operation timeout on the BufferedMutator, one that would override the Table's operation timeout? Would that help? That'd leave multiget. AP does this in parallel? Maybe then it could live within the operation timeout for the table as long as the multiget was many small ones rather than a fat one? Would allowing the setting of a timeout on each Get/Mutate object be crazy? This timeout would override the Table setting. Would 'work' for the single object case; e.g. a Get or Put. Harder to reason about for the Lists and batches. Could just get messy fast and be hard to reason about. > Timeout logic in AsyncProcess is broken > --- > > Key: HBASE-16664 > URL: https://issues.apache.org/jira/browse/HBASE-16664 > Project: HBase > Issue Type: Bug >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: 1.patch, HBASE-16664-branch-1-v1.patch, > HBASE-16664-branch-1-v1.patch, HBASE-16664-branch-1.1-v1.patch, > HBASE-16664-branch-1.2-v1.patch, HBASE-16664-branch-1.3-v1.patch, > HBASE-16664-v1.patch, HBASE-16664-v2.patch, HBASE-16664-v3.patch, > HBASE-16664-v4.patch, HBASE-16664-v5.patch, testhcm.patch > > > Have not checked the root cause, but I think timeout of all operations in > AsyncProcess is broken -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16811) Remove mob sweep job
[ https://issues.apache.org/jira/browse/HBASE-16811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567622#comment-15567622 ] Hudson commented on HBASE-16811: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1770 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1770/]) HBASE-16811 Remove mob sweep job. (appy: rev eb52e26822125028587201728f19caff4ae69976) * (delete) hbase-server/src/test/java/org/apache/hadoop/hbase/mob/mapreduce/TestMobSweepReducer.java * (delete) hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/SweepMapper.java * (delete) hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/SweepReducer.java * (delete) hbase-server/src/test/java/org/apache/hadoop/hbase/mob/mapreduce/TestMobSweepMapper.java * (delete) hbase-server/src/test/java/org/apache/hadoop/hbase/mob/mapreduce/TestMobSweepJob.java * (delete) hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/MobFilePathHashPartitioner.java * (delete) hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/Sweeper.java * (delete) hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/SweepJobNodeTracker.java * (delete) hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/SweepJob.java * (delete) hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/MemStoreWrapper.java > Remove mob sweep job > > > Key: HBASE-16811 > URL: https://issues.apache.org/jira/browse/HBASE-16811 > Project: HBase > Issue Type: Task >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-16811.master.001.patch > > > Discussed here: > http://mail-archives.apache.org/mod_mbox/hbase-dev/201610.mbox/%3CCAAjhxro%3Dt62K44dV2wUtq1hqYLogZ45M3oeNOFZPcnwcSY4_DQ%40mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15109) HM/RS failed to start when "fs.hdfs.impl.disable.cache" is set to true
[
https://issues.apache.org/jira/browse/HBASE-15109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567619#comment-15567619
]
Hudson commented on HBASE-15109:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1770 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1770/])
HBASE-15109 HM/RS failed to start when "fs.hdfs.impl.disable.cache" is (tedyu:
rev ec87b4bfe28fe4947b36172bdeba3bc61921e413)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ShutdownHook.java
> HM/RS failed to start when "fs.hdfs.impl.disable.cache" is set to true
> --
>
> Key: HBASE-15109
> URL: https://issues.apache.org/jira/browse/HBASE-15109
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15109-V2.patch, HBASE-15109-branch-1.patch,
> HBASE-15109.patch, HBASE-15109.patch
>
>
> HMaster failed to start during installing ShutdownHook when
> "fs.hdfs.impl.disable.cache" is set to true.
> {noformat}
> 2016-10-10 10:33:28,367 FATAL [master/HOST-10-18-106-146/10.18.106.146:16000]
> master.HMaster: Unhandled: Failed suppression of fs shutdown hook:
> org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@19a003b4
> java.lang.RuntimeException: Failed suppression of fs shutdown hook:
> org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@19a003b4
> at
> org.apache.hadoop.hbase.regionserver.ShutdownHook.suppressHdfsShutdownHook(ShutdownHook.java:204)
> at
> org.apache.hadoop.hbase.regionserver.ShutdownHook.install(ShutdownHook.java:84)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:950)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> And RegionServer is in blocking state until a master is available,
> {noformat}
> "regionserver/HOST-10-18-106-146/10.18.106.146:16020" #17 prio=5 os_prio=0
> tid=0x00e4e800 nid=0xdc38 in Object.wait() [0x7f329dac6000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:159)
> - locked <0xae25c0c8> (a
> org.apache.hadoop.hbase.zookeeper.MasterAddressTracker)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.blockAndCheckIfStopped(HRegionServer.java:870)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:809)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:783)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:943)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> During installation, first time it will try to suppress the HDFS shutdownhook
> by removing the hdfsclientfinalizer from HDFS. Since
> fs.hdfs.impl.disable.cache is enabled, so removal will be unsuccessful from
> HDFS (FS is not cached) and RuntimeException will be thrown with "Failed
> suppression of fs shutdown hook" message.
> In ShutdownHook,
> {code}
> if (!fsShutdownHooks.containsKey(hdfsClientFinalizer) &&
> !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> throw new RuntimeException("Failed suppression of fs shutdown hook: " +
> hdfsClientFinalizer);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16663) JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
[
https://issues.apache.org/jira/browse/HBASE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567625#comment-15567625
]
Hudson commented on HBASE-16663:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1770 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1770/])
HBASE-16663 JMX ConnectorServer stopped when unauthorized user try to
(apurtell: rev 0af10f89426dd97a1d599e29f1e301e74f9f9e99)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerCoprocessorHost.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
* (add)
hbase-server/src/test/java/org/apache/hadoop/hbase/TestJMXConnectorServer.java
> JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
>
>
> Key: HBASE-16663
> URL: https://issues.apache.org/jira/browse/HBASE-16663
> Project: HBase
> Issue Type: Bug
> Components: metrics, security
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 0.98.24
>
> Attachments: HBASE-16663-0.98-V4.patch, HBASE-16663-0.98.patch,
> HBASE-16663-V2.patch, HBASE-16663-V3.patch, HBASE-16663-V4.patch,
> HBASE-16663-branch-1.patch, HBASE-16663.patch
>
>
> After HBASE-16284, unauthorized user will not able allowed to stop
> HM/RS/cluster, but while executing "cpHost.preStopMaster()", ConnectorServer
> will be stopped before AccessController validation.
> hbase-site.xml,
> {noformat}
>
> hbase.coprocessor.master.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
>
> hbase.coprocessor.regionserver.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
> {noformat}
> HBaseAdmin.stopMaster(),
> {noformat}
> 2016-09-20 21:12:26,796 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 21:13:55,380 WARN
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 21:14:00,495 ERROR
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> master.MasterRpcServices: Exception occurred while stopping master
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopMaster(AccessController.java:1297)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost$68.call(MasterCoprocessorHost.java:821)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1188)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.preStopMaster(MasterCoprocessorHost.java:817)
> at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2352)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.stopMaster(MasterRpcServices.java:1364)
> {noformat}
> HBaseAdmin.stopRegionServer(rs-host-port),
> {noformat}
> 2016-09-20 20:59:01,234 INFO
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 20:59:01,250 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 20:59:01,253 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> regionserver.HRegionServer: The region server did not stop
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:84)
> at
>
[jira] [Commented] (HBASE-16146) Counters are expensive...
[ https://issues.apache.org/jira/browse/HBASE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567624#comment-15567624 ] Hudson commented on HBASE-16146: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1770 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1770/]) HBASE-16146 Remove thread local usage in Counter (garyh: rev 7b0acc292e1854b09c6cedc4dae1f6dae07779bf) * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/util/Counter.java > Counters are expensive... > - > > Key: HBASE-16146 > URL: https://issues.apache.org/jira/browse/HBASE-16146 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Gary Helmling > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16146.001.patch, HBASE-16146.branch-1.001.patch, > HBASE-16146.branch-1.3.001.patch, counters.patch, less_and_less_counters.png > > > Doing workloadc, perf shows 10%+ of CPU being spent on counter#add. If I > disable some of the hot ones -- see patch -- I can get 10% more throughput > (390k to 440k). Figure something better. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16622) Fix some issues with the HBase reference guide
[
https://issues.apache.org/jira/browse/HBASE-16622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567623#comment-15567623
]
Hudson commented on HBASE-16622:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1770 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1770/])
HBASE-16622 Fix some issues with the HBase reference guide (stack: rev
6b346ad046439e0b59cc4067a43208ab87cb173b)
* (edit) src/main/asciidoc/_chapters/hbase_apis.adoc
* (edit) src/main/asciidoc/_chapters/developer.adoc
> Fix some issues with the HBase reference guide
> --
>
> Key: HBASE-16622
> URL: https://issues.apache.org/jira/browse/HBASE-16622
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: alexxiyang
>Assignee: alexxiyang
> Fix For: 2.0.0
>
> Attachments: HBASE-16622-v0.diff, HBASE-16622-v1.diff,
> HBASE-16622-v2.diff
>
>
> 1.
> {code}
> if (admin.tableExists(tableName)) {
> System.out.println("Table does not exist.");
> System.exit(-1);
> }
> {code}
> This should be
> {code}
> if (!admin.tableExists(tableName)) {
> {code}
> 2.
> SNAPPY is not suitable for begginer. They may get exceptions like
> {code}
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.DoNotRetryIOException):
> org.apache.hadoop.hbase.DoNotRetryIOException: Compression algorithm
> 'snappy' previously failed test. Set hbase.table.sanity.checks to false at
> conf or table descriptor if you want to bypass sanity checks
> at
> org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:1701)
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1569)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1491)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:462)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55682)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2178)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> So the code below
> {code}
> table.addFamily(new
> HColumnDescriptor(CF_DEFAULT).setCompressionType(Algorithm.SNAPPY));
> {code}
> it better to change into
> {code}
> table.addFamily(new
> HColumnDescriptor(CF_DEFAULT).setCompressionType(Algorithm.NONE));
> {code}
> 3.
> Before modify column family , get the table from connection
> Change
> {code}
> HTableDescriptor table = new HTableDescriptor(tableName);
> {code}
> into
> {code}
> Table table = connection.getTable(TableName.valueOf(tablename));
> {code}
> 4.
> In 143.1.1. Code Formatting
> it just said
> {code}
> Still in Preferences, click . Be sure the following options are
> selected:Apache HBase ™ Reference Guide
> {code}
> But nothing after click. It should be Java->Editor->Save Actions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16802) Procedure v2 - group procedure cleaning
[ https://issues.apache.org/jira/browse/HBASE-16802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567621#comment-15567621 ] Hudson commented on HBASE-16802: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1770 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1770/]) HBASE-16802 Procedure v2 - group procedure cleaning (matteo.bertozzi: rev 662a1b241f05f93aee9a5b05d7929a482a5bfcd5) * (edit) hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/wal/TestWALProcedureStore.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/NoopProcedureStore.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/ProcedureStore.java * (edit) hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/ProcedureTestingUtility.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java > Procedure v2 - group procedure cleaning > --- > > Key: HBASE-16802 > URL: https://issues.apache.org/jira/browse/HBASE-16802 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-16802-v0.patch > > > group the cleaning of the evicted procedures -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16784) Make use of ExtendedCell#write(OutputStream os) for the default HFileWriter#append()
[ https://issues.apache.org/jira/browse/HBASE-16784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16784: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master. Thanks for the review. > Make use of ExtendedCell#write(OutputStream os) for the default > HFileWriter#append() > > > Key: HBASE-16784 > URL: https://issues.apache.org/jira/browse/HBASE-16784 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-16784.patch, HBASE-16784_1.patch, > HBASE-16784_2.patch, HBASE-16784_3.patch > > > Initially this I was thinking we need to add an interface to represent the > fact that the key is contiguous. But since Extendedcell is added > encapsulating all the internal interfaces and adds a write(OutputStream , > boolean) and tries to exploit the fact that the Cell is in KV serialized > format. Hence we can make use of it in HFileWriter#append() code in case of > No encoding case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567605#comment-15567605 ] Vladimir Rodionov commented on HBASE-14123: --- The last patch is on RB https://reviews.apache.org/r/52748 > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt, > 14123-master.v21.txt, 14123-master.v24.txt, 14123-master.v25.txt, > 14123-master.v27.txt, 14123-master.v28.txt, 14123-master.v29.full.txt, > 14123-master.v3.txt, 14123-master.v5.txt, 14123-master.v6.txt, > 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, > HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, > HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, > HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, > HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15513) hbase.hregion.memstore.chunkpool.maxsize is 0.0 by default
[ https://issues.apache.org/jira/browse/HBASE-15513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567599#comment-15567599 ] Vladimir Rodionov commented on HBASE-15513: --- 100% guarantees that we won't be reallocating buffers too often (may be never), anything less than 100% won't gives us such a guarantee. That is going to be partial chunk pool. > hbase.hregion.memstore.chunkpool.maxsize is 0.0 by default > -- > > Key: HBASE-15513 > URL: https://issues.apache.org/jira/browse/HBASE-15513 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-15513-v1.patch > > > That results in excessive MemStoreLAB chunk allocations because we can not > reuse them. Not sure, why it has been disabled, by default. May be the code > has not been tested well? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16799) CP exposed Store should not expose unwanted APIs
[ https://issues.apache.org/jira/browse/HBASE-16799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567597#comment-15567597 ] ramkrishna.s.vasudevan commented on HBASE-16799: Why to remove - whether we can add @PRivate tag to those APIs? If any LimitedPrivate interfaces are exposed to CPs that can contain some @Private methods in that? I think that is what we discussed too. > CP exposed Store should not expose unwanted APIs > - > > Key: HBASE-16799 > URL: https://issues.apache.org/jira/browse/HBASE-16799 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16799.patch, HBASE-16799.patch, > HBASE-16799_V2.patch > > > Store is exposed to CPs. The main use cases I can think of are getting store > scanner and other getters which return different states like memstore size, > max seqId etc. Those make sense. > But we added many other APIs which changes the state of the memstore, bulk > load files etc into this interface. Even an API which expose the memstore > itself!. This allow adding mutations into memstore bypassing all steps in > region. We track the memstore size per region level as well as globally. > These only allow us to flush region at sizes and/or flush selected regions > because of global heap pressure. Now if a CP get hold of store and/or > memstore, it can add mutations with out knowledge of these size accounting > and possibly OOME the RS. Similar way the bulk load related APIs. At HRegion > level, there are steps done (WAL write etc) after the bulk load HFile on > store. So bypassing these wont be correct. > In this jira, plan is to remove all such leaked APIs from Store. They are > called from HRegion and we can type cast to HStore to call them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit
[ https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567588#comment-15567588 ] Anoop Sam John commented on HBASE-16752: Right now what client is doing when it is seeing RequestTooBigException ? I dont mean in case of replication. (In normal user facing client). Trying to get what the callId is used for. What is async RPC client? So the idea on the replication part is that unless the src is also upgraded, the replication will fail (?) Do u plan to have some special mechanism at src end to temp halt it? > Upgrading from 1.2 to 1.3 can lead to replication failures due to difference > in RPC size limit > -- > > Key: HBASE-16752 > URL: https://issues.apache.org/jira/browse/HBASE-16752 > Project: HBase > Issue Type: Bug > Components: Replication, rpc >Affects Versions: 1.3.0 >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > > In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by > default to 256 MB. This means that during upgrade scenarios (or when source > is 1.2 peer is already on 1.3), it's possible to encounter a situation where > we try to send an rpc with size greater than 256 MB because we never unroll a > WALEdit while sending replication traffic. > RpcServer throws the underlying exception locally, but closes the connection > with returning the underlying error to the client, and client only sees a > "Broken pipe" error. > I am not sure what is the proper fix here (or if one is needed) to make sure > this does not happen, but we should return the underlying exception to the > RpcClient, because without it, it can be difficult to diagnose the problem, > especially for someone new to HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15513) hbase.hregion.memstore.chunkpool.maxsize is 0.0 by default
[ https://issues.apache.org/jira/browse/HBASE-15513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567582#comment-15567582 ] Anoop Sam John commented on HBASE-15513: Do we need 100% of the max global heap space size as def pool size? I feel it can be lesser. > hbase.hregion.memstore.chunkpool.maxsize is 0.0 by default > -- > > Key: HBASE-15513 > URL: https://issues.apache.org/jira/browse/HBASE-15513 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-15513-v1.patch > > > That results in excessive MemStoreLAB chunk allocations because we can not > reuse them. Not sure, why it has been disabled, by default. May be the code > has not been tested well? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16642) Use DelayQueue instead of TimeoutBlockingQueue
[ https://issues.apache.org/jira/browse/HBASE-16642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567573#comment-15567573 ] Hiroshi Ikeda commented on HBASE-16642: --- V2 mixes some other issues and I can't say for sure except that it still calls the current time twice per one comparison. > Use DelayQueue instead of TimeoutBlockingQueue > -- > > Key: HBASE-16642 > URL: https://issues.apache.org/jira/browse/HBASE-16642 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Hiroshi Ikeda >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16642-v2.patch, HBASE-16642.master.V1.patch > > > Enqueue poisons in order to wake up and end the internal threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15513) hbase.hregion.memstore.chunkpool.maxsize is 0.0 by default
[ https://issues.apache.org/jira/browse/HBASE-15513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567558#comment-15567558 ] ramkrishna.s.vasudevan commented on HBASE-15513: +1 to commit. > hbase.hregion.memstore.chunkpool.maxsize is 0.0 by default > -- > > Key: HBASE-15513 > URL: https://issues.apache.org/jira/browse/HBASE-15513 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-15513-v1.patch > > > That results in excessive MemStoreLAB chunk allocations because we can not > reuse them. Not sure, why it has been disabled, by default. May be the code > has not been tested well? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16809) Save one cell length calculation in HeapMemStoreLAB#copyCellInto
[ https://issues.apache.org/jira/browse/HBASE-16809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-16809: --- Resolution: Fixed Assignee: binlijin Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master. Thanks for the patch [~aoxiang]. Thanks for the review Stack. > Save one cell length calculation in HeapMemStoreLAB#copyCellInto > > > Key: HBASE-16809 > URL: https://issues.apache.org/jira/browse/HBASE-16809 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: binlijin >Assignee: binlijin >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16809_master.patch > > > HeapMemStoreLAB#copyCellInto have already calculate the cell's length, we can > pass it to KeyValueUtil#copyCellTo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16664) Timeout logic in AsyncProcess is broken
[
https://issues.apache.org/jira/browse/HBASE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567527#comment-15567527
]
Heng Chen commented on HBASE-16664:
---
{quote}
Then just set operation timeout to Long.MAX_VALUE and use retry times to
control the timeout under this scenario.
But what about the scenario that user knows how many entries he/she has?
{quote}
Although user knows how many entries, but they still not know how many entries
each RS will be requested. So it is also hard to decide the total time out it
is, maybe you can use the worst timeout but i still not sure how effective it
is and you can still use retry number and rpcTimeout to control it without
total timeout.
> Timeout logic in AsyncProcess is broken
> ---
>
> Key: HBASE-16664
> URL: https://issues.apache.org/jira/browse/HBASE-16664
> Project: HBase
> Issue Type: Bug
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: 1.patch, HBASE-16664-branch-1-v1.patch,
> HBASE-16664-branch-1-v1.patch, HBASE-16664-branch-1.1-v1.patch,
> HBASE-16664-branch-1.2-v1.patch, HBASE-16664-branch-1.3-v1.patch,
> HBASE-16664-v1.patch, HBASE-16664-v2.patch, HBASE-16664-v3.patch,
> HBASE-16664-v4.patch, HBASE-16664-v5.patch, testhcm.patch
>
>
> Have not checked the root cause, but I think timeout of all operations in
> AsyncProcess is broken
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15437) Response size calculated in RPCServer for warning tooLarge responses does NOT count CellScanner payload
[
https://issues.apache.org/jira/browse/HBASE-15437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567520#comment-15567520
]
Anoop Sam John commented on HBASE-15437:
It is metric on response data size not exactly like response RPC size right?
Seeing the use of too large, we add some logs. That means user is trying to
fetch too much of data in a single call. There is all chances of timeouts and
memory overuse. (We do have heartbeats, early part response based on size/time
BTW) So I feel we need continue with the old way of raw data size which we are
returning. At RPC side there might be encoding/compression based on the cluster
needs.
> Response size calculated in RPCServer for warning tooLarge responses does NOT
> count CellScanner payload
> ---
>
> Key: HBASE-15437
> URL: https://issues.apache.org/jira/browse/HBASE-15437
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC
>Reporter: deepankar
>Assignee: deepankar
> Attachments: HBASE-15437-v1.patch, HBASE-15437.patch
>
>
> After HBASE-13158 where we respond back to RPCs with cells in the payload ,
> the protobuf response will just have the count the cells to read from
> payload, but there are set of features where we log warn in RPCServer
> whenever the response is tooLarge, but this size now is not considering the
> sizes of the cells in the PayloadCellScanner. Code form RPCServer
> {code}
> long responseSize = result.getSerializedSize();
> // log any RPC responses that are slower than the configured warn
> // response time or larger than configured warning size
> boolean tooSlow = (processingTime > warnResponseTime &&
> warnResponseTime > -1);
> boolean tooLarge = (responseSize > warnResponseSize && warnResponseSize
> > -1);
> if (tooSlow || tooLarge) {
> // when tagging, we let TooLarge trump TooSmall to keep output simple
> // note that large responses will often also be slow.
> logResponse(new Object[]{param},
> md.getName(), md.getName() + "(" + param.getClass().getName() +
> ")",
> (tooLarge ? "TooLarge" : "TooSlow"),
> status.getClient(), startTime, processingTime, qTime,
> responseSize);
> }
> {code}
> Should this feature be not supported any more or should we add a method to
> CellScanner or a new interface which returns the serialized size (but this
> might not include the compression codecs which might be used during response
> ?) Any other Idea this could be fixed ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16799) CP exposed Store should not expose unwanted APIs
[ https://issues.apache.org/jira/browse/HBASE-16799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567511#comment-15567511 ] Anoop Sam John commented on HBASE-16799: Came across this while I was working on HBASE-16747. I had to change signature of APIs in Store and saw it is CP exposed. That made me to think this way. bq.We need to add a warning on the Store API? You mean on other APIs which will be there after this patch? Ya I will add comments in Store interface not to expose APIs that change the state of the store. Also any Store impl that we make have to extend HStore not directly impl this interface. That might happen very very rare. We have HMobStore which is any way extending HStore. > CP exposed Store should not expose unwanted APIs > - > > Key: HBASE-16799 > URL: https://issues.apache.org/jira/browse/HBASE-16799 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16799.patch, HBASE-16799.patch, > HBASE-16799_V2.patch > > > Store is exposed to CPs. The main use cases I can think of are getting store > scanner and other getters which return different states like memstore size, > max seqId etc. Those make sense. > But we added many other APIs which changes the state of the memstore, bulk > load files etc into this interface. Even an API which expose the memstore > itself!. This allow adding mutations into memstore bypassing all steps in > region. We track the memstore size per region level as well as globally. > These only allow us to flush region at sizes and/or flush selected regions > because of global heap pressure. Now if a CP get hold of store and/or > memstore, it can add mutations with out knowledge of these size accounting > and possibly OOME the RS. Similar way the bulk load related APIs. At HRegion > level, there are steps done (WAL write etc) after the bulk load HFile on > store. So bypassing these wont be correct. > In this jira, plan is to remove all such leaked APIs from Store. They are > called from HRegion and we can type cast to HStore to call them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16664) Timeout logic in AsyncProcess is broken
[
https://issues.apache.org/jira/browse/HBASE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567473#comment-15567473
]
Phil Yang commented on HBASE-16664:
---
{quote}
My understanding is single-get is one operation and one call for api, but multi
get is multi operations although it is also just one call for api
{quote}
Now matter what a "operation" is. All api in Table need a total timeout,
because it is more easy to understand for users. We can split into two
operation timeout settings for single and batch, but I am not a fan for this.
As I said, I think supporting read/write rpc timeout is not a good idea. We can
use different instances for different timeout settings.
{quote}
2 and 3 maybe in different threads
{quote}
It may be a wrong usage? Table is not thread-safe, although for some methods if
we access them in different threads we may not fail, but we don't have to keep
this "feature", right?
> Timeout logic in AsyncProcess is broken
> ---
>
> Key: HBASE-16664
> URL: https://issues.apache.org/jira/browse/HBASE-16664
> Project: HBase
> Issue Type: Bug
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: 1.patch, HBASE-16664-branch-1-v1.patch,
> HBASE-16664-branch-1-v1.patch, HBASE-16664-branch-1.1-v1.patch,
> HBASE-16664-branch-1.2-v1.patch, HBASE-16664-branch-1.3-v1.patch,
> HBASE-16664-v1.patch, HBASE-16664-v2.patch, HBASE-16664-v3.patch,
> HBASE-16664-v4.patch, HBASE-16664-v5.patch, testhcm.patch
>
>
> Have not checked the root cause, but I think timeout of all operations in
> AsyncProcess is broken
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16664) Timeout logic in AsyncProcess is broken
[ https://issues.apache.org/jira/browse/HBASE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567469#comment-15567469 ] Duo Zhang commented on HBASE-16664: --- Then just set operation timeout to Long.MAX_VALUE and use retry times to control the timeout under this scenario. But what about the scenario that user knows how many entries he/she has? > Timeout logic in AsyncProcess is broken > --- > > Key: HBASE-16664 > URL: https://issues.apache.org/jira/browse/HBASE-16664 > Project: HBase > Issue Type: Bug >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: 1.patch, HBASE-16664-branch-1-v1.patch, > HBASE-16664-branch-1-v1.patch, HBASE-16664-branch-1.1-v1.patch, > HBASE-16664-branch-1.2-v1.patch, HBASE-16664-branch-1.3-v1.patch, > HBASE-16664-v1.patch, HBASE-16664-v2.patch, HBASE-16664-v3.patch, > HBASE-16664-v4.patch, HBASE-16664-v5.patch, testhcm.patch > > > Have not checked the root cause, but I think timeout of all operations in > AsyncProcess is broken -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16664) Timeout logic in AsyncProcess is broken
[
https://issues.apache.org/jira/browse/HBASE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567465#comment-15567465
]
Heng Chen commented on HBASE-16664:
---
{quote}
We can not tell users "some operations don't support a total timeout"
{quote}
Normally, in batch request, users maybe not know how many put or get included,
it is hard for our user to define the total timeout
> Timeout logic in AsyncProcess is broken
> ---
>
> Key: HBASE-16664
> URL: https://issues.apache.org/jira/browse/HBASE-16664
> Project: HBase
> Issue Type: Bug
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: 1.patch, HBASE-16664-branch-1-v1.patch,
> HBASE-16664-branch-1-v1.patch, HBASE-16664-branch-1.1-v1.patch,
> HBASE-16664-branch-1.2-v1.patch, HBASE-16664-branch-1.3-v1.patch,
> HBASE-16664-v1.patch, HBASE-16664-v2.patch, HBASE-16664-v3.patch,
> HBASE-16664-v4.patch, HBASE-16664-v5.patch, testhcm.patch
>
>
> Have not checked the root cause, but I think timeout of all operations in
> AsyncProcess is broken
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16664) Timeout logic in AsyncProcess is broken
[
https://issues.apache.org/jira/browse/HBASE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567451#comment-15567451
]
Heng Chen commented on HBASE-16664:
---
{quote}
And I think for a user, I call the batch method, or the multi get method, to me
it is one operation. I can increase the operation timeout if I want.
{quote}
My understanding is single-get is one operation and one call for api, but
multi get is multi operations although it is also just one call for api
> Timeout logic in AsyncProcess is broken
> ---
>
> Key: HBASE-16664
> URL: https://issues.apache.org/jira/browse/HBASE-16664
> Project: HBase
> Issue Type: Bug
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: 1.patch, HBASE-16664-branch-1-v1.patch,
> HBASE-16664-branch-1-v1.patch, HBASE-16664-branch-1.1-v1.patch,
> HBASE-16664-branch-1.2-v1.patch, HBASE-16664-branch-1.3-v1.patch,
> HBASE-16664-v1.patch, HBASE-16664-v2.patch, HBASE-16664-v3.patch,
> HBASE-16664-v4.patch, HBASE-16664-v5.patch, testhcm.patch
>
>
> Have not checked the root cause, but I think timeout of all operations in
> AsyncProcess is broken
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16146) Counters are expensive...
[ https://issues.apache.org/jira/browse/HBASE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567444#comment-15567444 ] Hudson commented on HBASE-16146: FAILURE: Integrated in Jenkins build HBase-1.3-JDK7 #37 (See [https://builds.apache.org/job/HBase-1.3-JDK7/37/]) HBASE-16146 Remove thread local usage in Counter (garyh: rev dcb47c9b715c2331abe7e8ccbc1b69f24168dd97) * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/util/Counter.java > Counters are expensive... > - > > Key: HBASE-16146 > URL: https://issues.apache.org/jira/browse/HBASE-16146 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Gary Helmling > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16146.001.patch, HBASE-16146.branch-1.001.patch, > HBASE-16146.branch-1.3.001.patch, counters.patch, less_and_less_counters.png > > > Doing workloadc, perf shows 10%+ of CPU being spent on counter#add. If I > disable some of the hot ones -- see patch -- I can get 10% more throughput > (390k to 440k). Figure something better. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16663) JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
[
https://issues.apache.org/jira/browse/HBASE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567445#comment-15567445
]
Hudson commented on HBASE-16663:
FAILURE: Integrated in Jenkins build HBase-1.3-JDK7 #37 (See
[https://builds.apache.org/job/HBase-1.3-JDK7/37/])
HBASE-16663 JMX ConnectorServer stopped when unauthorized user try to
(apurtell: rev 3737c469606d3d17b15c9deea5114f92ba2aa461)
* (add)
hbase-server/src/test/java/org/apache/hadoop/hbase/TestJMXConnectorServer.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerCoprocessorHost.java
> JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
>
>
> Key: HBASE-16663
> URL: https://issues.apache.org/jira/browse/HBASE-16663
> Project: HBase
> Issue Type: Bug
> Components: metrics, security
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 0.98.24
>
> Attachments: HBASE-16663-0.98-V4.patch, HBASE-16663-0.98.patch,
> HBASE-16663-V2.patch, HBASE-16663-V3.patch, HBASE-16663-V4.patch,
> HBASE-16663-branch-1.patch, HBASE-16663.patch
>
>
> After HBASE-16284, unauthorized user will not able allowed to stop
> HM/RS/cluster, but while executing "cpHost.preStopMaster()", ConnectorServer
> will be stopped before AccessController validation.
> hbase-site.xml,
> {noformat}
>
> hbase.coprocessor.master.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
>
> hbase.coprocessor.regionserver.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
> {noformat}
> HBaseAdmin.stopMaster(),
> {noformat}
> 2016-09-20 21:12:26,796 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 21:13:55,380 WARN
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 21:14:00,495 ERROR
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> master.MasterRpcServices: Exception occurred while stopping master
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopMaster(AccessController.java:1297)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost$68.call(MasterCoprocessorHost.java:821)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1188)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.preStopMaster(MasterCoprocessorHost.java:817)
> at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2352)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.stopMaster(MasterRpcServices.java:1364)
> {noformat}
> HBaseAdmin.stopRegionServer(rs-host-port),
> {noformat}
> 2016-09-20 20:59:01,234 INFO
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 20:59:01,250 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 20:59:01,253 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> regionserver.HRegionServer: The region server did not stop
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:84)
> at
>
[jira] [Updated] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-15921: -- Attachment: HBASE-15921-v8.patch > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Jurriaan Mous >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15921-v2.patch, HBASE-15921-v3.patch, > HBASE-15921-v4.patch, HBASE-15921-v5.patch, HBASE-15921-v6.patch, > HBASE-15921-v7.patch, HBASE-15921-v8.patch, HBASE-15921.demo.patch, > HBASE-15921.patch, HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16664) Timeout logic in AsyncProcess is broken
[ https://issues.apache.org/jira/browse/HBASE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567420#comment-15567420 ] Heng Chen edited comment on HBASE-16664 at 10/12/16 3:34 AM: - The typical case in our application is something like TestFromClientSide did 1. get HTable object 2. call batch puts 3. call get 4. close HTable It is ok now. 2 and 3 maybe in different threads was (Author: chenheng): The typical case in our application is something like TestFromClientSide did 1. get HTable object 2. call batch puts 3. call get 4. close HTable It is ok now. > Timeout logic in AsyncProcess is broken > --- > > Key: HBASE-16664 > URL: https://issues.apache.org/jira/browse/HBASE-16664 > Project: HBase > Issue Type: Bug >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: 1.patch, HBASE-16664-branch-1-v1.patch, > HBASE-16664-branch-1-v1.patch, HBASE-16664-branch-1.1-v1.patch, > HBASE-16664-branch-1.2-v1.patch, HBASE-16664-branch-1.3-v1.patch, > HBASE-16664-v1.patch, HBASE-16664-v2.patch, HBASE-16664-v3.patch, > HBASE-16664-v4.patch, HBASE-16664-v5.patch, testhcm.patch > > > Have not checked the root cause, but I think timeout of all operations in > AsyncProcess is broken -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16664) Timeout logic in AsyncProcess is broken
[ https://issues.apache.org/jira/browse/HBASE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567420#comment-15567420 ] Heng Chen commented on HBASE-16664: --- The typical case in our application is something like TestFromClientSide did 1. get HTable object 2. call batch puts 3. call get 4. close HTable It is ok now. > Timeout logic in AsyncProcess is broken > --- > > Key: HBASE-16664 > URL: https://issues.apache.org/jira/browse/HBASE-16664 > Project: HBase > Issue Type: Bug >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: 1.patch, HBASE-16664-branch-1-v1.patch, > HBASE-16664-branch-1-v1.patch, HBASE-16664-branch-1.1-v1.patch, > HBASE-16664-branch-1.2-v1.patch, HBASE-16664-branch-1.3-v1.patch, > HBASE-16664-v1.patch, HBASE-16664-v2.patch, HBASE-16664-v3.patch, > HBASE-16664-v4.patch, HBASE-16664-v5.patch, testhcm.patch > > > Have not checked the root cause, but I think timeout of all operations in > AsyncProcess is broken -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16664) Timeout logic in AsyncProcess is broken
[ https://issues.apache.org/jira/browse/HBASE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567407#comment-15567407 ] Phil Yang commented on HBASE-16664: --- I think the best practice is using different Table instances for different timeout settings if users need.(And of course, it is not thread safe so we need them in each thread) We can not tell users "some operations don't support a total timeout" And a related issue is we have read/write rpc timeout now. In fact I think it is useless because we can set different timeout for different instance, and for batch or CAS operations, we can not say it is read or write... > Timeout logic in AsyncProcess is broken > --- > > Key: HBASE-16664 > URL: https://issues.apache.org/jira/browse/HBASE-16664 > Project: HBase > Issue Type: Bug >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: 1.patch, HBASE-16664-branch-1-v1.patch, > HBASE-16664-branch-1-v1.patch, HBASE-16664-branch-1.1-v1.patch, > HBASE-16664-branch-1.2-v1.patch, HBASE-16664-branch-1.3-v1.patch, > HBASE-16664-v1.patch, HBASE-16664-v2.patch, HBASE-16664-v3.patch, > HBASE-16664-v4.patch, HBASE-16664-v5.patch, testhcm.patch > > > Have not checked the root cause, but I think timeout of all operations in > AsyncProcess is broken -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16812) Cleanup deprecated compact() function
[
https://issues.apache.org/jira/browse/HBASE-16812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567397#comment-15567397
]
Hadoop QA commented on HBASE-16812:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
0s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 59s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 50s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 114m 52s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestModifyTableProcedure |
| |
org.apache.hadoop.hbase.master.procedure.TestMasterProcedureSchedulerConcurrency
|
| | org.apache.hadoop.hbase.master.procedure.TestRestoreSnapshotProcedure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832808/HBASE-16812.master.001.patch
|
| JIRA Issue | HBASE-16812 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 10d8ea4e56af 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0af10f8 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3950/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3950/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3950/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3950/console |
|
[jira] [Commented] (HBASE-16664) Timeout logic in AsyncProcess is broken
[
https://issues.apache.org/jira/browse/HBASE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567389#comment-15567389
]
Duo Zhang commented on HBASE-16664:
---
{noformat}
* This class is NOT thread safe for reads nor writes.
* In the case of writes (Put, Delete), the underlying write buffer can
* be corrupted if multiple threads contend over a single HTable instance.
* In the case of reads, some fields used by a Scan are shared among all
threads.
{noformat}
HTable is not thread safe so it is not a good idea to share one HTable. The
right approach is to share one Connection instance and use new Table instance
every time.
And I think for a user, I call the batch method, or the multi get method, to me
it is one operation. I can increase the operation timeout if I want.
> Timeout logic in AsyncProcess is broken
> ---
>
> Key: HBASE-16664
> URL: https://issues.apache.org/jira/browse/HBASE-16664
> Project: HBase
> Issue Type: Bug
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: 1.patch, HBASE-16664-branch-1-v1.patch,
> HBASE-16664-branch-1-v1.patch, HBASE-16664-branch-1.1-v1.patch,
> HBASE-16664-branch-1.2-v1.patch, HBASE-16664-branch-1.3-v1.patch,
> HBASE-16664-v1.patch, HBASE-16664-v2.patch, HBASE-16664-v3.patch,
> HBASE-16664-v4.patch, HBASE-16664-v5.patch, testhcm.patch
>
>
> Have not checked the root cause, but I think timeout of all operations in
> AsyncProcess is broken
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16664) Timeout logic in AsyncProcess is broken
[
https://issues.apache.org/jira/browse/HBASE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567379#comment-15567379
]
Heng Chen commented on HBASE-16664:
---
Normally we share one HTable object in our application. It is hard to avoid
race condition.
I think we should define the operation timeout firstly to avoid conflicts.
This is our description for it in hbase-default.xml
{code}
hbase.client.operation.timeout
120
Operation timeout is a top-level restriction (millisecond)
that makes sure a
blocking operation in Table will not be blocked more than this. In each
operation, if rpc
request fails because of timeout or other reason, it will retry until
success or throw
RetriesExhaustedException. But if the total time being blocking reach the
operation timeout
before retries exhausted, it will break early and throw
SocketTimeoutException.
{code}
what's the one operation? Batch is one operation? [~stack]
> Timeout logic in AsyncProcess is broken
> ---
>
> Key: HBASE-16664
> URL: https://issues.apache.org/jira/browse/HBASE-16664
> Project: HBase
> Issue Type: Bug
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: 1.patch, HBASE-16664-branch-1-v1.patch,
> HBASE-16664-branch-1-v1.patch, HBASE-16664-branch-1.1-v1.patch,
> HBASE-16664-branch-1.2-v1.patch, HBASE-16664-branch-1.3-v1.patch,
> HBASE-16664-v1.patch, HBASE-16664-v2.patch, HBASE-16664-v3.patch,
> HBASE-16664-v4.patch, HBASE-16664-v5.patch, testhcm.patch
>
>
> Have not checked the root cause, but I think timeout of all operations in
> AsyncProcess is broken
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16489) Configuration parsing
[
https://issues.apache.org/jira/browse/HBASE-16489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567351#comment-15567351
]
Enis Soztutar commented on HBASE-16489:
---
- {{hbase-native-client/core/configuration-test.cc}} does not assert anything,
just prints out values, so it is not a unit test. Please make sure that there
are at least some tests.
- Please use a better name for this map. It is not a single property. Why are
we typedefing this anyway?
+using HBASE_CONF_PROPERTY = std::map;
- This for loop is not how we do substitute variables in the java code:
{code}
+ for (int i = 0; i < MAX_SUBSTS; i++) {
{code}
Instead of blindly iterating over the values 20 times, we do substitute
matching at the get() time. There is a subtle difference in the case that
Configuration is a dynamic object in Java, so the substituted variables can
change on runtime.
- we need a version of Get without deprecated key handling:
{code}
+const std::string HBaseConfiguration::Get(const std::string ,
+ const std::string _value) {
{code}
> Configuration parsing
> -
>
> Key: HBASE-16489
> URL: https://issues.apache.org/jira/browse/HBASE-16489
> Project: HBase
> Issue Type: Sub-task
>Reporter: Sudeep Sunthankar
>Assignee: Sudeep Sunthankar
> Attachments: HBASE-16489.HBASE-14850.v1.patch
>
>
> Reading hbase-site.xml is required to read various properties viz.
> zookeeper-quorum, client retires etc. We can either use Apache Xerces or
> Boost libraries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16664) Timeout logic in AsyncProcess is broken
[ https://issues.apache.org/jira/browse/HBASE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567348#comment-15567348 ] Duo Zhang commented on HBASE-16664: --- There is a setOperationTimeout method in the Table interface, you can call it before doing any operation. > Timeout logic in AsyncProcess is broken > --- > > Key: HBASE-16664 > URL: https://issues.apache.org/jira/browse/HBASE-16664 > Project: HBase > Issue Type: Bug >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: 1.patch, HBASE-16664-branch-1-v1.patch, > HBASE-16664-branch-1-v1.patch, HBASE-16664-branch-1.1-v1.patch, > HBASE-16664-branch-1.2-v1.patch, HBASE-16664-branch-1.3-v1.patch, > HBASE-16664-v1.patch, HBASE-16664-v2.patch, HBASE-16664-v3.patch, > HBASE-16664-v4.patch, HBASE-16664-v5.patch, testhcm.patch > > > Have not checked the root cause, but I think timeout of all operations in > AsyncProcess is broken -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16664) Timeout logic in AsyncProcess is broken
[ https://issues.apache.org/jira/browse/HBASE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567343#comment-15567343 ] Heng Chen commented on HBASE-16664: --- For example, in our production cluster, we set operation timeout 3s for single get. So as the patch, the batch operations will be 3s timeout? I don't think it is a good idea. > Timeout logic in AsyncProcess is broken > --- > > Key: HBASE-16664 > URL: https://issues.apache.org/jira/browse/HBASE-16664 > Project: HBase > Issue Type: Bug >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: 1.patch, HBASE-16664-branch-1-v1.patch, > HBASE-16664-branch-1-v1.patch, HBASE-16664-branch-1.1-v1.patch, > HBASE-16664-branch-1.2-v1.patch, HBASE-16664-branch-1.3-v1.patch, > HBASE-16664-v1.patch, HBASE-16664-v2.patch, HBASE-16664-v3.patch, > HBASE-16664-v4.patch, HBASE-16664-v5.patch, testhcm.patch > > > Have not checked the root cause, but I think timeout of all operations in > AsyncProcess is broken -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16664) Timeout logic in AsyncProcess is broken
[ https://issues.apache.org/jira/browse/HBASE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567335#comment-15567335 ] Duo Zhang commented on HBASE-16664: --- Multi get also uses batch, and it will be executed with AsyncProcess, and I think there is no 'uncertain operations' here right? And if user think he/she does not need an operation timeout then just set it to Long.MAX_VALUE. But ignore operation timeout will hurt the user who use multi get and multi put. > Timeout logic in AsyncProcess is broken > --- > > Key: HBASE-16664 > URL: https://issues.apache.org/jira/browse/HBASE-16664 > Project: HBase > Issue Type: Bug >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: 1.patch, HBASE-16664-branch-1-v1.patch, > HBASE-16664-branch-1-v1.patch, HBASE-16664-branch-1.1-v1.patch, > HBASE-16664-branch-1.2-v1.patch, HBASE-16664-branch-1.3-v1.patch, > HBASE-16664-v1.patch, HBASE-16664-v2.patch, HBASE-16664-v3.patch, > HBASE-16664-v4.patch, HBASE-16664-v5.patch, testhcm.patch > > > Have not checked the root cause, but I think timeout of all operations in > AsyncProcess is broken -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15109) HM/RS failed to start when "fs.hdfs.impl.disable.cache" is set to true
[
https://issues.apache.org/jira/browse/HBASE-15109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567324#comment-15567324
]
Pankaj Kumar commented on HBASE-15109:
--
Thanks [~tedyu] and [~stack] for reviewing the patch. We should have this fix
in other branches as well, will add patch.
> HM/RS failed to start when "fs.hdfs.impl.disable.cache" is set to true
> --
>
> Key: HBASE-15109
> URL: https://issues.apache.org/jira/browse/HBASE-15109
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15109-V2.patch, HBASE-15109-branch-1.patch,
> HBASE-15109.patch, HBASE-15109.patch
>
>
> HMaster failed to start during installing ShutdownHook when
> "fs.hdfs.impl.disable.cache" is set to true.
> {noformat}
> 2016-10-10 10:33:28,367 FATAL [master/HOST-10-18-106-146/10.18.106.146:16000]
> master.HMaster: Unhandled: Failed suppression of fs shutdown hook:
> org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@19a003b4
> java.lang.RuntimeException: Failed suppression of fs shutdown hook:
> org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@19a003b4
> at
> org.apache.hadoop.hbase.regionserver.ShutdownHook.suppressHdfsShutdownHook(ShutdownHook.java:204)
> at
> org.apache.hadoop.hbase.regionserver.ShutdownHook.install(ShutdownHook.java:84)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:950)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> And RegionServer is in blocking state until a master is available,
> {noformat}
> "regionserver/HOST-10-18-106-146/10.18.106.146:16020" #17 prio=5 os_prio=0
> tid=0x00e4e800 nid=0xdc38 in Object.wait() [0x7f329dac6000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:159)
> - locked <0xae25c0c8> (a
> org.apache.hadoop.hbase.zookeeper.MasterAddressTracker)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.blockAndCheckIfStopped(HRegionServer.java:870)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:809)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:783)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:943)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> During installation, first time it will try to suppress the HDFS shutdownhook
> by removing the hdfsclientfinalizer from HDFS. Since
> fs.hdfs.impl.disable.cache is enabled, so removal will be unsuccessful from
> HDFS (FS is not cached) and RuntimeException will be thrown with "Failed
> suppression of fs shutdown hook" message.
> In ShutdownHook,
> {code}
> if (!fsShutdownHooks.containsKey(hdfsClientFinalizer) &&
> !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> throw new RuntimeException("Failed suppression of fs shutdown hook: " +
> hdfsClientFinalizer);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16812) Cleanup deprecated compact() function
[ https://issues.apache.org/jira/browse/HBASE-16812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567315#comment-15567315 ] Jingcheng Du edited comment on HBASE-16812 at 10/12/16 2:29 AM: Thanks a lot [~appy]. I am afraid this HMobStore.compact method cannot be deleted, instead we have to make this method be called in hbase compaction by adding a user parameter to this method. I guess this method is missed when modifying code in HStore.compact. This zk lock is used here to synchronize the major compaction and mob compaction, which can avoid the deleted cells being alive again. So I think the zk lock has to be there. I am thinking to implement update cells instead of put only for mob compaction, which is a update cell is regarded as deleted when there is no more non-update cells before it. This can remove all locks in mob compaction. How about removing the zk locks after this is implemented? was (Author: jingcheng...@intel.com): Thanks a lot [~appy]. I am afraid this HMobStore.compact method cannot be deleted, instead we have to make this method be called in hbase compaction by adding a user parameter to this method. I guess this method is missed when modifying code in HStore.compact. This zk lock is used here to synchronize the major compaction and mob compaction, which can avoid the deleted cells being alive again. So I think the zk lock has to be there. I am thinking to implement update cells instead of put only for mob compaction, which is a update cell is regarded as deleted when there is no more non-update cells. This can remove all locks in mob compaction. How about removing the zk locks after this is implemented? > Cleanup deprecated compact() function > - > > Key: HBASE-16812 > URL: https://issues.apache.org/jira/browse/HBASE-16812 > Project: HBase > Issue Type: Task >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-16812.master.001.patch > > > compact(CompactionContext compaction, CompactionThroughputController > throughputController) is [deprecated in 1.2.0 > release|https://github.com/apache/hbase/blob/rel/1.2.0/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java#L222]. > Store.java is also marked limited private. > Context: I was cleaning up zk table lock which is also used in that method's > [override|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HMobStore.java#L460] > in HMobStore. > This method isn't being called from anywhere except CompactionTool (which > creates HStore object, not HMobStore object). > [~jingcheng...@intel.com] Can you PTAL and help me understand what's going on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16813) Procedure v2 - Move ProcedureEvent to hbase-procedure module
Matteo Bertozzi created HBASE-16813: --- Summary: Procedure v2 - Move ProcedureEvent to hbase-procedure module Key: HBASE-16813 URL: https://issues.apache.org/jira/browse/HBASE-16813 Project: HBase Issue Type: Sub-task Components: proc-v2 Affects Versions: 2.0.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 2.0.0 ProcedureEvent was added in MasterProcedureScheduler, but it is generic enough to move to hbase-procedure module. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16642) Use DelayQueue instead of TimeoutBlockingQueue
[ https://issues.apache.org/jira/browse/HBASE-16642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16642: Issue Type: Sub-task (was: Improvement) Parent: HBASE-16617 > Use DelayQueue instead of TimeoutBlockingQueue > -- > > Key: HBASE-16642 > URL: https://issues.apache.org/jira/browse/HBASE-16642 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Hiroshi Ikeda >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16642-v2.patch, HBASE-16642.master.V1.patch > > > Enqueue poisons in order to wake up and end the internal threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16664) Timeout logic in AsyncProcess is broken
[ https://issues.apache.org/jira/browse/HBASE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567319#comment-15567319 ] Heng Chen commented on HBASE-16664: --- I think the reason we did not set total timeout for batch is that batch has uncertain operations, it is hard to set total timeout for it. But we could control retry times. > Timeout logic in AsyncProcess is broken > --- > > Key: HBASE-16664 > URL: https://issues.apache.org/jira/browse/HBASE-16664 > Project: HBase > Issue Type: Bug >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: 1.patch, HBASE-16664-branch-1-v1.patch, > HBASE-16664-branch-1-v1.patch, HBASE-16664-branch-1.1-v1.patch, > HBASE-16664-branch-1.2-v1.patch, HBASE-16664-branch-1.3-v1.patch, > HBASE-16664-v1.patch, HBASE-16664-v2.patch, HBASE-16664-v3.patch, > HBASE-16664-v4.patch, HBASE-16664-v5.patch, testhcm.patch > > > Have not checked the root cause, but I think timeout of all operations in > AsyncProcess is broken -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16812) Cleanup deprecated compact() function
[ https://issues.apache.org/jira/browse/HBASE-16812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567315#comment-15567315 ] Jingcheng Du commented on HBASE-16812: -- Thanks a lot [~appy]. I am afraid this HMobStore.compact method cannot be deleted, instead we have to make this method be called in hbase compaction by adding a user parameter to this method. I guess this method is missed when modifying code in HStore.compact. This zk lock is used here to synchronize the major compaction and mob compaction, which can avoid the deleted cells being alive again. So I think the zk lock has to be there. I am thinking to implement update cells instead of put only for mob compaction, which is a update cell is regarded as deleted when there is no more non-update cells. This can remove all locks in mob compaction. How about removing the zk locks after this is implemented? > Cleanup deprecated compact() function > - > > Key: HBASE-16812 > URL: https://issues.apache.org/jira/browse/HBASE-16812 > Project: HBase > Issue Type: Task >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-16812.master.001.patch > > > compact(CompactionContext compaction, CompactionThroughputController > throughputController) is [deprecated in 1.2.0 > release|https://github.com/apache/hbase/blob/rel/1.2.0/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java#L222]. > Store.java is also marked limited private. > Context: I was cleaning up zk table lock which is also used in that method's > [override|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HMobStore.java#L460] > in HMobStore. > This method isn't being called from anywhere except CompactionTool (which > creates HStore object, not HMobStore object). > [~jingcheng...@intel.com] Can you PTAL and help me understand what's going on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16664) Timeout logic in AsyncProcess is broken
[
https://issues.apache.org/jira/browse/HBASE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567296#comment-15567296
]
Duo Zhang commented on HBASE-16664:
---
{quote}
i think it is reasonable because operationTimeout should only affect one
operation NOT for batch operations.
{quote}
Why? Then how can I control the timeout for a batch operation? Only by retry
times?
> Timeout logic in AsyncProcess is broken
> ---
>
> Key: HBASE-16664
> URL: https://issues.apache.org/jira/browse/HBASE-16664
> Project: HBase
> Issue Type: Bug
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: 1.patch, HBASE-16664-branch-1-v1.patch,
> HBASE-16664-branch-1-v1.patch, HBASE-16664-branch-1.1-v1.patch,
> HBASE-16664-branch-1.2-v1.patch, HBASE-16664-branch-1.3-v1.patch,
> HBASE-16664-v1.patch, HBASE-16664-v2.patch, HBASE-16664-v3.patch,
> HBASE-16664-v4.patch, HBASE-16664-v5.patch, testhcm.patch
>
>
> Have not checked the root cause, but I think timeout of all operations in
> AsyncProcess is broken
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16146) Counters are expensive...
[ https://issues.apache.org/jira/browse/HBASE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567279#comment-15567279 ] Hudson commented on HBASE-16146: SUCCESS: Integrated in Jenkins build HBase-1.4 #461 (See [https://builds.apache.org/job/HBase-1.4/461/]) HBASE-16146 Remove thread local usage in Counter (garyh: rev 4f29c230384b82b64ef4ad9ba61497747436799f) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/util/Counter.java > Counters are expensive... > - > > Key: HBASE-16146 > URL: https://issues.apache.org/jira/browse/HBASE-16146 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Gary Helmling > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16146.001.patch, HBASE-16146.branch-1.001.patch, > HBASE-16146.branch-1.3.001.patch, counters.patch, less_and_less_counters.png > > > Doing workloadc, perf shows 10%+ of CPU being spent on counter#add. If I > disable some of the hot ones -- see patch -- I can get 10% more throughput > (390k to 440k). Figure something better. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12894) Upgrade Jetty to 9.2.6
[
https://issues.apache.org/jira/browse/HBASE-12894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567273#comment-15567273
]
Hadoop QA commented on HBASE-12894:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 10 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 51s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
41s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hbase-resource-bundle . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 0s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 6s
{color} | {color:red} hbase-thrift in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 7s
{color} | {color:red} hbase-rest in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 10s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 10s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
43s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 9s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 40s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 3m
3s {color} | {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hbase-resource-bundle . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 2s
{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 40s
{color} | {color:red} hbase-rest generated 5 new + 0 unchanged - 0 fixed = 5
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 5s
{color} | {color:green} hbase-resource-bundle in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 37s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 93m 57s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 42s
{color} | {color:green} hbase-thrift in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s
{color} | {color:green} hbase-it in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 41s
{color} |
[jira] [Commented] (HBASE-16146) Counters are expensive...
[ https://issues.apache.org/jira/browse/HBASE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567218#comment-15567218 ] Hudson commented on HBASE-16146: FAILURE: Integrated in Jenkins build HBase-1.3-JDK8 #42 (See [https://builds.apache.org/job/HBase-1.3-JDK8/42/]) HBASE-16146 Remove thread local usage in Counter (garyh: rev dcb47c9b715c2331abe7e8ccbc1b69f24168dd97) * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/util/Counter.java > Counters are expensive... > - > > Key: HBASE-16146 > URL: https://issues.apache.org/jira/browse/HBASE-16146 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Gary Helmling > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16146.001.patch, HBASE-16146.branch-1.001.patch, > HBASE-16146.branch-1.3.001.patch, counters.patch, less_and_less_counters.png > > > Doing workloadc, perf shows 10%+ of CPU being spent on counter#add. If I > disable some of the hot ones -- see patch -- I can get 10% more throughput > (390k to 440k). Figure something better. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16812) Cleanup deprecated compact() function
Appy created HBASE-16812: Summary: Cleanup deprecated compact() function Key: HBASE-16812 URL: https://issues.apache.org/jira/browse/HBASE-16812 Project: HBase Issue Type: Task Reporter: Appy Assignee: Appy Priority: Minor compact(CompactionContext compaction, CompactionThroughputController throughputController) is [deprecated in 1.2.0 release|https://github.com/apache/hbase/blob/rel/1.2.0/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java#L222]. Store.java is also marked limited private. Context: I was cleaning up zk table lock which is also used in that method's [override|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HMobStore.java#L460] in HMobStore. This method isn't being called from anywhere except CompactionTool (which creates HStore object, not HMobStore object). [~jingcheng...@intel.com] Can you PTAL and help me understand what's going on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16663) JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
[
https://issues.apache.org/jira/browse/HBASE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567140#comment-15567140
]
Andrew Purtell commented on HBASE-16663:
Pushed up to all branches but 1.1. I can commit there too. Please advise.
/cc [~ndimiduk]
> JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
>
>
> Key: HBASE-16663
> URL: https://issues.apache.org/jira/browse/HBASE-16663
> Project: HBase
> Issue Type: Bug
> Components: metrics, security
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 0.98.24
>
> Attachments: HBASE-16663-0.98-V4.patch, HBASE-16663-0.98.patch,
> HBASE-16663-V2.patch, HBASE-16663-V3.patch, HBASE-16663-V4.patch,
> HBASE-16663-branch-1.patch, HBASE-16663.patch
>
>
> After HBASE-16284, unauthorized user will not able allowed to stop
> HM/RS/cluster, but while executing "cpHost.preStopMaster()", ConnectorServer
> will be stopped before AccessController validation.
> hbase-site.xml,
> {noformat}
>
> hbase.coprocessor.master.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
>
> hbase.coprocessor.regionserver.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
> {noformat}
> HBaseAdmin.stopMaster(),
> {noformat}
> 2016-09-20 21:12:26,796 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 21:13:55,380 WARN
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 21:14:00,495 ERROR
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> master.MasterRpcServices: Exception occurred while stopping master
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopMaster(AccessController.java:1297)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost$68.call(MasterCoprocessorHost.java:821)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1188)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.preStopMaster(MasterCoprocessorHost.java:817)
> at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2352)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.stopMaster(MasterRpcServices.java:1364)
> {noformat}
> HBaseAdmin.stopRegionServer(rs-host-port),
> {noformat}
> 2016-09-20 20:59:01,234 INFO
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 20:59:01,250 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 20:59:01,253 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> regionserver.HRegionServer: The region server did not stop
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:84)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.execOperation(RegionServerCoprocessorHost.java:256)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.preStop(RegionServerCoprocessorHost.java:80)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1905)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.stopServer(RSRpcServices.java:1961)
> {noformat}
> HBaseAdmin.shutdown(),
> {noformat}
> 2016-09-21 12:09:08,259 INFO
>
[jira] [Commented] (HBASE-16731) Inconsistent results from the Get/Scan if we use the empty FilterList
[ https://issues.apache.org/jira/browse/HBASE-16731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567039#comment-15567039 ] stack commented on HBASE-16731: --- +1 Good. > Inconsistent results from the Get/Scan if we use the empty FilterList > - > > Key: HBASE-16731 > URL: https://issues.apache.org/jira/browse/HBASE-16731 > Project: HBase > Issue Type: Bug >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16731.v0.patch, HBASE-16731.v1.patch, > HBASE-16731.v2.patch > > > RSRpcServices#get() converts the Get to Scan without > scan#setLoadColumnFamiliesOnDemand. It causes that the result retrieved from > Get and Scan will be different if we use the empty filter. Scan doesn't > return any data but Get does. > see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729] > Any comments? Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16802) Procedure v2 - group procedure cleaning
[ https://issues.apache.org/jira/browse/HBASE-16802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16802: Resolution: Fixed Status: Resolved (was: Patch Available) > Procedure v2 - group procedure cleaning > --- > > Key: HBASE-16802 > URL: https://issues.apache.org/jira/browse/HBASE-16802 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-16802-v0.patch > > > group the cleaning of the evicted procedures -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16788) Race in compacted file deletion between HStore close() and closeAndArchiveCompactedFiles()
[
https://issues.apache.org/jira/browse/HBASE-16788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566990#comment-15566990
]
Hudson commented on HBASE-16788:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1769 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1769/])
HBASE-16788 addendum Account for HStore archiveLock in heap size (garyh: rev
bc7e034052dceddd21a8d45ea7b9b131f46a01f9)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
> Race in compacted file deletion between HStore close() and
> closeAndArchiveCompactedFiles()
> --
>
> Key: HBASE-16788
> URL: https://issues.apache.org/jira/browse/HBASE-16788
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.3.0
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Blocker
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: 16788-suggest.v2, HBASE-16788-addendum.patch,
> HBASE-16788.001.patch, HBASE-16788.002.patch, HBASE-16788_1.patch
>
>
> HBASE-13082 changed the way that compacted files are archived from being done
> inline on compaction completion to an async cleanup by the
> CompactedHFilesDischarger chore. It looks like the changes to HStore to
> support this introduced a race condition in the compacted HFile archiving.
> In the following sequence, we can wind up with two separate threads trying to
> archive the same HFiles, causing a regionserver abort:
> # compaction completes normally and the compacted files are added to
> {{compactedfiles}} in HStore's DefaultStoreFileManager
> # *threadA*: CompactedHFilesDischargeHandler runs in a RS executor service,
> calling closeAndArchiveCompactedFiles()
> ## obtains HStore readlock
> ## gets a copy of compactedfiles
> ## releases readlock
> # *threadB*: calls HStore.close() as part of region close
> ## obtains HStore writelock
> ## calls DefaultStoreFileManager.clearCompactedfiles(), getting a copy of
> same compactedfiles
> # *threadA*: calls HStore.removeCompactedfiles(compactedfiles)
> ## archives files in {compactedfiles} in HRegionFileSystem.removeStoreFiles()
> ## call HStore.clearCompactedFiles()
> ## waits on write lock
> # *threadB*: continues with close()
> ## calls removeCompactedfiles(compactedfiles)
> ## calls HRegionFIleSystem.removeStoreFiles() ->
> HFileArchiver.archiveStoreFiles()
> ## receives FileNotFoundException because the files have already been
> archived by threadA
> ## throws IOException
> # RS aborts
> I think the combination of fetching the compactedfiles list and removing the
> files needs to be covered by locking. Options I see are:
> * Modify HStore.closeAndArchiveCompactedFiles(): use writelock instead of
> readlock and move the call to removeCompactedfiles() inside the lock. This
> means the read operations will be blocked while the files are being archived,
> which is bad.
> * Synchronize closeAndArchiveCompactedFiles() and modify close() to call it
> instead of calling removeCompactedfiles() directly
> * Add a separate lock for compacted files removal and use in
> closeAndArchiveCompactedFiles() and close()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16803) Make hbase:acl table unsplittable
[ https://issues.apache.org/jira/browse/HBASE-16803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566989#comment-15566989 ] Hudson commented on HBASE-16803: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1769 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1769/]) HBASE-16803 Make hbase:acl table unsplittable (tedyu: rev bbc274626748b0b567116a3c7426b194c8a1ad97) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java HBASE-16803 Make hbase:acl table unsplittable - revert pending review (tedyu: rev fef3c908d33e6bc100a988e7ab9eeb6d4d2d61ea) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java > Make hbase:acl table unsplittable > - > > Key: HBASE-16803 > URL: https://issues.apache.org/jira/browse/HBASE-16803 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16803.v1.txt, 16803.v2.txt > > > HBASE-16773 fixed a case where PriorityRpcServer handler threads are all > occupied accessing hbase:acl table. > However, the fix relies on the fact that there is single region in hbase:acl > table so that the access can be local. > As discussed at the end of HBASE-16773, we should disable split of hbase:acl > table as well. > hbase:meta is normally much larger than hbase:acl table and it has only one > region. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15893) Get object
[ https://issues.apache.org/jira/browse/HBASE-15893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566985#comment-15566985 ] Enis Soztutar commented on HBASE-15893: --- It is fine to add test cases where the expectation is that an exception is thrown. However, the unit test itself should not fail. > Get object > -- > > Key: HBASE-15893 > URL: https://issues.apache.org/jira/browse/HBASE-15893 > Project: HBase > Issue Type: Sub-task >Reporter: Sudeep Sunthankar >Assignee: Sudeep Sunthankar > Attachments: HBASE-15893.HBASE-14850.v1.patch, > HBASE-15893.HBASE-14850.v2.patch, HBASE-15893.HBASE-14850.v3.patch, > HBASE-15893.HBASE-14850.v4.patch > > > Patch for creating Get objects. Get objects can be passed to the Table > implementation to fetch results for a given row. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16788) Race in compacted file deletion between HStore close() and closeAndArchiveCompactedFiles()
[
https://issues.apache.org/jira/browse/HBASE-16788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566979#comment-15566979
]
Hudson commented on HBASE-16788:
SUCCESS: Integrated in Jenkins build HBase-1.3-JDK7 #36 (See
[https://builds.apache.org/job/HBase-1.3-JDK7/36/])
HBASE-16788 addendum Account for HStore archiveLock in heap size (garyh: rev
cd3afa5a0d85751936c54fa2398b63ff2efa128c)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
> Race in compacted file deletion between HStore close() and
> closeAndArchiveCompactedFiles()
> --
>
> Key: HBASE-16788
> URL: https://issues.apache.org/jira/browse/HBASE-16788
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.3.0
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Blocker
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: 16788-suggest.v2, HBASE-16788-addendum.patch,
> HBASE-16788.001.patch, HBASE-16788.002.patch, HBASE-16788_1.patch
>
>
> HBASE-13082 changed the way that compacted files are archived from being done
> inline on compaction completion to an async cleanup by the
> CompactedHFilesDischarger chore. It looks like the changes to HStore to
> support this introduced a race condition in the compacted HFile archiving.
> In the following sequence, we can wind up with two separate threads trying to
> archive the same HFiles, causing a regionserver abort:
> # compaction completes normally and the compacted files are added to
> {{compactedfiles}} in HStore's DefaultStoreFileManager
> # *threadA*: CompactedHFilesDischargeHandler runs in a RS executor service,
> calling closeAndArchiveCompactedFiles()
> ## obtains HStore readlock
> ## gets a copy of compactedfiles
> ## releases readlock
> # *threadB*: calls HStore.close() as part of region close
> ## obtains HStore writelock
> ## calls DefaultStoreFileManager.clearCompactedfiles(), getting a copy of
> same compactedfiles
> # *threadA*: calls HStore.removeCompactedfiles(compactedfiles)
> ## archives files in {compactedfiles} in HRegionFileSystem.removeStoreFiles()
> ## call HStore.clearCompactedFiles()
> ## waits on write lock
> # *threadB*: continues with close()
> ## calls removeCompactedfiles(compactedfiles)
> ## calls HRegionFIleSystem.removeStoreFiles() ->
> HFileArchiver.archiveStoreFiles()
> ## receives FileNotFoundException because the files have already been
> archived by threadA
> ## throws IOException
> # RS aborts
> I think the combination of fetching the compactedfiles list and removing the
> files needs to be covered by locking. Options I see are:
> * Modify HStore.closeAndArchiveCompactedFiles(): use writelock instead of
> readlock and move the call to removeCompactedfiles() inside the lock. This
> means the read operations will be blocked while the files are being archived,
> which is bad.
> * Synchronize closeAndArchiveCompactedFiles() and modify close() to call it
> instead of calling removeCompactedfiles() directly
> * Add a separate lock for compacted files removal and use in
> closeAndArchiveCompactedFiles() and close()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16811) Remove mob sweep job
[ https://issues.apache.org/jira/browse/HBASE-16811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16811: - Resolution: Fixed Status: Resolved (was: Patch Available) > Remove mob sweep job > > > Key: HBASE-16811 > URL: https://issues.apache.org/jira/browse/HBASE-16811 > Project: HBase > Issue Type: Task >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-16811.master.001.patch > > > Discussed here: > http://mail-archives.apache.org/mod_mbox/hbase-dev/201610.mbox/%3CCAAjhxro%3Dt62K44dV2wUtq1hqYLogZ45M3oeNOFZPcnwcSY4_DQ%40mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16811) Remove mob sweep job
[ https://issues.apache.org/jira/browse/HBASE-16811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566964#comment-15566964 ] Appy commented on HBASE-16811: -- unrelated test. also run failed because of OOM. Since patch is just removing files and the code compiles, it should be fine from unit-test perspective. Committing. > Remove mob sweep job > > > Key: HBASE-16811 > URL: https://issues.apache.org/jira/browse/HBASE-16811 > Project: HBase > Issue Type: Task >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-16811.master.001.patch > > > Discussed here: > http://mail-archives.apache.org/mod_mbox/hbase-dev/201610.mbox/%3CCAAjhxro%3Dt62K44dV2wUtq1hqYLogZ45M3oeNOFZPcnwcSY4_DQ%40mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15893) Get object
[ https://issues.apache.org/jira/browse/HBASE-15893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566948#comment-15566948 ] Sudeep Sunthankar commented on HBASE-15893: --- Hi [~enis], I have added test cases which succeed as well as fail by throwing an exception.These exceptions include cases where either a negative timestamp is passed or if max_timestamp parameter is less than min_timestamp. Please let me know if it is ok to keep them. -- Best Regards, Sudeep S > Get object > -- > > Key: HBASE-15893 > URL: https://issues.apache.org/jira/browse/HBASE-15893 > Project: HBase > Issue Type: Sub-task >Reporter: Sudeep Sunthankar >Assignee: Sudeep Sunthankar > Attachments: HBASE-15893.HBASE-14850.v1.patch, > HBASE-15893.HBASE-14850.v2.patch, HBASE-15893.HBASE-14850.v3.patch, > HBASE-15893.HBASE-14850.v4.patch > > > Patch for creating Get objects. Get objects can be passed to the Table > implementation to fetch results for a given row. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16811) Remove mob sweep job
[
https://issues.apache.org/jira/browse/HBASE-16811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566920#comment-15566920
]
Hadoop QA commented on HBASE-16811:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 44s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 89m 17s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 135m 0s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegion |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832750/HBASE-16811.master.001.patch
|
| JIRA Issue | HBASE-16811 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux d51add626ec0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / fef3c90 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3946/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3946/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3946/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3946/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Remove mob sweep job
>
>
> Key: HBASE-16811
> URL: https://issues.apache.org/jira/browse/HBASE-16811
>
[jira] [Updated] (HBASE-16642) Use DelayQueue instead of TimeoutBlockingQueue
[ https://issues.apache.org/jira/browse/HBASE-16642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16642: Fix Version/s: 2.0.0 > Use DelayQueue instead of TimeoutBlockingQueue > -- > > Key: HBASE-16642 > URL: https://issues.apache.org/jira/browse/HBASE-16642 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Reporter: Hiroshi Ikeda >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16642-v2.patch, HBASE-16642.master.V1.patch > > > Enqueue poisons in order to wake up and end the internal threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16642) Use DelayQueue instead of TimeoutBlockingQueue
[ https://issues.apache.org/jira/browse/HBASE-16642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16642: Component/s: proc-v2 > Use DelayQueue instead of TimeoutBlockingQueue > -- > > Key: HBASE-16642 > URL: https://issues.apache.org/jira/browse/HBASE-16642 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Reporter: Hiroshi Ikeda >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16642-v2.patch, HBASE-16642.master.V1.patch > > > Enqueue poisons in order to wake up and end the internal threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16781) Fix flaky TestMasterProcedureWalLease
[ https://issues.apache.org/jira/browse/HBASE-16781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16781: Resolution: Fixed Status: Resolved (was: Patch Available) > Fix flaky TestMasterProcedureWalLease > - > > Key: HBASE-16781 > URL: https://issues.apache.org/jira/browse/HBASE-16781 > Project: HBase > Issue Type: Test > Components: proc-v2, test >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16781-v0.patch > > > Attempt to fix the flaky TestMasterProcedureWalLease. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16757) Integrate functionality currently done up as Coprocessor Endpoints into core.
[ https://issues.apache.org/jira/browse/HBASE-16757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566905#comment-15566905 ] stack commented on HBASE-16757: --- Thanks [~jerryhe] > Integrate functionality currently done up as Coprocessor Endpoints into core. > - > > Key: HBASE-16757 > URL: https://issues.apache.org/jira/browse/HBASE-16757 > Project: HBase > Issue Type: Task > Components: Coprocessors >Reporter: stack > > As part of the work over in HBASE-15638, "Shade Protobufs", I could not but > help noticing that of the seven or eight Coprocessor Endpoints bundled with > hbase, half should have been converted to be core long time again. In fact, > some of these core CPEPs are no longer viable as CPEPs, if they ever were, > given how intertwined with core they are. > For example, MultiRowMutation, the nice CPEP that allows us do cross-row > transactions used natively amending hbase:meta, has much of its facility > baked into core without which it could not run. In an exercise, I was able to > convert this one over without having to alter public APIs in Table or Admin. > Auth, as pointed out by [~mbertozzi], is not a Coprocessor Endpoint though it > is cast as one invoked natively by RPC. > VisibilityLabels is a CPEP but core types -- Query and Mutation -- actually > depend on VisibiltyLabel related classes. > SecureBulkLoad is not in any violation being a CPEP provided to add API > ahead-of-time since properly deprecated and already integrated to core but I > mention it here for completeness sake. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16799) CP exposed Store should not expose unwanted APIs
[ https://issues.apache.org/jira/browse/HBASE-16799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566897#comment-15566897 ] stack commented on HBASE-16799: --- Good one [~anoop.hbase] What brought this on? When were the methods added? We need to add a warning on the Store API? > CP exposed Store should not expose unwanted APIs > - > > Key: HBASE-16799 > URL: https://issues.apache.org/jira/browse/HBASE-16799 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16799.patch, HBASE-16799.patch, > HBASE-16799_V2.patch > > > Store is exposed to CPs. The main use cases I can think of are getting store > scanner and other getters which return different states like memstore size, > max seqId etc. Those make sense. > But we added many other APIs which changes the state of the memstore, bulk > load files etc into this interface. Even an API which expose the memstore > itself!. This allow adding mutations into memstore bypassing all steps in > region. We track the memstore size per region level as well as globally. > These only allow us to flush region at sizes and/or flush selected regions > because of global heap pressure. Now if a CP get hold of store and/or > memstore, it can add mutations with out knowledge of these size accounting > and possibly OOME the RS. Similar way the bulk load related APIs. At HRegion > level, there are steps done (WAL write etc) after the bulk load HFile on > store. So bypassing these wont be correct. > In this jira, plan is to remove all such leaked APIs from Store. They are > called from HRegion and we can type cast to HStore to call them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15924) Enhance hbase services autorestart capability to hbase-daemon.sh
[
https://issues.apache.org/jira/browse/HBASE-15924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566876#comment-15566876
]
Andrew Purtell commented on HBASE-15924:
In another test, I started the regionserver with {{./bin/hbase-daemon.sh
--autostart-window-retry-limit 3 autostart regionserver}} and in another SSH
session then attempted to stop the regionserver with {{./bin/hbase-daemon.sh
stop regionserver}}. This appears to work, although I can see by tailing the
regionserver log output file that the regionserver process is partially
restarted and rapidly killed. I tried again with {{./bin/hbase-daemon.sh
autostart regionserver}} i.e. no limit, then in another SSH session attempted
again to stop the regionserver with {{./bin/hbase-daemon.sh stop
regionserver}}, and it worked, but again with multiple partial restarts. Is
this supposed to work?
> Enhance hbase services autorestart capability to hbase-daemon.sh
> -
>
> Key: HBASE-15924
> URL: https://issues.apache.org/jira/browse/HBASE-15924
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.98.19
>Reporter: Loknath Priyatham Teja Singamsetty
>Assignee: Loknath Priyatham Teja Singamsetty
> Fix For: 0.98.24
>
> Attachments: HBASE-15924.master.0001.patch,
> HBASE-15924.master.0002.patch, HBASE-15924.master.0003.patch
>
>
> As part of HBASE-5939, the autorestart for hbase services has been added to
> deal with scenarios where hbase services (master/regionserver/master-backup)
> gets killed or goes down leading to unplanned outages. The changes were made
> to hbase-daemon.sh to support autorestart option.
> However, the autorestart implementation doesn't work in standalone mode and
> other than that have few gaps with the implementation as per release notes of
> HBASE-5939. Here is an attempt to re-design and fix the functionality
> considering all possible usecases with hbase service operations.
> Release Notes of HBASE-5939:
> --
> When launched with autorestart, HBase processes will automatically restart if
> they are not properly terminated, either by a "stop" command or by a cluster
> stop. To ensure that it does not overload the system when the server itself
> is corrupted and the process cannot be restarted, the server sleeps for 5
> minutes before restarting if it was already started 5 minutes ago previously.
> To use it, launch the process with "bin/start-hbase autorestart". This option
> is not fully compatible with the existing "restart" command: if you ask for a
> restart on a server launched with autorestart, the server will restart but
> the next server instance won't be automatically restarted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15109) HM/RS failed to start when "fs.hdfs.impl.disable.cache" is set to true
[
https://issues.apache.org/jira/browse/HBASE-15109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-15109:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks for the patch, Pankaj.
> HM/RS failed to start when "fs.hdfs.impl.disable.cache" is set to true
> --
>
> Key: HBASE-15109
> URL: https://issues.apache.org/jira/browse/HBASE-15109
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15109-V2.patch, HBASE-15109-branch-1.patch,
> HBASE-15109.patch, HBASE-15109.patch
>
>
> HMaster failed to start during installing ShutdownHook when
> "fs.hdfs.impl.disable.cache" is set to true.
> {noformat}
> 2016-10-10 10:33:28,367 FATAL [master/HOST-10-18-106-146/10.18.106.146:16000]
> master.HMaster: Unhandled: Failed suppression of fs shutdown hook:
> org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@19a003b4
> java.lang.RuntimeException: Failed suppression of fs shutdown hook:
> org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@19a003b4
> at
> org.apache.hadoop.hbase.regionserver.ShutdownHook.suppressHdfsShutdownHook(ShutdownHook.java:204)
> at
> org.apache.hadoop.hbase.regionserver.ShutdownHook.install(ShutdownHook.java:84)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:950)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> And RegionServer is in blocking state until a master is available,
> {noformat}
> "regionserver/HOST-10-18-106-146/10.18.106.146:16020" #17 prio=5 os_prio=0
> tid=0x00e4e800 nid=0xdc38 in Object.wait() [0x7f329dac6000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:159)
> - locked <0xae25c0c8> (a
> org.apache.hadoop.hbase.zookeeper.MasterAddressTracker)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.blockAndCheckIfStopped(HRegionServer.java:870)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:809)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:783)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:943)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> During installation, first time it will try to suppress the HDFS shutdownhook
> by removing the hdfsclientfinalizer from HDFS. Since
> fs.hdfs.impl.disable.cache is enabled, so removal will be unsuccessful from
> HDFS (FS is not cached) and RuntimeException will be thrown with "Failed
> suppression of fs shutdown hook" message.
> In ShutdownHook,
> {code}
> if (!fsShutdownHooks.containsKey(hdfsClientFinalizer) &&
> !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> throw new RuntimeException("Failed suppression of fs shutdown hook: " +
> hdfsClientFinalizer);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15893) Get object
[
https://issues.apache.org/jira/browse/HBASE-15893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566855#comment-15566855
]
Enis Soztutar commented on HBASE-15893:
---
[~sudeeps] the new tests are failing with the patch:
{code}
TESTING ALL TESTS
PASS<100ms 2 Passed 0 Skipped 0 Failed
//connection:connection-pool-test
PASS<100ms 3 Passed 0 Skipped 0 Failed
//serde:client-deserializer-test
PASS<100ms 4 Passed 0 Skipped 0 Failed //serde:client-serializer-test
PASS<100ms 1 Passed 0 Skipped 0 Failed
//serde:region-info-deserializer-test
PASS<100ms 4 Passed 0 Skipped 0 Failed //serde:server-name-test
PASS<100ms 3 Passed 0 Skipped 0 Failed //serde:table-name-test
PASS<100ms 3 Passed 0 Skipped 0 Failed //serde:zk-deserializer-test
PASS<100ms 1 Passed 0 Skipped 0 Failed //utils:user-util-test
PASS<100ms 7 Passed 0 Skipped 0 Failed //core:cell-test
PASS<100ms 2 Passed 0 Skipped 0 Failed //core:get-test
PASS 50.0s 2 Passed 0 Skipped 0 Failed //core:location-cache-test
FAIL<100ms 1 Passed 0 Skipped 3 Failed //core:time_range-test
FAILURE TimeRangeTest NegativeMinTime: unknown file
C++ exception with description "Timestamp cannot be negative. min_timestamp:
-100, max_timestamp:2000" thrown in the test body.
STANDARD OUT
unknown file: Failure
C++ exception with description "Timestamp cannot be negative. min_timestamp:
-100, max_timestamp:2000" thrown in the test body.
STANDARD ERR
FAILURE TimeRangeTest NegativeMaxTime: unknown file
C++ exception with description "Timestamp cannot be negative. min_timestamp:
100, max_timestamp:-2000" thrown in the test body.
STANDARD OUT
unknown file: Failure
C++ exception with description "Timestamp cannot be negative. min_timestamp:
100, max_timestamp:-2000" thrown in the test body.
STANDARD ERR
FAILURE TimeRangeTest MinTimeGreater: unknown file
C++ exception with description "max_timestamp [2000] should be greater than
min_timestamp [1]" thrown in the test body.
STANDARD OUT
unknown file: Failure
C++ exception with description "max_timestamp [2000] should be greater than
min_timestamp [1]" thrown in the test body.
{code}
Can you please take a look.
> Get object
> --
>
> Key: HBASE-15893
> URL: https://issues.apache.org/jira/browse/HBASE-15893
> Project: HBase
> Issue Type: Sub-task
>Reporter: Sudeep Sunthankar
>Assignee: Sudeep Sunthankar
> Attachments: HBASE-15893.HBASE-14850.v1.patch,
> HBASE-15893.HBASE-14850.v2.patch, HBASE-15893.HBASE-14850.v3.patch,
> HBASE-15893.HBASE-14850.v4.patch
>
>
> Patch for creating Get objects. Get objects can be passed to the Table
> implementation to fetch results for a given row.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15924) Enhance hbase services autorestart capability to hbase-daemon.sh
[
https://issues.apache.org/jira/browse/HBASE-15924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566852#comment-15566852
]
Andrew Purtell commented on HBASE-15924:
I took this for a spin.
{{./bin/hbase-daemon.sh autostart}} and {{autorestart}} works differently from
{{start}} in that we don't see a message printed like
{noformat}
echo starting $command, logging to $HBASE_LOGOUT
{noformat}
which lead me to believe the command silently failed to launch the requested
process. Please fix this, it's going to cause confusion.
There is an off-by-one error dealing with the autostart window retry limit. I
tried this with {{./bin/hbase-daemon.sh --autostart-window-retry-limit 3
autostart regionserver}} then in order to stop autostart had to kill the
regionserver 4 times:
{noformat}
1308 jps -l
1309 kill 11956
1310 jps -l
1311 kill 12760
1312 jps -l
1313 kill 13205
1314 jps -l
1315 kill 13645
{noformat}
until finally:
{noformat}
Tue Oct 11 15:41:20 PDT 2016 Autostart window retry limit: 3 exceeded for given
window size: 0 hours
{noformat}
> Enhance hbase services autorestart capability to hbase-daemon.sh
> -
>
> Key: HBASE-15924
> URL: https://issues.apache.org/jira/browse/HBASE-15924
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.98.19
>Reporter: Loknath Priyatham Teja Singamsetty
>Assignee: Loknath Priyatham Teja Singamsetty
> Fix For: 0.98.24
>
> Attachments: HBASE-15924.master.0001.patch,
> HBASE-15924.master.0002.patch, HBASE-15924.master.0003.patch
>
>
> As part of HBASE-5939, the autorestart for hbase services has been added to
> deal with scenarios where hbase services (master/regionserver/master-backup)
> gets killed or goes down leading to unplanned outages. The changes were made
> to hbase-daemon.sh to support autorestart option.
> However, the autorestart implementation doesn't work in standalone mode and
> other than that have few gaps with the implementation as per release notes of
> HBASE-5939. Here is an attempt to re-design and fix the functionality
> considering all possible usecases with hbase service operations.
> Release Notes of HBASE-5939:
> --
> When launched with autorestart, HBase processes will automatically restart if
> they are not properly terminated, either by a "stop" command or by a cluster
> stop. To ensure that it does not overload the system when the server itself
> is corrupted and the process cannot be restarted, the server sleeps for 5
> minutes before restarting if it was already started 5 minutes ago previously.
> To use it, launch the process with "bin/start-hbase autorestart". This option
> is not fully compatible with the existing "restart" command: if you ask for a
> restart on a server launched with autorestart, the server will restart but
> the next server instance won't be automatically restarted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15109) HM/RS failed to start when "fs.hdfs.impl.disable.cache" is set to true
[
https://issues.apache.org/jira/browse/HBASE-15109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-15109:
---
Hadoop Flags: Reviewed
Fix Version/s: 2.0.0
> HM/RS failed to start when "fs.hdfs.impl.disable.cache" is set to true
> --
>
> Key: HBASE-15109
> URL: https://issues.apache.org/jira/browse/HBASE-15109
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15109-V2.patch, HBASE-15109-branch-1.patch,
> HBASE-15109.patch, HBASE-15109.patch
>
>
> HMaster failed to start during installing ShutdownHook when
> "fs.hdfs.impl.disable.cache" is set to true.
> {noformat}
> 2016-10-10 10:33:28,367 FATAL [master/HOST-10-18-106-146/10.18.106.146:16000]
> master.HMaster: Unhandled: Failed suppression of fs shutdown hook:
> org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@19a003b4
> java.lang.RuntimeException: Failed suppression of fs shutdown hook:
> org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@19a003b4
> at
> org.apache.hadoop.hbase.regionserver.ShutdownHook.suppressHdfsShutdownHook(ShutdownHook.java:204)
> at
> org.apache.hadoop.hbase.regionserver.ShutdownHook.install(ShutdownHook.java:84)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:950)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> And RegionServer is in blocking state until a master is available,
> {noformat}
> "regionserver/HOST-10-18-106-146/10.18.106.146:16020" #17 prio=5 os_prio=0
> tid=0x00e4e800 nid=0xdc38 in Object.wait() [0x7f329dac6000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:159)
> - locked <0xae25c0c8> (a
> org.apache.hadoop.hbase.zookeeper.MasterAddressTracker)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.blockAndCheckIfStopped(HRegionServer.java:870)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:809)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:783)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:943)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> During installation, first time it will try to suppress the HDFS shutdownhook
> by removing the hdfsclientfinalizer from HDFS. Since
> fs.hdfs.impl.disable.cache is enabled, so removal will be unsuccessful from
> HDFS (FS is not cached) and RuntimeException will be thrown with "Failed
> suppression of fs shutdown hook" message.
> In ShutdownHook,
> {code}
> if (!fsShutdownHooks.containsKey(hdfsClientFinalizer) &&
> !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> throw new RuntimeException("Failed suppression of fs shutdown hook: " +
> hdfsClientFinalizer);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15109) HM/RS failed to start when "fs.hdfs.impl.disable.cache" is set to true
[
https://issues.apache.org/jira/browse/HBASE-15109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566831#comment-15566831
]
stack commented on HBASE-15109:
---
+1
> HM/RS failed to start when "fs.hdfs.impl.disable.cache" is set to true
> --
>
> Key: HBASE-15109
> URL: https://issues.apache.org/jira/browse/HBASE-15109
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Attachments: HBASE-15109-V2.patch, HBASE-15109-branch-1.patch,
> HBASE-15109.patch, HBASE-15109.patch
>
>
> HMaster failed to start during installing ShutdownHook when
> "fs.hdfs.impl.disable.cache" is set to true.
> {noformat}
> 2016-10-10 10:33:28,367 FATAL [master/HOST-10-18-106-146/10.18.106.146:16000]
> master.HMaster: Unhandled: Failed suppression of fs shutdown hook:
> org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@19a003b4
> java.lang.RuntimeException: Failed suppression of fs shutdown hook:
> org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@19a003b4
> at
> org.apache.hadoop.hbase.regionserver.ShutdownHook.suppressHdfsShutdownHook(ShutdownHook.java:204)
> at
> org.apache.hadoop.hbase.regionserver.ShutdownHook.install(ShutdownHook.java:84)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:950)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> And RegionServer is in blocking state until a master is available,
> {noformat}
> "regionserver/HOST-10-18-106-146/10.18.106.146:16020" #17 prio=5 os_prio=0
> tid=0x00e4e800 nid=0xdc38 in Object.wait() [0x7f329dac6000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:159)
> - locked <0xae25c0c8> (a
> org.apache.hadoop.hbase.zookeeper.MasterAddressTracker)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.blockAndCheckIfStopped(HRegionServer.java:870)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:809)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:783)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:943)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> During installation, first time it will try to suppress the HDFS shutdownhook
> by removing the hdfsclientfinalizer from HDFS. Since
> fs.hdfs.impl.disable.cache is enabled, so removal will be unsuccessful from
> HDFS (FS is not cached) and RuntimeException will be thrown with "Failed
> suppression of fs shutdown hook" message.
> In ShutdownHook,
> {code}
> if (!fsShutdownHooks.containsKey(hdfsClientFinalizer) &&
> !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> throw new RuntimeException("Failed suppression of fs shutdown hook: " +
> hdfsClientFinalizer);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16749) HBase root pom.xml contains repo from people.apache.org/~garyh
[
https://issues.apache.org/jira/browse/HBASE-16749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Helmling updated HBASE-16749:
--
Attachment: HBASE-16749.branch-1.2.patch
HBASE-16749.branch-1.1.patch
Attaching patches to remove the unused repo from branch-1.1 and branch-1.2.
> HBase root pom.xml contains repo from people.apache.org/~garyh
> --
>
> Key: HBASE-16749
> URL: https://issues.apache.org/jira/browse/HBASE-16749
> Project: HBase
> Issue Type: Task
>Reporter: Wangda Tan
>Assignee: Gary Helmling
> Attachments: HBASE-16749.branch-1.1.patch,
> HBASE-16749.branch-1.2.patch, HBASE-16749.branch-1.2.patch
>
>
> This is found when I was building Hadoop/YARN:
> {code}
> [INFO]
>
> [INFO] Building Apache Hadoop YARN Timeline Service 3.0.0-alpha2-SNAPSHOT
> [INFO]
>
> Downloading:
> http://conjars.org/repo/org/apache/hadoop/hadoop-client/3.0.0-alpha2-SNAPSHOT/maven-metadata.xml
> ...
> Downloading:
> http://people.apache.org/~garyh/mvn/org/apache/hadoop/hadoop-client/3.0.0-alpha2-SNAPSHOT/maven-metadata.xml
> ...
> Downloading:
> http://people.apache.org/~garyh/mvn/org/apache/hadoop/hadoop-client/3.0.0-alpha2-SNAPSHOT/maven-metadata.xml
> {code}
> [~te...@apache.org] mentioned:
> bq. Among hbase releases (1.1.x, 1.2.y), root pom.xml contains
> "ghelmling.testing" repo.
> This private repo sometimes extends the Hadoop build time a lot. It might be
> better to use ASF snapshot repo first before downloading from private repo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16810) HBase Balancer throws ArrayIndexOutOfBoundsException when regionservers in /hbase/draining znode and unloaded
[
https://issues.apache.org/jira/browse/HBASE-16810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566823#comment-15566823
]
Matteo Bertozzi commented on HBASE-16810:
-
MockNoopMasterServices is the generic one that can be used. the one in
TestCatalogJanitor is specific to that test
> HBase Balancer throws ArrayIndexOutOfBoundsException when regionservers in
> /hbase/draining znode and unloaded
> -
>
> Key: HBASE-16810
> URL: https://issues.apache.org/jira/browse/HBASE-16810
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Ashu Pachauri
>Assignee: David Pope
> Fix For: 1.3.0
>
> Attachments: master.patch
>
>
> 1. Add a regionserver znode under /hbase/draining znode.
> 2. Use RegionMover to unload all regions from the regionserver.
> 3. Run balancer.
> {code}
> 16/09/21 14:17:33 ERROR ipc.RpcServer: Unexpected throwable object
> java.lang.ArrayIndexOutOfBoundsException: 75
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.getLocalityOfRegion(BaseLoadBalancer.java:867)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer$LocalityCostFunction.cost(StochasticLoadBalancer.java:1186)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.computeCost(StochasticLoadBalancer.java:521)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:309)
> at
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:264)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1339)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.balance(MasterRpcServices.java:442)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58555)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2268)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16802) Procedure v2 - group procedure cleaning
[ https://issues.apache.org/jira/browse/HBASE-16802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566810#comment-15566810 ] stack commented on HBASE-16802: --- LGTM. I like the way you have single arg long delete and then a bulk delete when more than one proc to get rid of. > Procedure v2 - group procedure cleaning > --- > > Key: HBASE-16802 > URL: https://issues.apache.org/jira/browse/HBASE-16802 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-16802-v0.patch > > > group the cleaning of the evicted procedures -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16808) Included the generated, shaded java files from "HBASE-15638 Shade protobuf" in our src assembly
[
https://issues.apache.org/jira/browse/HBASE-16808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566805#comment-15566805
]
Hadoop QA commented on HBASE-16808:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 8s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 50s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 10s
{color} | {color:green} hbase-assembly in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 17s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.0 Server=1.12.0 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832773/HBASE-16808.master.001.patch
|
| JIRA Issue | HBASE-16808 |
| Optional Tests | asflicense javac javadoc unit xml |
| uname | Linux 0ac728efc84a 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 7b0acc2 |
| Default Java | 1.8.0_101 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3948/testReport/ |
| modules | C: hbase-assembly U: hbase-assembly |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3948/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Included the generated, shaded java files from "HBASE-15638 Shade protobuf"
> in our src assembly
> ---
>
> Key: HBASE-16808
> URL: https://issues.apache.org/jira/browse/HBASE-16808
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Attachments: HBASE-16808.master.001.patch
>
>
> [~Apache9] found that I forgot to include the generated, shaded files in our
> src tgz. Let me fix. This is a follow-on to HBASE-16793
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16622) Fix some issues with the HBase reference guide
[
https://issues.apache.org/jira/browse/HBASE-16622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-16622:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.0.0
Status: Resolved (was: Patch Available)
Pushed to master branch. Thanks for the patch [~alexxiyang]
> Fix some issues with the HBase reference guide
> --
>
> Key: HBASE-16622
> URL: https://issues.apache.org/jira/browse/HBASE-16622
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: alexxiyang
>Assignee: alexxiyang
> Fix For: 2.0.0
>
> Attachments: HBASE-16622-v0.diff, HBASE-16622-v1.diff,
> HBASE-16622-v2.diff
>
>
> 1.
> {code}
> if (admin.tableExists(tableName)) {
> System.out.println("Table does not exist.");
> System.exit(-1);
> }
> {code}
> This should be
> {code}
> if (!admin.tableExists(tableName)) {
> {code}
> 2.
> SNAPPY is not suitable for begginer. They may get exceptions like
> {code}
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.DoNotRetryIOException):
> org.apache.hadoop.hbase.DoNotRetryIOException: Compression algorithm
> 'snappy' previously failed test. Set hbase.table.sanity.checks to false at
> conf or table descriptor if you want to bypass sanity checks
> at
> org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:1701)
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1569)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1491)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:462)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55682)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2178)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> So the code below
> {code}
> table.addFamily(new
> HColumnDescriptor(CF_DEFAULT).setCompressionType(Algorithm.SNAPPY));
> {code}
> it better to change into
> {code}
> table.addFamily(new
> HColumnDescriptor(CF_DEFAULT).setCompressionType(Algorithm.NONE));
> {code}
> 3.
> Before modify column family , get the table from connection
> Change
> {code}
> HTableDescriptor table = new HTableDescriptor(tableName);
> {code}
> into
> {code}
> Table table = connection.getTable(TableName.valueOf(tablename));
> {code}
> 4.
> In 143.1.1. Code Formatting
> it just said
> {code}
> Still in Preferences, click . Be sure the following options are
> selected:Apache HBase ™ Reference Guide
> {code}
> But nothing after click. It should be Java->Editor->Save Actions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16801) The Append/Increment may return the data from future
[ https://issues.apache.org/jira/browse/HBASE-16801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16801: -- Priority: Critical (was: Minor) I marked this critical. I see misreading if read wrong mvcc. In fact, should the mvcc be volatile since it could be set by one thread but read by another setting up a Scanner? > The Append/Increment may return the data from future > > > Key: HBASE-16801 > URL: https://issues.apache.org/jira/browse/HBASE-16801 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16801.v0.patch > > > OperationContext maintains the mvcc as a static member, so any Append’s and > Increment’s read point will be changed by others. That is, a retrying > Append/Increment may “see” the future data. > This is a master only issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16803) Make hbase:acl table unsplittable
[ https://issues.apache.org/jira/browse/HBASE-16803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566767#comment-15566767 ] Hudson commented on HBASE-16803: SUCCESS: Integrated in Jenkins build HBase-1.4 #460 (See [https://builds.apache.org/job/HBase-1.4/460/]) HBASE-16803 Make hbase:acl table unsplittable (tedyu: rev 408a9eb8a303e9f0db5eecaa77c30f28ff521d07) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java HBASE-16803 Make hbase:acl table unsplittable - revert pending review (tedyu: rev b47ded3b4226e290953cf433829fdc1ee25c08fd) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java > Make hbase:acl table unsplittable > - > > Key: HBASE-16803 > URL: https://issues.apache.org/jira/browse/HBASE-16803 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16803.v1.txt, 16803.v2.txt > > > HBASE-16773 fixed a case where PriorityRpcServer handler threads are all > occupied accessing hbase:acl table. > However, the fix relies on the fact that there is single region in hbase:acl > table so that the access can be local. > As discussed at the end of HBASE-16773, we should disable split of hbase:acl > table as well. > hbase:meta is normally much larger than hbase:acl table and it has only one > region. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16788) Race in compacted file deletion between HStore close() and closeAndArchiveCompactedFiles()
[
https://issues.apache.org/jira/browse/HBASE-16788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566768#comment-15566768
]
Hudson commented on HBASE-16788:
SUCCESS: Integrated in Jenkins build HBase-1.4 #460 (See
[https://builds.apache.org/job/HBase-1.4/460/])
HBASE-16788 addendum Account for HStore archiveLock in heap size (garyh: rev
f13a21696f2bbd4f572eb35c15282835998d4b34)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
> Race in compacted file deletion between HStore close() and
> closeAndArchiveCompactedFiles()
> --
>
> Key: HBASE-16788
> URL: https://issues.apache.org/jira/browse/HBASE-16788
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.3.0
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Blocker
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: 16788-suggest.v2, HBASE-16788-addendum.patch,
> HBASE-16788.001.patch, HBASE-16788.002.patch, HBASE-16788_1.patch
>
>
> HBASE-13082 changed the way that compacted files are archived from being done
> inline on compaction completion to an async cleanup by the
> CompactedHFilesDischarger chore. It looks like the changes to HStore to
> support this introduced a race condition in the compacted HFile archiving.
> In the following sequence, we can wind up with two separate threads trying to
> archive the same HFiles, causing a regionserver abort:
> # compaction completes normally and the compacted files are added to
> {{compactedfiles}} in HStore's DefaultStoreFileManager
> # *threadA*: CompactedHFilesDischargeHandler runs in a RS executor service,
> calling closeAndArchiveCompactedFiles()
> ## obtains HStore readlock
> ## gets a copy of compactedfiles
> ## releases readlock
> # *threadB*: calls HStore.close() as part of region close
> ## obtains HStore writelock
> ## calls DefaultStoreFileManager.clearCompactedfiles(), getting a copy of
> same compactedfiles
> # *threadA*: calls HStore.removeCompactedfiles(compactedfiles)
> ## archives files in {compactedfiles} in HRegionFileSystem.removeStoreFiles()
> ## call HStore.clearCompactedFiles()
> ## waits on write lock
> # *threadB*: continues with close()
> ## calls removeCompactedfiles(compactedfiles)
> ## calls HRegionFIleSystem.removeStoreFiles() ->
> HFileArchiver.archiveStoreFiles()
> ## receives FileNotFoundException because the files have already been
> archived by threadA
> ## throws IOException
> # RS aborts
> I think the combination of fetching the compactedfiles list and removing the
> files needs to be covered by locking. Options I see are:
> * Modify HStore.closeAndArchiveCompactedFiles(): use writelock instead of
> readlock and move the call to removeCompactedfiles() inside the lock. This
> means the read operations will be blocked while the files are being archived,
> which is bad.
> * Synchronize closeAndArchiveCompactedFiles() and modify close() to call it
> instead of calling removeCompactedfiles() directly
> * Add a separate lock for compacted files removal and use in
> closeAndArchiveCompactedFiles() and close()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16801) The Append/Increment may return the data from future
[ https://issues.apache.org/jira/browse/HBASE-16801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566759#comment-15566759 ] stack commented on HBASE-16801: --- Nice find [~chia7712] > The Append/Increment may return the data from future > > > Key: HBASE-16801 > URL: https://issues.apache.org/jira/browse/HBASE-16801 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16801.v0.patch > > > OperationContext maintains the mvcc as a static member, so any Append’s and > Increment’s read point will be changed by others. That is, a retrying > Append/Increment may “see” the future data. > This is a master only issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16146) Counters are expensive...
[ https://issues.apache.org/jira/browse/HBASE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling resolved HBASE-16146. --- Resolution: Fixed Assignee: Gary Helmling Hadoop Flags: Reviewed Fix Version/s: 1.4.0 1.3.0 2.0.0 Committed to branch-1.3, branch-1, and master. Counter is no longer used in master, but still present as a deprecated class, so included for consistency. Thanks, [~stack], [~mantonov], and [~enis] for reviews. > Counters are expensive... > - > > Key: HBASE-16146 > URL: https://issues.apache.org/jira/browse/HBASE-16146 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Gary Helmling > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16146.001.patch, HBASE-16146.branch-1.001.patch, > HBASE-16146.branch-1.3.001.patch, counters.patch, less_and_less_counters.png > > > Doing workloadc, perf shows 10%+ of CPU being spent on counter#add. If I > disable some of the hot ones -- see patch -- I can get 10% more throughput > (390k to 440k). Figure something better. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16146) Counters are expensive...
[ https://issues.apache.org/jira/browse/HBASE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling updated HBASE-16146: -- Attachment: HBASE-16146.branch-1.001.patch HBASE-16146.001.patch The patches for master and branch-1 each wound up being slightly different. Attaching all here. > Counters are expensive... > - > > Key: HBASE-16146 > URL: https://issues.apache.org/jira/browse/HBASE-16146 > Project: HBase > Issue Type: Sub-task >Reporter: stack > Attachments: HBASE-16146.001.patch, HBASE-16146.branch-1.001.patch, > HBASE-16146.branch-1.3.001.patch, counters.patch, less_and_less_counters.png > > > Doing workloadc, perf shows 10%+ of CPU being spent on counter#add. If I > disable some of the hot ones -- see patch -- I can get 10% more throughput > (390k to 440k). Figure something better. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[
https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566740#comment-15566740
]
stack commented on HBASE-16807:
---
Nice analysis. Have a patch [~pankaj_kumar]?
> RegionServer will fail to report new active Hmaster until
> HMaster/RegionServer failover
> ---
>
> Key: HBASE-16807
> URL: https://issues.apache.org/jira/browse/HBASE-16807
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>
> It's little weird, but it happened in the product environment that few
> RegionServer missed master znode create notification on master failover. In
> that case ZooKeeperNodeTracker will not refresh the cached data and
> MasterAddressTracker
> will always return old active HM detail to Region server on ServiceException.
> Though We create region server stub on failure but without refreshing the
> MasterAddressTracker data.
> In HRegionServer.createRegionServerStatusStub()
> {code}
> boolean refresh = false; // for the first time, use cached data
> RegionServerStatusService.BlockingInterface intf = null;
> boolean interrupted = false;
> try {
> while (keepLooping()) {
> sn = this.masterAddressTracker.getMasterAddress(refresh);
> if (sn == null) {
> if (!keepLooping()) {
> // give up with no connection.
> LOG.debug("No master found and cluster is stopped; bailing out");
> return null;
> }
> if (System.currentTimeMillis() > (previousLogTime + 1000)) {
> LOG.debug("No master found; retry");
> previousLogTime = System.currentTimeMillis();
> }
> refresh = true; // let's try pull it from ZK directly
> if (sleep(200)) {
> interrupted = true;
> }
> continue;
> }
> {code}
> Here we refresh node only when 'sn' is NULL otherwise it will use same cached
> data.
> So in above case RegionServer will never report active HMaster successfully
> until HMaster failover or RegionServer restart.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16469) Several log refactoring/improvement suggestions
[
https://issues.apache.org/jira/browse/HBASE-16469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566732#comment-15566732
]
stack commented on HBASE-16469:
---
+1
Waiting on qa build.
> Several log refactoring/improvement suggestions
> ---
>
> Key: HBASE-16469
> URL: https://issues.apache.org/jira/browse/HBASE-16469
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.2
>Reporter: Nemo Chen
> Labels: easyfix, easytest
> Fix For: 1.2.2
>
> Attachments: HBASE-16469.git3f671c1ead.001.patch
>
>
> *method invocation replaced by variable*
> hbase-1.2.2/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/LongTermArchivingHFileCleaner.java
> line 57: {code}Path file = fStat.getPath();{code}
> line 74: {code}LOG.error("Failed to lookup status of:" + fStat.getPath() + ",
> keeping it just incase.", e); {code}
> hbase-1.2.2/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/CloseRegionHandler.java
> line 118: {code}String name = regionInfo.getRegionNameAsString();{code}
> line 142: {code}LOG.warn("Can't close region: was already closed during
> close(): " +
> regionInfo.getRegionNameAsString()); {code}
> In the above two examples, the method invocations are assigned to the
> variables before the logging code. These method invocations should be
> replaced by variables in case of simplicity and readability
>
> *method invocation in return statement*
> hbase-1.2.2/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> line 5455:
> {code}
> public String toString() {
> return getRegionInfo().getRegionNameAsString();
> }
> {code}
> line 1260:
> {code}
> LOG.debug("Region " + getRegionInfo().getRegionNameAsString()
> + " is not mergeable because it is closing or closed");
> {code}
> line 1265:
> {code}
> LOG.debug("Region " + getRegionInfo().getRegionNameAsString()
> + " is not mergeable because it has references");
> {code}
> line 1413:
> {code}
> LOG.info("Running close preflush of " +
> getRegionInfo().getRegionNameAsString());
> {code}
> In these above examples, the "getRegionInfo().getRegionNameAsString())" is
> the return statement of method "toString" in the same class. They should be
> replaced with “this” in case of simplicity and readability.
>
> *check the logged variable if it is null*
> hbase-1.2.2/hbase-it/src/test/java/org/apache/hadoop/hbase/HBaseClusterManager.java
> line 88:
> {code}
> if ((sshUserName != null && sshUserName.length() > 0) ||
> (sshOptions != null && sshOptions.length() > 0)) {
> LOG.info("Running with SSH user [" + sshUserName + "] and options [" +
> sshOptions + "]");
> }
> {code}
> hbase-1.2.2/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
> line 980:
> {code}
> if ((regionState == null && latestState != null)
> || (regionState != null && latestState == null)
> || (regionState != null && latestState != null
> && latestState.getState() != regionState.getState())) {
> LOG.warn("Region state changed from " + regionState + " to "
> + latestState + ", while acquiring lock");
> }
> {code}
> In the above example, the logging variable could null at run time. It is a
> bad practice to include null variables inside logs.
>
> *variable in byte printed directly*
> hbase-1.2.2/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedUpdater.java
> line 145:
> {code}
> byte[] rowKey = dataGenerator.getDeterministicUniqueKey(rowKeyBase);
> {code}
> line 184:
> {code}
> LOG.error("Failed to update the row with key = [" + rowKey
> + "], since we could not get the original row");
> {code}
> rowKey should be printed as Bytes.toString(rowKey).
>
> *object toString contain mi*
> hbase-1.2.2/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java
> {code}
> LOG.warn("#" + id + ", the task was rejected by the pool. This is
> unexpected."+ " Server is "+ server.getServerName(),t);
> {code}
> server is an instance of class ServerName, we found ServerName.java:
> hbase-client/src/main/java/org/apache/hadoop/hbase/ServerName.java
> {code}
> @Override
> public String toString() {
> return getServerName();
> }
> {code}
> the toString method returns getServerName(), so the "server.getServerName()"
> should be replaced with "server" in case of simplicity and readability
> Similar examples are in:
> hbase-1.2.2/hbase-client/src/main/java/org/apache/hadoop/hbase/client/PreemptiveFastFailInterceptor.java
> {code}
> LOG.info("Clearing out PFFE for server " + server.getServerName());
> return getServerName();
> {code}
> hbase-1.2.2/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
> line 705:
> {code}
[jira] [Commented] (HBASE-16808) Included the generated, shaded java files from "HBASE-15638 Shade protobuf" in our src assembly
[ https://issues.apache.org/jira/browse/HBASE-16808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566723#comment-15566723 ] stack commented on HBASE-16808: --- I tested by building source tgz ensuring I could then build src tgz from what was in first tgz and then starting an hbase instance seems to work. > Included the generated, shaded java files from "HBASE-15638 Shade protobuf" > in our src assembly > --- > > Key: HBASE-16808 > URL: https://issues.apache.org/jira/browse/HBASE-16808 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack >Priority: Blocker > Attachments: HBASE-16808.master.001.patch > > > [~Apache9] found that I forgot to include the generated, shaded files in our > src tgz. Let me fix. This is a follow-on to HBASE-16793 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15437) Response size calculated in RPCServer for warning tooLarge responses does NOT count CellScanner payload
[
https://issues.apache.org/jira/browse/HBASE-15437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566722#comment-15566722
]
Jerry He commented on HBASE-15437:
--
In the old case, the reponseSize is raw result size. If we use the size after
encode/compression, it is a good change? [~anoop.hbase], [~enis]?
> Response size calculated in RPCServer for warning tooLarge responses does NOT
> count CellScanner payload
> ---
>
> Key: HBASE-15437
> URL: https://issues.apache.org/jira/browse/HBASE-15437
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC
>Reporter: deepankar
>Assignee: deepankar
> Attachments: HBASE-15437-v1.patch, HBASE-15437.patch
>
>
> After HBASE-13158 where we respond back to RPCs with cells in the payload ,
> the protobuf response will just have the count the cells to read from
> payload, but there are set of features where we log warn in RPCServer
> whenever the response is tooLarge, but this size now is not considering the
> sizes of the cells in the PayloadCellScanner. Code form RPCServer
> {code}
> long responseSize = result.getSerializedSize();
> // log any RPC responses that are slower than the configured warn
> // response time or larger than configured warning size
> boolean tooSlow = (processingTime > warnResponseTime &&
> warnResponseTime > -1);
> boolean tooLarge = (responseSize > warnResponseSize && warnResponseSize
> > -1);
> if (tooSlow || tooLarge) {
> // when tagging, we let TooLarge trump TooSmall to keep output simple
> // note that large responses will often also be slow.
> logResponse(new Object[]{param},
> md.getName(), md.getName() + "(" + param.getClass().getName() +
> ")",
> (tooLarge ? "TooLarge" : "TooSlow"),
> status.getClient(), startTime, processingTime, qTime,
> responseSize);
> }
> {code}
> Should this feature be not supported any more or should we add a method to
> CellScanner or a new interface which returns the serialized size (but this
> might not include the compression codecs which might be used during response
> ?) Any other Idea this could be fixed ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16808) Included the generated, shaded java files from "HBASE-15638 Shade protobuf" in our src assembly
[ https://issues.apache.org/jira/browse/HBASE-16808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16808: -- Status: Patch Available (was: Open) > Included the generated, shaded java files from "HBASE-15638 Shade protobuf" > in our src assembly > --- > > Key: HBASE-16808 > URL: https://issues.apache.org/jira/browse/HBASE-16808 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack >Priority: Blocker > Attachments: HBASE-16808.master.001.patch > > > [~Apache9] found that I forgot to include the generated, shaded files in our > src tgz. Let me fix. This is a follow-on to HBASE-16793 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16808) Included the generated, shaded java files from "HBASE-15638 Shade protobuf" in our src assembly
[ https://issues.apache.org/jira/browse/HBASE-16808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16808: -- Attachment: HBASE-16808.master.001.patch > Included the generated, shaded java files from "HBASE-15638 Shade protobuf" > in our src assembly > --- > > Key: HBASE-16808 > URL: https://issues.apache.org/jira/browse/HBASE-16808 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack >Priority: Blocker > Attachments: HBASE-16808.master.001.patch > > > [~Apache9] found that I forgot to include the generated, shaded files in our > src tgz. Let me fix. This is a follow-on to HBASE-16793 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16664) Timeout logic in AsyncProcess is broken
[ https://issues.apache.org/jira/browse/HBASE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566707#comment-15566707 ] stack commented on HBASE-16664: --- Patch LGTM. Needs a fat release note. Good by you [~chenheng]? > Timeout logic in AsyncProcess is broken > --- > > Key: HBASE-16664 > URL: https://issues.apache.org/jira/browse/HBASE-16664 > Project: HBase > Issue Type: Bug >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: 1.patch, HBASE-16664-branch-1-v1.patch, > HBASE-16664-branch-1-v1.patch, HBASE-16664-branch-1.1-v1.patch, > HBASE-16664-branch-1.2-v1.patch, HBASE-16664-branch-1.3-v1.patch, > HBASE-16664-v1.patch, HBASE-16664-v2.patch, HBASE-16664-v3.patch, > HBASE-16664-v4.patch, HBASE-16664-v5.patch, testhcm.patch > > > Have not checked the root cause, but I think timeout of all operations in > AsyncProcess is broken -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16146) Counters are expensive...
[ https://issues.apache.org/jira/browse/HBASE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566686#comment-15566686 ] stack commented on HBASE-16146: --- +1 > Counters are expensive... > - > > Key: HBASE-16146 > URL: https://issues.apache.org/jira/browse/HBASE-16146 > Project: HBase > Issue Type: Sub-task >Reporter: stack > Attachments: HBASE-16146.branch-1.3.001.patch, counters.patch, > less_and_less_counters.png > > > Doing workloadc, perf shows 10%+ of CPU being spent on counter#add. If I > disable some of the hot ones -- see patch -- I can get 10% more throughput > (390k to 440k). Figure something better. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
