[jira] [Commented] (HBASE-28508) Remove the need for ADMIN permissions for RSRpcServices#execRegionServerService
[
https://issues.apache.org/jira/browse/HBASE-28508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836720#comment-17836720
]
Rushabh Shah commented on HBASE-28508:
--
> For region server level CPs, I think requiring ADMIN permission is
> reasonable...
IMO it depends on the implementation of the region server CPs to decide whether
it is reasonable.
> Remove the need for ADMIN permissions for
> RSRpcServices#execRegionServerService
> ---
>
> Key: HBASE-28508
> URL: https://issues.apache.org/jira/browse/HBASE-28508
> Project: HBase
> Issue Type: Bug
> Components: acl
>Affects Versions: 2.4.17, 2.5.8
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Labels: pull-request-available
>
> We have introduced a new regionserver coproc within phoenix and all the
> permission related tests are failing with the following exception.
> {noformat}
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.security.AccessDeniedException):
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'groupUser_N42' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessChecker.requireGlobalPermission(AccessChecker.java:152)
> at
> org.apache.hadoop.hbase.security.access.AccessChecker.requirePermission(AccessChecker.java:125)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.requirePermission(RSRpcServices.java:1318)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.rpcPreCheck(RSRpcServices.java:584)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execRegionServerService(RSRpcServices.java:3804)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45016)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:415)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
> {noformat}
> This check is failing.
> [RSRpcServices|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L3815]
> {code}
> @Override
> public CoprocessorServiceResponse execRegionServerService(RpcController
> controller,
> CoprocessorServiceRequest request) throws ServiceException {
> rpcPreCheck("execRegionServerService");
> return server.execRegionServerService(controller, request);
> }
> private void rpcPreCheck(String requestName) throws ServiceException {
> try {
> checkOpen();
> requirePermission(requestName, Permission.Action.ADMIN);
> } catch (IOException ioe) {
> throw new ServiceException(ioe);
> }
> }
> {code}
> Why do we need ADMIN permissions to call region server coproc? We don't need
> ADMIN permissions to call all region co-procs. We require ADMIN permissions
> to execute some region coprocs (compactionSwitch, clearRegionBlockCache).
> Can we change the permission to READ?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28508) Remove the need for ADMIN permissions for RSRpcServices#execRegionServerService
[
https://issues.apache.org/jira/browse/HBASE-28508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836356#comment-17836356
]
Rushabh Shah commented on HBASE-28508:
--
> The problem is, lacking such a framework, we need to assume the maximum
> privilege possible.
Created HBASE-28515 for overriding access check per RS coproc endpoint.
> Remove the need for ADMIN permissions for
> RSRpcServices#execRegionServerService
> ---
>
> Key: HBASE-28508
> URL: https://issues.apache.org/jira/browse/HBASE-28508
> Project: HBase
> Issue Type: Bug
> Components: acl
>Affects Versions: 2.4.17, 2.5.8
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Labels: pull-request-available
>
> We have introduced a new regionserver coproc within phoenix and all the
> permission related tests are failing with the following exception.
> {noformat}
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.security.AccessDeniedException):
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'groupUser_N42' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessChecker.requireGlobalPermission(AccessChecker.java:152)
> at
> org.apache.hadoop.hbase.security.access.AccessChecker.requirePermission(AccessChecker.java:125)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.requirePermission(RSRpcServices.java:1318)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.rpcPreCheck(RSRpcServices.java:584)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execRegionServerService(RSRpcServices.java:3804)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45016)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:415)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
> {noformat}
> This check is failing.
> [RSRpcServices|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L3815]
> {code}
> @Override
> public CoprocessorServiceResponse execRegionServerService(RpcController
> controller,
> CoprocessorServiceRequest request) throws ServiceException {
> rpcPreCheck("execRegionServerService");
> return server.execRegionServerService(controller, request);
> }
> private void rpcPreCheck(String requestName) throws ServiceException {
> try {
> checkOpen();
> requirePermission(requestName, Permission.Action.ADMIN);
> } catch (IOException ioe) {
> throw new ServiceException(ioe);
> }
> }
> {code}
> Why do we need ADMIN permissions to call region server coproc? We don't need
> ADMIN permissions to call all region co-procs. We require ADMIN permissions
> to execute some region coprocs (compactionSwitch, clearRegionBlockCache).
> Can we change the permission to READ?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HBASE-28515) Validate access check for each RegionServerCoprocessor Endpoint.
Rushabh Shah created HBASE-28515: Summary: Validate access check for each RegionServerCoprocessor Endpoint. Key: HBASE-28515 URL: https://issues.apache.org/jira/browse/HBASE-28515 Project: HBase Issue Type: Improvement Components: Coprocessors Reporter: Rushabh Shah Currently we enforce ADMIN permissions for each regionserver coprocessor method. See HBASE-28508 for more details. There can be regionserver endpoint implementation which don't want to enforce calling users to have ADMIN permissions. So there needs to be a way for RS coproc implementation to override the access checks, with the default being requiring ADMIN permissions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28508) Remove the need for ADMIN permissions for RSRpcServices#execRegionServerService
[
https://issues.apache.org/jira/browse/HBASE-28508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835914#comment-17835914
]
Rushabh Shah commented on HBASE-28508:
--
> You might have a coprocessor which does writes, or even alters things about
> the cluster.
Not all the co-processors alters things about the cluster. It should be upto
the coproc implementation to enforce permissions. For the RS coproc
implementation in phoenix, we just validate timestamps of the table and make a
decision whether the cached table is stale or not. Also since this co proc
invocation happens on the client side, we have to enforce all the clients have
ADMIN permissions which might not be true in current deployments. In short for
custom coprocs, we need to leave upto implementation to enforce permissions.
[~bbeaudreault] [~vjasani]
> Remove the need for ADMIN permissions for
> RSRpcServices#execRegionServerService
> ---
>
> Key: HBASE-28508
> URL: https://issues.apache.org/jira/browse/HBASE-28508
> Project: HBase
> Issue Type: Bug
> Components: acl
>Affects Versions: 2.4.17, 2.5.8
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Labels: pull-request-available
>
> We have introduced a new regionserver coproc within phoenix and all the
> permission related tests are failing with the following exception.
> {noformat}
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.security.AccessDeniedException):
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'groupUser_N42' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessChecker.requireGlobalPermission(AccessChecker.java:152)
> at
> org.apache.hadoop.hbase.security.access.AccessChecker.requirePermission(AccessChecker.java:125)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.requirePermission(RSRpcServices.java:1318)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.rpcPreCheck(RSRpcServices.java:584)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execRegionServerService(RSRpcServices.java:3804)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45016)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:415)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
> {noformat}
> This check is failing.
> [RSRpcServices|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L3815]
> {code}
> @Override
> public CoprocessorServiceResponse execRegionServerService(RpcController
> controller,
> CoprocessorServiceRequest request) throws ServiceException {
> rpcPreCheck("execRegionServerService");
> return server.execRegionServerService(controller, request);
> }
> private void rpcPreCheck(String requestName) throws ServiceException {
> try {
> checkOpen();
> requirePermission(requestName, Permission.Action.ADMIN);
> } catch (IOException ioe) {
> throw new ServiceException(ioe);
> }
> }
> {code}
> Why do we need ADMIN permissions to call region server coproc? We don't need
> ADMIN permissions to call all region co-procs. We require ADMIN permissions
> to execute some region coprocs (compactionSwitch, clearRegionBlockCache).
> Can we change the permission to READ?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HBASE-28508) Remove the need for ADMIN permissions for RSRpcServices#execRegionServerService
[
https://issues.apache.org/jira/browse/HBASE-28508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh Shah updated HBASE-28508:
-
Affects Version/s: 2.5.8
2.4.17
> Remove the need for ADMIN permissions for
> RSRpcServices#execRegionServerService
> ---
>
> Key: HBASE-28508
> URL: https://issues.apache.org/jira/browse/HBASE-28508
> Project: HBase
> Issue Type: Bug
> Components: acl
>Affects Versions: 2.4.17, 2.5.8
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
>
> We have introduced a new regionserver coproc within phoenix and all the
> permission related tests are failing with the following exception.
> {noformat}
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.security.AccessDeniedException):
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'groupUser_N42' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessChecker.requireGlobalPermission(AccessChecker.java:152)
> at
> org.apache.hadoop.hbase.security.access.AccessChecker.requirePermission(AccessChecker.java:125)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.requirePermission(RSRpcServices.java:1318)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.rpcPreCheck(RSRpcServices.java:584)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execRegionServerService(RSRpcServices.java:3804)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45016)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:415)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
> {noformat}
> This check is failing.
> [RSRpcServices|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L3815]
> {code}
> @Override
> public CoprocessorServiceResponse execRegionServerService(RpcController
> controller,
> CoprocessorServiceRequest request) throws ServiceException {
> rpcPreCheck("execRegionServerService");
> return server.execRegionServerService(controller, request);
> }
> private void rpcPreCheck(String requestName) throws ServiceException {
> try {
> checkOpen();
> requirePermission(requestName, Permission.Action.ADMIN);
> } catch (IOException ioe) {
> throw new ServiceException(ioe);
> }
> }
> {code}
> Why do we need ADMIN permissions to call region server coproc? We don't need
> ADMIN permissions to call all region co-procs. We require ADMIN permissions
> to execute some region coprocs (compactionSwitch, clearRegionBlockCache).
> Can we change the permission to READ?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HBASE-28508) Remove the need for ADMIN permissions for RSRpcServices#execRegionServerService
Rushabh Shah created HBASE-28508:
Summary: Remove the need for ADMIN permissions for
RSRpcServices#execRegionServerService
Key: HBASE-28508
URL: https://issues.apache.org/jira/browse/HBASE-28508
Project: HBase
Issue Type: Bug
Components: acl
Reporter: Rushabh Shah
Assignee: Rushabh Shah
We have introduced a new regionserver coproc within phoenix and all the
permission related tests are failing with the following exception.
{noformat}
Caused by:
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.security.AccessDeniedException):
org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions for user 'groupUser_N42' (global, action=ADMIN)
at
org.apache.hadoop.hbase.security.access.AccessChecker.requireGlobalPermission(AccessChecker.java:152)
at
org.apache.hadoop.hbase.security.access.AccessChecker.requirePermission(AccessChecker.java:125)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.requirePermission(RSRpcServices.java:1318)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.rpcPreCheck(RSRpcServices.java:584)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.execRegionServerService(RSRpcServices.java:3804)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45016)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:415)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
{noformat}
This check is failing.
[RSRpcServices|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L3815]
{code}
@Override
public CoprocessorServiceResponse execRegionServerService(RpcController
controller,
CoprocessorServiceRequest request) throws ServiceException {
rpcPreCheck("execRegionServerService");
return server.execRegionServerService(controller, request);
}
private void rpcPreCheck(String requestName) throws ServiceException {
try {
checkOpen();
requirePermission(requestName, Permission.Action.ADMIN);
} catch (IOException ioe) {
throw new ServiceException(ioe);
}
}
{code}
Why do we need ADMIN permissions to call region server coproc? We don't need
ADMIN permissions to call all region co-procs. We require ADMIN permissions to
execute some region coprocs (compactionSwitch, clearRegionBlockCache).
Can we change the permission to READ?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28438) Add support spitting region into multiple regions(more than 2)
[ https://issues.apache.org/jira/browse/HBASE-28438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17827536#comment-17827536 ] Rushabh Shah commented on HBASE-28438: -- [~rajeshbabu] We had a similar discussion couple of months ago on hbase-dev mailing thread [here|https://lists.apache.org/thread/chhbfb2f1hlwnn89rll7pjyg8zkq5yl9]. Glad to see that you also need similar feature. > Add support spitting region into multiple regions(more than 2) > -- > > Key: HBASE-28438 > URL: https://issues.apache.org/jira/browse/HBASE-28438 > Project: HBase > Issue Type: Improvement >Reporter: Rajeshbabu Chintaguntla >Assignee: Rajeshbabu Chintaguntla >Priority: Major > > We have a requirement of splitting one region into multiple hundreds of > regions at a time to distribute loading hot data. To do that we need split a > region and wait for the completion of it and then again split the two regions > etc..which is time consuming activity. > Would be better to support splitting region into multiple regions more than > two so that in single operation we can split the region. > Todo that we need to take care > 1)Supporting admin APIs to take multiple split keys > 2)Implement new procedure to create new regions, creating meta entries and > udpating them to meta > 3) close the parent region and open split regions. > 4) Update the compaction of post split and readers also to use the portion > store file reader based on the range to scan than half store reader. > 5) make sure the catalog jonitor also cleaning the parent regions when there > are all the regions split properly. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28437) Region Server crash in our production environment.
[
https://issues.apache.org/jira/browse/HBASE-28437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17825468#comment-17825468
]
Rushabh Shah commented on HBASE-28437:
--
Just before the crash, saw the following logline from thread:
RS-EventLoopGroup-1-92
{noformat}
2024-03-08 16:50:35,401 WARN [RS-EventLoopGroup-1-92]
client.AsyncBatchRpcRetryingCaller - Process batch for [''] in
from failed, tries=11
org.apache.hadoop.hbase.RegionTooBusyException:
org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=1.0 G,
regionName=16226d6d057ca2d10b0377e6596ff291, server=
at
org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4969)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4438)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4369)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1033)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:951)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:916)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2892)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45961)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:415)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
at sun.reflect.GeneratedConstructorAccessor162.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toException(ProtobufUtil.java:1606)
at
org.apache.hadoop.hbase.shaded.protobuf.ResponseConverter.getResults(ResponseConverter.java:189)
at
org.apache.hadoop.hbase.client.AsyncBatchRpcRetryingCaller.lambda$sendToServer$11(AsyncBatchRpcRetryingCaller.java:410)
at
org.apache.hbase.thirdparty.com.google.protobuf.RpcUtil$1.run(RpcUtil.java:79)
at
org.apache.hbase.thirdparty.com.google.protobuf.RpcUtil$1.run(RpcUtil.java:70)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:396)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:92)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:425)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:420)
at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:114)
at org.apache.hadoop.hbase.ipc.Call.setResponse(Call.java:146)
at
org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.finishCall(NettyRpcDuplexHandler.java:157)
at
org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:201)
at
org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:218)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at
org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at
org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at
org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at
[jira] [Created] (HBASE-28437) Region Server crash in our production environment.
Rushabh Shah created HBASE-28437:
Summary: Region Server crash in our production environment.
Key: HBASE-28437
URL: https://issues.apache.org/jira/browse/HBASE-28437
Project: HBase
Issue Type: Bug
Reporter: Rushabh Shah
Recently we are seeing lot of RS crash in our production environment creating
core dump file and hs_err_pid.log file.
HBase: hbase-2.5
Java: openjdk 1.8
Copying contents from hs_err_pid.log below:
{noformat}
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x7f9fb1415ba2, pid=50172, tid=0x7f92a97ec700
#
# JRE version: OpenJDK Runtime Environment (Zulu 8.76.0.18-SA-linux64)
(8.0_402-b06) (build 1.8.0_402-b06)
# Java VM: OpenJDK 64-Bit Server VM (25.402-b06 mixed mode linux-amd64 )
# Problematic frame:
# J 19801 C2
org.apache.hadoop.hbase.util.ByteBufferUtils.copyBufferToStream(Ljava/io/OutputStream;Ljava/nio/ByteBuffer;II)V
(75 bytes) @ 0x7f9fb1415ba2 [0x7f9fb14159a0+0x202]
#
# Core dump written. Default location: /home/sfdc/core or core.50172
#
# If you would like to submit a bug report, please visit:
# http://www.azul.com/support/
#
--- T H R E A D ---
Current thread (0x7f9fa2d13000): JavaThread "RS-EventLoopGroup-1-92"
daemon [_thread_in_Java, id=54547, stack(0x7f92a96ec000,0x7f92a97ed000)]
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr:
0x559869daf000
Registers:
RAX=0x7f9dbd8b6460, RBX=0x0008, RCX=0x0005c86b,
RDX=0x7f9dbd8b6460
RSP=0x7f92a97eaf20, RBP=0x0002, RSI=0x7f92d225e970,
RDI=0x0069
R8 =0x55986975f028, R9 =0x0064ffd8, R10=0x005f,
R11=0x7f94a778b290
R12=0x7f9e62855ae8, R13=0x, R14=0x7f9e5a14b1e0,
R15=0x7f9fa2d13000
RIP=0x7f9fb1415ba2, EFLAGS=0x00010216, CSGSFS=0x0033,
ERR=0x0004
TRAPNO=0x000e
Top of Stack: (sp=0x7f92a97eaf20)
0x7f92a97eaf20: 00690064ff79 7f9dbd8b6460
0x7f92a97eaf30: 7f9dbd8b6460 00570003
0x7f92a97eaf40: 7f94a778b290 000400010004
0x7f92a97eaf50: 0004d090c130 7f9db550
0x7f92a97eaf60: 000800040001 7f92a97eaf90
0x7f92a97eaf70: 7f92d0908648 0001
0x7f92a97eaf80: 0001 005c
0x7f92a97eaf90: 7f94ee8078d0 0206
0x7f92a97eafa0: 7f9db5545a00 7f9fafb63670
0x7f92a97eafb0: 7f9e5a13ed70 00690001
0x7f92a97eafc0: 7f93ab8965b8 7f93b9959210
0x7f92a97eafd0: 7f9db5545a00 7f9fb04b3e30
0x7f92a97eafe0: 7f9e5a13ed70 7f930001
0x7f92a97eaff0: 7f93ab8965b8 7f93a8ae3920
0x7f92a97eb000: 7f93b9959210 7f94a778b290
0x7f92a97eb010: 7f9b60707c20 7f93a8938c28
0x7f92a97eb020: 7f94ee8078d0 7f9b60708608
0x7f92a97eb030: 7f9b60707bc0 7f9b60707c20
0x7f92a97eb040: 0069 7f93ab8965b8
0x7f92a97eb050: 7f94a778b290 7f94a778b290
0x7f92a97eb060: 0005c80d0005c80c a828a590
0x7f92a97eb070: 7f9e5a13ed70 0001270e
0x7f92a97eb080: 7f9db5545790 01440022
0x7f92a97eb090: 7f95ddc800c0 7f93ab89a6c8
0x7f92a97eb0a0: 7f93ae65c270 7f9fb24af990
0x7f92a97eb0b0: 7f93ae65c290 7f93ae65c270
0x7f92a97eb0c0: 7f9e5a13ed70 7f92ca328528
0x7f92a97eb0d0: 7f9e5a13ed98 7f9e5e1e88b0
0x7f92a97eb0e0: 7f92ca32d870 7f9e5a13ed98
0x7f92a97eb0f0: 7f9e5e1e88b0 7f93b9956288
0x7f92a97eb100: 7f9e5a13ed70 7f9fb23c3aac
0x7f92a97eb110: 7f9317c9c8d0 7f9b60708608
Instructions: (pc=0x7f9fb1415ba2)
0x7f9fb1415b82: 44 3b d7 0f 8d 6d fe ff ff 4c 8b 40 10 45 8b ca
0x7f9fb1415b92: 44 03 0c 24 c4 c1 f9 7e c3 4d 8b 5b 18 4d 63 c9
0x7f9fb1415ba2: 47 0f be 04 08 4d 85 db 0f 84 49 03 00 00 4d 8b
0x7f9fb1415bb2: 4b 08 48 b9 10 1c be 10 93 7f 00 00 4c 3b c9 0f
Register to memory mapping:
RAX=0x7f9dbd8b6460 is an oop
java.nio.DirectByteBuffer
- klass: 'java/nio/DirectByteBuffer'
RBX=0x0008 is an unknown value
RCX=0x0005c86b is an unknown value
RDX=0x7f9dbd8b6460 is an oop
java.nio.DirectByteBuffer
- klass: 'java/nio/DirectByteBuffer'
RSP=0x7f92a97eaf20 is pointing into the stack for thread: 0x7f9fa2d13000
RBP=0x0002 is an unknown value
RSI=0x7f92d225e970 is pointing into metadata
RDI=0x0069 is an unknown value
R8 =0x55986975f028 is an unknown value
R9 =0x0064ffd8 is an unknown value
R10=0x005f is an unknown value
R11=0x7f94a778b290 is an oop
org.apache.hbase.thirdparty.io.netty.buffer.PooledUnsafeDirectByteBuf
- klass:
[jira] [Commented] (HBASE-28391) Remove the need for ADMIN permissions for listDecommissionedRegionServers
[
https://issues.apache.org/jira/browse/HBASE-28391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821400#comment-17821400
]
Rushabh Shah commented on HBASE-28391:
--
Thank you [~nihaljain.cs] and [~zhangduo] for the review. Merged the patch to
all active branches: trunk, branch-3, branch-2, branch-2.6, branch-2.5 and
branch-2.4
> Remove the need for ADMIN permissions for listDecommissionedRegionServers
> -
>
> Key: HBASE-28391
> URL: https://issues.apache.org/jira/browse/HBASE-28391
> Project: HBase
> Issue Type: Bug
> Components: Admin
>Affects Versions: 2.4.17, 2.5.7
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 2.4.18, 4.0.0-alpha-1, 2.7.0, 2.5.8, 3.0.0-beta-2
>
>
> Why we need {{ADMIN}} permissions for
> {{AccessController#preListDecommissionedRegionServers}} ?
> From Phoenix, we are calling {{Admin#getRegionServers(true)}} where the
> argument {{excludeDecommissionedRS}} is set to true. Refer
> [here|https://github.com/apache/hbase/blob/branch-2.5/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java#L1721-L1730].
> If {{excludeDecommissionedRS}} is set to true and if we have
> {{AccessController}} co-proc attached, it requires ADMIN permissions to
> execute {{listDecommissionedRegionServers}} RPC. Refer
> [here|https://github.com/apache/hbase/blob/branch-2.5/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java#L1205-L1207].
>
> {code:java}
> @Override
> public void
> preListDecommissionedRegionServers(ObserverContext
> ctx)
> throws IOException {
> requirePermission(ctx, "listDecommissionedRegionServers", Action.ADMIN);
> }
> {code}
> I understand that we need ADMIN permissions for
> _preDecommissionRegionServers_ and _preRecommissionRegionServer_ because it
> changes the membership of regionservers but I don’t see any need for ADMIN
> permissions for _listDecommissionedRegionServers_. Do you think we can
> remove need for ADMIN permissions for _listDecommissionedRegionServers_ RPC?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-28391) Remove the need for ADMIN permissions for listDecommissionedRegionServers
[
https://issues.apache.org/jira/browse/HBASE-28391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh Shah resolved HBASE-28391.
--
Fix Version/s: 2.6.0
2.4.18
4.0.0-alpha-1
2.7.0
2.5.8
3.0.0-beta-2
Resolution: Fixed
> Remove the need for ADMIN permissions for listDecommissionedRegionServers
> -
>
> Key: HBASE-28391
> URL: https://issues.apache.org/jira/browse/HBASE-28391
> Project: HBase
> Issue Type: Bug
> Components: Admin
>Affects Versions: 2.4.17, 2.5.7
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 2.4.18, 4.0.0-alpha-1, 2.7.0, 2.5.8, 3.0.0-beta-2
>
>
> Why we need {{ADMIN}} permissions for
> {{AccessController#preListDecommissionedRegionServers}} ?
> From Phoenix, we are calling {{Admin#getRegionServers(true)}} where the
> argument {{excludeDecommissionedRS}} is set to true. Refer
> [here|https://github.com/apache/hbase/blob/branch-2.5/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java#L1721-L1730].
> If {{excludeDecommissionedRS}} is set to true and if we have
> {{AccessController}} co-proc attached, it requires ADMIN permissions to
> execute {{listDecommissionedRegionServers}} RPC. Refer
> [here|https://github.com/apache/hbase/blob/branch-2.5/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java#L1205-L1207].
>
> {code:java}
> @Override
> public void
> preListDecommissionedRegionServers(ObserverContext
> ctx)
> throws IOException {
> requirePermission(ctx, "listDecommissionedRegionServers", Action.ADMIN);
> }
> {code}
> I understand that we need ADMIN permissions for
> _preDecommissionRegionServers_ and _preRecommissionRegionServer_ because it
> changes the membership of regionservers but I don’t see any need for ADMIN
> permissions for _listDecommissionedRegionServers_. Do you think we can
> remove need for ADMIN permissions for _listDecommissionedRegionServers_ RPC?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (HBASE-28391) Remove the need for ADMIN permissions for listDecommissionedRegionServers
[
https://issues.apache.org/jira/browse/HBASE-28391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820958#comment-17820958
]
Rushabh Shah edited comment on HBASE-28391 at 2/27/24 4:59 AM:
---
> Better start a discussion thread on dev list about this, if no objections,
>let's change it to Action.READ
Thank you [~zhangduo] for the reply. Created discussion thread
[here|https://lists.apache.org/thread/vcf50plmsx59yh4fyvsfpo7xht4rbhz8]
was (Author: shahrs87):
> Better start a discussion thread on dev list about this, if no objections,
>let's change it to Action.READ
Thank you [~zhangduo] for the reply. Created discussion thread [here|]
> Remove the need for ADMIN permissions for listDecommissionedRegionServers
> -
>
> Key: HBASE-28391
> URL: https://issues.apache.org/jira/browse/HBASE-28391
> Project: HBase
> Issue Type: Bug
> Components: Admin
>Affects Versions: 2.4.17, 2.5.7
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Labels: pull-request-available
>
> Why we need {{ADMIN}} permissions for
> {{AccessController#preListDecommissionedRegionServers}} ?
> From Phoenix, we are calling {{Admin#getRegionServers(true)}} where the
> argument {{excludeDecommissionedRS}} is set to true. Refer
> [here|https://github.com/apache/hbase/blob/branch-2.5/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java#L1721-L1730].
> If {{excludeDecommissionedRS}} is set to true and if we have
> {{AccessController}} co-proc attached, it requires ADMIN permissions to
> execute {{listDecommissionedRegionServers}} RPC. Refer
> [here|https://github.com/apache/hbase/blob/branch-2.5/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java#L1205-L1207].
>
> {code:java}
> @Override
> public void
> preListDecommissionedRegionServers(ObserverContext
> ctx)
> throws IOException {
> requirePermission(ctx, "listDecommissionedRegionServers", Action.ADMIN);
> }
> {code}
> I understand that we need ADMIN permissions for
> _preDecommissionRegionServers_ and _preRecommissionRegionServer_ because it
> changes the membership of regionservers but I don’t see any need for ADMIN
> permissions for _listDecommissionedRegionServers_. Do you think we can
> remove need for ADMIN permissions for _listDecommissionedRegionServers_ RPC?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28391) Remove the need for ADMIN permissions for listDecommissionedRegionServers
[
https://issues.apache.org/jira/browse/HBASE-28391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820958#comment-17820958
]
Rushabh Shah commented on HBASE-28391:
--
> Better start a discussion thread on dev list about this, if no objections,
>let's change it to Action.READ
Thank you [~zhangduo] for the reply. Created discussion thread [here|]
> Remove the need for ADMIN permissions for listDecommissionedRegionServers
> -
>
> Key: HBASE-28391
> URL: https://issues.apache.org/jira/browse/HBASE-28391
> Project: HBase
> Issue Type: Bug
> Components: Admin
>Affects Versions: 2.4.17, 2.5.7
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Labels: pull-request-available
>
> Why we need {{ADMIN}} permissions for
> {{AccessController#preListDecommissionedRegionServers}} ?
> From Phoenix, we are calling {{Admin#getRegionServers(true)}} where the
> argument {{excludeDecommissionedRS}} is set to true. Refer
> [here|https://github.com/apache/hbase/blob/branch-2.5/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java#L1721-L1730].
> If {{excludeDecommissionedRS}} is set to true and if we have
> {{AccessController}} co-proc attached, it requires ADMIN permissions to
> execute {{listDecommissionedRegionServers}} RPC. Refer
> [here|https://github.com/apache/hbase/blob/branch-2.5/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java#L1205-L1207].
>
> {code:java}
> @Override
> public void
> preListDecommissionedRegionServers(ObserverContext
> ctx)
> throws IOException {
> requirePermission(ctx, "listDecommissionedRegionServers", Action.ADMIN);
> }
> {code}
> I understand that we need ADMIN permissions for
> _preDecommissionRegionServers_ and _preRecommissionRegionServer_ because it
> changes the membership of regionservers but I don’t see any need for ADMIN
> permissions for _listDecommissionedRegionServers_. Do you think we can
> remove need for ADMIN permissions for _listDecommissionedRegionServers_ RPC?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HBASE-28391) Remove the need for ADMIN permissions for listDecommissionedRegionServers
Rushabh Shah created HBASE-28391:
Summary: Remove the need for ADMIN permissions for
listDecommissionedRegionServers
Key: HBASE-28391
URL: https://issues.apache.org/jira/browse/HBASE-28391
Project: HBase
Issue Type: Bug
Components: Admin
Affects Versions: 2.5.7, 2.4.17
Reporter: Rushabh Shah
Assignee: Rushabh Shah
Why we need {{ADMIN}} permissions for
{{AccessController#preListDecommissionedRegionServers}} ?
>From Phoenix, we are calling {{Admin#getRegionServers(true)}} where the
>argument {{excludeDecommissionedRS}} is set to true. Refer
>[here|https://github.com/apache/hbase/blob/branch-2.5/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java#L1721-L1730].
If {{excludeDecommissionedRS}} is set to true and if we have
{{AccessController}} co-proc attached, it requires ADMIN permissions to execute
{{listDecommissionedRegionServers}} RPC. Refer
[here|https://github.com/apache/hbase/blob/branch-2.5/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java#L1205-L1207].
{code:java}
@Override
public void
preListDecommissionedRegionServers(ObserverContext
ctx)
throws IOException {
requirePermission(ctx, "listDecommissionedRegionServers", Action.ADMIN);
}
{code}
I understand that we need ADMIN permissions for _preDecommissionRegionServers_
and _preRecommissionRegionServer_ because it changes the membership of
regionservers but I don’t see any need for ADMIN permissions for
_listDecommissionedRegionServers_. Do you think we can remove need for ADMIN
permissions for _listDecommissionedRegionServers_ RPC?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HBASE-28293) Add metric for GetClusterStatus request count.
Rushabh Shah created HBASE-28293: Summary: Add metric for GetClusterStatus request count. Key: HBASE-28293 URL: https://issues.apache.org/jira/browse/HBASE-28293 Project: HBase Issue Type: Bug Reporter: Rushabh Shah We have been bitten multiple times by GetClusterStatus request overwhelming HMaster's memory usage. It would be good to add a metric for the total GetClusterStatus requests count. In almost all of our production incidents involving GetClusterStatus request, HMaster were running out of memory with many clients call this RPC in parallel and the response size is very big. In hbase2 we have [ClusterMetrics.Option|https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/ClusterMetrics.java#L164-L224] which can reduce the size of the response. It would be nice to add another metric to indicate if the response size of GetClusterStatus is greater than some threshold (like 5MB) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28271) Infinite waiting on lock acquisition by snapshot can result in unresponsive master
[ https://issues.apache.org/jira/browse/HBASE-28271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798655#comment-17798655 ] Rushabh Shah commented on HBASE-28271: -- Isn't the problem that we don't have timeout for acquiring master lock. Snapshot creation is just 1 caller of this lock. Maybe have some timeout on acquiring master lock so that all callers can benefit from this? Also lets not couple rpc timeout with this timeout. We can create a new config for this, if not already present. > Infinite waiting on lock acquisition by snapshot can result in unresponsive > master > -- > > Key: HBASE-28271 > URL: https://issues.apache.org/jira/browse/HBASE-28271 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0-alpha-4, 2.4.17, 2.5.7 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Attachments: image.png > > > When a region is stuck in transition for significant time, any attempt to > take snapshot on the table would keep master handler thread in forever > waiting state. As part of the creating snapshot on enabled or disabled table, > in order to get the table level lock, LockProcedure is executed but if any > region of the table is in transition, LockProcedure could not be executed by > the snapshot handler, resulting in forever waiting until the region > transition is completed, allowing the table level lock to be acquired by the > snapshot handler. > In cases where a region stays in RIT for considerable time, if enough > attempts are made by the client to create snapshots on the table, it can > easily exhaust all handler threads, leading to potentially unresponsive > master. Attached a sample thread dump. > Proposal: The snapshot handler should not stay stuck forever if it cannot > take table level lock, it should fail-fast. > !image.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28204) Canary can take lot more time If any region (except the first region) starts with delete markers
[ https://issues.apache.org/jira/browse/HBASE-28204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792193#comment-17792193 ] Rushabh Shah commented on HBASE-28204: -- > FYI Bryan Beaudreault, we have seen a bit of perf regression so need to > revert the commit. Just wanted to keep you in loop in case you have started > preparing RC0 already. [~vjasani] Can you share more details where you saw this regression and/or if there is any thread I can follow? > Canary can take lot more time If any region (except the first region) starts > with delete markers > > > Key: HBASE-28204 > URL: https://issues.apache.org/jira/browse/HBASE-28204 > Project: HBase > Issue Type: Bug > Components: canary >Reporter: Mihir Monani >Assignee: Mihir Monani >Priority: Major > Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.7, 2.7.0 > > > In CanaryTool.java, Canary reads only the first row of the region using > [Get|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L520C33-L520C33] > for any region of the table. Canary uses [Scan with FirstRowKeyFilter for > table > scan|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L530] > If the said region has empty start key (This will only happen when region is > the first region for a table) > With -[HBASE-16091|https://issues.apache.org/jira/browse/HBASE-16091]- > RawScan was > [implemented|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L519-L534] > to improve performance for regions which can have high number of delete > markers. Based on currently implementation, [RawScan is only > enabled|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L519] > if region has empty start-key (or region is first region for the table). > RawScan doesn't work for rest of the regions in the table except first > region. Also If the region has all the rows or majority of the rows with > delete markers, Get Operation can take a lot of time. This is can cause > timeouts for CanaryTool. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28204) Canary can take lot more time If any region (except the first region) starts with delete markers
[ https://issues.apache.org/jira/browse/HBASE-28204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786513#comment-17786513 ] Rushabh Shah commented on HBASE-28204: -- Thank you [~mihir6692] for the PR. Committed to branch-2.4, branch-2.5, branch-2, branch-3 and master. > Canary can take lot more time If any region (except the first region) starts > with delete markers > > > Key: HBASE-28204 > URL: https://issues.apache.org/jira/browse/HBASE-28204 > Project: HBase > Issue Type: Bug > Components: canary >Reporter: Mihir Monani >Assignee: Mihir Monani >Priority: Major > Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.7 > > > In CanaryTool.java, Canary reads only the first row of the region using > [Get|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L520C33-L520C33] > for any region of the table. Canary uses [Scan with FirstRowKeyFilter for > table > scan|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L530] > If the said region has empty start key (This will only happen when region is > the first region for a table) > With -[HBASE-16091|https://issues.apache.org/jira/browse/HBASE-16091]- > RawScan was > [implemented|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L519-L534] > to improve performance for regions which can have high number of delete > markers. Based on currently implementation, [RawScan is only > enabled|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L519] > if region has empty start-key (or region is first region for the table). > RawScan doesn't work for rest of the regions in the table except first > region. Also If the region has all the rows or majority of the rows with > delete markers, Get Operation can take a lot of time. This is can cause > timeouts for CanaryTool. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-28204) Canary can take lot more time If any region (except the first region) starts with delete markers
[ https://issues.apache.org/jira/browse/HBASE-28204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh Shah resolved HBASE-28204. -- Fix Version/s: 2.6.0 2.4.18 3.0.0-beta-1 2.5.7 Resolution: Fixed > Canary can take lot more time If any region (except the first region) starts > with delete markers > > > Key: HBASE-28204 > URL: https://issues.apache.org/jira/browse/HBASE-28204 > Project: HBase > Issue Type: Bug > Components: canary >Reporter: Mihir Monani >Assignee: Mihir Monani >Priority: Major > Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.7 > > > In CanaryTool.java, Canary reads only the first row of the region using > [Get|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L520C33-L520C33] > for any region of the table. Canary uses [Scan with FirstRowKeyFilter for > table > scan|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L530] > If the said region has empty start key (This will only happen when region is > the first region for a table) > With -[HBASE-16091|https://issues.apache.org/jira/browse/HBASE-16091]- > RawScan was > [implemented|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L519-L534] > to improve performance for regions which can have high number of delete > markers. Based on currently implementation, [RawScan is only > enabled|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L519] > if region has empty start-key (or region is first region for the table). > RawScan doesn't work for rest of the regions in the table except first > region. Also If the region has all the rows or majority of the rows with > delete markers, Get Operation can take a lot of time. This is can cause > timeouts for CanaryTool. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28184) Tailing the WAL is very slow if there are multiple peers.
[
https://issues.apache.org/jira/browse/HBASE-28184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784216#comment-17784216
]
Rushabh Shah commented on HBASE-28184:
--
There is one test failure
TestBasicWALEntryStreamFSHLog#testCleanClosedWALs in branch-2.5 (and in
branch-2.4 also I assume). Don' know why I didn't see in the jenkins build
report for branch-2.5 PR.
It fails with this error:
{noformat}
java.lang.AssertionError:
Expected :0
Actual :1
{noformat}
I can see the following exception:
{noformat}
2023-11-08T13:50:40,578 DEBUG [Listener at localhost/50204]
wal.ProtobufLogReader(442): EOF at position 210
2023-11-08T13:50:40,791 DEBUG [Listener at localhost/50204]
wal.ProtobufLogReader(447): Encountered a malformed edit, seeking back to last
good position in file, from 218 to 210
java.io.EOFException: Invalid PB, EOF? Ignoring; originalPosition=210,
currentPosition=218
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:376)
~[classes/:?]
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:104)
~[classes/:?]
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:92)
~[classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:259)
~[classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:181)
~[classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:102)
~[classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.peek(WALEntryStream.java:111)
~[classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.next(WALEntryStream.java:118)
~[classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStreamTestBase$WALEntryStreamWithRetries.access$001(WALEntryStreamTestBase.java:82)
~[test-classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStreamTestBase$WALEntryStreamWithRetries.lambda$next$0(WALEntryStreamTestBase.java:95)
~[test-classes/:?]
at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:184)
~[test-classes/:?]
at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:135)
~[test-classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStreamTestBase$WALEntryStreamWithRetries.next(WALEntryStreamTestBase.java:94)
~[test-classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.TestBasicWALEntryStream.testCleanClosedWALs(TestBasicWALEntryStream.java:726)
~[test-classes/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_292]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_292]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_292]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_292]
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
~[junit-4.13.2.jar:4.13.2]
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
~[junit-4.13.2.jar:4.13.2]
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
~[junit-4.13.2.jar:4.13.2]
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
~[junit-4.13.2.jar:4.13.2]
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
~[junit-4.13.2.jar:4.13.2]
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
~[junit-4.13.2.jar:4.13.2]
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
~[junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
~[junit-4.13.2.jar:4.13.2]
at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
~[junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
~[junit-4.13.2.jar:4.13.2]
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
~[junit-4.13.2.jar:4.13.2]
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
~[junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
~[junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
~[junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
~[junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
~[junit-4.13.2.jar:4.13.2]
[jira] [Resolved] (HBASE-28184) Tailing the WAL is very slow if there are multiple peers.
[
https://issues.apache.org/jira/browse/HBASE-28184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh Shah resolved HBASE-28184.
--
Resolution: Fixed
> Tailing the WAL is very slow if there are multiple peers.
> -
>
> Key: HBASE-28184
> URL: https://issues.apache.org/jira/browse/HBASE-28184
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.7
>
>
> Noticed in one of our production clusters which has 4 peers.
> Due to sudden ingestion of data, the size of log queue increased to a peak of
> 506. We have configured log roll size to 256 MB. Most of the edits in the WAL
> were from a table for which replication is disabled.
> So all ReplicationSourceWALReader thread had to do was to replay the WAL and
> NOT replicate them. Still it took 12 hours to drain the queue.
> Took few jstacks and found that ReplicationSourceWALReader was waiting to
> acquire rollWriterLock
> [here|https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java#L1231]
> {noformat}
> "regionserver/,1" #1036 daemon prio=5 os_prio=0 tid=0x7f44b374e800
> nid=0xbd7f waiting on condition [0x7f37b4d19000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x7f3897a3e150> (a
> java.util.concurrent.locks.ReentrantLock$FairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:837)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:872)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1202)
> at
> java.util.concurrent.locks.ReentrantLock$FairSync.lock(ReentrantLock.java:228)
> at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.getLogFileSizeIfBeingWritten(AbstractFSWAL.java:1102)
> at
> org.apache.hadoop.hbase.wal.WALProvider.lambda$null$0(WALProvider.java:128)
> at
> org.apache.hadoop.hbase.wal.WALProvider$$Lambda$177/1119730685.apply(Unknown
> Source)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1361)
> at
> java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
> at
> java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499)
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at
> java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.findAny(ReferencePipeline.java:536)
> at
> org.apache.hadoop.hbase.wal.WALProvider.lambda$getWALFileLengthProvider$2(WALProvider.java:129)
> at
> org.apache.hadoop.hbase.wal.WALProvider$$Lambda$140/1246380717.getLogFileSizeIfBeingWritten(Unknown
> Source)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:260)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:172)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:101)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:222)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:157)
> {noformat}
> All the peers will contend for this lock during every batch read.
> Look at the code snippet below. We are guarding this section with
> rollWriterLock if we are replicating the active WAL file. But in our case we
> are NOT replicating active WAL file but still we acquire this lock only to
> return OptionalLong.empty();
> {noformat}
> /**
>* if the given {@code path} is being written currently, then return its
> length.
>*
>* This is used by replication to prevent replicating unacked log entries.
> See
>*
[jira] [Commented] (HBASE-28184) Tailing the WAL is very slow if there are multiple peers.
[
https://issues.apache.org/jira/browse/HBASE-28184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17783777#comment-17783777
]
Rushabh Shah commented on HBASE-28184:
--
Committed the change to branch-2.4. branch-2.5, branch-2, branch-3 and master.
Thank you [~zhangduo] [~vjasani] [~Xiaolin Ha] for the reviews!
> Tailing the WAL is very slow if there are multiple peers.
> -
>
> Key: HBASE-28184
> URL: https://issues.apache.org/jira/browse/HBASE-28184
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.7
>
>
> Noticed in one of our production clusters which has 4 peers.
> Due to sudden ingestion of data, the size of log queue increased to a peak of
> 506. We have configured log roll size to 256 MB. Most of the edits in the WAL
> were from a table for which replication is disabled.
> So all ReplicationSourceWALReader thread had to do was to replay the WAL and
> NOT replicate them. Still it took 12 hours to drain the queue.
> Took few jstacks and found that ReplicationSourceWALReader was waiting to
> acquire rollWriterLock
> [here|https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java#L1231]
> {noformat}
> "regionserver/,1" #1036 daemon prio=5 os_prio=0 tid=0x7f44b374e800
> nid=0xbd7f waiting on condition [0x7f37b4d19000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x7f3897a3e150> (a
> java.util.concurrent.locks.ReentrantLock$FairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:837)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:872)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1202)
> at
> java.util.concurrent.locks.ReentrantLock$FairSync.lock(ReentrantLock.java:228)
> at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.getLogFileSizeIfBeingWritten(AbstractFSWAL.java:1102)
> at
> org.apache.hadoop.hbase.wal.WALProvider.lambda$null$0(WALProvider.java:128)
> at
> org.apache.hadoop.hbase.wal.WALProvider$$Lambda$177/1119730685.apply(Unknown
> Source)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1361)
> at
> java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
> at
> java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499)
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at
> java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.findAny(ReferencePipeline.java:536)
> at
> org.apache.hadoop.hbase.wal.WALProvider.lambda$getWALFileLengthProvider$2(WALProvider.java:129)
> at
> org.apache.hadoop.hbase.wal.WALProvider$$Lambda$140/1246380717.getLogFileSizeIfBeingWritten(Unknown
> Source)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:260)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:172)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:101)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:222)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:157)
> {noformat}
> All the peers will contend for this lock during every batch read.
> Look at the code snippet below. We are guarding this section with
> rollWriterLock if we are replicating the active WAL file. But in our case we
> are NOT replicating active WAL file but still we acquire this lock only to
> return OptionalLong.empty();
> {noformat}
> /**
>* if the given {@code path} is being written currently,
[jira] [Updated] (HBASE-28184) Tailing the WAL is very slow if there are multiple peers.
[
https://issues.apache.org/jira/browse/HBASE-28184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh Shah updated HBASE-28184:
-
Fix Version/s: 2.6.0
2.4.18
3.0.0-beta-1
2.5.7
> Tailing the WAL is very slow if there are multiple peers.
> -
>
> Key: HBASE-28184
> URL: https://issues.apache.org/jira/browse/HBASE-28184
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.7
>
>
> Noticed in one of our production clusters which has 4 peers.
> Due to sudden ingestion of data, the size of log queue increased to a peak of
> 506. We have configured log roll size to 256 MB. Most of the edits in the WAL
> were from a table for which replication is disabled.
> So all ReplicationSourceWALReader thread had to do was to replay the WAL and
> NOT replicate them. Still it took 12 hours to drain the queue.
> Took few jstacks and found that ReplicationSourceWALReader was waiting to
> acquire rollWriterLock
> [here|https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java#L1231]
> {noformat}
> "regionserver/,1" #1036 daemon prio=5 os_prio=0 tid=0x7f44b374e800
> nid=0xbd7f waiting on condition [0x7f37b4d19000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x7f3897a3e150> (a
> java.util.concurrent.locks.ReentrantLock$FairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:837)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:872)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1202)
> at
> java.util.concurrent.locks.ReentrantLock$FairSync.lock(ReentrantLock.java:228)
> at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.getLogFileSizeIfBeingWritten(AbstractFSWAL.java:1102)
> at
> org.apache.hadoop.hbase.wal.WALProvider.lambda$null$0(WALProvider.java:128)
> at
> org.apache.hadoop.hbase.wal.WALProvider$$Lambda$177/1119730685.apply(Unknown
> Source)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1361)
> at
> java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
> at
> java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499)
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at
> java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.findAny(ReferencePipeline.java:536)
> at
> org.apache.hadoop.hbase.wal.WALProvider.lambda$getWALFileLengthProvider$2(WALProvider.java:129)
> at
> org.apache.hadoop.hbase.wal.WALProvider$$Lambda$140/1246380717.getLogFileSizeIfBeingWritten(Unknown
> Source)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:260)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:172)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:101)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:222)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:157)
> {noformat}
> All the peers will contend for this lock during every batch read.
> Look at the code snippet below. We are guarding this section with
> rollWriterLock if we are replicating the active WAL file. But in our case we
> are NOT replicating active WAL file but still we acquire this lock only to
> return OptionalLong.empty();
> {noformat}
> /**
>* if the given {@code path} is being written currently, then return its
> length.
>*
>* This is used by replication to prevent
[jira] [Comment Edited] (HBASE-28184) Tailing the WAL is very slow if there are multiple peers.
[
https://issues.apache.org/jira/browse/HBASE-28184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17782690#comment-17782690
]
Rushabh Shah edited comment on HBASE-28184 at 11/3/23 6:11 PM:
---
Thank you [~zhangduo] for your reply.
I can see disk busy at 100% for first 6-7 hours but don't see any read/write
load for the next 5 hours.
Maybe once we remove acquiring rollWriter lock, we will see where it is
actually waiting?
was (Author: shahrs87):
Thank you [~zhangduo] for your reply.
I can see disk busy at 100% for first 6-7 hours but don't see any read/write
load for the next 5 hours.
> Tailing the WAL is very slow if there are multiple peers.
> -
>
> Key: HBASE-28184
> URL: https://issues.apache.org/jira/browse/HBASE-28184
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
>
> Noticed in one of our production clusters which has 4 peers.
> Due to sudden ingestion of data, the size of log queue increased to a peak of
> 506. We have configured log roll size to 256 MB. Most of the edits in the WAL
> were from a table for which replication is disabled.
> So all ReplicationSourceWALReader thread had to do was to replay the WAL and
> NOT replicate them. Still it took 12 hours to drain the queue.
> Took few jstacks and found that ReplicationSourceWALReader was waiting to
> acquire rollWriterLock
> [here|https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java#L1231]
> {noformat}
> "regionserver/,1" #1036 daemon prio=5 os_prio=0 tid=0x7f44b374e800
> nid=0xbd7f waiting on condition [0x7f37b4d19000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x7f3897a3e150> (a
> java.util.concurrent.locks.ReentrantLock$FairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:837)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:872)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1202)
> at
> java.util.concurrent.locks.ReentrantLock$FairSync.lock(ReentrantLock.java:228)
> at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.getLogFileSizeIfBeingWritten(AbstractFSWAL.java:1102)
> at
> org.apache.hadoop.hbase.wal.WALProvider.lambda$null$0(WALProvider.java:128)
> at
> org.apache.hadoop.hbase.wal.WALProvider$$Lambda$177/1119730685.apply(Unknown
> Source)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1361)
> at
> java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
> at
> java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499)
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at
> java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.findAny(ReferencePipeline.java:536)
> at
> org.apache.hadoop.hbase.wal.WALProvider.lambda$getWALFileLengthProvider$2(WALProvider.java:129)
> at
> org.apache.hadoop.hbase.wal.WALProvider$$Lambda$140/1246380717.getLogFileSizeIfBeingWritten(Unknown
> Source)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:260)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:172)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:101)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:222)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:157)
> {noformat}
> All the peers will contend for this lock during every batch read.
> Look at the code snippet below. We are guarding this section with
> rollWriterLock if
[jira] [Commented] (HBASE-28184) Tailing the WAL is very slow if there are multiple peers.
[
https://issues.apache.org/jira/browse/HBASE-28184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17782690#comment-17782690
]
Rushabh Shah commented on HBASE-28184:
--
Thank you [~zhangduo] for your reply.
I can see disk busy at 100% for first 6-7 hours but don't see any read/write
load for the next 5 hours.
> Tailing the WAL is very slow if there are multiple peers.
> -
>
> Key: HBASE-28184
> URL: https://issues.apache.org/jira/browse/HBASE-28184
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
>
> Noticed in one of our production clusters which has 4 peers.
> Due to sudden ingestion of data, the size of log queue increased to a peak of
> 506. We have configured log roll size to 256 MB. Most of the edits in the WAL
> were from a table for which replication is disabled.
> So all ReplicationSourceWALReader thread had to do was to replay the WAL and
> NOT replicate them. Still it took 12 hours to drain the queue.
> Took few jstacks and found that ReplicationSourceWALReader was waiting to
> acquire rollWriterLock
> [here|https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java#L1231]
> {noformat}
> "regionserver/,1" #1036 daemon prio=5 os_prio=0 tid=0x7f44b374e800
> nid=0xbd7f waiting on condition [0x7f37b4d19000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x7f3897a3e150> (a
> java.util.concurrent.locks.ReentrantLock$FairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:837)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:872)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1202)
> at
> java.util.concurrent.locks.ReentrantLock$FairSync.lock(ReentrantLock.java:228)
> at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.getLogFileSizeIfBeingWritten(AbstractFSWAL.java:1102)
> at
> org.apache.hadoop.hbase.wal.WALProvider.lambda$null$0(WALProvider.java:128)
> at
> org.apache.hadoop.hbase.wal.WALProvider$$Lambda$177/1119730685.apply(Unknown
> Source)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1361)
> at
> java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
> at
> java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499)
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at
> java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.findAny(ReferencePipeline.java:536)
> at
> org.apache.hadoop.hbase.wal.WALProvider.lambda$getWALFileLengthProvider$2(WALProvider.java:129)
> at
> org.apache.hadoop.hbase.wal.WALProvider$$Lambda$140/1246380717.getLogFileSizeIfBeingWritten(Unknown
> Source)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:260)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:172)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:101)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:222)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:157)
> {noformat}
> All the peers will contend for this lock during every batch read.
> Look at the code snippet below. We are guarding this section with
> rollWriterLock if we are replicating the active WAL file. But in our case we
> are NOT replicating active WAL file but still we acquire this lock only to
> return OptionalLong.empty();
> {noformat}
> /**
>* if the given {@code path} is being written currently, then return its
> length.
>*
>* This is used by
[jira] [Created] (HBASE-28184) Tailing the WAL is very slow if there are multiple peers.
Rushabh Shah created HBASE-28184:
Summary: Tailing the WAL is very slow if there are multiple peers.
Key: HBASE-28184
URL: https://issues.apache.org/jira/browse/HBASE-28184
Project: HBase
Issue Type: Bug
Components: Replication
Affects Versions: 2.0.0
Reporter: Rushabh Shah
Assignee: Rushabh Shah
Noticed in one of our production clusters which has 4 peers.
Due to sudden ingestion of data, the size of log queue increased to a peak of
506. We have configured log roll size to 256 MB. Most of the edits in the WAL
were from a table for which replication is disabled.
So all ReplicationSourceWALReader thread had to do was to replay the WAL and
NOT replicate them. Still it took 12 hours to drain the queue.
Took few jstacks and found that ReplicationSourceWALReader was waiting to
acquire rollWriterLock
[here|https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java#L1231]
{noformat}
"regionserver/,1" #1036 daemon prio=5 os_prio=0 tid=0x7f44b374e800
nid=0xbd7f waiting on condition [0x7f37b4d19000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x7f3897a3e150> (a
java.util.concurrent.locks.ReentrantLock$FairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:837)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:872)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1202)
at
java.util.concurrent.locks.ReentrantLock$FairSync.lock(ReentrantLock.java:228)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.getLogFileSizeIfBeingWritten(AbstractFSWAL.java:1102)
at
org.apache.hadoop.hbase.wal.WALProvider.lambda$null$0(WALProvider.java:128)
at
org.apache.hadoop.hbase.wal.WALProvider$$Lambda$177/1119730685.apply(Unknown
Source)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at
java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1361)
at
java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
at
java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.util.stream.ReferencePipeline.findAny(ReferencePipeline.java:536)
at
org.apache.hadoop.hbase.wal.WALProvider.lambda$getWALFileLengthProvider$2(WALProvider.java:129)
at
org.apache.hadoop.hbase.wal.WALProvider$$Lambda$140/1246380717.getLogFileSizeIfBeingWritten(Unknown
Source)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:260)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:172)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:101)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:222)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:157)
{noformat}
All the peers will contend for this lock during every batch read.
Look at the code snippet below. We are guarding this section with
rollWriterLock if we are replicating the active WAL file. But in our case we
are NOT replicating active WAL file but still we acquire this lock only to
return OptionalLong.empty();
{noformat}
/**
* if the given {@code path} is being written currently, then return its
length.
*
* This is used by replication to prevent replicating unacked log entries. See
* https://issues.apache.org/jira/browse/HBASE-14004 for more details.
*/
@Override
public OptionalLong getLogFileSizeIfBeingWritten(Path path) {
rollWriterLock.lock();
try {
...
...
} finally {
rollWriterLock.unlock();
}
{noformat}
We can check the size of log queue and if it is greater than 1 then we can
return early without acquiring the lock.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28045) Sort on store file size on hmaster page is broken.
[ https://issues.apache.org/jira/browse/HBASE-28045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759080#comment-17759080 ] Rushabh Shah commented on HBASE-28045: -- [~nihaljain.cs] You are right. We have some test clusters running hbase-1.6 version. I mistakenly thought this is upgraded to 2.5. Sorry for the noise. Closing this as Invalid. > Sort on store file size on hmaster page is broken. > -- > > Key: HBASE-28045 > URL: https://issues.apache.org/jira/browse/HBASE-28045 > Project: HBase > Issue Type: Bug > Components: UI >Affects Versions: 2.5.2 >Reporter: Rushabh Shah >Priority: Major > Labels: newbie, starter > Attachments: Screenshot 2023-08-24 at 11.08.54 AM.png, Screenshot > 2023-08-25 at 1.02.49 PM.png > > > An image is worth 1000 words. > !Screenshot 2023-08-24 at 11.08.54 AM.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-28045) Sort on store file size on hmaster page is broken.
[ https://issues.apache.org/jira/browse/HBASE-28045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh Shah resolved HBASE-28045. -- Resolution: Invalid > Sort on store file size on hmaster page is broken. > -- > > Key: HBASE-28045 > URL: https://issues.apache.org/jira/browse/HBASE-28045 > Project: HBase > Issue Type: Bug > Components: UI >Affects Versions: 2.5.2 >Reporter: Rushabh Shah >Priority: Major > Labels: newbie, starter > Attachments: Screenshot 2023-08-24 at 11.08.54 AM.png, Screenshot > 2023-08-25 at 1.02.49 PM.png > > > An image is worth 1000 words. > !Screenshot 2023-08-24 at 11.08.54 AM.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-28045) Sort on store file size on hmaster page is broken.
[ https://issues.apache.org/jira/browse/HBASE-28045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh Shah updated HBASE-28045: - Labels: newbie starter (was: ) > Sort on store file size on hmaster page is broken. > -- > > Key: HBASE-28045 > URL: https://issues.apache.org/jira/browse/HBASE-28045 > Project: HBase > Issue Type: Bug > Components: UI >Affects Versions: 2.5.2 >Reporter: Rushabh Shah >Priority: Major > Labels: newbie, starter > Attachments: Screenshot 2023-08-24 at 11.08.54 AM.png > > > An image is worth 1000 words. > !Screenshot 2023-08-24 at 11.08.54 AM.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-28045) Sort on store file size on hmaster page is broken.
[ https://issues.apache.org/jira/browse/HBASE-28045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh Shah updated HBASE-28045: - Summary: Sort on store file size on hmaster page is broken. (was: Sort on store file size on hmaster page is borken.) > Sort on store file size on hmaster page is broken. > -- > > Key: HBASE-28045 > URL: https://issues.apache.org/jira/browse/HBASE-28045 > Project: HBase > Issue Type: Bug > Components: UI >Affects Versions: 2.5.2 >Reporter: Rushabh Shah >Priority: Major > Attachments: Screenshot 2023-08-24 at 11.08.54 AM.png > > > An image is worth 1000 words. > !Screenshot 2023-08-24 at 11.08.54 AM.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-28045) Sort on store file size on hmaster page is borken.
Rushabh Shah created HBASE-28045: Summary: Sort on store file size on hmaster page is borken. Key: HBASE-28045 URL: https://issues.apache.org/jira/browse/HBASE-28045 Project: HBase Issue Type: Bug Components: UI Affects Versions: 2.5.2 Reporter: Rushabh Shah Attachments: Screenshot 2023-08-24 at 11.08.54 AM.png An image is worth 1000 words. !Screenshot 2023-08-24 at 11.08.54 AM.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-28039) Create metric for region in transition count per table.
Rushabh Shah created HBASE-28039: Summary: Create metric for region in transition count per table. Key: HBASE-28039 URL: https://issues.apache.org/jira/browse/HBASE-28039 Project: HBase Issue Type: Bug Reporter: Rushabh Shah Currently we have ritCount and ritCountOverThreshold metric for the whole cluster. It would be nice to have the ritCount and ritCountOverThreshold metric per table. It helps in debugging failed queries for a given table due to RIT. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HBASE-27965) Change the label from InterfaceAudience.Private to InterfaceAudience.LimitedPrivate for org.apache.hadoop.hbase.Server.java
[
https://issues.apache.org/jira/browse/HBASE-27965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh Shah reassigned HBASE-27965:
Assignee: Shubham Roy
> Change the label from InterfaceAudience.Private to
> InterfaceAudience.LimitedPrivate for org.apache.hadoop.hbase.Server.java
>
>
> Key: HBASE-27965
> URL: https://issues.apache.org/jira/browse/HBASE-27965
> Project: HBase
> Issue Type: Task
> Components: Coprocessors
>Affects Versions: 2.4.4
>Reporter: Shubham Roy
>Assignee: Shubham Roy
>Priority: Minor
>
> Currently the class org.apache.hadoop.hbase.Server.java is marked
> InterfaceAudience.Private.
> {{{}This prevents in getting shared zookeeper watcher (ZKWatcher) instances
> from org.apache.hadoop.hbase.regionserver.RegionServerServices.java (extends
> org.apache.hadoop.hbase.Server.java{}}}{{{}) using the method
> getZooKeeper().{}}}
>
> {{This creates a problem in writing custom Coprocessors because now we don't
> have a shared ZKWatcher instance.}}
>
> {{The proposed solution is to use the InterfaceAudience.LimitedPrivate. }}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-27963) Replication stuck when switch to new reader
[
https://issues.apache.org/jira/browse/HBASE-27963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17740668#comment-17740668
]
Rushabh Shah commented on HBASE-27963:
--
We are also seeing similar errors in our production environment. We are running
some version of 1.7 version. As a work around we restart the regionserver and
the new regionserver is able to replicate. So some in-memory data structure is
out of sync.
> Replication stuck when switch to new reader
> ---
>
> Key: HBASE-27963
> URL: https://issues.apache.org/jira/browse/HBASE-27963
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 3.0.0-alpha-4, 2.4.17, 2.5.5
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
>
> After creating new reader for next WAL, it immediately seek() to the
> currentPositionOfEntry, but this position may be spill over the length of
> current WAL.
> {code:java}
> WARN
> [RpcServer.default.FPRWQ.Fifo.read.handler=101,queue=1,port=16020.replicationSource.wal-reader.XXX]
> regionserver.ReplicationSourceWALReader: Failed to read stream of
> replication entries
> java.io.EOFException: Cannot seek after EOF
> at
> org.apache.hadoop.hdfs.DFSInputStream.seek(DFSInputStream.java:1488)
> at
> org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:62)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.seekOnFs(ProtobufLogReader.java:495)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.seek(ReaderBase.java:138)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.seek(WALEntryStream.java:399)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.openReader(WALEntryStream.java:341)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.handleFileNotFound(WALEntryStream.java:328)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.openReader(WALEntryStream.java:347)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.openNextLog(WALEntryStream.java:310)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.checkReader(WALEntryStream.java:300)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:176)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:102)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.tryAdvanceStreamAndCreateWALBatch(ReplicationSourceWALReader.java:260)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:142)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HBASE-27957) HConnection (and ZookeeprWatcher threads) leak in case of AUTH_FAILED exception.
Rushabh Shah created HBASE-27957:
Summary: HConnection (and ZookeeprWatcher threads) leak in case of
AUTH_FAILED exception.
Key: HBASE-27957
URL: https://issues.apache.org/jira/browse/HBASE-27957
Project: HBase
Issue Type: Bug
Affects Versions: 2.4.17, 1.7.2
Reporter: Rushabh Shah
Observed this in production environment running some version of 1.7 release.
Application didn't had the right keytab setup for authentication. Application
was trying to create HConnection and zookeeper server threw AUTH_FAILED
exception.
After few hours of application in this state, saw thousands of
zk-event-processor thread with below stack trace.
{noformat}
"zk-event-processor-pool1-t1" #1275 daemon prio=5 os_prio=0 cpu=1.04ms
elapsed=41794.58s tid=0x7fd7805066d0 nid=0x1245 waiting on condition
[0x7fd75df01000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18.0.102/Native Method)
- parking to wait for <0x7fd9874a85e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(java.base@11.0.18.0.102/LockSupport.java:194)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18.0.102/AbstractQueuedSynchronizer.java:2081)
at
java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.18.0.102/LinkedBlockingQueue.java:433)
at
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18.0.102/ThreadPoolExecutor.java:1054)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18.0.102/ThreadPoolExecutor.java:1114)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18.0.102/ThreadPoolExecutor.java:628)
{noformat}
{code:java|title=ConnectionManager.java|borderStyle=solid}
HConnectionImplementation(Configuration conf, boolean managed,
ExecutorService pool, User user, String clusterId) throws IOException {
...
...
try {
this.registry = setupRegistry();
retrieveClusterId();
...
...
} catch (Throwable e) {
// avoid leaks: registry, rpcClient, ...
LOG.debug("connection construction failed", e);
close();
throw e;
}
{code}
retrieveClusterId internally calls ZKConnectionRegistry#getClusterId
{code:java|title=ZKConnectionRegistry.java|borderStyle=solid}
private String clusterId = null;
@Override
public String getClusterId() {
if (this.clusterId != null) return this.clusterId;
// No synchronized here, worse case we will retrieve it twice, that's
// not an issue.
try (ZooKeeperKeepAliveConnection zkw = hci.getKeepAliveZooKeeperWatcher())
{
this.clusterId = ZKClusterId.readClusterIdZNode(zkw);
if (this.clusterId == null) {
LOG.info("ClusterId read in ZooKeeper is null");
}
} catch (KeeperException | IOException e) { ---> WE ARE SWALLOWING
THIS EXCEPTION AND RETURNING NULL.
LOG.warn("Can't retrieve clusterId from Zookeeper", e);
}
return this.clusterId;
}
{code}
ZkConnectionRegistry#getClusterId threw the following exception.(Our logging
system trims stack traces longer than 5 lines.)
{noformat}
Cause: org.apache.zookeeper.KeeperException$AuthFailedException:
KeeperErrorCode = AuthFailed for /hbase/hbaseid
StackTrace:
org.apache.zookeeper.KeeperException.create(KeeperException.java:126)
org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1213)
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:285)
org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:470)
{noformat}
We should throw KeeperException from ZKConnectionRegistry#getClusterId all the
way back to HConnectionImplementation constructor to close all the watcher
threads and throw the exception back to the caller.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-27908) Can't get connection to ZooKeeper
[
https://issues.apache.org/jira/browse/HBASE-27908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729358#comment-17729358
]
Rushabh Shah commented on HBASE-27908:
--
[~ibrar1761] JIRA is not a forum to seek help regarding configuration issues or
dev setup. Please send an email to d...@hbase.apache.org
or u...@hbase.apache.org and hopefully someone will reply there. I am closing
this bug as INVALID.
> Can't get connection to ZooKeeper
> -
>
> Key: HBASE-27908
> URL: https://issues.apache.org/jira/browse/HBASE-27908
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 1.4.13
>Reporter: Ibrar Ahmed
>Priority: Major
>
> I am using Hbase cluster along with apache kylin, the connection between Edge
> node and the Hbase cluster is good.
> following are the logs from Kylin side which shows Error exception:
> {code:java}
> java.net.SocketTimeoutException: callTimeout=120, callDuration=1275361:
> org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to
> ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:178)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4551)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:561)
> at
> org.apache.hadoop.hbase.client.HTable.getTableDescriptor(HTable.java:585)
> at
> org.apache.kylin.storage.hbase.steps.HFileOutputFormat3.configureIncrementalLoad(HFileOutputFormat3.java:328)
> at
> org.apache.kylin.storage.hbase.steps.CubeHFileJob.run(CubeHFileJob.java:101)
> at
> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:144)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:179)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:179)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> Caused by: org.apache.hadoop.hbase.MasterNotRunningException:
> org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to
> ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1618)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1638)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1795)
> at
> org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:140)
> ... 13 more
> Caused by: org.apache.hadoop.hbase.MasterNotRunningException: Can't get
> connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:971)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.access$400(ConnectionManager.java:566)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1567)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1609)
> ... 17 more
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:220)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:425)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:960)
> ... 20 more {code}
> Following are the logs from Hbase cluster master NOde which accepts the
> connection from Edge NOde(Kylin):
> {code:java}
> 2023-06-05 10:00:30,336 [myid:0]
[jira] [Resolved] (HBASE-27908) Can't get connection to ZooKeeper
[
https://issues.apache.org/jira/browse/HBASE-27908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh Shah resolved HBASE-27908.
--
Resolution: Invalid
> Can't get connection to ZooKeeper
> -
>
> Key: HBASE-27908
> URL: https://issues.apache.org/jira/browse/HBASE-27908
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 1.4.13
>Reporter: Ibrar Ahmed
>Priority: Major
>
> I am using Hbase cluster along with apache kylin, the connection between Edge
> node and the Hbase cluster is good.
> following are the logs from Kylin side which shows Error exception:
> {code:java}
> java.net.SocketTimeoutException: callTimeout=120, callDuration=1275361:
> org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to
> ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:178)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4551)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:561)
> at
> org.apache.hadoop.hbase.client.HTable.getTableDescriptor(HTable.java:585)
> at
> org.apache.kylin.storage.hbase.steps.HFileOutputFormat3.configureIncrementalLoad(HFileOutputFormat3.java:328)
> at
> org.apache.kylin.storage.hbase.steps.CubeHFileJob.run(CubeHFileJob.java:101)
> at
> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:144)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:179)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:179)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> Caused by: org.apache.hadoop.hbase.MasterNotRunningException:
> org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to
> ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1618)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1638)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1795)
> at
> org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:140)
> ... 13 more
> Caused by: org.apache.hadoop.hbase.MasterNotRunningException: Can't get
> connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:971)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.access$400(ConnectionManager.java:566)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1567)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1609)
> ... 17 more
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:220)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:425)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:960)
> ... 20 more {code}
> Following are the logs from Hbase cluster master NOde which accepts the
> connection from Edge NOde(Kylin):
> {code:java}
> 2023-06-05 10:00:30,336 [myid:0] - INFO
> [CommitProcessor:0:NIOServerCnxn@1056] - Closed socket connection for client
> /10.127.2.201:37328 which had sessionid 0x7311c000c
> 2023-06-05 13:14:48,346 [myid:0] - INFO
> [PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task
[jira] [Resolved] (HBASE-26913) Replication Observability Framework
[ https://issues.apache.org/jira/browse/HBASE-26913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh Shah resolved HBASE-26913. -- Resolution: Fixed > Replication Observability Framework > --- > > Key: HBASE-26913 > URL: https://issues.apache.org/jira/browse/HBASE-26913 > Project: HBase > Issue Type: New Feature > Components: regionserver, Replication >Reporter: Rushabh Shah >Assignee: Rushabh Shah >Priority: Major > Fix For: 2.6.0, 3.0.0-alpha-4 > > > In our production clusters, we have seen cases where data is present in > source cluster but not in the sink cluster and 1 case where data is present > in sink cluster but not in source cluster. > We have internal tools where we take incremental backup every day on both > source and sink clusters and we compare the hash of the data in both the > backups. We have seen many cases where hash doesn't match which means data is > not consistent between source and sink for that given day. The Mean Time To > Detect (MTTD) these inconsistencies is atleast 2 days and requires lot of > manual debugging. > We need some tool where we can reduce MTTD and requires less manual debugging. > I have attached design doc. Huge thanks to [~bharathv] to come up with this > design at my work place. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27697) Create a dummy metric implementation in ConnectionImplementation.
Rushabh Shah created HBASE-27697:
Summary: Create a dummy metric implementation in
ConnectionImplementation.
Key: HBASE-27697
URL: https://issues.apache.org/jira/browse/HBASE-27697
Project: HBase
Issue Type: Bug
Components: metrics
Affects Versions: 2.5.0
Reporter: Rushabh Shah
This Jira is for branch-2 only.
If CLIENT_SIDE_METRICS_ENABLED_KEY conf is set to true, then we initialize
metrics to MetricsConnection, otherwise it is set to null.
{code:java}
if (conf.getBoolean(CLIENT_SIDE_METRICS_ENABLED_KEY, false)) {
this.metricsScope = MetricsConnection.getScope(conf, clusterId,
this);
this.metrics =
MetricsConnection.getMetricsConnection(this.metricsScope, this::getBatchPool,
this::getMetaLookupPool);
} else {
this.metrics = null;
}
{code}
Whenever we want to update metrics, we always do a null check. We can improve
this by creating a dummy metrics object and have an empty implementation. When
we want to populate the metrics, we can check if metrics is an instance of
dummy metrics.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-26913) Replication Observability Framework
[ https://issues.apache.org/jira/browse/HBASE-26913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17629946#comment-17629946 ] Rushabh Shah commented on HBASE-26913: -- > After looking at the current PR, I think maybe we could introduce a special >region info, for handling the region server level markers, for example, let's >call the table 'hbase:replication_marker_placeholder', and we will always use >this table's first region info, i.e, creating by >RegionInfoBuilder.newBuilder(tableName).build(), to write region server level >markers. And it will be replicated to remote peers, but when splitting, we >will just drop it, which is almost the same with the current implementation. In this case also, we will create an edit with a region which will not reside on any region server. In the first proposal, we were re-using an existing table we created for this framework instead of creating yet another table. [~zhangduo] Am I missing something? > Replication Observability Framework > --- > > Key: HBASE-26913 > URL: https://issues.apache.org/jira/browse/HBASE-26913 > Project: HBase > Issue Type: New Feature > Components: regionserver, Replication >Reporter: Rushabh Shah >Assignee: Rushabh Shah >Priority: Major > Fix For: 2.6.0, 3.0.0-alpha-4 > > > In our production clusters, we have seen cases where data is present in > source cluster but not in the sink cluster and 1 case where data is present > in sink cluster but not in source cluster. > We have internal tools where we take incremental backup every day on both > source and sink clusters and we compare the hash of the data in both the > backups. We have seen many cases where hash doesn't match which means data is > not consistent between source and sink for that given day. The Mean Time To > Detect (MTTD) these inconsistencies is atleast 2 days and requires lot of > manual debugging. > We need some tool where we can reduce MTTD and requires less manual debugging. > I have attached design doc. Huge thanks to [~bharathv] to come up with this > design at my work place. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27100) Add documentation for Replication Observability Framework in hbase book.
[ https://issues.apache.org/jira/browse/HBASE-27100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh Shah resolved HBASE-27100. -- Fix Version/s: 3.0.0-alpha-4 Resolution: Fixed > Add documentation for Replication Observability Framework in hbase book. > > > Key: HBASE-27100 > URL: https://issues.apache.org/jira/browse/HBASE-27100 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Rushabh Shah >Assignee: Rushabh Shah >Priority: Major > Fix For: 3.0.0-alpha-4 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-26925) Create WAL event tracker table to track all the WAL events.
[ https://issues.apache.org/jira/browse/HBASE-26925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh Shah resolved HBASE-26925. -- Fix Version/s: 3.0.0-alpha-4 Resolution: Fixed > Create WAL event tracker table to track all the WAL events. > --- > > Key: HBASE-26925 > URL: https://issues.apache.org/jira/browse/HBASE-26925 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: Rushabh Shah >Assignee: Rushabh Shah >Priority: Major > Fix For: 3.0.0-alpha-4 > > > Design Doc: > [https://docs.google.com/document/d/14oZ5ssY28hvJaQD_Jg9kWX7LfUKUyyU2PCA93PPzVko/edit#] > Create wal event tracker table to track WAL events. Whenever we roll the WAL, > we will save the WAL name, WAL length, region server, timestamp in a table. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27085) Create REPLICATION_SINK_TRACKER table to persist sentinel rows coming from source cluster.
[ https://issues.apache.org/jira/browse/HBASE-27085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh Shah resolved HBASE-27085. -- Fix Version/s: 3.0.0-alpha-4 Resolution: Fixed > Create REPLICATION_SINK_TRACKER table to persist sentinel rows coming from > source cluster. > -- > > Key: HBASE-27085 > URL: https://issues.apache.org/jira/browse/HBASE-27085 > Project: HBase > Issue Type: Sub-task >Reporter: Rushabh Shah >Assignee: Rushabh Shah >Priority: Major > Fix For: 3.0.0-alpha-4 > > > This work is to create sink tracker table to persist tracker rows coming from > replication source cluster. > Create ReplicationMarkerChore to create replication marker rows periodically. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27461) Add multiWAL support for Replication Observability framework.
Rushabh Shah created HBASE-27461: Summary: Add multiWAL support for Replication Observability framework. Key: HBASE-27461 URL: https://issues.apache.org/jira/browse/HBASE-27461 Project: HBase Issue Type: Sub-task Components: regionserver Reporter: Rushabh Shah HBASE-26913 added a new framework for observing health of replication subsystem. Currently we add replication marker row just to one WAL (i.e. default WAL). We need to add support for multi WAL implementation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HBASE-27383) Add dead region server to SplitLogManager#deadWorkers set as the first step.
[ https://issues.apache.org/jira/browse/HBASE-27383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh Shah reassigned HBASE-27383: Assignee: (was: Rushabh Shah) > Add dead region server to SplitLogManager#deadWorkers set as the first step. > > > Key: HBASE-27383 > URL: https://issues.apache.org/jira/browse/HBASE-27383 > Project: HBase > Issue Type: Bug >Affects Versions: 2.5.0, 1.7.2, 2.4.14 >Reporter: Rushabh Shah >Priority: Major > > Currently we add a dead region server to +SplitLogManager#deadWorkers+ set in > SERVER_CRASH_SPLIT_LOGS state. > Consider a case where a region server is handling split log task for > hbase:meta table and SplitLogManager has exhausted all the retries and won't > try any more region server. > The region server which is handling split log task has died. > We have a check in SplitLogManager where if a region server is declared dead > and if that region server is responsible for split log task then we > forcefully resubmit split log task. See the code > [here|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java#L721-L726]. > But we add a region server to SplitLogManager#deadWorkers set in > [SERVER_CRASH_SPLIT_LOGS|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java#L252] > state. > Before that it runs > [SERVER_CRASH_GET_REGIONS|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java#L214] > state and checks if hbase:meta table is up. In this case, hbase:meta table > was not online and that prevented SplitLogManager to add this RS to > deadWorkers list. This created a deadlock and hbase cluster was completely > down for an extended period of time until we failed over active hmaster. See > HBASE-27382 for more details. > Improvements: > 1. We should a dead region server to +SplitLogManager#deadWorkers+ list as > the first step. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27382) Cluster completely down due to WAL splitting failing for hbase:meta table.
[
https://issues.apache.org/jira/browse/HBASE-27382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608796#comment-17608796
]
Rushabh Shah commented on HBASE-27382:
--
> I prefer we just abort the master under this scenario.
[~zhangduo] Just to confirm, you mean abort master if retries are exhausted
only for WALs belonging to hbase:meta table or for WALs belonging to any table
?
> Cluster completely down due to WAL splitting failing for hbase:meta table.
> --
>
> Key: HBASE-27382
> URL: https://issues.apache.org/jira/browse/HBASE-27382
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.5.0, 1.7.2, 2.4.14
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Fix For: 2.6.0, 2.5.1, 3.0.0-alpha-4, 2.4.15
>
>
> We are running some version of 1.7.2 in our production environment. We
> encountered this issue recently.
> We colocate namenode and region server holding hbase:meta table on a set of 5
> master nodes. Co-incidentally active namenode and region server holding meta
> table were on the same physical node and that node went down due to hardware
> issue. We have sub optimal hdfs level timeouts configured so whenever active
> namenode goes down, it takes around 12-15 minutes for hdfs client within
> hbase to connect to new active namenode. So all the region servers were
> having problems for about 15 minutes to connect to new active namenode.
> Below are the sequence of events:
> 1. Host running active namenode and hbase:meta went down at +2022-09-09
> 16:56:56,878+
> 2. HMaster started running ServerCrashProcedure at +2022-09-09 16:59:05,696+
> {noformat}
> 2022-09-09 16:59:05,696 DEBUG [t-processor-pool2-t1]
> procedure2.ProcedureExecutor - Procedure ServerCrashProcedure
> serverName=,61020,1662714013670, shouldSplitWal=true,
> carryingMeta=true id=1 owner=dummy state=RUNNABLE:SERVER_CRASH_START added to
> the store.
> 2022-09-09 16:59:05,702 DEBUG [t-processor-pool2-t1] master.ServerManager -
> Added=,61020,1662714013670 to dead servers, submitted shutdown
> handler to be executed meta=true
> 2022-09-09 16:59:05,707 DEBUG [ProcedureExecutor-0] master.DeadServer -
> Started processing ,61020,1662714013670; numProcessing=1
> 2022-09-09 16:59:05,712 INFO [ProcedureExecutor-0]
> procedure.ServerCrashProcedure - Start processing crashed
> ,61020,1662714013670
> {noformat}
> 3. SplitLogManager created 2 split log tasks in zookeeper.
> {noformat}
> 2022-09-09 16:59:06,049 INFO [ProcedureExecutor-1] master.SplitLogManager -
> Started splitting 2 logs in
> [hdfs:///hbase/WALs/,61020,1662714013670-splitting]
> for [,61020,1662714013670]
> 2022-09-09 16:59:06,081 DEBUG [main-EventThread]
> coordination.SplitLogManagerCoordination - put up splitlog task at znode
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
> 2022-09-09 16:59:06,093 DEBUG [main-EventThread]
> coordination.SplitLogManagerCoordination - put up splitlog task at znode
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662739251611.meta
> {noformat}
> 4. The first split log task is more interesting:
> +/hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta+
> 5. Since all the region servers were having problems connecting to active
> namenode, SplitLogManager tried total of 4 times to assign this task (3
> resubmits, configured by hbase.splitlog.max.resubmit) and then finally gave
> up.
> {noformat}
> -- try 1 -
> 2022-09-09 16:59:06,205 INFO [main-EventThread]
> coordination.SplitLogManagerCoordination - task
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
> acquired by ,61020,1662540522069
> -- try 2 -
> 2022-09-09 17:01:06,642 INFO [ager__ChoreService_1]
> coordination.SplitLogManagerCoordination - resubmitting task
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
> 2022-09-09 17:01:06,666 DEBUG [main-EventThread]
> coordination.SplitLogManagerCoordination - task not yet acquired
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
> ver = 2
> 2022-09-09 17:01:06,715 INFO [main-EventThread]
> coordination.SplitLogManagerCoordination - task
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
> acquired by ,61020,1662530684713
> -- try 3 -
> 2022-09-09 17:03:07,643 INFO [ager__ChoreService_1]
> coordination.SplitLogManagerCoordination -
[jira] [Updated] (HBASE-27383) Add dead region server to SplitLogManager#deadWorkers set as the first step.
[ https://issues.apache.org/jira/browse/HBASE-27383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh Shah updated HBASE-27383: - Affects Version/s: 2.5.0 (was: 1.6.0) > Add dead region server to SplitLogManager#deadWorkers set as the first step. > > > Key: HBASE-27383 > URL: https://issues.apache.org/jira/browse/HBASE-27383 > Project: HBase > Issue Type: Bug >Affects Versions: 2.5.0, 1.7.2, 2.4.14 >Reporter: Rushabh Shah >Assignee: Rushabh Shah >Priority: Major > > Currently we add a dead region server to +SplitLogManager#deadWorkers+ set in > SERVER_CRASH_SPLIT_LOGS state. > Consider a case where a region server is handling split log task for > hbase:meta table and SplitLogManager has exhausted all the retries and won't > try any more region server. > The region server which is handling split log task has died. > We have a check in SplitLogManager where if a region server is declared dead > and if that region server is responsible for split log task then we > forcefully resubmit split log task. See the code > [here|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java#L721-L726]. > But we add a region server to SplitLogManager#deadWorkers set in > [SERVER_CRASH_SPLIT_LOGS|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java#L252] > state. > Before that it runs > [SERVER_CRASH_GET_REGIONS|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java#L214] > state and checks if hbase:meta table is up. In this case, hbase:meta table > was not online and that prevented SplitLogManager to add this RS to > deadWorkers list. This created a deadlock and hbase cluster was completely > down for an extended period of time until we failed over active hmaster. See > HBASE-27382 for more details. > Improvements: > 1. We should a dead region server to +SplitLogManager#deadWorkers+ list as > the first step. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27383) Add dead region server to SplitLogManager#deadWorkers set as the first step.
[ https://issues.apache.org/jira/browse/HBASE-27383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh Shah updated HBASE-27383: - Affects Version/s: 2.4.14 > Add dead region server to SplitLogManager#deadWorkers set as the first step. > > > Key: HBASE-27383 > URL: https://issues.apache.org/jira/browse/HBASE-27383 > Project: HBase > Issue Type: Bug >Affects Versions: 1.6.0, 1.7.2, 2.4.14 >Reporter: Rushabh Shah >Assignee: Rushabh Shah >Priority: Major > > Currently we add a dead region server to +SplitLogManager#deadWorkers+ set in > SERVER_CRASH_SPLIT_LOGS state. > Consider a case where a region server is handling split log task for > hbase:meta table and SplitLogManager has exhausted all the retries and won't > try any more region server. > The region server which is handling split log task has died. > We have a check in SplitLogManager where if a region server is declared dead > and if that region server is responsible for split log task then we > forcefully resubmit split log task. See the code > [here|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java#L721-L726]. > But we add a region server to SplitLogManager#deadWorkers set in > [SERVER_CRASH_SPLIT_LOGS|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java#L252] > state. > Before that it runs > [SERVER_CRASH_GET_REGIONS|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java#L214] > state and checks if hbase:meta table is up. In this case, hbase:meta table > was not online and that prevented SplitLogManager to add this RS to > deadWorkers list. This created a deadlock and hbase cluster was completely > down for an extended period of time until we failed over active hmaster. See > HBASE-27382 for more details. > Improvements: > 1. We should a dead region server to +SplitLogManager#deadWorkers+ list as > the first step. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27382) Cluster completely down due to WAL splitting failing for hbase:meta table.
[
https://issues.apache.org/jira/browse/HBASE-27382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17607987#comment-17607987
]
Rushabh Shah commented on HBASE-27382:
--
[~apurtell] This is an issue in branch-2 as well. We still have
ZKSplitLogManagerCoordination present in branch-2 even if the default is not to
use zookeeper as co-ordination engine.
> Cluster completely down due to WAL splitting failing for hbase:meta table.
> --
>
> Key: HBASE-27382
> URL: https://issues.apache.org/jira/browse/HBASE-27382
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.6.0, 1.7.1, 1.7.2
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
>
> We are running some version of 1.7.2 in our production environment. We
> encountered this issue recently.
> We colocate namenode and region server holding hbase:meta table on a set of 5
> master nodes. Co-incidentally active namenode and region server holding meta
> table were on the same physical node and that node went down due to hardware
> issue. We have sub optimal hdfs level timeouts configured so whenever active
> namenode goes down, it takes around 12-15 minutes for hdfs client within
> hbase to connect to new active namenode. So all the region servers were
> having problems for about 15 minutes to connect to new active namenode.
> Below are the sequence of events:
> 1. Host running active namenode and hbase:meta went down at +2022-09-09
> 16:56:56,878+
> 2. HMaster started running ServerCrashProcedure at +2022-09-09 16:59:05,696+
> {noformat}
> 2022-09-09 16:59:05,696 DEBUG [t-processor-pool2-t1]
> procedure2.ProcedureExecutor - Procedure ServerCrashProcedure
> serverName=,61020,1662714013670, shouldSplitWal=true,
> carryingMeta=true id=1 owner=dummy state=RUNNABLE:SERVER_CRASH_START added to
> the store.
> 2022-09-09 16:59:05,702 DEBUG [t-processor-pool2-t1] master.ServerManager -
> Added=,61020,1662714013670 to dead servers, submitted shutdown
> handler to be executed meta=true
> 2022-09-09 16:59:05,707 DEBUG [ProcedureExecutor-0] master.DeadServer -
> Started processing ,61020,1662714013670; numProcessing=1
> 2022-09-09 16:59:05,712 INFO [ProcedureExecutor-0]
> procedure.ServerCrashProcedure - Start processing crashed
> ,61020,1662714013670
> {noformat}
> 3. SplitLogManager created 2 split log tasks in zookeeper.
> {noformat}
> 2022-09-09 16:59:06,049 INFO [ProcedureExecutor-1] master.SplitLogManager -
> Started splitting 2 logs in
> [hdfs:///hbase/WALs/,61020,1662714013670-splitting]
> for [,61020,1662714013670]
> 2022-09-09 16:59:06,081 DEBUG [main-EventThread]
> coordination.SplitLogManagerCoordination - put up splitlog task at znode
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
> 2022-09-09 16:59:06,093 DEBUG [main-EventThread]
> coordination.SplitLogManagerCoordination - put up splitlog task at znode
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662739251611.meta
> {noformat}
> 4. The first split log task is more interesting:
> +/hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta+
> 5. Since all the region servers were having problems connecting to active
> namenode, SplitLogManager tried total of 4 times to assign this task (3
> resubmits, configured by hbase.splitlog.max.resubmit) and then finally gave
> up.
> {noformat}
> -- try 1 -
> 2022-09-09 16:59:06,205 INFO [main-EventThread]
> coordination.SplitLogManagerCoordination - task
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
> acquired by ,61020,1662540522069
> -- try 2 -
> 2022-09-09 17:01:06,642 INFO [ager__ChoreService_1]
> coordination.SplitLogManagerCoordination - resubmitting task
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
> 2022-09-09 17:01:06,666 DEBUG [main-EventThread]
> coordination.SplitLogManagerCoordination - task not yet acquired
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
> ver = 2
> 2022-09-09 17:01:06,715 INFO [main-EventThread]
> coordination.SplitLogManagerCoordination - task
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
> acquired by ,61020,1662530684713
> -- try 3 -
> 2022-09-09 17:03:07,643 INFO [ager__ChoreService_1]
> coordination.SplitLogManagerCoordination - resubmitting task
>
[jira] [Commented] (HBASE-27383) Add dead region server to SplitLogManager#deadWorkers set as the first step.
[ https://issues.apache.org/jira/browse/HBASE-27383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17607969#comment-17607969 ] Rushabh Shah commented on HBASE-27383: -- Cc [~dmanning] > Add dead region server to SplitLogManager#deadWorkers set as the first step. > > > Key: HBASE-27383 > URL: https://issues.apache.org/jira/browse/HBASE-27383 > Project: HBase > Issue Type: Bug >Affects Versions: 1.6.0, 1.7.2 >Reporter: Rushabh Shah >Assignee: Rushabh Shah >Priority: Major > > Currently we add a dead region server to +SplitLogManager#deadWorkers+ set in > SERVER_CRASH_SPLIT_LOGS state. > Consider a case where a region server is handling split log task for > hbase:meta table and SplitLogManager has exhausted all the retries and won't > try any more region server. > The region server which is handling split log task has died. > We have a check in SplitLogManager where if a region server is declared dead > and if that region server is responsible for split log task then we > forcefully resubmit split log task. See the code > [here|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java#L721-L726]. > But we add a region server to SplitLogManager#deadWorkers set in > [SERVER_CRASH_SPLIT_LOGS|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java#L252] > state. > Before that it runs > [SERVER_CRASH_GET_REGIONS|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java#L214] > state and checks if hbase:meta table is up. In this case, hbase:meta table > was not online and that prevented SplitLogManager to add this RS to > deadWorkers list. This created a deadlock and hbase cluster was completely > down for an extended period of time until we failed over active hmaster. See > HBASE-27382 for more details. > Improvements: > 1. We should a dead region server to +SplitLogManager#deadWorkers+ list as > the first step. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27383) Add dead region server to SplitLogManager#deadWorkers set as the first step.
Rushabh Shah created HBASE-27383: Summary: Add dead region server to SplitLogManager#deadWorkers set as the first step. Key: HBASE-27383 URL: https://issues.apache.org/jira/browse/HBASE-27383 Project: HBase Issue Type: Bug Affects Versions: 1.7.2, 1.6.0 Reporter: Rushabh Shah Assignee: Rushabh Shah Currently we add a dead region server to +SplitLogManager#deadWorkers+ set in SERVER_CRASH_SPLIT_LOGS state. Consider a case where a region server is handling split log task for hbase:meta table and SplitLogManager has exhausted all the retries and won't try any more region server. The region server which is handling split log task has died. We have a check in SplitLogManager where if a region server is declared dead and if that region server is responsible for split log task then we forcefully resubmit split log task. See the code [here|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java#L721-L726]. But we add a region server to SplitLogManager#deadWorkers set in [SERVER_CRASH_SPLIT_LOGS|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java#L252] state. Before that it runs [SERVER_CRASH_GET_REGIONS|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java#L214] state and checks if hbase:meta table is up. In this case, hbase:meta table was not online and that prevented SplitLogManager to add this RS to deadWorkers list. This created a deadlock and hbase cluster was completely down for an extended period of time until we failed over active hmaster. See HBASE-27382 for more details. Improvements: 1. We should a dead region server to +SplitLogManager#deadWorkers+ list as the first step. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27382) Cluster completely down due to WAL splitting failing for hbase:meta table.
[
https://issues.apache.org/jira/browse/HBASE-27382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17607960#comment-17607960
]
Rushabh Shah commented on HBASE-27382:
--
I haven't looked at the proc v2 based WAL split implementation but we still
have config properties like hbase.splitlog.max.resubmit which dictates this
behavior.
> Cluster completely down due to WAL splitting failing for hbase:meta table.
> --
>
> Key: HBASE-27382
> URL: https://issues.apache.org/jira/browse/HBASE-27382
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.6.0, 1.7.1, 1.7.2
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
>
> We are running some version of 1.7.2 in our production environment. We
> encountered this issue recently.
> We colocate namenode and region server holding hbase:meta table on a set of 5
> master nodes. Co-incidentally active namenode and region server holding meta
> table were on the same physical node and that node went down due to hardware
> issue. We have sub optimal hdfs level timeouts configured so whenever active
> namenode goes down, it takes around 12-15 minutes for hdfs client within
> hbase to connect to new active namenode. So all the region servers were
> having problems for about 15 minutes to connect to new active namenode.
> Below are the sequence of events:
> 1. Host running active namenode and hbase:meta went down at +2022-09-09
> 16:56:56,878+
> 2. HMaster started running ServerCrashProcedure at +2022-09-09 16:59:05,696+
> {noformat}
> 2022-09-09 16:59:05,696 DEBUG [t-processor-pool2-t1]
> procedure2.ProcedureExecutor - Procedure ServerCrashProcedure
> serverName=,61020,1662714013670, shouldSplitWal=true,
> carryingMeta=true id=1 owner=dummy state=RUNNABLE:SERVER_CRASH_START added to
> the store.
> 2022-09-09 16:59:05,702 DEBUG [t-processor-pool2-t1] master.ServerManager -
> Added=,61020,1662714013670 to dead servers, submitted shutdown
> handler to be executed meta=true
> 2022-09-09 16:59:05,707 DEBUG [ProcedureExecutor-0] master.DeadServer -
> Started processing ,61020,1662714013670; numProcessing=1
> 2022-09-09 16:59:05,712 INFO [ProcedureExecutor-0]
> procedure.ServerCrashProcedure - Start processing crashed
> ,61020,1662714013670
> {noformat}
> 3. SplitLogManager created 2 split log tasks in zookeeper.
> {noformat}
> 2022-09-09 16:59:06,049 INFO [ProcedureExecutor-1] master.SplitLogManager -
> Started splitting 2 logs in
> [hdfs:///hbase/WALs/,61020,1662714013670-splitting]
> for [,61020,1662714013670]
> 2022-09-09 16:59:06,081 DEBUG [main-EventThread]
> coordination.SplitLogManagerCoordination - put up splitlog task at znode
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
> 2022-09-09 16:59:06,093 DEBUG [main-EventThread]
> coordination.SplitLogManagerCoordination - put up splitlog task at znode
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662739251611.meta
> {noformat}
> 4. The first split log task is more interesting:
> +/hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta+
> 5. Since all the region servers were having problems connecting to active
> namenode, SplitLogManager tried total of 4 times to assign this task (3
> resubmits, configured by hbase.splitlog.max.resubmit) and then finally gave
> up.
> {noformat}
> -- try 1 -
> 2022-09-09 16:59:06,205 INFO [main-EventThread]
> coordination.SplitLogManagerCoordination - task
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
> acquired by ,61020,1662540522069
> -- try 2 -
> 2022-09-09 17:01:06,642 INFO [ager__ChoreService_1]
> coordination.SplitLogManagerCoordination - resubmitting task
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
> 2022-09-09 17:01:06,666 DEBUG [main-EventThread]
> coordination.SplitLogManagerCoordination - task not yet acquired
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
> ver = 2
> 2022-09-09 17:01:06,715 INFO [main-EventThread]
> coordination.SplitLogManagerCoordination - task
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
> acquired by ,61020,1662530684713
> -- try 3 -
> 2022-09-09 17:03:07,643 INFO [ager__ChoreService_1]
> coordination.SplitLogManagerCoordination - resubmitting task
>
[jira] [Commented] (HBASE-27382) Cluster completely down due to WAL splitting failing for hbase:meta table.
[
https://issues.apache.org/jira/browse/HBASE-27382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17607950#comment-17607950
]
Rushabh Shah commented on HBASE-27382:
--
Cc [~apurt...@yahoo.com] [~vjasani] [~dmanning]
> Cluster completely down due to WAL splitting failing for hbase:meta table.
> --
>
> Key: HBASE-27382
> URL: https://issues.apache.org/jira/browse/HBASE-27382
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.7.2
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
>
> We are running some version of 1.7.2 in our production environment. We
> encountered this issue recently.
> We colocate namenode and region server holding hbase:meta table on a set of 5
> master nodes. Co-incidentally active namenode and region server holding meta
> table were on the same physical node and that node went down due to hardware
> issue. We have sub optimal hdfs level timeouts configured so whenever active
> namenode goes down, it takes around 12-15 minutes for hdfs client within
> hbase to connect to new active namenode. So all the region servers were
> having problems for about 15 minutes to connect to new active namenode.
> Below are the sequence of events:
> 1. Host running active namenode and hbase:meta went down at +2022-09-09
> 16:56:56,878+
> 2. HMaster started running ServerCrashProcedure at +2022-09-09 16:59:05,696+
> {noformat}
> 2022-09-09 16:59:05,696 DEBUG [t-processor-pool2-t1]
> procedure2.ProcedureExecutor - Procedure ServerCrashProcedure
> serverName=,61020,1662714013670, shouldSplitWal=true,
> carryingMeta=true id=1 owner=dummy state=RUNNABLE:SERVER_CRASH_START added to
> the store.
> 2022-09-09 16:59:05,702 DEBUG [t-processor-pool2-t1] master.ServerManager -
> Added=,61020,1662714013670 to dead servers, submitted shutdown
> handler to be executed meta=true
> 2022-09-09 16:59:05,707 DEBUG [ProcedureExecutor-0] master.DeadServer -
> Started processing ,61020,1662714013670; numProcessing=1
> 2022-09-09 16:59:05,712 INFO [ProcedureExecutor-0]
> procedure.ServerCrashProcedure - Start processing crashed
> ,61020,1662714013670
> {noformat}
> 3. SplitLogManager created 2 split log tasks in zookeeper.
> {noformat}
> 2022-09-09 16:59:06,049 INFO [ProcedureExecutor-1] master.SplitLogManager -
> Started splitting 2 logs in
> [hdfs:///hbase/WALs/,61020,1662714013670-splitting]
> for [,61020,1662714013670]
> 2022-09-09 16:59:06,081 DEBUG [main-EventThread]
> coordination.SplitLogManagerCoordination - put up splitlog task at znode
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
> 2022-09-09 16:59:06,093 DEBUG [main-EventThread]
> coordination.SplitLogManagerCoordination - put up splitlog task at znode
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662739251611.meta
> {noformat}
> 4. The first split log task is more interesting:
> +/hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta+
> 5. Since all the region servers were having problems connecting to active
> namenode, SplitLogManager tried total of 4 times to assign this task (3
> resubmits, configured by hbase.splitlog.max.resubmit) and then finally gave
> up.
> {noformat}
> -- try 1 -
> 2022-09-09 16:59:06,205 INFO [main-EventThread]
> coordination.SplitLogManagerCoordination - task
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
> acquired by ,61020,1662540522069
> -- try 2 -
> 2022-09-09 17:01:06,642 INFO [ager__ChoreService_1]
> coordination.SplitLogManagerCoordination - resubmitting task
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
> 2022-09-09 17:01:06,666 DEBUG [main-EventThread]
> coordination.SplitLogManagerCoordination - task not yet acquired
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
> ver = 2
> 2022-09-09 17:01:06,715 INFO [main-EventThread]
> coordination.SplitLogManagerCoordination - task
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
> acquired by ,61020,1662530684713
> -- try 3 -
> 2022-09-09 17:03:07,643 INFO [ager__ChoreService_1]
> coordination.SplitLogManagerCoordination - resubmitting task
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
> 2022-09-09 17:03:07,687 DEBUG [main-EventThread]
> coordination.SplitLogManagerCoordination - task not yet
[jira] [Updated] (HBASE-27382) Cluster completely down due to WAL splitting failing for hbase:meta table.
[
https://issues.apache.org/jira/browse/HBASE-27382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh Shah updated HBASE-27382:
-
Description:
We are running some version of 1.7.2 in our production environment. We
encountered this issue recently.
We colocate namenode and region server holding hbase:meta table on a set of 5
master nodes. Co-incidentally active namenode and region server holding meta
table were on the same physical node and that node went down due to hardware
issue. We have sub optimal hdfs level timeouts configured so whenever active
namenode goes down, it takes around 12-15 minutes for hdfs client within hbase
to connect to new active namenode. So all the region servers were having
problems for about 15 minutes to connect to new active namenode.
Below are the sequence of events:
1. Host running active namenode and hbase:meta went down at +2022-09-09
16:56:56,878+
2. HMaster started running ServerCrashProcedure at +2022-09-09 16:59:05,696+
{noformat}
2022-09-09 16:59:05,696 DEBUG [t-processor-pool2-t1]
procedure2.ProcedureExecutor - Procedure ServerCrashProcedure
serverName=,61020,1662714013670, shouldSplitWal=true,
carryingMeta=true id=1 owner=dummy state=RUNNABLE:SERVER_CRASH_START added to
the store.
2022-09-09 16:59:05,702 DEBUG [t-processor-pool2-t1] master.ServerManager -
Added=,61020,1662714013670 to dead servers, submitted shutdown
handler to be executed meta=true
2022-09-09 16:59:05,707 DEBUG [ProcedureExecutor-0] master.DeadServer - Started
processing ,61020,1662714013670; numProcessing=1
2022-09-09 16:59:05,712 INFO [ProcedureExecutor-0]
procedure.ServerCrashProcedure - Start processing crashed
,61020,1662714013670
{noformat}
3. SplitLogManager created 2 split log tasks in zookeeper.
{noformat}
2022-09-09 16:59:06,049 INFO [ProcedureExecutor-1] master.SplitLogManager -
Started splitting 2 logs in
[hdfs:///hbase/WALs/,61020,1662714013670-splitting]
for [,61020,1662714013670]
2022-09-09 16:59:06,081 DEBUG [main-EventThread]
coordination.SplitLogManagerCoordination - put up splitlog task at znode
/hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
2022-09-09 16:59:06,093 DEBUG [main-EventThread]
coordination.SplitLogManagerCoordination - put up splitlog task at znode
/hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662739251611.meta
{noformat}
4. The first split log task is more interesting:
+/hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta+
5. Since all the region servers were having problems connecting to active
namenode, SplitLogManager tried total of 4 times to assign this task (3
resubmits, configured by hbase.splitlog.max.resubmit) and then finally gave up.
{noformat}
-- try 1 -
2022-09-09 16:59:06,205 INFO [main-EventThread]
coordination.SplitLogManagerCoordination - task
/hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
acquired by ,61020,1662540522069
-- try 2 -
2022-09-09 17:01:06,642 INFO [ager__ChoreService_1]
coordination.SplitLogManagerCoordination - resubmitting task
/hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
2022-09-09 17:01:06,666 DEBUG [main-EventThread]
coordination.SplitLogManagerCoordination - task not yet acquired
/hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
ver = 2
2022-09-09 17:01:06,715 INFO [main-EventThread]
coordination.SplitLogManagerCoordination - task
/hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
acquired by ,61020,1662530684713
-- try 3 -
2022-09-09 17:03:07,643 INFO [ager__ChoreService_1]
coordination.SplitLogManagerCoordination - resubmitting task
/hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
2022-09-09 17:03:07,687 DEBUG [main-EventThread]
coordination.SplitLogManagerCoordination - task not yet acquired
/hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
ver = 4
2022-09-09 17:03:07,738 INFO [main-EventThread]
coordination.SplitLogManagerCoordination - task
/hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
acquired by ,61020,1662542355806
-- try 4 -
2022-09-09 17:05:08,684 INFO [ager__ChoreService_1]
coordination.SplitLogManagerCoordination - resubmitting task
/hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
[jira] [Created] (HBASE-27382) Cluster completely down due to wal splitting failing for hbase:meta table.
Rushabh Shah created HBASE-27382:
Summary: Cluster completely down due to wal splitting failing for
hbase:meta table.
Key: HBASE-27382
URL: https://issues.apache.org/jira/browse/HBASE-27382
Project: HBase
Issue Type: Bug
Affects Versions: 1.7.2
Reporter: Rushabh Shah
Assignee: Rushabh Shah
We are running some version of 1.7.2 in our production environment. We
encountered this issue recently.
We colocate namenode and region server holding hbase:meta table on a set of 5
master nodes. Co-incidentally active namenode and region server holding meta
table were on the same physical node and that node went down due to hardware
issue. We have sub optimal hdfs level timeouts configured so whenever active
namenode goes down, it takes around 12-15 minutes for hdfs client within hbase
to connect to new active namenode. So all the region servers were having
problems for about 15 minutes to connect to new active namenode.
Below are the sequence of events:
1. Host running active namenode and hbase:meta went down at +2022-09-09
16:56:56,878_
2. HMaster started running ServerCrashProcedure at +2022-09-09 16:59:05,696+
{noformat}
2022-09-09 16:59:05,696 DEBUG [t-processor-pool2-t1]
procedure2.ProcedureExecutor - Procedure ServerCrashProcedure
serverName=,61020,1662714013670, shouldSplitWal=true,
carryingMeta=true id=1 owner=dummy state=RUNNABLE:SERVER_CRASH_START added to
the store.
2022-09-09 16:59:05,702 DEBUG [t-processor-pool2-t1] master.ServerManager -
Added=,61020,1662714013670 to dead servers, submitted shutdown
handler to be executed meta=true
2022-09-09 16:59:05,707 DEBUG [ProcedureExecutor-0] master.DeadServer - Started
processing ,61020,1662714013670; numProcessing=1
2022-09-09 16:59:05,712 INFO [ProcedureExecutor-0]
procedure.ServerCrashProcedure - Start processing crashed
,61020,1662714013670
{noformat}
3. SplitLogManager created 2 split log tasks in zookeeper.
{noformat}
2022-09-09 16:59:06,049 INFO [ProcedureExecutor-1] master.SplitLogManager -
Started splitting 2 logs in
[hdfs:///hbase/WALs/,61020,1662714013670-splitting]
for [,61020,1662714013670]
2022-09-09 16:59:06,081 DEBUG [main-EventThread]
coordination.SplitLogManagerCoordination - put up splitlog task at znode
/hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
2022-09-09 16:59:06,093 DEBUG [main-EventThread]
coordination.SplitLogManagerCoordination - put up splitlog task at znode
/hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662739251611.meta
{noformat}
4. The first split log task is more interesting:
+/hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta+
5. Since all the region servers were having problems connecting to active
namenode, SplitLogManager tried total of 4 times to assign this task (3
resubmits, configured by hbase.splitlog.max.resubmit) and then finally gave up.
{noformat}
-- try 1 -
2022-09-09 16:59:06,205 INFO [main-EventThread]
coordination.SplitLogManagerCoordination - task
/hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
acquired by ,61020,1662540522069
-- try 2 -
2022-09-09 17:01:06,642 INFO [ager__ChoreService_1]
coordination.SplitLogManagerCoordination - resubmitting task
/hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
2022-09-09 17:01:06,666 DEBUG [main-EventThread]
coordination.SplitLogManagerCoordination - task not yet acquired
/hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
ver = 2
2022-09-09 17:01:06,715 INFO [main-EventThread]
coordination.SplitLogManagerCoordination - task
/hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
acquired by ,61020,1662530684713
-- try 3 -
2022-09-09 17:03:07,643 INFO [ager__ChoreService_1]
coordination.SplitLogManagerCoordination - resubmitting task
/hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
2022-09-09 17:03:07,687 DEBUG [main-EventThread]
coordination.SplitLogManagerCoordination - task not yet acquired
/hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
ver = 4
2022-09-09 17:03:07,738 INFO [main-EventThread]
coordination.SplitLogManagerCoordination - task
/hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
acquired by ,61020,1662542355806
-- try 4 -
2022-09-09
[jira] [Updated] (HBASE-27382) Cluster completely down due to WAL splitting failing for hbase:meta table.
[
https://issues.apache.org/jira/browse/HBASE-27382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh Shah updated HBASE-27382:
-
Summary: Cluster completely down due to WAL splitting failing for
hbase:meta table. (was: Cluster completely down due to wal splitting failing
for hbase:meta table.)
> Cluster completely down due to WAL splitting failing for hbase:meta table.
> --
>
> Key: HBASE-27382
> URL: https://issues.apache.org/jira/browse/HBASE-27382
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.7.2
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
>
> We are running some version of 1.7.2 in our production environment. We
> encountered this issue recently.
> We colocate namenode and region server holding hbase:meta table on a set of 5
> master nodes. Co-incidentally active namenode and region server holding meta
> table were on the same physical node and that node went down due to hardware
> issue. We have sub optimal hdfs level timeouts configured so whenever active
> namenode goes down, it takes around 12-15 minutes for hdfs client within
> hbase to connect to new active namenode. So all the region servers were
> having problems for about 15 minutes to connect to new active namenode.
> Below are the sequence of events:
> 1. Host running active namenode and hbase:meta went down at +2022-09-09
> 16:56:56,878_
> 2. HMaster started running ServerCrashProcedure at +2022-09-09 16:59:05,696+
> {noformat}
> 2022-09-09 16:59:05,696 DEBUG [t-processor-pool2-t1]
> procedure2.ProcedureExecutor - Procedure ServerCrashProcedure
> serverName=,61020,1662714013670, shouldSplitWal=true,
> carryingMeta=true id=1 owner=dummy state=RUNNABLE:SERVER_CRASH_START added to
> the store.
> 2022-09-09 16:59:05,702 DEBUG [t-processor-pool2-t1] master.ServerManager -
> Added=,61020,1662714013670 to dead servers, submitted shutdown
> handler to be executed meta=true
> 2022-09-09 16:59:05,707 DEBUG [ProcedureExecutor-0] master.DeadServer -
> Started processing ,61020,1662714013670; numProcessing=1
> 2022-09-09 16:59:05,712 INFO [ProcedureExecutor-0]
> procedure.ServerCrashProcedure - Start processing crashed
> ,61020,1662714013670
> {noformat}
> 3. SplitLogManager created 2 split log tasks in zookeeper.
> {noformat}
> 2022-09-09 16:59:06,049 INFO [ProcedureExecutor-1] master.SplitLogManager -
> Started splitting 2 logs in
> [hdfs:///hbase/WALs/,61020,1662714013670-splitting]
> for [,61020,1662714013670]
> 2022-09-09 16:59:06,081 DEBUG [main-EventThread]
> coordination.SplitLogManagerCoordination - put up splitlog task at znode
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
> 2022-09-09 16:59:06,093 DEBUG [main-EventThread]
> coordination.SplitLogManagerCoordination - put up splitlog task at znode
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662739251611.meta
> {noformat}
> 4. The first split log task is more interesting:
> +/hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta+
> 5. Since all the region servers were having problems connecting to active
> namenode, SplitLogManager tried total of 4 times to assign this task (3
> resubmits, configured by hbase.splitlog.max.resubmit) and then finally gave
> up.
> {noformat}
> -- try 1 -
> 2022-09-09 16:59:06,205 INFO [main-EventThread]
> coordination.SplitLogManagerCoordination - task
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
> acquired by ,61020,1662540522069
> -- try 2 -
> 2022-09-09 17:01:06,642 INFO [ager__ChoreService_1]
> coordination.SplitLogManagerCoordination - resubmitting task
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
> 2022-09-09 17:01:06,666 DEBUG [main-EventThread]
> coordination.SplitLogManagerCoordination - task not yet acquired
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
> ver = 2
> 2022-09-09 17:01:06,715 INFO [main-EventThread]
> coordination.SplitLogManagerCoordination - task
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
> acquired by ,61020,1662530684713
> -- try 3 -
> 2022-09-09 17:03:07,643 INFO [ager__ChoreService_1]
> coordination.SplitLogManagerCoordination - resubmitting task
> /hbase/splitWAL/WALs%2F%2C61020%2C1662714013670-splitting%2F%252C61020%252C1662714013670.meta.1662735651285.meta
> 2022-09-09 17:03:07,687
[jira] [Commented] (HBASE-26913) Replication Observability Framework
[ https://issues.apache.org/jira/browse/HBASE-26913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17556980#comment-17556980 ] Rushabh Shah commented on HBASE-26913: -- Thank you [~vjasani] for merging all the sub-tasks for this feature. What should be the fix versions for the sub-tasks ? Since all the sub-tasks are merged to feature branch, should we close them with HBASE-26913 as the fix version ? Please advise. > Replication Observability Framework > --- > > Key: HBASE-26913 > URL: https://issues.apache.org/jira/browse/HBASE-26913 > Project: HBase > Issue Type: New Feature > Components: regionserver, Replication >Reporter: Rushabh Shah >Assignee: Rushabh Shah >Priority: Major > > In our production clusters, we have seen cases where data is present in > source cluster but not in the sink cluster and 1 case where data is present > in sink cluster but not in source cluster. > We have internal tools where we take incremental backup every day on both > source and sink clusters and we compare the hash of the data in both the > backups. We have seen many cases where hash doesn't match which means data is > not consistent between source and sink for that given day. The Mean Time To > Detect (MTTD) these inconsistencies is atleast 2 days and requires lot of > manual debugging. > We need some tool where we can reduce MTTD and requires less manual debugging. > I have attached design doc. Huge thanks to [~bharathv] to come up with this > design at my work place. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (HBASE-27100) Add documentation for Replication Observability Framework in hbase book.
Rushabh Shah created HBASE-27100: Summary: Add documentation for Replication Observability Framework in hbase book. Key: HBASE-27100 URL: https://issues.apache.org/jira/browse/HBASE-27100 Project: HBase Issue Type: Sub-task Components: documentation Reporter: Rushabh Shah Assignee: Rushabh Shah -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HBASE-27085) Create REPLICATION_SINK_TRACKER table to persist sentinel rows coming from source cluster.
[ https://issues.apache.org/jira/browse/HBASE-27085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17545656#comment-17545656 ] Rushabh Shah commented on HBASE-27085: -- Cc [~vjasani] > Create REPLICATION_SINK_TRACKER table to persist sentinel rows coming from > source cluster. > -- > > Key: HBASE-27085 > URL: https://issues.apache.org/jira/browse/HBASE-27085 > Project: HBase > Issue Type: Sub-task >Reporter: Rushabh Shah >Assignee: Rushabh Shah >Priority: Major > > This work is to create sink tracker table to persist tracker rows coming from > replication source cluster. > Create ReplicationMarkerChore to create replication marker rows periodically. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (HBASE-27085) Create REPLICATION_SINK_TRACKER table to persist sentinel rows coming from source cluster.
Rushabh Shah created HBASE-27085: Summary: Create REPLICATION_SINK_TRACKER table to persist sentinel rows coming from source cluster. Key: HBASE-27085 URL: https://issues.apache.org/jira/browse/HBASE-27085 Project: HBase Issue Type: Sub-task Reporter: Rushabh Shah Assignee: Rushabh Shah This work is to create sink tracker table to persist tracker rows coming from replication source cluster. Create ReplicationMarkerChore to create replication marker rows periodically. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (HBASE-26905) ReplicationPeerManager#checkPeerExists should throw ReplicationPeerNotFoundException if peer doesn't exists
[
https://issues.apache.org/jira/browse/HBASE-26905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh Shah reassigned HBASE-26905:
Assignee: Rushabh Shah
> ReplicationPeerManager#checkPeerExists should throw
> ReplicationPeerNotFoundException if peer doesn't exists
> ---
>
> Key: HBASE-26905
> URL: https://issues.apache.org/jira/browse/HBASE-26905
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
>
> ReplicationPeerManager#checkPeerExists should throw
> ReplicationPeerNotFoundException if peer doesn't exists. Currently it throws
> generic DoNotRetryIOException.
> {code:java}
> private ReplicationPeerDescription checkPeerExists(String peerId) throws
> DoNotRetryIOException {
> ReplicationPeerDescription desc = peers.get(peerId);
> if (desc == null) {
> throw new DoNotRetryIOException("Replication peer " + peerId + " does not
> exist");
> }
> return desc;
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (HBASE-26953) CallDroppedException message says "server 0.0.0.0 is overloaded"
[ https://issues.apache.org/jira/browse/HBASE-26953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh Shah updated HBASE-26953: - Labels: newbie (was: ) > CallDroppedException message says "server 0.0.0.0 is overloaded" > > > Key: HBASE-26953 > URL: https://issues.apache.org/jira/browse/HBASE-26953 > Project: HBase > Issue Type: Bug >Reporter: Bryan Beaudreault >Assignee: Ayesha Mosaddeque >Priority: Trivial > Labels: newbie > > This is not a super useful error when seen on the client side. Maybe we can > have it pull in the ServerName instead, or at the very least resolve to the > hostname. > > https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallRunner.java#L213 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (HBASE-26963) ReplicationSource#removePeer hangs if we try to remove bad peer.
[
https://issues.apache.org/jira/browse/HBASE-26963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh Shah updated HBASE-26963:
-
Affects Version/s: 3.0.0-alpha-2
2.5.0
> ReplicationSource#removePeer hangs if we try to remove bad peer.
>
>
> Key: HBASE-26963
> URL: https://issues.apache.org/jira/browse/HBASE-26963
> Project: HBase
> Issue Type: Bug
> Components: regionserver, Replication
>Affects Versions: 2.5.0, 3.0.0-alpha-2, 2.4.11
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Attachments: HBASE-26963.patch
>
>
> ReplicationSource#removePeer hangs if we try to remove bad peer.
> Steps to reproduce:
> 1. Set config replication.source.regionserver.abort to false so that it
> doesn't abort regionserver.
> 2. Add a dummy peer.
> 2. Remove that peer.
> RemovePeer call will hang indefinitely until the test times out.
> Attached a patch to reproduce the above behavior.
> I can see following threads in the stack trace:
> {noformat}
> "RS_REFRESH_PEER-regionserver/rushabh-ltmflld:0-0.replicationSource,dummypeer_1"
> #339 daemon prio=5 os_prio=31 tid=0x7f8caa
> 44a800 nid=0x22107 waiting on condition [0x7000107e5000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.sleepForRetries(ReplicationSource.java:511)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.initialize(ReplicationSource.java:577)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.lambda$startup$4(ReplicationSource.java:633)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$$Lambda$350/89698794.uncaughtException(Unknown
> Source)
> at java.lang.Thread.dispatchUncaughtException(Thread.java:1959)
> {noformat}
> {noformat}
> "RS_REFRESH_PEER-regionserver/rushabh-ltmflld:0-0" #338 daemon prio=5
> os_prio=31 tid=0x7f8ca82fa800 nid=0x22307 in Object.wait()
> [0x7000106e2000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Thread.join(Thread.java:1260)
> - locked <0x000799975ea0> (a java.lang.Thread)
> at org.apache.hadoop.hbase.util.Threads.shutdown(Threads.java:106)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:674)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:657)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:652)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:647)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.removePeer(ReplicationSourceManager.java:330)
> at
> org.apache.hadoop.hbase.replication.regionserver.PeerProcedureHandlerImpl.removePeer(PeerProcedureHandlerImpl.java:56)
> at
> org.apache.hadoop.hbase.replication.regionserver.RefreshPeerCallable.call(RefreshPeerCallable.java:61)
> at
> org.apache.hadoop.hbase.replication.regionserver.RefreshPeerCallable.call(RefreshPeerCallable.java:35)
> at
> org.apache.hadoop.hbase.regionserver.handler.RSProcedureHandler.process(RSProcedureHandler.java:49)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> {noformat}
> "Listener at localhost/55013" #20 daemon prio=5 os_prio=31
> tid=0x7f8caf95a000 nid=0x6703 waiting on condition [0x72
> 544000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:3442)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:3372)
> at org.apache.hadoop.hbase.util.FutureUtils.get(FutureUtils.java:182)
> at
> org.apache.hadoop.hbase.client.Admin.removeReplicationPeer(Admin.java:2861)
> at
> org.apache.hadoop.hbase.client.replication.TestBadReplicationPeer.cleanPeer(TestBadReplicationPeer.java:74)
> at
> org.apache.hadoop.hbase.client.replication.TestBadReplicationPeer.testWrongReplicationEndpoint(TestBadReplicationPeer.java:66)
>
[jira] [Assigned] (HBASE-26963) ReplicationSource#removePeer hangs if we try to remove bad peer.
[
https://issues.apache.org/jira/browse/HBASE-26963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh Shah reassigned HBASE-26963:
Assignee: Rushabh Shah
> ReplicationSource#removePeer hangs if we try to remove bad peer.
>
>
> Key: HBASE-26963
> URL: https://issues.apache.org/jira/browse/HBASE-26963
> Project: HBase
> Issue Type: Bug
> Components: regionserver, Replication
>Affects Versions: 2.4.11
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Attachments: HBASE-26963.patch
>
>
> ReplicationSource#removePeer hangs if we try to remove bad peer.
> Steps to reproduce:
> 1. Set config replication.source.regionserver.abort to false so that it
> doesn't abort regionserver.
> 2. Add a dummy peer.
> 2. Remove that peer.
> RemovePeer call will hang indefinitely until the test times out.
> Attached a patch to reproduce the above behavior.
> I can see following threads in the stack trace:
> {noformat}
> "RS_REFRESH_PEER-regionserver/rushabh-ltmflld:0-0.replicationSource,dummypeer_1"
> #339 daemon prio=5 os_prio=31 tid=0x7f8caa
> 44a800 nid=0x22107 waiting on condition [0x7000107e5000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.sleepForRetries(ReplicationSource.java:511)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.initialize(ReplicationSource.java:577)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.lambda$startup$4(ReplicationSource.java:633)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$$Lambda$350/89698794.uncaughtException(Unknown
> Source)
> at java.lang.Thread.dispatchUncaughtException(Thread.java:1959)
> {noformat}
> {noformat}
> "RS_REFRESH_PEER-regionserver/rushabh-ltmflld:0-0" #338 daemon prio=5
> os_prio=31 tid=0x7f8ca82fa800 nid=0x22307 in Object.wait()
> [0x7000106e2000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Thread.join(Thread.java:1260)
> - locked <0x000799975ea0> (a java.lang.Thread)
> at org.apache.hadoop.hbase.util.Threads.shutdown(Threads.java:106)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:674)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:657)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:652)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:647)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.removePeer(ReplicationSourceManager.java:330)
> at
> org.apache.hadoop.hbase.replication.regionserver.PeerProcedureHandlerImpl.removePeer(PeerProcedureHandlerImpl.java:56)
> at
> org.apache.hadoop.hbase.replication.regionserver.RefreshPeerCallable.call(RefreshPeerCallable.java:61)
> at
> org.apache.hadoop.hbase.replication.regionserver.RefreshPeerCallable.call(RefreshPeerCallable.java:35)
> at
> org.apache.hadoop.hbase.regionserver.handler.RSProcedureHandler.process(RSProcedureHandler.java:49)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> {noformat}
> "Listener at localhost/55013" #20 daemon prio=5 os_prio=31
> tid=0x7f8caf95a000 nid=0x6703 waiting on condition [0x72
> 544000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:3442)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:3372)
> at org.apache.hadoop.hbase.util.FutureUtils.get(FutureUtils.java:182)
> at
> org.apache.hadoop.hbase.client.Admin.removeReplicationPeer(Admin.java:2861)
> at
> org.apache.hadoop.hbase.client.replication.TestBadReplicationPeer.cleanPeer(TestBadReplicationPeer.java:74)
> at
> org.apache.hadoop.hbase.client.replication.TestBadReplicationPeer.testWrongReplicationEndpoint(TestBadReplicationPeer.java:66)
> {noformat}
> The main thread
[jira] [Updated] (HBASE-26963) ReplicationSource#removePeer hangs if we try to remove bad peer.
[
https://issues.apache.org/jira/browse/HBASE-26963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh Shah updated HBASE-26963:
-
Attachment: HBASE-26963.patch
> ReplicationSource#removePeer hangs if we try to remove bad peer.
>
>
> Key: HBASE-26963
> URL: https://issues.apache.org/jira/browse/HBASE-26963
> Project: HBase
> Issue Type: Bug
> Components: regionserver, Replication
>Affects Versions: 2.4.11
>Reporter: Rushabh Shah
>Priority: Major
> Attachments: HBASE-26963.patch
>
>
> ReplicationSource#removePeer hangs if we try to remove bad peer.
> Steps to reproduce:
> 1. Set config replication.source.regionserver.abort to false so that it
> doesn't abort regionserver.
> 2. Add a dummy peer.
> 2. Remove that peer.
> RemovePeer call will hang indefinitely until the test times out.
> Attached a patch to reproduce the above behavior.
> I can see following threads in the stack trace:
> {noformat}
> "RS_REFRESH_PEER-regionserver/rushabh-ltmflld:0-0.replicationSource,dummypeer_1"
> #339 daemon prio=5 os_prio=31 tid=0x7f8caa
> 44a800 nid=0x22107 waiting on condition [0x7000107e5000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.sleepForRetries(ReplicationSource.java:511)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.initialize(ReplicationSource.java:577)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.lambda$startup$4(ReplicationSource.java:633)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$$Lambda$350/89698794.uncaughtException(Unknown
> Source)
> at java.lang.Thread.dispatchUncaughtException(Thread.java:1959)
> {noformat}
> {noformat}
> "RS_REFRESH_PEER-regionserver/rushabh-ltmflld:0-0" #338 daemon prio=5
> os_prio=31 tid=0x7f8ca82fa800 nid=0x22307 in Object.wait()
> [0x7000106e2000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Thread.join(Thread.java:1260)
> - locked <0x000799975ea0> (a java.lang.Thread)
> at org.apache.hadoop.hbase.util.Threads.shutdown(Threads.java:106)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:674)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:657)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:652)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:647)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.removePeer(ReplicationSourceManager.java:330)
> at
> org.apache.hadoop.hbase.replication.regionserver.PeerProcedureHandlerImpl.removePeer(PeerProcedureHandlerImpl.java:56)
> at
> org.apache.hadoop.hbase.replication.regionserver.RefreshPeerCallable.call(RefreshPeerCallable.java:61)
> at
> org.apache.hadoop.hbase.replication.regionserver.RefreshPeerCallable.call(RefreshPeerCallable.java:35)
> at
> org.apache.hadoop.hbase.regionserver.handler.RSProcedureHandler.process(RSProcedureHandler.java:49)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> {noformat}
> "Listener at localhost/55013" #20 daemon prio=5 os_prio=31
> tid=0x7f8caf95a000 nid=0x6703 waiting on condition [0x72
> 544000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:3442)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:3372)
> at org.apache.hadoop.hbase.util.FutureUtils.get(FutureUtils.java:182)
> at
> org.apache.hadoop.hbase.client.Admin.removeReplicationPeer(Admin.java:2861)
> at
> org.apache.hadoop.hbase.client.replication.TestBadReplicationPeer.cleanPeer(TestBadReplicationPeer.java:74)
> at
> org.apache.hadoop.hbase.client.replication.TestBadReplicationPeer.testWrongReplicationEndpoint(TestBadReplicationPeer.java:66)
> {noformat}
> The main thread "TestBadReplicationPeer.testWrongReplicationEndpoint" is
> waiting
[jira] [Updated] (HBASE-26963) ReplicationSource#removePeer hangs if we try to remove bad peer.
[
https://issues.apache.org/jira/browse/HBASE-26963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh Shah updated HBASE-26963:
-
Component/s: regionserver
Replication
> ReplicationSource#removePeer hangs if we try to remove bad peer.
>
>
> Key: HBASE-26963
> URL: https://issues.apache.org/jira/browse/HBASE-26963
> Project: HBase
> Issue Type: Bug
> Components: regionserver, Replication
>Affects Versions: 2.4.11
>Reporter: Rushabh Shah
>Priority: Major
>
> ReplicationSource#removePeer hangs if we try to remove bad peer.
> Steps to reproduce:
> 1. Set config replication.source.regionserver.abort to false so that it
> doesn't abort regionserver.
> 2. Add a dummy peer.
> 2. Remove that peer.
> RemovePeer call will hang indefinitely until the test times out.
> Attached a patch to reproduce the above behavior.
> I can see following threads in the stack trace:
> {noformat}
> "RS_REFRESH_PEER-regionserver/rushabh-ltmflld:0-0.replicationSource,dummypeer_1"
> #339 daemon prio=5 os_prio=31 tid=0x7f8caa
> 44a800 nid=0x22107 waiting on condition [0x7000107e5000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.sleepForRetries(ReplicationSource.java:511)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.initialize(ReplicationSource.java:577)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.lambda$startup$4(ReplicationSource.java:633)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$$Lambda$350/89698794.uncaughtException(Unknown
> Source)
> at java.lang.Thread.dispatchUncaughtException(Thread.java:1959)
> {noformat}
> {noformat}
> "RS_REFRESH_PEER-regionserver/rushabh-ltmflld:0-0" #338 daemon prio=5
> os_prio=31 tid=0x7f8ca82fa800 nid=0x22307 in Object.wait()
> [0x7000106e2000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Thread.join(Thread.java:1260)
> - locked <0x000799975ea0> (a java.lang.Thread)
> at org.apache.hadoop.hbase.util.Threads.shutdown(Threads.java:106)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:674)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:657)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:652)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:647)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.removePeer(ReplicationSourceManager.java:330)
> at
> org.apache.hadoop.hbase.replication.regionserver.PeerProcedureHandlerImpl.removePeer(PeerProcedureHandlerImpl.java:56)
> at
> org.apache.hadoop.hbase.replication.regionserver.RefreshPeerCallable.call(RefreshPeerCallable.java:61)
> at
> org.apache.hadoop.hbase.replication.regionserver.RefreshPeerCallable.call(RefreshPeerCallable.java:35)
> at
> org.apache.hadoop.hbase.regionserver.handler.RSProcedureHandler.process(RSProcedureHandler.java:49)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> {noformat}
> "Listener at localhost/55013" #20 daemon prio=5 os_prio=31
> tid=0x7f8caf95a000 nid=0x6703 waiting on condition [0x72
> 544000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:3442)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:3372)
> at org.apache.hadoop.hbase.util.FutureUtils.get(FutureUtils.java:182)
> at
> org.apache.hadoop.hbase.client.Admin.removeReplicationPeer(Admin.java:2861)
> at
> org.apache.hadoop.hbase.client.replication.TestBadReplicationPeer.cleanPeer(TestBadReplicationPeer.java:74)
> at
> org.apache.hadoop.hbase.client.replication.TestBadReplicationPeer.testWrongReplicationEndpoint(TestBadReplicationPeer.java:66)
> {noformat}
> The main thread "TestBadReplicationPeer.testWrongReplicationEndpoint" is
> waiting for
[jira] [Created] (HBASE-26963) ReplicationSource#removePeer hangs if we try to remove bad peer.
Rushabh Shah created HBASE-26963:
Summary: ReplicationSource#removePeer hangs if we try to remove
bad peer.
Key: HBASE-26963
URL: https://issues.apache.org/jira/browse/HBASE-26963
Project: HBase
Issue Type: Bug
Affects Versions: 2.4.11
Reporter: Rushabh Shah
ReplicationSource#removePeer hangs if we try to remove bad peer.
Steps to reproduce:
1. Set config replication.source.regionserver.abort to false so that it doesn't
abort regionserver.
2. Add a dummy peer.
2. Remove that peer.
RemovePeer call will hang indefinitely until the test times out.
Attached a patch to reproduce the above behavior.
I can see following threads in the stack trace:
{noformat}
"RS_REFRESH_PEER-regionserver/rushabh-ltmflld:0-0.replicationSource,dummypeer_1"
#339 daemon prio=5 os_prio=31 tid=0x7f8caa
44a800 nid=0x22107 waiting on condition [0x7000107e5000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.sleepForRetries(ReplicationSource.java:511)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.initialize(ReplicationSource.java:577)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.lambda$startup$4(ReplicationSource.java:633)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$$Lambda$350/89698794.uncaughtException(Unknown
Source)
at java.lang.Thread.dispatchUncaughtException(Thread.java:1959)
{noformat}
{noformat}
"RS_REFRESH_PEER-regionserver/rushabh-ltmflld:0-0" #338 daemon prio=5
os_prio=31 tid=0x7f8ca82fa800 nid=0x22307 in Object.wait()
[0x7000106e2000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1260)
- locked <0x000799975ea0> (a java.lang.Thread)
at org.apache.hadoop.hbase.util.Threads.shutdown(Threads.java:106)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:674)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:657)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:652)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.terminate(ReplicationSource.java:647)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.removePeer(ReplicationSourceManager.java:330)
at
org.apache.hadoop.hbase.replication.regionserver.PeerProcedureHandlerImpl.removePeer(PeerProcedureHandlerImpl.java:56)
at
org.apache.hadoop.hbase.replication.regionserver.RefreshPeerCallable.call(RefreshPeerCallable.java:61)
at
org.apache.hadoop.hbase.replication.regionserver.RefreshPeerCallable.call(RefreshPeerCallable.java:35)
at
org.apache.hadoop.hbase.regionserver.handler.RSProcedureHandler.process(RSProcedureHandler.java:49)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
{noformat}
"Listener at localhost/55013" #20 daemon prio=5 os_prio=31
tid=0x7f8caf95a000 nid=0x6703 waiting on condition [0x72
544000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:3442)
at
org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:3372)
at org.apache.hadoop.hbase.util.FutureUtils.get(FutureUtils.java:182)
at
org.apache.hadoop.hbase.client.Admin.removeReplicationPeer(Admin.java:2861)
at
org.apache.hadoop.hbase.client.replication.TestBadReplicationPeer.cleanPeer(TestBadReplicationPeer.java:74)
at
org.apache.hadoop.hbase.client.replication.TestBadReplicationPeer.testWrongReplicationEndpoint(TestBadReplicationPeer.java:66)
{noformat}
The main thread "TestBadReplicationPeer.testWrongReplicationEndpoint" is
waiting for Admin#removeReplicationPeer.
The refreshPeer thread (PeerProcedureHandlerImpl#removePeer) responsible to
terminate peer (#338) is waiting on ReplicationSource thread to be terminated.
The ReplicateSource thread (#339) is in sleeping state. Notice that this
thread's stack trace is in ReplicationSource#uncaughtException method.
When we call ReplicationSourceManager#removePeer, we set sourceRunning flag to
false, send an interrupt signal to ReplicationSource thread
[jira] [Created] (HBASE-26957) Add support to hbase shell to remove coproc by its class name instead of coproc ID
Rushabh Shah created HBASE-26957: Summary: Add support to hbase shell to remove coproc by its class name instead of coproc ID Key: HBASE-26957 URL: https://issues.apache.org/jira/browse/HBASE-26957 Project: HBase Issue Type: Bug Components: Coprocessors, shell Affects Versions: 1.7.1 Reporter: Rushabh Shah The syntax for removing coproc is as below: hbase> alter 't1', METHOD => 'table_att_unset', NAME => 'coprocessor$1' We have to use coproc id to remove a coproc from a given table. Consider the following scenario. Due to some bug in a coproc, we have to remove a given coproc from all the tables in a cluster. Every table can have different set of co-procs. For a given co-proc class, the coproc ID will not be same for all the tables in a cluster. This gets more complex if we want to remove co-proc from all the production clusters. Instead we can pass a co-proc class name to alter table command. So if a table has that co-proc, it will remove otherwise do nothing. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26925) Create WAL event tracker table to track all the WAL events.
Rushabh Shah created HBASE-26925: Summary: Create WAL event tracker table to track all the WAL events. Key: HBASE-26925 URL: https://issues.apache.org/jira/browse/HBASE-26925 Project: HBase Issue Type: Sub-task Components: wal Reporter: Rushabh Shah Assignee: Rushabh Shah Design Doc: [https://docs.google.com/document/d/14oZ5ssY28hvJaQD_Jg9kWX7LfUKUyyU2PCA93PPzVko/edit#] Create wal event tracker table to track WAL events. Whenever we roll the WAL, we will save the WAL name, WAL length, region server, timestamp in a table. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26913) Replication Observability Framework
[
https://issues.apache.org/jira/browse/HBASE-26913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh Shah updated HBASE-26913:
-
Description:
In our production clusters, we have seen cases where data is present in source
cluster but not in the sink cluster and 1 case where data is present in sink
cluster but not in source cluster.
We have internal tools where we take incremental backup every day on both
source and sink clusters and we compare the hash of the data in both the
backups. We have seen many cases where hash doesn't match which means data is
not consistent between source and sink for that given day. The Mean Time To
Detect (MTTD) these inconsistencies is atleast 2 days and requires lot of
manual debugging.
We need some tool where we can reduce MTTD and requires less manual debugging.
I have attached design doc. Huge thanks to [~bharathv] to come up with this
design at my work place.
was:
{*}{*}In our production clusters, we have seen cases where data is present in
source cluster but not in the sink cluster and 1 case where data is present in
sink cluster but not in source cluster.
We have internal tools where we take incremental backup every day on both
source and sink clusters and we compare the hash of the data in both the
backups. We have seen many cases where hash doesn't match which means data is
not consistent between source and sink for that given day. The Mean Time To
Detect (MTTD) these inconsistencies is atleast 2 days and requires lot of
manual debugging.
We need some tool where we can reduce MTTD and requires less manual debugging.
I have attached design doc. Huge thanks to [~bharathv] to come up with this
design at my work place.
> Replication Observability Framework
> ---
>
> Key: HBASE-26913
> URL: https://issues.apache.org/jira/browse/HBASE-26913
> Project: HBase
> Issue Type: New Feature
> Components: regionserver, Replication
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
>
> In our production clusters, we have seen cases where data is present in
> source cluster but not in the sink cluster and 1 case where data is present
> in sink cluster but not in source cluster.
> We have internal tools where we take incremental backup every day on both
> source and sink clusters and we compare the hash of the data in both the
> backups. We have seen many cases where hash doesn't match which means data is
> not consistent between source and sink for that given day. The Mean Time To
> Detect (MTTD) these inconsistencies is atleast 2 days and requires lot of
> manual debugging.
> We need some tool where we can reduce MTTD and requires less manual debugging.
> I have attached design doc. Huge thanks to [~bharathv] to come up with this
> design at my work place.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26913) Replication Observability Framework
Rushabh Shah created HBASE-26913:
Summary: Replication Observability Framework
Key: HBASE-26913
URL: https://issues.apache.org/jira/browse/HBASE-26913
Project: HBase
Issue Type: New Feature
Components: regionserver, Replication
Reporter: Rushabh Shah
Assignee: Rushabh Shah
{*}{*}In our production clusters, we have seen cases where data is present in
source cluster but not in the sink cluster and 1 case where data is present in
sink cluster but not in source cluster.
We have internal tools where we take incremental backup every day on both
source and sink clusters and we compare the hash of the data in both the
backups. We have seen many cases where hash doesn't match which means data is
not consistent between source and sink for that given day. The Mean Time To
Detect (MTTD) these inconsistencies is atleast 2 days and requires lot of
manual debugging.
We need some tool where we can reduce MTTD and requires less manual debugging.
I have attached design doc. Huge thanks to [~bharathv] to come up with this
design at my work place.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26905) ReplicationPeerManager#checkPeerExists should throw ReplicationPeerNotFoundException if peer doesn't exists
Rushabh Shah created HBASE-26905:
Summary: ReplicationPeerManager#checkPeerExists should throw
ReplicationPeerNotFoundException if peer doesn't exists
Key: HBASE-26905
URL: https://issues.apache.org/jira/browse/HBASE-26905
Project: HBase
Issue Type: Bug
Components: Replication
Reporter: Rushabh Shah
ReplicationPeerManager#checkPeerExists should throw
ReplicationPeerNotFoundException if peer doesn't exists. Currently it throws
generic DoNotRetryIOException.
{code:java}
private ReplicationPeerDescription checkPeerExists(String peerId) throws
DoNotRetryIOException {
ReplicationPeerDescription desc = peers.get(peerId);
if (desc == null) {
throw new DoNotRetryIOException("Replication peer " + peerId + " does not
exist");
}
return desc;
} {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26792) Implement ScanInfo#toString
Rushabh Shah created HBASE-26792: Summary: Implement ScanInfo#toString Key: HBASE-26792 URL: https://issues.apache.org/jira/browse/HBASE-26792 Project: HBase Issue Type: Improvement Reporter: Rushabh Shah Assignee: Rushabh Shah We don't have ScanInfo#toString. We use ScanInfo while creating StoreScanner which is used in preFlushScannerOpen in co-proc. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26702) Make ageOfLastShip, ageOfLastApplied extend TimeHistogram instead of plain histogram.
[
https://issues.apache.org/jira/browse/HBASE-26702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489610#comment-17489610
]
Rushabh Shah commented on HBASE-26702:
--
Thank you [~zhangduo] for the review and commit !
> Make ageOfLastShip, ageOfLastApplied extend TimeHistogram instead of plain
> histogram.
> --
>
> Key: HBASE-26702
> URL: https://issues.apache.org/jira/browse/HBASE-26702
> Project: HBase
> Issue Type: Improvement
> Components: metrics, Replication
>Affects Versions: 1.7.1, 2.3.7, 3.0.0-alpha-3
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Minor
> Fix For: 2.5.0, 1.7.2, 3.0.0-alpha-3, 2.4.10
>
>
> Currently age of last ship metric is an instance of an Histogram type.
> [Here|https://github.com/apache/hbase/blob/master/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSourceImpl.java#L58]
> {quote}
>ageOfLastShippedOpHist =
> rms.getMetricsRegistry().getHistogram(SOURCE_AGE_OF_LAST_SHIPPED_OP);
> {quote}
> We can change it to TimeHistogram so that we get the range information also.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26702) Make ageOfLastShip, ageOfLastApplied extend TimeHistogram instead of plain histogram.
[
https://issues.apache.org/jira/browse/HBASE-26702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh Shah updated HBASE-26702:
-
Summary: Make ageOfLastShip, ageOfLastApplied extend TimeHistogram instead
of plain histogram. (was: Make ageOfLastShip extend TimeHistogram instead of
plain histogram.)
> Make ageOfLastShip, ageOfLastApplied extend TimeHistogram instead of plain
> histogram.
> --
>
> Key: HBASE-26702
> URL: https://issues.apache.org/jira/browse/HBASE-26702
> Project: HBase
> Issue Type: Improvement
> Components: metrics, Replication
>Affects Versions: 1.7.1, 2.3.7, 3.0.0-alpha-3
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Minor
>
> Currently age of last ship metric is an instance of an Histogram type.
> [Here|https://github.com/apache/hbase/blob/master/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSourceImpl.java#L58]
> {quote}
>ageOfLastShippedOpHist =
> rms.getMetricsRegistry().getHistogram(SOURCE_AGE_OF_LAST_SHIPPED_OP);
> {quote}
> We can change it to TimeHistogram so that we get the range information also.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26702) Make ageOfLastShip extend TimeHistogram instead of plain histogram.
Rushabh Shah created HBASE-26702:
Summary: Make ageOfLastShip extend TimeHistogram instead of plain
histogram.
Key: HBASE-26702
URL: https://issues.apache.org/jira/browse/HBASE-26702
Project: HBase
Issue Type: Improvement
Components: metrics, Replication
Affects Versions: 2.3.7, 1.7.1, 3.0.0-alpha-3
Reporter: Rushabh Shah
Assignee: Rushabh Shah
Currently age of last ship metric is an instance of an Histogram type.
[Here|https://github.com/apache/hbase/blob/master/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSourceImpl.java#L58]
{quote}
ageOfLastShippedOpHist =
rms.getMetricsRegistry().getHistogram(SOURCE_AGE_OF_LAST_SHIPPED_OP);
{quote}
We can change it to TimeHistogram so that we get the range information also.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26581) Add metrics around failed replication edits
[ https://issues.apache.org/jira/browse/HBASE-26581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh Shah updated HBASE-26581: - Component/s: metrics Replication > Add metrics around failed replication edits > --- > > Key: HBASE-26581 > URL: https://issues.apache.org/jira/browse/HBASE-26581 > Project: HBase > Issue Type: Improvement > Components: metrics, Replication >Reporter: Bryan Beaudreault >Priority: Minor > > We ran into HBASE-26575 recently, but had a hard time tracking down where the > edits were coming from. The only real evidence of a failure in replication > was the ageOfLastShipped metric. It would be nice to add an explicit metric > around source.failedBatches and sink.failedBatches, incremented whenever an > exception is encountered. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26468) Region Server doesn't exit cleanly incase it crashes.
[
https://issues.apache.org/jira/browse/HBASE-26468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451737#comment-17451737
]
Rushabh Shah commented on HBASE-26468:
--
Thank you [~vjasani] for the review and the merge !
Thank you [~zhangduo] [~gjacoby] for the review and feedback !
> Region Server doesn't exit cleanly incase it crashes.
> -
>
> Key: HBASE-26468
> URL: https://issues.apache.org/jira/browse/HBASE-26468
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.6.0
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2, 1.7.2, 2.4.9
>
>
> Observed this in our production cluster running 1.6 version.
> RS crashed due to some reason but the process was still running. On debugging
> more, found out there was 1 non-daemon thread running and that was not
> allowing RS to exit cleanly. Our clusters are managed by Ambari and have auto
> restart capability within them. But since the process was running and pid
> file was present, Ambari also couldn't do much. There will be some bug where
> we will miss to stop some non daemon thread. Shutdown hook will not be called
> unless one of the following 2 conditions are met:
> # The Java virtual machine shuts down in response to two kinds of events:
> The program exits normally, when the last non-daemon thread exits or when the
> exit (equivalently, System.exit) method is invoked, or
> # The virtual machine is terminated in response to a user interrupt, such as
> typing ^C, or a system-wide event, such as user logoff or system shutdown.
> Considering the first condition, when the last non-daemon thread exits or
> when the exit method is invoked.
> Below is the code snippet from
> [HRegionServerCommandLine.java|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServerCommandLine.java#L51]
> {code:java}
> private int start() throws Exception {
> try {
> if (LocalHBaseCluster.isLocal(conf)) {
> // Ignore this.
> } else {
> HRegionServer hrs =
> HRegionServer.constructRegionServer(regionServerClass, conf);
> hrs.start();
> hrs.join();
> if (hrs.isAborted()) {
> throw new RuntimeException("HRegionServer Aborted");
> }
> }
> } catch (Throwable t) {
> LOG.error("Region server exiting", t);
> return 1;
> }
> return 0;
> }
> {code}
> Within HRegionServer, there is a subtle difference between when a server is
> aborted v/s when it is stopped. If it is stopped, then isAborted will return
> false and it will exit with return code 0.
> Below is the code from
> [ServerCommandLine.java|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerCommandLine.java#L147]
> {code:java}
> public void doMain(String args[]) {
> try {
> int ret = ToolRunner.run(HBaseConfiguration.create(), this, args);
> if (ret != 0) {
> System.exit(ret);
> }
> } catch (Exception e) {
> LOG.error("Failed to run", e);
> System.exit(-1);
> }
> }
> {code}
> If return code is 0, then it won't call System.exit. This means JVM will wait
> to call ShutdownHook until all non daemon threads are stopped which means
> infinite wait if we don't close all non-daemon threads cleanly.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26480) Close NamedQueueRecorder to allow HMaster/RS to shutdown gracefully
[
https://issues.apache.org/jira/browse/HBASE-26480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17449234#comment-17449234
]
Rushabh Shah commented on HBASE-26480:
--
Thank you [~Xiaolin Ha] for the review and merge !
> Close NamedQueueRecorder to allow HMaster/RS to shutdown gracefully
> ---
>
> Key: HBASE-26480
> URL: https://issues.apache.org/jira/browse/HBASE-26480
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Fix For: 1.8.0
>
>
> Saw one case in our production cluster where RS was not exiting. Saw this
> non-daemon thread in hung RS stack trace:
> {noformat}
> "main.slowlog.append-pool-pool1-t1" #26 prio=5 os_prio=31
> tid=0x7faf23bf7800 nid=0x6d07 waiting on condition [0x73f4d000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0004039e3840> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:47)
> at
> com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
> at
> com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
>
> This is coming from
> [NamedQueueRecorder|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/namequeues/NamedQueueRecorder.java#L65]
> implementation.
> This bug doesn't exists in branch-2 and master since the Disruptor
> initialization has changed and we set daemon=true also. See [this
> code|https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/namequeues/NamedQueueRecorder.java#L68]
>
> FYI [~vjasani] [~zhangduo]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26482) HMaster may clean wals that is replicating in rare cases
[
https://issues.apache.org/jira/browse/HBASE-26482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448709#comment-17448709
]
Rushabh Shah commented on HBASE-26482:
--
[~zhengzhuobinzzb] Can this occur in branch-1 also ?
> HMaster may clean wals that is replicating in rare cases
>
>
> Key: HBASE-26482
> URL: https://issues.apache.org/jira/browse/HBASE-26482
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: zhuobin zheng
>Assignee: zhuobin zheng
>Priority: Critical
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.9
>
>
> In our cluster, i can found some FileNotFoundException when
> ReplicationSourceWALReader running for replication recovery queue.
> I guss the wal most likely removed by hmaste. And i found something to
> support it.
> The method getAllWALs:
> [https://github.com/apache/hbase/blob/master/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationQueueStorage.java#L509
>
> |https://github.com/apache/hbase/blob/master/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationQueueStorage.java#L509]Use
> zk cversion of /hbase/replication/rs as an optimistic lock to control
> concurrent ops.
> But, zk cversion *only can only reflect the changes of child nodes, but not
> the changes of grandchildren.*
> So, HMaster may loss some wal from this method in follow situation.
> # HMaster do log clean , and invoke getAllWALs to filter log which should
> not be deleted.
> # HMaster cache current cversion of /hbase/replication/rs as *v0*
> # HMaster cache all RS server name, and traverse them, get the WAL in each
> Queue
> # *RS2* dead after HMaster traverse {*}RS1{*}, and before traverse *RS2*
> # *RS1* claim one queue of *RS2,* which named *peerid-RS2* now
> # By the way , the cversion of /hbase/replication/rs not changed before all
> of *RS2* queue is removed, because the children of /hbase/replication/rs not
> change.
> # So, Hmaster will lost the wals in *peerid-RS2,* because we have already
> traversed *RS1 ,* and ** this queue not exists in *RS2*
> The above expression is currently only speculation, not confirmed
> Flie Not Found Log.
>
> {code:java}
> // code placeholder
> 2021-11-22 15:18:39,593 ERROR
> [ReplicationExecutor-0.replicationSource,peer_id-hostname,60020,1636802867348.replicationSource.wal-reader.hostname%2C60020%2C1636802867348,peer_id-hostname,60020,1636802867348]
> regionserver.WALEntryStream: Couldn't locate log:
> hdfs://namenode/hbase/oldWALs/hostname%2C60020%2C1636802867348.1636944748704
> 2021-11-22 15:18:39,593 ERROR
> [ReplicationExecutor-0.replicationSource,peer_id-hostname,60020,1636802867348.replicationSource.wal-reader.hostname%2C60020%2C1636802867348,peer_id-hostname,60020,1636802867348]
> regionserver.ReplicationSourceWALReader: Failed to read stream of
> replication entries
> java.io.FileNotFoundException: File does not exist:
> hdfs://namenode/hbase/oldWALs/hostname%2C60020%2C1636802867348.1636944748704
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1612)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1605)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1620)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:64)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:168)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:321)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:303)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:291)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:427)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.openReader(WALEntryStream.java:355)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.openNextLog(WALEntryStream.java:303)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.checkReader(WALEntryStream.java:294)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:175)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:101)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:192)
> at
>
[jira] [Commented] (HBASE-26468) Region Server doesn't exit cleanly incase it crashes.
[
https://issues.apache.org/jira/browse/HBASE-26468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448017#comment-17448017
]
Rushabh Shah commented on HBASE-26468:
--
> Maybe we could add a delay? For example, if the process does not exit for 30
> seconds, we call System.exit to force quit, and the return value should be
> something other than 0 to indicate that this is a force terminate.
Sounds like a good idea. Thank you [~zhangduo] !
> Region Server doesn't exit cleanly incase it crashes.
> -
>
> Key: HBASE-26468
> URL: https://issues.apache.org/jira/browse/HBASE-26468
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.6.0
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2, 1.7.2, 2.3.8, 2.4.9
>
>
> Observed this in our production cluster running 1.6 version.
> RS crashed due to some reason but the process was still running. On debugging
> more, found out there was 1 non-daemon thread running and that was not
> allowing RS to exit cleanly. Our clusters are managed by Ambari and have auto
> restart capability within them. But since the process was running and pid
> file was present, Ambari also couldn't do much. There will be some bug where
> we will miss to stop some non daemon thread. Shutdown hook will not be called
> unless one of the following 2 conditions are met:
> # The Java virtual machine shuts down in response to two kinds of events:
> The program exits normally, when the last non-daemon thread exits or when the
> exit (equivalently, System.exit) method is invoked, or
> # The virtual machine is terminated in response to a user interrupt, such as
> typing ^C, or a system-wide event, such as user logoff or system shutdown.
> Considering the first condition, when the last non-daemon thread exits or
> when the exit method is invoked.
> Below is the code snippet from
> [HRegionServerCommandLine.java|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServerCommandLine.java#L51]
> {code:java}
> private int start() throws Exception {
> try {
> if (LocalHBaseCluster.isLocal(conf)) {
> // Ignore this.
> } else {
> HRegionServer hrs =
> HRegionServer.constructRegionServer(regionServerClass, conf);
> hrs.start();
> hrs.join();
> if (hrs.isAborted()) {
> throw new RuntimeException("HRegionServer Aborted");
> }
> }
> } catch (Throwable t) {
> LOG.error("Region server exiting", t);
> return 1;
> }
> return 0;
> }
> {code}
> Within HRegionServer, there is a subtle difference between when a server is
> aborted v/s when it is stopped. If it is stopped, then isAborted will return
> false and it will exit with return code 0.
> Below is the code from
> [ServerCommandLine.java|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerCommandLine.java#L147]
> {code:java}
> public void doMain(String args[]) {
> try {
> int ret = ToolRunner.run(HBaseConfiguration.create(), this, args);
> if (ret != 0) {
> System.exit(ret);
> }
> } catch (Exception e) {
> LOG.error("Failed to run", e);
> System.exit(-1);
> }
> }
> {code}
> If return code is 0, then it won't call System.exit. This means JVM will wait
> to call ShutdownHook until all non daemon threads are stopped which means
> infinite wait if we don't close all non-daemon threads cleanly.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26468) Region Server doesn't exit cleanly incase it crashes.
[
https://issues.apache.org/jira/browse/HBASE-26468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447429#comment-17447429
]
Rushabh Shah commented on HBASE-26468:
--
[~zhangduo] Created this jira with more details on which non daemon thread:
HBASE-26480
> Region Server doesn't exit cleanly incase it crashes.
> -
>
> Key: HBASE-26468
> URL: https://issues.apache.org/jira/browse/HBASE-26468
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.6.0
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2, 1.7.2, 2.3.8, 2.4.9
>
>
> Observed this in our production cluster running 1.6 version.
> RS crashed due to some reason but the process was still running. On debugging
> more, found out there was 1 non-daemon thread running and that was not
> allowing RS to exit cleanly. Our clusters are managed by Ambari and have auto
> restart capability within them. But since the process was running and pid
> file was present, Ambari also couldn't do much. There will be some bug where
> we will miss to stop some non daemon thread. Shutdown hook will not be called
> unless one of the following 2 conditions are met:
> # The Java virtual machine shuts down in response to two kinds of events:
> The program exits normally, when the last non-daemon thread exits or when the
> exit (equivalently, System.exit) method is invoked, or
> # The virtual machine is terminated in response to a user interrupt, such as
> typing ^C, or a system-wide event, such as user logoff or system shutdown.
> Considering the first condition, when the last non-daemon thread exits or
> when the exit method is invoked.
> Below is the code snippet from
> [HRegionServerCommandLine.java|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServerCommandLine.java#L51]
> {code:java}
> private int start() throws Exception {
> try {
> if (LocalHBaseCluster.isLocal(conf)) {
> // Ignore this.
> } else {
> HRegionServer hrs =
> HRegionServer.constructRegionServer(regionServerClass, conf);
> hrs.start();
> hrs.join();
> if (hrs.isAborted()) {
> throw new RuntimeException("HRegionServer Aborted");
> }
> }
> } catch (Throwable t) {
> LOG.error("Region server exiting", t);
> return 1;
> }
> return 0;
> }
> {code}
> Within HRegionServer, there is a subtle difference between when a server is
> aborted v/s when it is stopped. If it is stopped, then isAborted will return
> false and it will exit with return code 0.
> Below is the code from
> [ServerCommandLine.java|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerCommandLine.java#L147]
> {code:java}
> public void doMain(String args[]) {
> try {
> int ret = ToolRunner.run(HBaseConfiguration.create(), this, args);
> if (ret != 0) {
> System.exit(ret);
> }
> } catch (Exception e) {
> LOG.error("Failed to run", e);
> System.exit(-1);
> }
> }
> {code}
> If return code is 0, then it won't call System.exit. This means JVM will wait
> to call ShutdownHook until all non daemon threads are stopped which means
> infinite wait if we don't close all non-daemon threads cleanly.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26480) Close NamedQueueRecorder to allow HMaster/RS to shutdown gracefully
Rushabh Shah created HBASE-26480:
Summary: Close NamedQueueRecorder to allow HMaster/RS to shutdown
gracefully
Key: HBASE-26480
URL: https://issues.apache.org/jira/browse/HBASE-26480
Project: HBase
Issue Type: Bug
Affects Versions: 1.7.0
Reporter: Rushabh Shah
Assignee: Rushabh Shah
Saw one case in our production cluster where RS was not exiting. Saw this
non-daemon thread in hung RS stack trace:
{noformat}
"main.slowlog.append-pool-pool1-t1" #26 prio=5 os_prio=31
tid=0x7faf23bf7800 nid=0x6d07 waiting on condition [0x73f4d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0004039e3840> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:47)
at
com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
at
com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
at
com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
This is coming from
[NamedQueueRecorder|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/namequeues/NamedQueueRecorder.java#L65]
implementation.
This bug doesn't exists in branch-2 and master since the Disruptor
initialization has changed and we set daemon=true also. See [this
code|https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/namequeues/NamedQueueRecorder.java#L68]
FYI [~vjasani] [~zhangduo]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26468) Region Server doesn't exit cleanly incase it crashes.
[
https://issues.apache.org/jira/browse/HBASE-26468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17446829#comment-17446829
]
Rushabh Shah commented on HBASE-26468:
--
Created PR for master, branch-2 and branch-1. Please review.
> Region Server doesn't exit cleanly incase it crashes.
> -
>
> Key: HBASE-26468
> URL: https://issues.apache.org/jira/browse/HBASE-26468
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.6.0
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
>
> Observed this in our production cluster running 1.6 version.
> RS crashed due to some reason but the process was still running. On debugging
> more, found out there was 1 non-daemon thread running and that was not
> allowing RS to exit cleanly. Our clusters are managed by Ambari and have auto
> restart capability within them. But since the process was running and pid
> file was present, Ambari also couldn't do much. There will be some bug where
> we will miss to stop some non daemon thread. Shutdown hook will not be called
> unless one of the following 2 conditions are met:
> # The Java virtual machine shuts down in response to two kinds of events:
> The program exits normally, when the last non-daemon thread exits or when the
> exit (equivalently, System.exit) method is invoked, or
> # The virtual machine is terminated in response to a user interrupt, such as
> typing ^C, or a system-wide event, such as user logoff or system shutdown.
> Considering the first condition, when the last non-daemon thread exits or
> when the exit method is invoked.
> Below is the code snippet from
> [HRegionServerCommandLine.java|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServerCommandLine.java#L51]
> {code:java}
> private int start() throws Exception {
> try {
> if (LocalHBaseCluster.isLocal(conf)) {
> // Ignore this.
> } else {
> HRegionServer hrs =
> HRegionServer.constructRegionServer(regionServerClass, conf);
> hrs.start();
> hrs.join();
> if (hrs.isAborted()) {
> throw new RuntimeException("HRegionServer Aborted");
> }
> }
> } catch (Throwable t) {
> LOG.error("Region server exiting", t);
> return 1;
> }
> return 0;
> }
> {code}
> Within HRegionServer, there is a subtle difference between when a server is
> aborted v/s when it is stopped. If it is stopped, then isAborted will return
> false and it will exit with return code 0.
> Below is the code from
> [ServerCommandLine.java|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerCommandLine.java#L147]
> {code:java}
> public void doMain(String args[]) {
> try {
> int ret = ToolRunner.run(HBaseConfiguration.create(), this, args);
> if (ret != 0) {
> System.exit(ret);
> }
> } catch (Exception e) {
> LOG.error("Failed to run", e);
> System.exit(-1);
> }
> }
> {code}
> If return code is 0, then it won't call System.exit. This means JVM will wait
> to call ShutdownHook until all non daemon threads are stopped which means
> infinite wait if we don't close all non-daemon threads cleanly.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (HBASE-26468) Region Server doesn't exit cleanly incase it crashes.
[
https://issues.apache.org/jira/browse/HBASE-26468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh Shah reassigned HBASE-26468:
Assignee: Rushabh Shah
> Region Server doesn't exit cleanly incase it crashes.
> -
>
> Key: HBASE-26468
> URL: https://issues.apache.org/jira/browse/HBASE-26468
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.6.0
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
>
> Observed this in our production cluster running 1.6 version.
> RS crashed due to some reason but the process was still running. On debugging
> more, found out there was 1 non-daemon thread running and that was not
> allowing RS to exit cleanly. Our clusters are managed by Ambari and have auto
> restart capability within them. But since the process was running and pid
> file was present, Ambari also couldn't do much. There will be some bug where
> we will miss to stop some non daemon thread. Shutdown hook will not be called
> unless one of the following 2 conditions are met:
> # The Java virtual machine shuts down in response to two kinds of events:
> The program exits normally, when the last non-daemon thread exits or when the
> exit (equivalently, System.exit) method is invoked, or
> # The virtual machine is terminated in response to a user interrupt, such as
> typing ^C, or a system-wide event, such as user logoff or system shutdown.
> Considering the first condition, when the last non-daemon thread exits or
> when the exit method is invoked.
> Below is the code snippet from
> [HRegionServerCommandLine.java|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServerCommandLine.java#L51]
> {code:java}
> private int start() throws Exception {
> try {
> if (LocalHBaseCluster.isLocal(conf)) {
> // Ignore this.
> } else {
> HRegionServer hrs =
> HRegionServer.constructRegionServer(regionServerClass, conf);
> hrs.start();
> hrs.join();
> if (hrs.isAborted()) {
> throw new RuntimeException("HRegionServer Aborted");
> }
> }
> } catch (Throwable t) {
> LOG.error("Region server exiting", t);
> return 1;
> }
> return 0;
> }
> {code}
> Within HRegionServer, there is a subtle difference between when a server is
> aborted v/s when it is stopped. If it is stopped, then isAborted will return
> false and it will exit with return code 0.
> Below is the code from
> [ServerCommandLine.java|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerCommandLine.java#L147]
> {code:java}
> public void doMain(String args[]) {
> try {
> int ret = ToolRunner.run(HBaseConfiguration.create(), this, args);
> if (ret != 0) {
> System.exit(ret);
> }
> } catch (Exception e) {
> LOG.error("Failed to run", e);
> System.exit(-1);
> }
> }
> {code}
> If return code is 0, then it won't call System.exit. This means JVM will wait
> to call ShutdownHook until all non daemon threads are stopped which means
> infinite wait if we don't close all non-daemon threads cleanly.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26468) Region Server doesn't exit cleanly incase it crashes.
[
https://issues.apache.org/jira/browse/HBASE-26468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh Shah updated HBASE-26468:
-
Description:
Observed this in our production cluster running 1.6 version.
RS crashed due to some reason but the process was still running. On debugging
more, found out there was 1 non-daemon thread running and that was not allowing
RS to exit cleanly. Our clusters are managed by Ambari and have auto restart
capability within them. But since the process was running and pid file was
present, Ambari also couldn't do much. There will be some bug where we will
miss to stop some non daemon thread. Shutdown hook will not be called unless
one of the following 2 conditions are met:
# The Java virtual machine shuts down in response to two kinds of events:
The program exits normally, when the last non-daemon thread exits or when the
exit (equivalently, System.exit) method is invoked, or
# The virtual machine is terminated in response to a user interrupt, such as
typing ^C, or a system-wide event, such as user logoff or system shutdown.
Considering the first condition, when the last non-daemon thread exits or when
the exit method is invoked.
Below is the code snippet from
[HRegionServerCommandLine.java|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServerCommandLine.java#L51]
{code:java}
private int start() throws Exception {
try {
if (LocalHBaseCluster.isLocal(conf)) {
// Ignore this.
} else {
HRegionServer hrs =
HRegionServer.constructRegionServer(regionServerClass, conf);
hrs.start();
hrs.join();
if (hrs.isAborted()) {
throw new RuntimeException("HRegionServer Aborted");
}
}
} catch (Throwable t) {
LOG.error("Region server exiting", t);
return 1;
}
return 0;
}
{code}
Within HRegionServer, there is a subtle difference between when a server is
aborted v/s when it is stopped. If it is stopped, then isAborted will return
false and it will exit with return code 0.
Below is the code from
[ServerCommandLine.java|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerCommandLine.java#L147]
{code:java}
public void doMain(String args[]) {
try {
int ret = ToolRunner.run(HBaseConfiguration.create(), this, args);
if (ret != 0) {
System.exit(ret);
}
} catch (Exception e) {
LOG.error("Failed to run", e);
System.exit(-1);
}
}
{code}
If return code is 0, then it won't call System.exit. This means JVM will wait
to call ShutdownHook until all non daemon threads are stopped which means
infinite wait if we don't close all non-daemon threads cleanly.
was:
Observed this in our production cluster running 1.6 version.
RS crashed due to some reason but the process was still running. On debugging
more, found out there was 1 non-daemon thread running and that was not allowing
RS to exit cleanly. Our clusters are managed by Ambari and have auto restart
capability within them. But since the process was running and pid file was
present, Ambari also couldn't do much. There will be some bug where we will
miss to stop some non daemon thread but there should be some maximum amount of
time we should wait before exiting the thread.
Relevant code:
[HRegionServerCommandLine.java|https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServerCommandLine.java]
{code:java}
logProcessInfo(getConf());
HRegionServer hrs =
HRegionServer.constructRegionServer(regionServerClass, conf);
hrs.start();
hrs.join(); -> This should be a timed join.
if (hrs.isAborted()) {
throw new RuntimeException("HRegionServer Aborted");
}
}
{code}
> Region Server doesn't exit cleanly incase it crashes.
> -
>
> Key: HBASE-26468
> URL: https://issues.apache.org/jira/browse/HBASE-26468
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.6.0
>Reporter: Rushabh Shah
>Priority: Major
>
> Observed this in our production cluster running 1.6 version.
> RS crashed due to some reason but the process was still running. On debugging
> more, found out there was 1 non-daemon thread running and that was not
> allowing RS to exit cleanly. Our clusters are managed by Ambari and have auto
> restart capability within them. But since the process was running and pid
> file was present, Ambari also couldn't do much. There will be some bug where
> we will miss to stop some non daemon thread. Shutdown hook will not be called
> unless one of the following 2 conditions are met:
> # The Java
[jira] [Updated] (HBASE-26468) Region Server doesn't exit cleanly incase it crashes.
[
https://issues.apache.org/jira/browse/HBASE-26468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh Shah updated HBASE-26468:
-
Description:
Observed this in our production cluster running 1.6 version.
RS crashed due to some reason but the process was still running. On debugging
more, found out there was 1 non-daemon thread running and that was not allowing
RS to exit cleanly. Our clusters are managed by Ambari and have auto restart
capability within them. But since the process was running and pid file was
present, Ambari also couldn't do much. There will be some bug where we will
miss to stop some non daemon thread but there should be some maximum amount of
time we should wait before exiting the thread.
Relevant code:
[HRegionServerCommandLine.java|https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServerCommandLine.java]
{code:java}
logProcessInfo(getConf());
HRegionServer hrs =
HRegionServer.constructRegionServer(regionServerClass, conf);
hrs.start();
hrs.join(); -> This should be a timed join.
if (hrs.isAborted()) {
throw new RuntimeException("HRegionServer Aborted");
}
}
{code}
was:
Observed this in our production cluster running 1.6 version.
RS crashed due to some reason but the process was still running. On debugging
more, found out there was 1 non-daemon thread running and that was not allowing
RS to exit cleanly. Our clusters are managed by Ambari and have auto restart
capability within them. But since the process was running and pid file was
present, Ambari also couldn't do much. There will be some bug where we will
miss to stop some daemon thread but there should be some maximum amount of time
we should wait before exiting the thread.
Relevant code:
[HRegionServerCommandLine.java|https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServerCommandLine.java]
{code:java}
logProcessInfo(getConf());
HRegionServer hrs =
HRegionServer.constructRegionServer(regionServerClass, conf);
hrs.start();
hrs.join(); -> This should be a timed join.
if (hrs.isAborted()) {
throw new RuntimeException("HRegionServer Aborted");
}
}
{code}
> Region Server doesn't exit cleanly incase it crashes.
> -
>
> Key: HBASE-26468
> URL: https://issues.apache.org/jira/browse/HBASE-26468
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.6.0
>Reporter: Rushabh Shah
>Priority: Major
>
> Observed this in our production cluster running 1.6 version.
> RS crashed due to some reason but the process was still running. On debugging
> more, found out there was 1 non-daemon thread running and that was not
> allowing RS to exit cleanly. Our clusters are managed by Ambari and have auto
> restart capability within them. But since the process was running and pid
> file was present, Ambari also couldn't do much. There will be some bug where
> we will miss to stop some non daemon thread but there should be some maximum
> amount of time we should wait before exiting the thread.
> Relevant code:
> [HRegionServerCommandLine.java|https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServerCommandLine.java]
> {code:java}
> logProcessInfo(getConf());
> HRegionServer hrs =
> HRegionServer.constructRegionServer(regionServerClass, conf);
> hrs.start();
> hrs.join(); -> This should be a timed join.
> if (hrs.isAborted()) {
> throw new RuntimeException("HRegionServer Aborted");
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26468) Region Server doesn't exit cleanly incase it crashes.
[
https://issues.apache.org/jira/browse/HBASE-26468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh Shah updated HBASE-26468:
-
Description:
Observed this in our production cluster running 1.6 version.
RS crashed due to some reason but the process was still running. On debugging
more, found out there was 1 non-daemon thread running and that was not allowing
RS to exit cleanly. Our clusters are managed by Ambari and have auto restart
capability within them. But since the process was running and pid file was
present, Ambari also couldn't do much. There will be some bug where we will
miss to stop some daemon thread but there should be some maximum amount of time
we should wait before exiting the thread.
Relevant code:
[HRegionServerCommandLine.java|https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServerCommandLine.java]
{code:java}
logProcessInfo(getConf());
HRegionServer hrs =
HRegionServer.constructRegionServer(regionServerClass, conf);
hrs.start();
hrs.join(); -> This should be a timed join.
if (hrs.isAborted()) {
throw new RuntimeException("HRegionServer Aborted");
}
}
{code}
was:
Observed this in our production cluster running 1.6 version.
RS crashed due to some reason but the process was still running. On debugging
more, found out there was 1 non-daemon thread running and that was not allowing
RS to stop cleanly. Our clusters are managed by Ambari and have auto restart
capability within them. But since the process was running and pid file was
present, Ambari also couldn't do much. There will be some bug where we will
miss to stop some daemon thread but there should be some maximum amount of time
we should wait before exiting the thread.
Relevant code:
[HRegionServerCommandLine.java|https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServerCommandLine.java]
{code:java}
logProcessInfo(getConf());
HRegionServer hrs =
HRegionServer.constructRegionServer(regionServerClass, conf);
hrs.start();
hrs.join(); -> This should be a timed join.
if (hrs.isAborted()) {
throw new RuntimeException("HRegionServer Aborted");
}
}
{code}
> Region Server doesn't exit cleanly incase it crashes.
> -
>
> Key: HBASE-26468
> URL: https://issues.apache.org/jira/browse/HBASE-26468
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.6.0
>Reporter: Rushabh Shah
>Priority: Major
>
> Observed this in our production cluster running 1.6 version.
> RS crashed due to some reason but the process was still running. On debugging
> more, found out there was 1 non-daemon thread running and that was not
> allowing RS to exit cleanly. Our clusters are managed by Ambari and have auto
> restart capability within them. But since the process was running and pid
> file was present, Ambari also couldn't do much. There will be some bug where
> we will miss to stop some daemon thread but there should be some maximum
> amount of time we should wait before exiting the thread.
> Relevant code:
> [HRegionServerCommandLine.java|https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServerCommandLine.java]
> {code:java}
> logProcessInfo(getConf());
> HRegionServer hrs =
> HRegionServer.constructRegionServer(regionServerClass, conf);
> hrs.start();
> hrs.join(); -> This should be a timed join.
> if (hrs.isAborted()) {
> throw new RuntimeException("HRegionServer Aborted");
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26468) Region Server doesn't exit cleanly incase it crashes.
Rushabh Shah created HBASE-26468:
Summary: Region Server doesn't exit cleanly incase it crashes.
Key: HBASE-26468
URL: https://issues.apache.org/jira/browse/HBASE-26468
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 1.6.0
Reporter: Rushabh Shah
Observed this in our production cluster running 1.6 version.
RS crashed due to some reason but the process was still running. On debugging
more, found out there was 1 non-daemon thread running and that was not allowing
RS to stop cleanly. Our clusters are managed by Ambari and have auto restart
capability within them. But since the process was running and pid file was
present, Ambari also couldn't do much. There will be some bug where we will
miss to stop some daemon thread but there should be some maximum amount of time
we should wait before exiting the thread.
Relevant code:
[HRegionServerCommandLine.java|https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServerCommandLine.java]
{code:java}
logProcessInfo(getConf());
HRegionServer hrs =
HRegionServer.constructRegionServer(regionServerClass, conf);
hrs.start();
hrs.join(); -> This should be a timed join.
if (hrs.isAborted()) {
throw new RuntimeException("HRegionServer Aborted");
}
}
{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26442) TestReplicationEndpoint#testInterClusterReplication fails in branch-1
[
https://issues.apache.org/jira/browse/HBASE-26442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh Shah updated HBASE-26442:
-
Description:
{noformat}
[INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ hbase-server ---
[INFO]
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.hadoop.hbase.replication.TestReplicationEndpoint
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 20.978
s <<< FAILURE! - in org.apache.hadoop.hbase.replication.TestReplicationEndpoint
[ERROR] org.apache.hadoop.hbase.replication.TestReplicationEndpoint Time
elapsed: 3.921 s <<< FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.hbase.replication.TestReplicationEndpoint.tearDownAfterClass(TestReplicationEndpoint.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestReplicationEndpoint.tearDownAfterClass:88
[INFO]
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0
{noformat}
It fails if testInterClusterReplication is run alone but succeeds (depending on
the order of test it ran) as a test suite.
was:
{noformat}
[INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ hbase-server ---
[INFO]
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.hadoop.hbase.replication.TestReplicationEndpoint
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 20.978
s <<< FAILURE! - in org.apache.hadoop.hbase.replication.TestReplicationEndpoint
[ERROR] org.apache.hadoop.hbase.replication.TestReplicationEndpoint Time
elapsed: 3.921 s <<< FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.hbase.replication.TestReplicationEndpoint.tearDownAfterClass(TestReplicationEndpoint.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
[jira] [Updated] (HBASE-26442) TestReplicationEndpoint#testInterClusterReplication fails in branch-1
[
https://issues.apache.org/jira/browse/HBASE-26442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh Shah updated HBASE-26442:
-
Description:
{noformat}
[INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ hbase-server ---
[INFO]
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.hadoop.hbase.replication.TestReplicationEndpoint
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 20.978
s <<< FAILURE! - in org.apache.hadoop.hbase.replication.TestReplicationEndpoint
[ERROR] org.apache.hadoop.hbase.replication.TestReplicationEndpoint Time
elapsed: 3.921 s <<< FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.hbase.replication.TestReplicationEndpoint.tearDownAfterClass(TestReplicationEndpoint.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestReplicationEndpoint.tearDownAfterClass:88
[INFO]
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0
{noformat}
It fails if
was:
{noformat}
[INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ hbase-server ---
[INFO]
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.hadoop.hbase.replication.TestReplicationEndpoint
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 20.978
s <<< FAILURE! - in org.apache.hadoop.hbase.replication.TestReplicationEndpoint
[ERROR] org.apache.hadoop.hbase.replication.TestReplicationEndpoint Time
elapsed: 3.921 s <<< FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.hbase.replication.TestReplicationEndpoint.tearDownAfterClass(TestReplicationEndpoint.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
[jira] [Created] (HBASE-26442) TestReplicationEndpoint#testInterClusterReplication fails in branch-1
Rushabh Shah created HBASE-26442:
Summary: TestReplicationEndpoint#testInterClusterReplication fails
in branch-1
Key: HBASE-26442
URL: https://issues.apache.org/jira/browse/HBASE-26442
Project: HBase
Issue Type: Bug
Affects Versions: 1.7.1
Reporter: Rushabh Shah
Assignee: Rushabh Shah
{noformat}
[INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ hbase-server ---
[INFO]
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.hadoop.hbase.replication.TestReplicationEndpoint
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 20.978
s <<< FAILURE! - in org.apache.hadoop.hbase.replication.TestReplicationEndpoint
[ERROR] org.apache.hadoop.hbase.replication.TestReplicationEndpoint Time
elapsed: 3.921 s <<< FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.hbase.replication.TestReplicationEndpoint.tearDownAfterClass(TestReplicationEndpoint.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestReplicationEndpoint.tearDownAfterClass:88
[INFO]
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26435) [branch-1] The log rolling request maybe canceled immediately in LogRoller due to a race
[
https://issues.apache.org/jira/browse/HBASE-26435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh Shah updated HBASE-26435:
-
Description:
Saw this issue in our internal 1.6 branch.
All the writes to this RS were getting failing since the underlying hdfs file
was corrupt. This healed after 1 hour (equivalent to
hbase.regionserver.logroll.period conf key).
The WAL was rolled but the new WAL file was not writable and it logged the
following error also.
{noformat}
2021-11-03 19:20:19,503 WARN [.168:60020.logRoller] hdfs.DFSClient - Error
while syncing
java.io.IOException: Could not get block locations. Source file
"/hbase/WALs/,60020,1635567166484/%2C60020%2C1635567166484.1635967219389"
- Aborting...
at
org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1466)
at
org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1251)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:670)
2021-11-03 19:20:19,507 WARN [.168:60020.logRoller] wal.FSHLog - pre-sync
failed but an optimization so keep going
java.io.IOException: Could not get block locations. Source file
"/hbase/WALs/,60020,1635567166484/%2C60020%2C1635567166484.1635967219389"
- Aborting...
at
org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1466)
at
org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1251)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:670)
{noformat}
Since the new WAL file was not writable, appends to that file started failing
immediately it was rolled.
{noformat}
2021-11-03 19:20:19,677 INFO [.168:60020.logRoller] wal.FSHLog - Rolled WAL
/hbase/WALs/,60020,1635567166484/%2C60020%2C1635567166484.1635965392022
with entries=253234, filesize=425.67 MB; new WAL
/hbase/WALs/,60020,1635567166484/%2C60020%2C1635567166484.1635967219389
2021-11-03 19:20:19,690 WARN [020.append-pool17-t1] wal.FSHLog - Append
sequenceId=1962661783, requesting roll of WAL
java.io.IOException: Could not get block locations. Source file
"/hbase/WALs/,60020,1635567166484/%2C60020%2C1635567166484.1635967219389"
- Aborting...
at
org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1466)
at
org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1251)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:670)
2021-11-03 19:20:19,690 INFO [.168:60020.logRoller] wal.FSHLog - Archiving
hdfs://prod-EMPTY-hbase2a/hbase/WALs/,60020,1635567166484/%2C60020%2C1635567166484.1635960792837
to
hdfs://prod-EMPTY-hbase2a/hbase/oldWALs/hbase2a-dnds1-232-ukb.ops.sfdc.net%2C60020%2C1635567166484.1635960792837
{noformat}
We always reset the rollLog flag within LogRoller thread after the rollWal call
is complete.
Within FSHLog#rollWriter method, it does many things, like replacing the writer
and archiving old logs. If append thread fails to write to new file while
logRoller thread is cleaning old logs, we will miss the rollLog flag since
LogRoller will reset the flag to false while the previous rollWriter call is
going on.
Relevant code:
https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java#L183-L203
We need to reset rollLog flag before we start rolling the wal.
This is fixed in branch-2 and master via HBASE-22684 but we didn't fix it in
branch-1
Also branch-2 has multi wal implementation so it can apply cleanly in branch-1.
was:
Saw this issue in our internal 1.6 branch.
The WAL was rolled but the new WAL file was not writable and it logged the
following error also
{noformat}
2021-11-03 19:20:19,503 WARN [.168:60020.logRoller] hdfs.DFSClient - Error
while syncing
java.io.IOException: Could not get block locations. Source file
"/hbase/WALs/,60020,1635567166484/%2C60020%2C1635567166484.1635967219389"
- Aborting...
at
org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1466)
at
org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1251)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:670)
2021-11-03 19:20:19,507 WARN [.168:60020.logRoller] wal.FSHLog - pre-sync
failed but an optimization so keep going
java.io.IOException: Could not get block locations. Source file
"/hbase/WALs/,60020,1635567166484/%2C60020%2C1635567166484.1635967219389"
- Aborting...
at
org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1466)
at
org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1251)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:670)
{noformat}
Since the new WAL file was not writable, appends to that file started failing
immediately it was
[jira] [Assigned] (HBASE-26435) [branch-1] The log rolling request maybe canceled immediately in LogRoller due to a race
[
https://issues.apache.org/jira/browse/HBASE-26435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh Shah reassigned HBASE-26435:
Assignee: Rushabh Shah
> [branch-1] The log rolling request maybe canceled immediately in LogRoller
> due to a race
> -
>
> Key: HBASE-26435
> URL: https://issues.apache.org/jira/browse/HBASE-26435
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Affects Versions: 1.6.0
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Fix For: 1.7.2
>
>
> Saw this issue in our internal 1.6 branch.
> The WAL was rolled but the new WAL file was not writable and it logged the
> following error also
> {noformat}
> 2021-11-03 19:20:19,503 WARN [.168:60020.logRoller] hdfs.DFSClient - Error
> while syncing
> java.io.IOException: Could not get block locations. Source file
> "/hbase/WALs/,60020,1635567166484/%2C60020%2C1635567166484.1635967219389"
> - Aborting...
> at
> org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1466)
> at
> org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1251)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:670)
> 2021-11-03 19:20:19,507 WARN [.168:60020.logRoller] wal.FSHLog - pre-sync
> failed but an optimization so keep going
> java.io.IOException: Could not get block locations. Source file
> "/hbase/WALs/,60020,1635567166484/%2C60020%2C1635567166484.1635967219389"
> - Aborting...
> at
> org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1466)
> at
> org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1251)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:670)
> {noformat}
> Since the new WAL file was not writable, appends to that file started failing
> immediately it was rolled.
> {noformat}
> 2021-11-03 19:20:19,677 INFO [.168:60020.logRoller] wal.FSHLog - Rolled WAL
> /hbase/WALs/,60020,1635567166484/%2C60020%2C1635567166484.1635965392022
> with entries=253234, filesize=425.67 MB; new WAL
> /hbase/WALs/,60020,1635567166484/%2C60020%2C1635567166484.1635967219389
> 2021-11-03 19:20:19,690 WARN [020.append-pool17-t1] wal.FSHLog - Append
> sequenceId=1962661783, requesting roll of WAL
> java.io.IOException: Could not get block locations. Source file
> "/hbase/WALs/,60020,1635567166484/%2C60020%2C1635567166484.1635967219389"
> - Aborting...
> at
> org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1466)
> at
> org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1251)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:670)
> 2021-11-03 19:20:19,690 INFO [.168:60020.logRoller] wal.FSHLog - Archiving
> hdfs://prod-EMPTY-hbase2a/hbase/WALs/,60020,1635567166484/%2C60020%2C1635567166484.1635960792837
> to
> hdfs://prod-EMPTY-hbase2a/hbase/oldWALs/hbase2a-dnds1-232-ukb.ops.sfdc.net%2C60020%2C1635567166484.1635960792837
> {noformat}
> We always reset the rollLog flag within LogRoller thread after the rollWal
> call is complete.
> Within FSHLog#rollWriter method, it does many things, like replacing the
> writer and archiving old logs. If append thread fails to write to new file
> while logRoller thread is cleaning old logs, we will miss the rollLog flag
> since LogRoller will reset the flag to false while the previous rollWriter
> call is going on.
> Relevant code:
> https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java#L183-L203
> We need to reset rollLog flag before we start rolling the wal.
> This is fixed in branch-2 and master via HBASE-22684 but we didn't fix it in
> branch-1
> Also branch-2 has multi wal implementation so it can apply cleanly in
> branch-1.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26435) [branch-1] The log rolling request maybe canceled immediately in LogRoller due to a race
Rushabh Shah created HBASE-26435:
Summary: [branch-1] The log rolling request maybe canceled
immediately in LogRoller due to a race
Key: HBASE-26435
URL: https://issues.apache.org/jira/browse/HBASE-26435
Project: HBase
Issue Type: Sub-task
Components: wal
Affects Versions: 1.6.0
Reporter: Rushabh Shah
Fix For: 1.7.2
Saw this issue in our internal 1.6 branch.
The WAL was rolled but the new WAL file was not writable and it logged the
following error also
{noformat}
2021-11-03 19:20:19,503 WARN [.168:60020.logRoller] hdfs.DFSClient - Error
while syncing
java.io.IOException: Could not get block locations. Source file
"/hbase/WALs/,60020,1635567166484/%2C60020%2C1635567166484.1635967219389"
- Aborting...
at
org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1466)
at
org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1251)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:670)
2021-11-03 19:20:19,507 WARN [.168:60020.logRoller] wal.FSHLog - pre-sync
failed but an optimization so keep going
java.io.IOException: Could not get block locations. Source file
"/hbase/WALs/,60020,1635567166484/%2C60020%2C1635567166484.1635967219389"
- Aborting...
at
org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1466)
at
org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1251)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:670)
{noformat}
Since the new WAL file was not writable, appends to that file started failing
immediately it was rolled.
{noformat}
2021-11-03 19:20:19,677 INFO [.168:60020.logRoller] wal.FSHLog - Rolled WAL
/hbase/WALs/,60020,1635567166484/%2C60020%2C1635567166484.1635965392022
with entries=253234, filesize=425.67 MB; new WAL
/hbase/WALs/,60020,1635567166484/%2C60020%2C1635567166484.1635967219389
2021-11-03 19:20:19,690 WARN [020.append-pool17-t1] wal.FSHLog - Append
sequenceId=1962661783, requesting roll of WAL
java.io.IOException: Could not get block locations. Source file
"/hbase/WALs/,60020,1635567166484/%2C60020%2C1635567166484.1635967219389"
- Aborting...
at
org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1466)
at
org.apache.hadoop.hdfs.DataStreamer.processDatanodeError(DataStreamer.java:1251)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:670)
2021-11-03 19:20:19,690 INFO [.168:60020.logRoller] wal.FSHLog - Archiving
hdfs://prod-EMPTY-hbase2a/hbase/WALs/,60020,1635567166484/%2C60020%2C1635567166484.1635960792837
to
hdfs://prod-EMPTY-hbase2a/hbase/oldWALs/hbase2a-dnds1-232-ukb.ops.sfdc.net%2C60020%2C1635567166484.1635960792837
{noformat}
We always reset the rollLog flag within LogRoller thread after the rollWal call
is complete.
Within FSHLog#rollWriter method, it does many things, like replacing the writer
and archiving old logs. If append thread fails to write to new file while
logRoller thread is cleaning old logs, we will miss the rollLog flag since
LogRoller will reset the flag to false while the previous rollWriter call is
going on.
Relevant code:
https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java#L183-L203
We need to reset rollLog flag before we start rolling the wal.
This is fixed in branch-2 and master via HBASE-22684 but we didn't fix it in
branch-1
Also branch-2 has multi wal implementation so it can apply cleanly in branch-1.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (HBASE-26408) Aborting to preserve WAL as source of truth can abort in recoverable situations
[
https://issues.apache.org/jira/browse/HBASE-26408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437423#comment-17437423
]
Rushabh Shah edited comment on HBASE-26408 at 11/2/21, 3:27 PM:
> I wonder if we should wrap thepostWALWrite in a try/catch so that we can
>have more control over how exceptions from that call get wrapped.
I was thinking on similar lines. We can still have DamagedWALException thrown
from append method but we should have different checked exceptions in the
underlying cause and then we can take better decisions on which ones to
abort/not abort. Today we will have to do string comparisons and that would be
very brittle.
was (Author: shahrs87):
> I wonder if we should wrap thepostWALWrite in a try/catch so that we can
>have more control over how exceptions from that call get wrapped.
I was thinking on similar lines. We can still have DamagedWALException thrown
from append method but we should have different checked exceptions in the
underlying cause and then we can take better decisions on which ones to
abort/not abort. Today we will have to do string comparisons and that would be
very brittle.
[|https://issues.apache.org/jira/secure/AddComment!default.jspa?id=13409189]
> Aborting to preserve WAL as source of truth can abort in recoverable
> situations
> ---
>
> Key: HBASE-26408
> URL: https://issues.apache.org/jira/browse/HBASE-26408
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Bryan Beaudreault
>Priority: Major
>
> HBASE-26195 added an important feature to avoid data corruption by preserving
> the WAL as a source of truth when WAL sync fails. See that issue for
> background.
> That issue's primary driver was a TimeoutIOException, but the solution was to
> catch and abort on Throwable. The idea here was that we can't anticipate all
> possible failures, so we should err on the side of data correctness. As
> pointed out by [~rushabh.shah] in his comments, this solution has the
> potential to lose HBase capacity quickly in "not very grave" situations. It
> would be good to add an escape hatch for those explicit known cases, of which
> I recently encountered:
> I recently rolled this out to some of our test clusters, most of which are
> small. Afterward, doing a rolling restart of DataNodes caused the following
> IOException: "Failed to replace a bad datanode on the existing pipeline due
> to no more good datanodes being available to try..."
> If you're familiar with HDFS pipeline recovery, this error will be familiar.
> Basically the restarted DataNodes caused pipeline failures, those datanodes
> were added to an internal exclude list that never gets cleared, and
> eventually there were no more nodes to choose from resulting in an error.
> This error is pretty explicit, and at this point the DFSOutputStream for the
> WAL is dead. I think this error is a reasonable one to simply bubble up and
> not abort the RegionServer on, instead just failing and rolling back the
> writes.
> What do people think about starting an allowlist of known good error messages
> for which we do not trigger an abort of the RS? Something like this:
> {{} catch (Throwable t) {}}
> {{ // WAL sync failed. Aborting to avoid a mismatch between the memstore,
> WAL,}}
> {{ // and any replicated clusters.}}
> {{ if (!walSyncSuccess && !allowedException(t)) {}}
> {{ rsServices.abort("WAL sync failed, aborting to preserve WAL as source of
> truth", t);}}
> \{{ }}}
> {{... snip ..}}
> {{private boolean allowedException(Throwable t) {}}{\{ }}
> {{ return t.getMessage().startsWith("Failed to replace a bad datanode");}}
> {{}}}
> We could of course make configurable if people like, or just add to it over
> time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-26408) Aborting to preserve WAL as source of truth can abort in recoverable situations
[
https://issues.apache.org/jira/browse/HBASE-26408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437423#comment-17437423
]
Rushabh Shah commented on HBASE-26408:
--
> I wonder if we should wrap thepostWALWrite in a try/catch so that we can
>have more control over how exceptions from that call get wrapped.
I was thinking on similar lines. We can still have DamagedWALException thrown
from append method but we should have different checked exceptions in the
underlying cause and then we can take better decisions on which ones to
abort/not abort. Today we will have to do string comparisons and that would be
very brittle.
[|https://issues.apache.org/jira/secure/AddComment!default.jspa?id=13409189]
> Aborting to preserve WAL as source of truth can abort in recoverable
> situations
> ---
>
> Key: HBASE-26408
> URL: https://issues.apache.org/jira/browse/HBASE-26408
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Bryan Beaudreault
>Priority: Major
>
> HBASE-26195 added an important feature to avoid data corruption by preserving
> the WAL as a source of truth when WAL sync fails. See that issue for
> background.
> That issue's primary driver was a TimeoutIOException, but the solution was to
> catch and abort on Throwable. The idea here was that we can't anticipate all
> possible failures, so we should err on the side of data correctness. As
> pointed out by [~rushabh.shah] in his comments, this solution has the
> potential to lose HBase capacity quickly in "not very grave" situations. It
> would be good to add an escape hatch for those explicit known cases, of which
> I recently encountered:
> I recently rolled this out to some of our test clusters, most of which are
> small. Afterward, doing a rolling restart of DataNodes caused the following
> IOException: "Failed to replace a bad datanode on the existing pipeline due
> to no more good datanodes being available to try..."
> If you're familiar with HDFS pipeline recovery, this error will be familiar.
> Basically the restarted DataNodes caused pipeline failures, those datanodes
> were added to an internal exclude list that never gets cleared, and
> eventually there were no more nodes to choose from resulting in an error.
> This error is pretty explicit, and at this point the DFSOutputStream for the
> WAL is dead. I think this error is a reasonable one to simply bubble up and
> not abort the RegionServer on, instead just failing and rolling back the
> writes.
> What do people think about starting an allowlist of known good error messages
> for which we do not trigger an abort of the RS? Something like this:
> {{} catch (Throwable t) {}}
> {{ // WAL sync failed. Aborting to avoid a mismatch between the memstore,
> WAL,}}
> {{ // and any replicated clusters.}}
> {{ if (!walSyncSuccess && !allowedException(t)) {}}
> {{ rsServices.abort("WAL sync failed, aborting to preserve WAL as source of
> truth", t);}}
> \{{ }}}
> {{... snip ..}}
> {{private boolean allowedException(Throwable t) {}}{\{ }}
> {{ return t.getMessage().startsWith("Failed to replace a bad datanode");}}
> {{}}}
> We could of course make configurable if people like, or just add to it over
> time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-26408) Aborting to preserve WAL as source of truth can abort in recoverable situations
[
https://issues.apache.org/jira/browse/HBASE-26408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437418#comment-17437418
]
Rushabh Shah commented on HBASE-26408:
--
> I agree that it's possible for postWALWrite to fail, and that should also
> probably not abort.
[~bbeaudreault] Trying to understand why it shouldn't abort ? postWALWrite
failed but the entry is written to HDFS/WAL. But in HRegion#append, the write
will fail causing it to roll back from memstore and again primary and
replicated cluster will be out of sync.
> Aborting to preserve WAL as source of truth can abort in recoverable
> situations
> ---
>
> Key: HBASE-26408
> URL: https://issues.apache.org/jira/browse/HBASE-26408
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Bryan Beaudreault
>Priority: Major
>
> HBASE-26195 added an important feature to avoid data corruption by preserving
> the WAL as a source of truth when WAL sync fails. See that issue for
> background.
> That issue's primary driver was a TimeoutIOException, but the solution was to
> catch and abort on Throwable. The idea here was that we can't anticipate all
> possible failures, so we should err on the side of data correctness. As
> pointed out by [~rushabh.shah] in his comments, this solution has the
> potential to lose HBase capacity quickly in "not very grave" situations. It
> would be good to add an escape hatch for those explicit known cases, of which
> I recently encountered:
> I recently rolled this out to some of our test clusters, most of which are
> small. Afterward, doing a rolling restart of DataNodes caused the following
> IOException: "Failed to replace a bad datanode on the existing pipeline due
> to no more good datanodes being available to try..."
> If you're familiar with HDFS pipeline recovery, this error will be familiar.
> Basically the restarted DataNodes caused pipeline failures, those datanodes
> were added to an internal exclude list that never gets cleared, and
> eventually there were no more nodes to choose from resulting in an error.
> This error is pretty explicit, and at this point the DFSOutputStream for the
> WAL is dead. I think this error is a reasonable one to simply bubble up and
> not abort the RegionServer on, instead just failing and rolling back the
> writes.
> What do people think about starting an allowlist of known good error messages
> for which we do not trigger an abort of the RS? Something like this:
> {{} catch (Throwable t) {}}
> {{ // WAL sync failed. Aborting to avoid a mismatch between the memstore,
> WAL,}}
> {{ // and any replicated clusters.}}
> {{ if (!walSyncSuccess && !allowedException(t)) {}}
> {{ rsServices.abort("WAL sync failed, aborting to preserve WAL as source of
> truth", t);}}
> \{{ }}}
> {{... snip ..}}
> {{private boolean allowedException(Throwable t) {}}{\{ }}
> {{ return t.getMessage().startsWith("Failed to replace a bad datanode");}}
> {{}}}
> We could of course make configurable if people like, or just add to it over
> time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-26408) Aborting to preserve WAL as source of truth can abort in recoverable situations
[
https://issues.apache.org/jira/browse/HBASE-26408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437393#comment-17437393
]
Rushabh Shah commented on HBASE-26408:
--
Thank you [~bbeaudreault] for reporting the issue.
This is the code snippet from
[+FSHLog#RingBufferEventHandler#append+|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java#L2081-L2129]
method
{code:java}
void append(final FSWALEntry entry) throws Exception {
// TODO: WORK ON MAKING THIS APPEND FASTER. DOING WAY TOO MUCH WORK WITH
CPs, PBing, etc.
atHeadOfRingBufferEventHandlerAppend();
long start = EnvironmentEdgeManager.currentTime();
byte [] encodedRegionName = entry.getKey().getEncodedRegionName();
long regionSequenceId = WALKey.NO_SEQUENCE_ID;
try {
regionSequenceId = entry.getKey().getSequenceId();
// Edits are empty, there is nothing to append. Maybe empty when we
are looking for a
// region sequence id only, a region edit/sequence id that is not
associated with an actual
// edit. It has to go through all the rigmarole to be sure we have the
right ordering.
if (entry.getEdit().isEmpty()) {
return;
}
// Coprocessor hook.
if (!coprocessorHost.preWALWrite(entry.getHRegionInfo(), entry.getKey(),
entry.getEdit())) {
if (entry.getEdit().isReplay()) {
// Set replication scope null so that this won't be replicated
entry.getKey().setScopes(null);
}
}
if (!listeners.isEmpty()) {
for (WALActionsListener i: listeners) {
// TODO: Why does listener take a table description and CPs take a
regioninfo? Fix.
i.visitLogEntryBeforeWrite(entry.getHTableDescriptor(),
entry.getKey(),
entry.getEdit());
}
}
writer.append(entry);
assert highestUnsyncedSequence < entry.getSequence();
highestUnsyncedSequence = entry.getSequence();
sequenceIdAccounting.update(encodedRegionName, entry.getFamilyNames(),
regionSequenceId,
entry.isInMemstore());
coprocessorHost.postWALWrite(entry.getHRegionInfo(), entry.getKey(),
entry.getEdit());
// Update metrics.
postAppend(entry, EnvironmentEdgeManager.currentTime() - start);
} catch (Exception e) {
String msg = "Append sequenceId=" + regionSequenceId + ", requesting
roll of WAL";
LOG.warn(msg, e);
requestLogRoll();
throw new DamagedWALException(msg, e);
}
numEntries.incrementAndGet();
}
{code}
We catch all the Exception coming out of append method and throw blanket
DamagedWALException. It is possible that writer#append succeeds at L#2114 but
something fails afterwards in coprocessorHost#postWALWrite method.
If that happens again we will be in an inconsistent state between primary and
replicated cluster. Instead of skip aborting on DamagedWALException, we should
unwrap DamagedWALException and see the underlying cause and skip aborting if
the cause is in allowedList.
> Aborting to preserve WAL as source of truth can abort in recoverable
> situations
> ---
>
> Key: HBASE-26408
> URL: https://issues.apache.org/jira/browse/HBASE-26408
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Bryan Beaudreault
>Priority: Major
>
> HBASE-26195 added an important feature to avoid data corruption by preserving
> the WAL as a source of truth when WAL sync fails. See that issue for
> background.
> That issue's primary driver was a TimeoutIOException, but the solution was to
> catch and abort on Throwable. The idea here was that we can't anticipate all
> possible failures, so we should err on the side of data correctness. As
> pointed out by [~rushabh.shah] in his comments, this solution has the
> potential to lose HBase capacity quickly in "not very grave" situations. It
> would be good to add an escape hatch for those explicit known cases, of which
> I recently encountered:
> I recently rolled this out to some of our test clusters, most of which are
> small. Afterward, doing a rolling restart of DataNodes caused the following
> IOException: "Failed to replace a bad datanode on the existing pipeline due
> to no more good datanodes being available to try..."
> If you're familiar with HDFS pipeline recovery, this error will be familiar.
> Basically the restarted DataNodes caused pipeline failures, those datanodes
> were added to an internal exclude list that never gets cleared, and
> eventually there were no more nodes to choose from resulting in an error.
> This error is pretty explicit, and at this point the
[jira] [Commented] (HBASE-26404) Update javadoc for CellUtil#createCell with tags methods.
[
https://issues.apache.org/jira/browse/HBASE-26404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17436796#comment-17436796
]
Rushabh Shah commented on HBASE-26404:
--
Thank you [~zhangduo] for the review and commit !
> Update javadoc for CellUtil#createCell with tags methods.
> --
>
> Key: HBASE-26404
> URL: https://issues.apache.org/jira/browse/HBASE-26404
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.8
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Fix For: 2.5.0, 2.4.9
>
>
> We have the following methods in CellUtil class which are deprecated. We used
> to use this method within custom COPROC to create cells with custom tags. We
> deprecated them in 2.0.0 version and created a new class called RawCell which
> is LimitedPrivate with visibility set to COPROC. There is no reference to
> RawCell#createCell(Cell cell, List tags) method in javadoc of
> CellUtil#createCell methods which are now deprecated. This is not user
> friendly. We should improve the javadoc within CellUtil#createCell(Cell,
> tags) method.
> {noformat}
> /**
>* Note : Now only CPs can create cell with tags using the CP environment
>* @return A new cell which is having the extra tags also added to it.
>* @deprecated As of release 2.0.0, this will be removed in HBase 3.0.0.
>*
>*/
> @Deprecated
> public static Cell createCell(Cell cell, List tags) {
> return PrivateCellUtil.createCell(cell, tags);
> }
> {noformat}
> {noformat}
> public static Cell createCell(Cell cell, byte[] tags)
> public static Cell createCell(Cell cell, byte[] value, byte[] tags) {
> {noformat}
> This improvement is only applicable to branch-2 since we have removed these
> methods in branch-3.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
