[jira] [Created] (HBASE-27878) balance_rsgroup NullPointerException
zhengsicheng created HBASE-27878: Summary: balance_rsgroup NullPointerException Key: HBASE-27878 URL: https://issues.apache.org/jira/browse/HBASE-27878 Project: HBase Issue Type: Bug Reporter: zhengsicheng Assignee: zhengsicheng hbase(main):001:0> balance_rsgroup 'default' ERROR: java.io.IOException: Cannot invoke "org.apache.hadoop.hbase.ServerName.getAddress()" because "currentHostServer" is null at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:466) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318) Caused by: java.lang.NullPointerException: Cannot invoke "org.apache.hadoop.hbase.ServerName.getAddress()" because "currentHostServer" is null at org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.correctAssignments(RSGroupBasedLoadBalancer.java:320) at org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.balanceCluster(RSGroupBasedLoadBalancer.java:126) at org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.balanceRSGroup(RSGroupAdminServer.java:461) at org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl.balanceRSGroup(RSGroupAdminEndpoint.java:301) at org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos$RSGroupAdminService.callMethod(RSGroupAdminProtos.java:14948) at org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:921) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:394) ... 3 more For usage try 'help "balance_rsgroup"' -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HBASE-27763) Recover WAL encounter KeeperErrorCode = NoNode cause RegionServer crash
[ https://issues.apache.org/jira/browse/HBASE-27763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng reassigned HBASE-27763: Assignee: (was: zhengsicheng) > Recover WAL encounter KeeperErrorCode = NoNode cause RegionServer crash > > > Key: HBASE-27763 > URL: https://issues.apache.org/jira/browse/HBASE-27763 > Project: HBase > Issue Type: Bug >Reporter: guoxiaojiao >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HBASE-27763) Recover WAL encounter KeeperErrorCode = NoNode cause RegionServer crash
[ https://issues.apache.org/jira/browse/HBASE-27763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng reassigned HBASE-27763: Assignee: zhengsicheng > Recover WAL encounter KeeperErrorCode = NoNode cause RegionServer crash > > > Key: HBASE-27763 > URL: https://issues.apache.org/jira/browse/HBASE-27763 > Project: HBase > Issue Type: Bug >Reporter: guoxiaojiao >Assignee: zhengsicheng >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HBASE-27387) MetricsSource lastShippedTimeStamps ConcurrentModificationException cause RegionServer crash
[ https://issues.apache.org/jira/browse/HBASE-27387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng reassigned HBASE-27387: Assignee: (was: zhengsicheng) > MetricsSource lastShippedTimeStamps ConcurrentModificationException cause > RegionServer crash > - > > Key: HBASE-27387 > URL: https://issues.apache.org/jira/browse/HBASE-27387 > Project: HBase > Issue Type: Bug >Reporter: zhengsicheng >Priority: Minor > > 022-09-20 14:14:40,332 ERROR [regionserver/hostname1:16020] > regionserver.HRegionServer: * ABORTING region server > hostname1,16020,1663147531495: Unhandled: null * > 8587 java.util.ConcurrentModificationException > 8588 at java.util.HashMap$HashIterator.nextNode(HashMap.java:1442) > 8589 at java.util.HashMap$ValueIterator.next(HashMap.java:1471) > 8590 at > org.apache.hadoop.hbase.replication.regionserver.MetricsSource.getTimestampOfLastShippedOp(MetricsSource.java:321) > 8591 at > org.apache.hadoop.hbase.replication.regionserver.ReplicationLoad.buildReplicationLoad(ReplicationLoad.java:80) > 8592 at > org.apache.hadoop.hbase.replication.regionserver.Replication.buildReplicationLoad(Replication.java:264) > 8593 at > org.apache.hadoop.hbase.replication.regionserver.Replication.refreshAndGetReplicationLoad(Replication.java:253) > 8594 at > org.apache.hadoop.hbase.regionserver.HRegionServer.buildServerLoad(HRegionServer.java:1436) > 8595 at > org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1243) > 8596 at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1065) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27664) RS crash on ipc request to big
zhengsicheng created HBASE-27664: Summary: RS crash on ipc request to big Key: HBASE-27664 URL: https://issues.apache.org/jira/browse/HBASE-27664 Project: HBase Issue Type: Bug Reporter: zhengsicheng Assignee: zhengsicheng 2023-02-17 16:44:09,601 WARN [RS-EventLoopGroup-1-46] ipc.NettyRpcServer: RPC data length of 825701220 received from client_ip1 is greater than max allowed 268435456. Set "hba se.ipc.max.request.size" on server to override this limit (not recommended) 9423 2023-02-17 16:44:12,668 ERROR [RpcServer.default.RWQ.Fifo.write.handler=62,queue=62,port=16020] ipc.RpcServer: Unexpected throwable object 9424 java.lang.RuntimeException: Unknown code 98 9425 at org.apache.hadoop.hbase.KeyValue$Type.codeToType(KeyValue.java:276) 9426 at org.apache.hadoop.hbase.CellUtil.getCellKeyAsString(CellUtil.java:1340) 9427 at org.apache.hadoop.hbase.CellUtil.getCellKeyAsString(CellUtil.java:1318) 9428 at org.apache.hadoop.hbase.CellUtil.toString(CellUtil.java:1512) 9429 at org.apache.hadoop.hbase.ByteBufferKeyValue.toString(ByteBufferKeyValue.java:301) 9430 at org.apache.hadoop.hbase.client.Mutation.add(Mutation.java:786) 9431 at org.apache.hadoop.hbase.client.Put.add(Put.java:282) 9432 at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toPut(ProtobufUtil.java:656) 9433 at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1033) 9434 at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:987) 9435 at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:950) 9436 at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2949) 9437 at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45265) 9438 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:394) 9439 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) 9440 at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) 9441 at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318) 9442 2023-02-17 16:44:19,430 WARN [RS-EventLoopGroup-1-60] ipc.RpcServer: Invalid request header: , should have param set in it 9443 2023-02-17 16:44:19,431 WARN [RS-EventLoopGroup-1-60] ipc.RpcServer: /hostname_rs1:16020 is unable to read call parameter from client client_ip1 9444 org.apache.hadoop.hbase.DoNotRetryIOException: Invalid request header: , should have param set in it 9445 at org.apache.hadoop.hbase.ipc.ServerRpcConnection.processRequest(ServerRpcConnection.java:654) 9446 at org.apache.hadoop.hbase.ipc.ServerRpcConnection.processOneRpc(ServerRpcConnection.java:448) 9447 at org.apache.hadoop.hbase.ipc.NettyServerRpcConnection.process(NettyServerRpcConnection.java:89) 9448 at org.apache.hadoop.hbase.ipc.NettyServerRpcConnection.process(NettyServerRpcConnection.java:63) 9449 at org.apache.hadoop.hbase.ipc.NettyRpcServerRequestDecoder.channelRead(NettyRpcServerRequestDecoder.java:62) 9450 at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) 9451 at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) 9452 at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) 9453 at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) 9454 at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:314) 9455 at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:435) 9456 at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) 9457 at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) 9458 at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) 9459 at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) 9460 at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) 9461 at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[jira] [Commented] (HBASE-27307) Add Move table to Target Group on master web
[ https://issues.apache.org/jira/browse/HBASE-27307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692525#comment-17692525 ] zhengsicheng commented on HBASE-27307: -- [~zhangduo] PR updated > Add Move table to Target Group on master web > > > Key: HBASE-27307 > URL: https://issues.apache.org/jira/browse/HBASE-27307 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: move.png > > > Move table to Target Group on master web -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27307) Add Move table to Target Group on master web
[ https://issues.apache.org/jira/browse/HBASE-27307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27307: - Description: Move table to Target Group > Add Move table to Target Group on master web > > > Key: HBASE-27307 > URL: https://issues.apache.org/jira/browse/HBASE-27307 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: move.png > > > Move table to Target Group -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27307) Add Move table to Target Group on master web
[ https://issues.apache.org/jira/browse/HBASE-27307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27307: - Description: Move table to Target Group on master web (was: Move table to Target Group) > Add Move table to Target Group on master web > > > Key: HBASE-27307 > URL: https://issues.apache.org/jira/browse/HBASE-27307 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: move.png > > > Move table to Target Group on master web -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27307) Add Move table to Target Group on master web
[ https://issues.apache.org/jira/browse/HBASE-27307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27307: - Attachment: move.png > Add Move table to Target Group on master web > > > Key: HBASE-27307 > URL: https://issues.apache.org/jira/browse/HBASE-27307 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: move.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27307) Add Move table to Target Group on master web
[ https://issues.apache.org/jira/browse/HBASE-27307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27307: - Description: (was: !image-2023-02-22-01-09-41-209.png|width=2073,height=1127!) > Add Move table to Target Group on master web > > > Key: HBASE-27307 > URL: https://issues.apache.org/jira/browse/HBASE-27307 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27307) Add Move table to Target Group on master web
[ https://issues.apache.org/jira/browse/HBASE-27307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27307: - Attachment: (was: image-2023-02-22-01-09-41-209.png) > Add Move table to Target Group on master web > > > Key: HBASE-27307 > URL: https://issues.apache.org/jira/browse/HBASE-27307 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > > !image-2023-02-22-01-09-41-209.png|width=2073,height=1127! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27307) Add Move table to Target Group on master web
[ https://issues.apache.org/jira/browse/HBASE-27307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27307: - Description: !image-2023-02-22-01-09-41-209.png|width=2073,height=1127! (was: !image-2023-02-22-01-09-41-209.png!) > Add Move table to Target Group on master web > > > Key: HBASE-27307 > URL: https://issues.apache.org/jira/browse/HBASE-27307 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: image-2023-02-22-01-09-41-209.png > > > !image-2023-02-22-01-09-41-209.png|width=2073,height=1127! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27307) Add Move table to Target Group on master web
[ https://issues.apache.org/jira/browse/HBASE-27307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27307: - Description: !image-2023-02-22-01-09-41-209.png! > Add Move table to Target Group on master web > > > Key: HBASE-27307 > URL: https://issues.apache.org/jira/browse/HBASE-27307 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: image-2023-02-22-01-09-41-209.png > > > !image-2023-02-22-01-09-41-209.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27307) Add Move table to Target Group on master web
[ https://issues.apache.org/jira/browse/HBASE-27307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27307: - Attachment: image-2023-02-22-01-09-41-209.png > Add Move table to Target Group on master web > > > Key: HBASE-27307 > URL: https://issues.apache.org/jira/browse/HBASE-27307 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: image-2023-02-22-01-09-41-209.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27523) Add BulkLoad bandwidth throttling
[ https://issues.apache.org/jira/browse/HBASE-27523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683227#comment-17683227 ] zhengsicheng commented on HBASE-27523: -- [~zhangduo] Added description RP available > Add BulkLoad bandwidth throttling > - > > Key: HBASE-27523 > URL: https://issues.apache.org/jira/browse/HBASE-27523 > Project: HBase > Issue Type: New Feature >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: image-2023-02-02-13-10-06-619.png > > > # Background: > During bulk load HFile copy, the server performance is affected to prevent > the bandwidth of the server from being full. We can design a throttler to > limit the speed of copy and thus limit the bandwidth used by copy > # Function: > 2.1 Dynamically Updating Traffic Limiting Configurations > 2.2 Node copy HFile Traffic Limiting > 2.3 Synchronous copy HFile of the Active and Standby Clusters is unlimited > # Mode of Use: > hbase.regionserver.bulkload.node.bandwidth is greater than 0 open unit > Byte/sec > Update the limit value with update_all_config or update_config 'node' > # Effect: > // Begin 5M/sec > > hbase.regionserver.bulkload.node.bandwidth > 5242880 > hbase-site.xml > > // After update_all_config to 10M/sec > > hbase.regionserver.bulkload.node.bandwidth > 5242880 > hbase-site.xml > > !image-2023-02-02-13-10-06-619.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27523) Add BulkLoad bandwidth throttling
[ https://issues.apache.org/jira/browse/HBASE-27523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27523: - Issue Type: New Feature (was: Task) > Add BulkLoad bandwidth throttling > - > > Key: HBASE-27523 > URL: https://issues.apache.org/jira/browse/HBASE-27523 > Project: HBase > Issue Type: New Feature >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: image-2023-02-02-13-10-06-619.png > > > # Background: > During bulk load HFile copy, the server performance is affected to prevent > the bandwidth of the server from being full. We can design a throttler to > limit the speed of copy and thus limit the bandwidth used by copy > # Function: > 2.1 Dynamically Updating Traffic Limiting Configurations > 2.2 Node copy HFile Traffic Limiting > 2.3 Synchronous copy HFile of the Active and Standby Clusters is unlimited > # Mode of Use: > hbase.regionserver.bulkload.node.bandwidth is greater than 0 open unit > Byte/sec > Update the limit value with update_all_config or update_config 'node' > # Effect: > // Begin 5M/sec > > hbase.regionserver.bulkload.node.bandwidth > 5242880 > hbase-site.xml > > // After update_all_config to 10M/sec > > hbase.regionserver.bulkload.node.bandwidth > 5242880 > hbase-site.xml > > !image-2023-02-02-13-10-06-619.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27523) Add BulkLoad bandwidth throttling
[ https://issues.apache.org/jira/browse/HBASE-27523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27523: - Attachment: image-2023-02-02-13-10-06-619.png > Add BulkLoad bandwidth throttling > - > > Key: HBASE-27523 > URL: https://issues.apache.org/jira/browse/HBASE-27523 > Project: HBase > Issue Type: Task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: image-2023-02-02-13-10-06-619.png > > > # Background: > During bulk load HFile copy, the server performance is affected to prevent > the bandwidth of the server from being full. We can design a throttler to > limit the speed of copy and thus limit the bandwidth used by copy > # Function: > 2.1 Dynamically Updating Traffic Limiting Configurations > 2.2 Node copy HFile Traffic Limiting > 2.3 Synchronous copy HFile of the Active and Standby Clusters is unlimited > # Mode of Use: > hbase.regionserver.bulkload.node.bandwidth is greater than 0 open unit > Byte/sec > Update the limit value with update_all_config or update_config 'node' > # Effect: > // Begin 5M/sec > > hbase.regionserver.bulkload.node.bandwidth > 5242880 > hbase-site.xml > > // After update_all_config to 10M/sec > > hbase.regionserver.bulkload.node.bandwidth > 5242880 > hbase-site.xml > > !image-2023-02-02-11-08-58-479.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27523) Add BulkLoad bandwidth throttling
[ https://issues.apache.org/jira/browse/HBASE-27523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27523: - Description: # Background: During bulk load HFile copy, the server performance is affected to prevent the bandwidth of the server from being full. We can design a throttler to limit the speed of copy and thus limit the bandwidth used by copy # Function: 2.1 Dynamically Updating Traffic Limiting Configurations 2.2 Node copy HFile Traffic Limiting 2.3 Synchronous copy HFile of the Active and Standby Clusters is unlimited # Mode of Use: hbase.regionserver.bulkload.node.bandwidth is greater than 0 open unit Byte/sec Update the limit value with update_all_config or update_config 'node' # Effect: // Begin 5M/sec hbase.regionserver.bulkload.node.bandwidth 5242880 hbase-site.xml // After update_all_config to 10M/sec hbase.regionserver.bulkload.node.bandwidth 5242880 hbase-site.xml !image-2023-02-02-13-10-06-619.png! was: # Background: During bulk load HFile copy, the server performance is affected to prevent the bandwidth of the server from being full. We can design a throttler to limit the speed of copy and thus limit the bandwidth used by copy # Function: 2.1 Dynamically Updating Traffic Limiting Configurations 2.2 Node copy HFile Traffic Limiting 2.3 Synchronous copy HFile of the Active and Standby Clusters is unlimited # Mode of Use: hbase.regionserver.bulkload.node.bandwidth is greater than 0 open unit Byte/sec Update the limit value with update_all_config or update_config 'node' # Effect: // Begin 5M/sec hbase.regionserver.bulkload.node.bandwidth 5242880 hbase-site.xml // After update_all_config to 10M/sec hbase.regionserver.bulkload.node.bandwidth 5242880 hbase-site.xml !image-2023-02-02-11-08-58-479.png! > Add BulkLoad bandwidth throttling > - > > Key: HBASE-27523 > URL: https://issues.apache.org/jira/browse/HBASE-27523 > Project: HBase > Issue Type: Task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: image-2023-02-02-13-10-06-619.png > > > # Background: > During bulk load HFile copy, the server performance is affected to prevent > the bandwidth of the server from being full. We can design a throttler to > limit the speed of copy and thus limit the bandwidth used by copy > # Function: > 2.1 Dynamically Updating Traffic Limiting Configurations > 2.2 Node copy HFile Traffic Limiting > 2.3 Synchronous copy HFile of the Active and Standby Clusters is unlimited > # Mode of Use: > hbase.regionserver.bulkload.node.bandwidth is greater than 0 open unit > Byte/sec > Update the limit value with update_all_config or update_config 'node' > # Effect: > // Begin 5M/sec > > hbase.regionserver.bulkload.node.bandwidth > 5242880 > hbase-site.xml > > // After update_all_config to 10M/sec > > hbase.regionserver.bulkload.node.bandwidth > 5242880 > hbase-site.xml > > !image-2023-02-02-13-10-06-619.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27523) Add BulkLoad bandwidth throttling
[ https://issues.apache.org/jira/browse/HBASE-27523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27523: - Attachment: (was: image-2023-02-02-11-08-58-479.png) > Add BulkLoad bandwidth throttling > - > > Key: HBASE-27523 > URL: https://issues.apache.org/jira/browse/HBASE-27523 > Project: HBase > Issue Type: Task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: image-2023-02-02-13-10-06-619.png > > > # Background: > During bulk load HFile copy, the server performance is affected to prevent > the bandwidth of the server from being full. We can design a throttler to > limit the speed of copy and thus limit the bandwidth used by copy > # Function: > 2.1 Dynamically Updating Traffic Limiting Configurations > 2.2 Node copy HFile Traffic Limiting > 2.3 Synchronous copy HFile of the Active and Standby Clusters is unlimited > # Mode of Use: > hbase.regionserver.bulkload.node.bandwidth is greater than 0 open unit > Byte/sec > Update the limit value with update_all_config or update_config 'node' > # Effect: > // Begin 5M/sec > > hbase.regionserver.bulkload.node.bandwidth > 5242880 > hbase-site.xml > > // After update_all_config to 10M/sec > > hbase.regionserver.bulkload.node.bandwidth > 5242880 > hbase-site.xml > > !image-2023-02-02-13-10-06-619.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27523) Add BulkLoad bandwidth throttling
[ https://issues.apache.org/jira/browse/HBASE-27523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27523: - Description: # Background: During bulk load HFile copy, the server performance is affected to prevent the bandwidth of the server from being full. We can design a throttler to limit the speed of copy and thus limit the bandwidth used by copy # Function: 2.1 Dynamically Updating Traffic Limiting Configurations 2.2 Node copy HFile Traffic Limiting 2.3 Synchronous copy HFile of the Active and Standby Clusters is unlimited # Mode of Use: hbase.regionserver.bulkload.node.bandwidth is greater than 0 open unit Byte/sec Update the limit value with update_all_config or update_config 'node' # Effect: // Begin 5M/sec hbase.regionserver.bulkload.node.bandwidth 5242880 hbase-site.xml // After update_all_config to 10M/sec hbase.regionserver.bulkload.node.bandwidth 5242880 hbase-site.xml !image-2023-02-02-11-08-58-479.png! was:!image-2023-02-02-11-08-58-479.png! > Add BulkLoad bandwidth throttling > - > > Key: HBASE-27523 > URL: https://issues.apache.org/jira/browse/HBASE-27523 > Project: HBase > Issue Type: Task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: image-2023-02-02-11-08-58-479.png > > > # Background: > During bulk load HFile copy, the server performance is affected to prevent > the bandwidth of the server from being full. We can design a throttler to > limit the speed of copy and thus limit the bandwidth used by copy > # Function: > 2.1 Dynamically Updating Traffic Limiting Configurations > 2.2 Node copy HFile Traffic Limiting > 2.3 Synchronous copy HFile of the Active and Standby Clusters is unlimited > # Mode of Use: > hbase.regionserver.bulkload.node.bandwidth is greater than 0 open unit > Byte/sec > Update the limit value with update_all_config or update_config 'node' > # Effect: > // Begin 5M/sec > > hbase.regionserver.bulkload.node.bandwidth > 5242880 > hbase-site.xml > > // After update_all_config to 10M/sec > > hbase.regionserver.bulkload.node.bandwidth > 5242880 > hbase-site.xml > > !image-2023-02-02-11-08-58-479.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27523) Add BulkLoad bandwidth throttling
[ https://issues.apache.org/jira/browse/HBASE-27523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27523: - Description: !image-2023-02-02-11-08-58-479.png! > Add BulkLoad bandwidth throttling > - > > Key: HBASE-27523 > URL: https://issues.apache.org/jira/browse/HBASE-27523 > Project: HBase > Issue Type: Task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: image-2023-02-02-11-08-58-479.png > > > !image-2023-02-02-11-08-58-479.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27523) Add BulkLoad bandwidth throttling
[ https://issues.apache.org/jira/browse/HBASE-27523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27523: - Attachment: image-2023-02-02-11-08-58-479.png > Add BulkLoad bandwidth throttling > - > > Key: HBASE-27523 > URL: https://issues.apache.org/jira/browse/HBASE-27523 > Project: HBase > Issue Type: Task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: image-2023-02-02-11-08-58-479.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27569) balance_rsgroup NullPointerException
[ https://issues.apache.org/jira/browse/HBASE-27569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27569: - Description: hbase shell hbase(main):001:0> balance_rsgroup 'default' ERROR: java.io.IOException at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:469) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318) Caused by: java.lang.NullPointerException hbase server log 2023-01-16 18:53:51,418 INFO [RpcServer.default.RWQ.Fifo.read.handler=315,queue=85,port=16000] rsgroup.RSGroupAdminEndpoint: Client=x balance rsgroup, group=default 2023-01-16 18:53:51,421 ERROR [RpcServer.default.RWQ.Fifo.read.handler=315,queue=85,port=16000] ipc.RpcServer: Unexpected throwable object java.lang.NullPointerException was: hbase(main):001:0> balance_rsgroup 'default' ERROR: java.io.IOException at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:469) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318) Caused by: java.lang.NullPointerException > balance_rsgroup NullPointerException > > > Key: HBASE-27569 > URL: https://issues.apache.org/jira/browse/HBASE-27569 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > > hbase shell > hbase(main):001:0> balance_rsgroup 'default' > ERROR: java.io.IOException > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:469) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318) > Caused by: java.lang.NullPointerException > > hbase server log > 2023-01-16 18:53:51,418 INFO > [RpcServer.default.RWQ.Fifo.read.handler=315,queue=85,port=16000] > rsgroup.RSGroupAdminEndpoint: Client=x balance rsgroup, group=default > 2023-01-16 18:53:51,421 ERROR > [RpcServer.default.RWQ.Fifo.read.handler=315,queue=85,port=16000] > ipc.RpcServer: Unexpected throwable object > java.lang.NullPointerException -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27569) balance_rsgroup NullPointerException
zhengsicheng created HBASE-27569: Summary: balance_rsgroup NullPointerException Key: HBASE-27569 URL: https://issues.apache.org/jira/browse/HBASE-27569 Project: HBase Issue Type: Bug Components: Client Reporter: zhengsicheng Assignee: zhengsicheng hbase(main):001:0> balance_rsgroup 'default' ERROR: java.io.IOException at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:469) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318) Caused by: java.lang.NullPointerException -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HBASE-27523) Add BulkLoad bandwidth throttling
[ https://issues.apache.org/jira/browse/HBASE-27523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-27523 started by zhengsicheng. > Add BulkLoad bandwidth throttling > - > > Key: HBASE-27523 > URL: https://issues.apache.org/jira/browse/HBASE-27523 > Project: HBase > Issue Type: Task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27307) Add Move table to Target Group on master web
[ https://issues.apache.org/jira/browse/HBASE-27307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17651123#comment-17651123 ] zhengsicheng commented on HBASE-27307: -- [~zhangduo] PR is available > Add Move table to Target Group on master web > > > Key: HBASE-27307 > URL: https://issues.apache.org/jira/browse/HBASE-27307 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HBASE-27307) Add Move table to Target Group on master web
[ https://issues.apache.org/jira/browse/HBASE-27307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-27307 started by zhengsicheng. > Add Move table to Target Group on master web > > > Key: HBASE-27307 > URL: https://issues.apache.org/jira/browse/HBASE-27307 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27523) Add BulkLoad bandwidth throttling
zhengsicheng created HBASE-27523: Summary: Add BulkLoad bandwidth throttling Key: HBASE-27523 URL: https://issues.apache.org/jira/browse/HBASE-27523 Project: HBase Issue Type: Task Reporter: zhengsicheng Assignee: zhengsicheng -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27309) Add major compact table or region operation on master web table page
[ https://issues.apache.org/jira/browse/HBASE-27309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17631109#comment-17631109 ] zhengsicheng commented on HBASE-27309: -- [~zhangduo] [PR to brach-2|https://github.com/apache/hbase/pull/4870/files] is available > Add major compact table or region operation on master web table page > > > Key: HBASE-27309 > URL: https://issues.apache.org/jira/browse/HBASE-27309 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: image-2022-09-22-02-32-36-619.png > > > Add major compact table or region operation on master web table page > !image-2022-09-22-02-32-36-619.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27309) Add major compact table or region operation on master web table page
[ https://issues.apache.org/jira/browse/HBASE-27309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610569#comment-17610569 ] zhengsicheng commented on HBASE-27309: -- [~zhangduo] PR available > Add major compact table or region operation on master web table page > > > Key: HBASE-27309 > URL: https://issues.apache.org/jira/browse/HBASE-27309 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: image-2022-09-22-02-32-36-619.png > > > Add major compact table or region operation on master web table page > !image-2022-09-22-02-32-36-619.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HBASE-27387) MetricsSource lastShippedTimeStamps ConcurrentModificationException cause RegionServer crash
[ https://issues.apache.org/jira/browse/HBASE-27387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-27387 started by zhengsicheng. > MetricsSource lastShippedTimeStamps ConcurrentModificationException cause > RegionServer crash > - > > Key: HBASE-27387 > URL: https://issues.apache.org/jira/browse/HBASE-27387 > Project: HBase > Issue Type: Bug >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > > 022-09-20 14:14:40,332 ERROR [regionserver/hostname1:16020] > regionserver.HRegionServer: * ABORTING region server > hostname1,16020,1663147531495: Unhandled: null * > 8587 java.util.ConcurrentModificationException > 8588 at java.util.HashMap$HashIterator.nextNode(HashMap.java:1442) > 8589 at java.util.HashMap$ValueIterator.next(HashMap.java:1471) > 8590 at > org.apache.hadoop.hbase.replication.regionserver.MetricsSource.getTimestampOfLastShippedOp(MetricsSource.java:321) > 8591 at > org.apache.hadoop.hbase.replication.regionserver.ReplicationLoad.buildReplicationLoad(ReplicationLoad.java:80) > 8592 at > org.apache.hadoop.hbase.replication.regionserver.Replication.buildReplicationLoad(Replication.java:264) > 8593 at > org.apache.hadoop.hbase.replication.regionserver.Replication.refreshAndGetReplicationLoad(Replication.java:253) > 8594 at > org.apache.hadoop.hbase.regionserver.HRegionServer.buildServerLoad(HRegionServer.java:1436) > 8595 at > org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1243) > 8596 at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1065) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27387) MetricsSource lastShippedTimeStamps ConcurrentModificationException cause RegionServer crash
zhengsicheng created HBASE-27387: Summary: MetricsSource lastShippedTimeStamps ConcurrentModificationException cause RegionServer crash Key: HBASE-27387 URL: https://issues.apache.org/jira/browse/HBASE-27387 Project: HBase Issue Type: Bug Reporter: zhengsicheng Assignee: zhengsicheng 022-09-20 14:14:40,332 ERROR [regionserver/hostname1:16020] regionserver.HRegionServer: * ABORTING region server hostname1,16020,1663147531495: Unhandled: null * 8587 java.util.ConcurrentModificationException 8588 at java.util.HashMap$HashIterator.nextNode(HashMap.java:1442) 8589 at java.util.HashMap$ValueIterator.next(HashMap.java:1471) 8590 at org.apache.hadoop.hbase.replication.regionserver.MetricsSource.getTimestampOfLastShippedOp(MetricsSource.java:321) 8591 at org.apache.hadoop.hbase.replication.regionserver.ReplicationLoad.buildReplicationLoad(ReplicationLoad.java:80) 8592 at org.apache.hadoop.hbase.replication.regionserver.Replication.buildReplicationLoad(Replication.java:264) 8593 at org.apache.hadoop.hbase.replication.regionserver.Replication.refreshAndGetReplicationLoad(Replication.java:253) 8594 at org.apache.hadoop.hbase.regionserver.HRegionServer.buildServerLoad(HRegionServer.java:1436) 8595 at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1243) 8596 at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1065) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27308) Add balance rsgroup operation on master web table page
[ https://issues.apache.org/jira/browse/HBASE-27308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27308: - Summary: Add balance rsgroup operation on master web table page (was: Add split table or region on master web) > Add balance rsgroup operation on master web table page > -- > > Key: HBASE-27308 > URL: https://issues.apache.org/jira/browse/HBASE-27308 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27309) Add major compact table or region operation on master web table page
[ https://issues.apache.org/jira/browse/HBASE-27309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27309: - Description: !image-2022-09-22-02-32-36-619.png! > Add major compact table or region operation on master web table page > > > Key: HBASE-27309 > URL: https://issues.apache.org/jira/browse/HBASE-27309 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: image-2022-09-22-02-32-36-619.png > > > > !image-2022-09-22-02-32-36-619.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27309) Add major compact table or region operation on master web table page
[ https://issues.apache.org/jira/browse/HBASE-27309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27309: - Description: Add major compact table or region operation on master web table page !image-2022-09-22-02-32-36-619.png! was: !image-2022-09-22-02-32-36-619.png! > Add major compact table or region operation on master web table page > > > Key: HBASE-27309 > URL: https://issues.apache.org/jira/browse/HBASE-27309 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: image-2022-09-22-02-32-36-619.png > > > Add major compact table or region operation on master web table page > !image-2022-09-22-02-32-36-619.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27309) Add major compact table or region operation on master web table page
[ https://issues.apache.org/jira/browse/HBASE-27309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27309: - Attachment: image-2022-09-22-02-32-36-619.png > Add major compact table or region operation on master web table page > > > Key: HBASE-27309 > URL: https://issues.apache.org/jira/browse/HBASE-27309 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: image-2022-09-22-02-32-36-619.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27309) Add major compact table or region operation on master web table page
[ https://issues.apache.org/jira/browse/HBASE-27309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27309: - Summary: Add major compact table or region operation on master web table page (was: Add compact table or region on master web) > Add major compact table or region operation on master web table page > > > Key: HBASE-27309 > URL: https://issues.apache.org/jira/browse/HBASE-27309 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27310) Add disabled table is displayed red on master web tables list
[ https://issues.apache.org/jira/browse/HBASE-27310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27310: - Summary: Add disabled table is displayed red on master web tables list (was: Add color show tables on master web) > Add disabled table is displayed red on master web tables list > - > > Key: HBASE-27310 > URL: https://issues.apache.org/jira/browse/HBASE-27310 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: image-2022-09-01-14-27-56-145.png > > > The table of the disabled state is displayed in red > The effect is as follows： > !image-2022-09-01-14-27-56-145.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27310) Add color show tables on master web
[ https://issues.apache.org/jira/browse/HBASE-27310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606827#comment-17606827 ] zhengsicheng commented on HBASE-27310: -- ok ！Thank you for your guidance 发自我的小米在 "Duo Zhang (Jira)" ，2022年9月15日 22:48写道： [ [1]https://issues.apache.org/jira/browse/HBASE-27310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17605387#comment-17605387 ] Duo Zhang commented on HBASE-27310: --- So you mean change the color to red for disabled table? I think it is fine, but let's change the title of this issue to make it more accurate? -- This message was sent by Atlassian Jira (v8.20.10#820010) [1] https://issues.apache.org/jira/browse/HBASE-27310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17605387#comment-17605387 [2] https://issues.apache.org/jira/browse/HBASE-27310 > Add color show tables on master web > --- > > Key: HBASE-27310 > URL: https://issues.apache.org/jira/browse/HBASE-27310 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: image-2022-09-01-14-27-56-145.png > > > The table of the disabled state is displayed in red > The effect is as follows： > !image-2022-09-01-14-27-56-145.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27307) Add Move table to Target Group on master web
[ https://issues.apache.org/jira/browse/HBASE-27307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27307: - Summary: Add Move table to Target Group on master web (was: Add Move table to Target Group) > Add Move table to Target Group on master web > > > Key: HBASE-27307 > URL: https://issues.apache.org/jira/browse/HBASE-27307 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27309) Add compact table or region on master web
[ https://issues.apache.org/jira/browse/HBASE-27309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27309: - Summary: Add compact table or region on master web (was: Add compact table or region) > Add compact table or region on master web > - > > Key: HBASE-27309 > URL: https://issues.apache.org/jira/browse/HBASE-27309 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27308) Add split table or region on master web
[ https://issues.apache.org/jira/browse/HBASE-27308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27308: - Summary: Add split table or region on master web (was: Add split table or region) > Add split table or region on master web > --- > > Key: HBASE-27308 > URL: https://issues.apache.org/jira/browse/HBASE-27308 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27310) Add color show tables on master web
[ https://issues.apache.org/jira/browse/HBASE-27310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27310: - Summary: Add color show tables on master web (was: Add color show tables) > Add color show tables on master web > --- > > Key: HBASE-27310 > URL: https://issues.apache.org/jira/browse/HBASE-27310 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: image-2022-09-01-14-27-56-145.png > > > The table of the disabled state is displayed in red > The effect is as follows： > !image-2022-09-01-14-27-56-145.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27267) Delete causes timestamp to be negative
[
https://issues.apache.org/jira/browse/HBASE-27267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhengsicheng resolved HBASE-27267.
--
Fix Version/s: 2.4.5
2.5.0
3.0.0-alpha-1
Resolution: Resolved
> Delete causes timestamp to be negative
> --
>
> Key: HBASE-27267
> URL: https://issues.apache.org/jira/browse/HBASE-27267
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.3.4
>Reporter: zhengsicheng
>Assignee: zhengsicheng
>Priority: Major
> Fix For: 2.4.5, 2.5.0, 3.0.0-alpha-1
>
> Attachments: image-2022-08-11-17-10-00-389.png, screenshot-1.png
>
>
> When client-1.1.6 and server-2.3.4 there is a case where the batch delete
> timestamp is negative
> # 1. RegionServer log message:
> {code:java}
> 2022-07-19 12:13:29,324 WARN
> [RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
> hbase.KeyValueUtil: Timestamp cannot be negative, ts=-4323977095312258207,
> KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
> 2022-07-19 12:13:29,324 WARN
> [RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
> wal.ProtobufLogReader: Encountered a malformed edit, seeking back to last
> good position in file, from 1099261 to 1078224
> java.io.EOFException: EOF while reading 660 WAL KVs; started reading at
> 1078317 and read up to 1099261
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:403)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:97)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:85)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:264)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:178)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:103)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:230)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:145)
> Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative,
> ts=-4323977095312258207, KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
> at
> org.apache.hadoop.hbase.KeyValueUtil.checkKeyValueBytes(KeyValueUtil.java:612)
> at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:346)
> at
> org.apache.hadoop.hbase.KeyValueUtil.createKeyValueFromInputStream(KeyValueUtil.java:717)
> at
> org.apache.hadoop.hbase.codec.KeyValueCodecWithTags$KeyValueDecoder.parseCell(KeyValueCodecWithTags.java:81)
> at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:68)
> at org.apache.hadoop.hbase.wal.WALEdit.readFromCells(WALEdit.java:276)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:387)
> ... 7 more
> {code}
> # 2. Debug WAL file ,found that the delete operation is caused
> {code:java}
> Sequence=365693989, table=tableA, region=148cedb7b8ca3145690800fd650e084d, at
> write timestamp=Sat Jul 16 00:50:01 CST 2022
> 2022-07-22 22:09:43,244 ERROR [main] wal.WALPrettyPrinter: Timestamp is
> negative row=rowkey1, column=d:act, timestamp=-4323977095312258207,
> type=Delete
> {code}
> # 3. User use spark read/write hbase
> batchsize is 1
> {code:scala}
> def dataDeleteFromHbase(rdd: RDD[(String, String)], hbase_table: String,
> hbase_instance: String, hbase_accesskey: String, accumulator:
> LongAccumulator, buffersize: String, batchsize: Int): Unit = {
> rdd.foreachPartition(iterator => {
> val partitionId = TaskContext.getPartitionId()
> val conf = HBaseConfiguration.create()
> val connection = SparkHbaseUtils.getconnection(conf)
> val table = connection.getTable(TableName.valueOf(hbase_table))
> var deleteList = new util.LinkedList[Delete]()
> var count = 0
> var batchCount = 0
> while (iterator.hasNext) {
> val element = iterator.next
> val crc32 = new CRC32()
> crc32.update(s"${element._1}_${element._2}".getBytes())
> val crcArr = convertLow4bit2SmallEndan(crc32.getValue)
> val key = concat(DigestUtils.md5(s"${element._1}_${element._2}"),
> crcArr)
> val delete = new Delete(key)
> deleteList.add(delete)
> count
[jira] [Commented] (HBASE-27267) Delete causes timestamp to be negative
[
https://issues.apache.org/jira/browse/HBASE-27267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17600816#comment-17600816
]
zhengsicheng commented on HBASE-27267:
--
HBASE-2.3.4 merge https://issues.apache.org/jira/browse/HBASE-26036 solve the
problem。
> Delete causes timestamp to be negative
> --
>
> Key: HBASE-27267
> URL: https://issues.apache.org/jira/browse/HBASE-27267
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.3.4
>Reporter: zhengsicheng
>Assignee: zhengsicheng
>Priority: Major
> Attachments: image-2022-08-11-17-10-00-389.png, screenshot-1.png
>
>
> When client-1.1.6 and server-2.3.4 there is a case where the batch delete
> timestamp is negative
> # 1. RegionServer log message:
> {code:java}
> 2022-07-19 12:13:29,324 WARN
> [RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
> hbase.KeyValueUtil: Timestamp cannot be negative, ts=-4323977095312258207,
> KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
> 2022-07-19 12:13:29,324 WARN
> [RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
> wal.ProtobufLogReader: Encountered a malformed edit, seeking back to last
> good position in file, from 1099261 to 1078224
> java.io.EOFException: EOF while reading 660 WAL KVs; started reading at
> 1078317 and read up to 1099261
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:403)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:97)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:85)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:264)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:178)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:103)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:230)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:145)
> Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative,
> ts=-4323977095312258207, KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
> at
> org.apache.hadoop.hbase.KeyValueUtil.checkKeyValueBytes(KeyValueUtil.java:612)
> at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:346)
> at
> org.apache.hadoop.hbase.KeyValueUtil.createKeyValueFromInputStream(KeyValueUtil.java:717)
> at
> org.apache.hadoop.hbase.codec.KeyValueCodecWithTags$KeyValueDecoder.parseCell(KeyValueCodecWithTags.java:81)
> at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:68)
> at org.apache.hadoop.hbase.wal.WALEdit.readFromCells(WALEdit.java:276)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:387)
> ... 7 more
> {code}
> # 2. Debug WAL file ,found that the delete operation is caused
> {code:java}
> Sequence=365693989, table=tableA, region=148cedb7b8ca3145690800fd650e084d, at
> write timestamp=Sat Jul 16 00:50:01 CST 2022
> 2022-07-22 22:09:43,244 ERROR [main] wal.WALPrettyPrinter: Timestamp is
> negative row=rowkey1, column=d:act, timestamp=-4323977095312258207,
> type=Delete
> {code}
> # 3. User use spark read/write hbase
> batchsize is 1
> {code:scala}
> def dataDeleteFromHbase(rdd: RDD[(String, String)], hbase_table: String,
> hbase_instance: String, hbase_accesskey: String, accumulator:
> LongAccumulator, buffersize: String, batchsize: Int): Unit = {
> rdd.foreachPartition(iterator => {
> val partitionId = TaskContext.getPartitionId()
> val conf = HBaseConfiguration.create()
> val connection = SparkHbaseUtils.getconnection(conf)
> val table = connection.getTable(TableName.valueOf(hbase_table))
> var deleteList = new util.LinkedList[Delete]()
> var count = 0
> var batchCount = 0
> while (iterator.hasNext) {
> val element = iterator.next
> val crc32 = new CRC32()
> crc32.update(s"${element._1}_${element._2}".getBytes())
> val crcArr = convertLow4bit2SmallEndan(crc32.getValue)
> val key = concat(DigestUtils.md5(s"${element._1}_${element._2}"),
> crcArr)
> val delete = new Delete(key)
> deleteList.add(delete)
> count += 1
> if (count %
[jira] [Work started] (HBASE-27309) Add compact table or region
[ https://issues.apache.org/jira/browse/HBASE-27309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-27309 started by zhengsicheng. > Add compact table or region > --- > > Key: HBASE-27309 > URL: https://issues.apache.org/jira/browse/HBASE-27309 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27310) Add color show tables
[ https://issues.apache.org/jira/browse/HBASE-27310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17598760#comment-17598760 ] zhengsicheng commented on HBASE-27310: -- [https://github.com/apache/hbase/pull/4755/files] [~zhangduo] This PR is available > Add color show tables > - > > Key: HBASE-27310 > URL: https://issues.apache.org/jira/browse/HBASE-27310 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: image-2022-09-01-14-27-56-145.png > > > The table of the disabled state is displayed in red > The effect is as follows： > !image-2022-09-01-14-27-56-145.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HBASE-27310) Add color show tables
[ https://issues.apache.org/jira/browse/HBASE-27310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-27310 started by zhengsicheng. > Add color show tables > - > > Key: HBASE-27310 > URL: https://issues.apache.org/jira/browse/HBASE-27310 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: image-2022-09-01-14-27-56-145.png > > > The table of the disabled state is displayed in red > The effect is as follows： > !image-2022-09-01-14-27-56-145.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27310) Add color show tables
[ https://issues.apache.org/jira/browse/HBASE-27310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27310: - Description: The table of the disabled state is displayed in red The effect is as follows： !image-2022-09-01-14-27-56-145.png! was:The table of the disabled state is displayed in red > Add color show tables > - > > Key: HBASE-27310 > URL: https://issues.apache.org/jira/browse/HBASE-27310 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: image-2022-09-01-14-27-56-145.png > > > The table of the disabled state is displayed in red > The effect is as follows： > !image-2022-09-01-14-27-56-145.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27310) Add color show tables
[ https://issues.apache.org/jira/browse/HBASE-27310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27310: - Attachment: image-2022-09-01-14-27-56-145.png > Add color show tables > - > > Key: HBASE-27310 > URL: https://issues.apache.org/jira/browse/HBASE-27310 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: image-2022-09-01-14-27-56-145.png > > > The table of the disabled state is displayed in red -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HBASE-27306) HBase-web adds shortcut operations
[ https://issues.apache.org/jira/browse/HBASE-27306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-27306 started by zhengsicheng. > HBase-web adds shortcut operations > -- > > Key: HBASE-27306 > URL: https://issues.apache.org/jira/browse/HBASE-27306 > Project: HBase > Issue Type: Umbrella >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > > hbase-web adds shortcut operations to improve the operational efficiency of > DBA personnel. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27310) Add color show tables
zhengsicheng created HBASE-27310: Summary: Add color show tables Key: HBASE-27310 URL: https://issues.apache.org/jira/browse/HBASE-27310 Project: HBase Issue Type: Sub-task Reporter: zhengsicheng Assignee: zhengsicheng The table of the disabled state is displayed in red -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27309) Add compact table or region
zhengsicheng created HBASE-27309: Summary: Add compact table or region Key: HBASE-27309 URL: https://issues.apache.org/jira/browse/HBASE-27309 Project: HBase Issue Type: Sub-task Reporter: zhengsicheng Assignee: zhengsicheng -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27308) Add split table or region
[ https://issues.apache.org/jira/browse/HBASE-27308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27308: - Summary: Add split table or region (was: Add split region) > Add split table or region > - > > Key: HBASE-27308 > URL: https://issues.apache.org/jira/browse/HBASE-27308 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27308) Add split region
zhengsicheng created HBASE-27308: Summary: Add split region Key: HBASE-27308 URL: https://issues.apache.org/jira/browse/HBASE-27308 Project: HBase Issue Type: Sub-task Reporter: zhengsicheng Assignee: zhengsicheng -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27307) Add Move table to Target Group
zhengsicheng created HBASE-27307: Summary: Add Move table to Target Group Key: HBASE-27307 URL: https://issues.apache.org/jira/browse/HBASE-27307 Project: HBase Issue Type: Sub-task Reporter: zhengsicheng Assignee: zhengsicheng -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27306) HBase-web adds shortcut operations
[ https://issues.apache.org/jira/browse/HBASE-27306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27306: - Summary: HBase-web adds shortcut operations (was: hbase-web adds shortcut operations) > HBase-web adds shortcut operations > -- > > Key: HBASE-27306 > URL: https://issues.apache.org/jira/browse/HBASE-27306 > Project: HBase > Issue Type: Umbrella >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > > hbase-web adds shortcut operations to improve the operational efficiency of > DBA personnel. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27306) hbase-web adds shortcut operations
zhengsicheng created HBASE-27306: Summary: hbase-web adds shortcut operations Key: HBASE-27306 URL: https://issues.apache.org/jira/browse/HBASE-27306 Project: HBase Issue Type: Umbrella Reporter: zhengsicheng Assignee: zhengsicheng hbase-web adds shortcut operations to improve the operational efficiency of DBA personnel. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (HBASE-27249) Remove invalid peer RegionServer crash
[
https://issues.apache.org/jira/browse/HBASE-27249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17572199#comment-17572199
]
zhengsicheng edited comment on HBASE-27249 at 8/16/22 12:24 PM:
[~zhangduo] When add peer cluster, but peer cluster zookeeper shuotdown or
invalid. remove peer cause source cluster RS abort.
was (Author: zhengsicheng):
[~zhangduo] When add peer cluster, but peer cluster zookeeper shuotdown or
invalid cause source cluster RS abort.
> Remove invalid peer RegionServer crash
> --
>
> Key: HBASE-27249
> URL: https://issues.apache.org/jira/browse/HBASE-27249
> Project: HBase
> Issue Type: Bug
>Reporter: zhengsicheng
>Assignee: zhengsicheng
>Priority: Major
>
> add_peer 'test', CLUSTER_KEY => "zookeeper-01:2181:/hbase_01"
> remove_peer 'test'
> find add peer wrong, remove peer but regionserver crash
> The log information is as follows:
> 2022-07-18 13:26:11,016 ERROR
> [ReadOnlyZKClient-zookeeper-01:2181@0x44281bff-SendThread(zookeeper-01:2181)]
> client.StaticHostProvider: Unable to resolve address:
> zookeeper-01/:2181
> java.net.UnknownHostException: zookeeper-01
> at
> java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:800)
> at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1507)
> at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1366)
> at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1300)
> at
> org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
> at
> org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
> at
> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
> 2022-07-18 13:26:11,016 WARN
> [ReadOnlyZKClient-zookeeper-01:2181@0x44281bff-SendThread(zookeeper-01:2181)]
> zookeeper.ClientCnxn: Session 0x0 for server zookeeper-01/:2181,
> unexpected error, closing socket connection and attempting reconnect
> java.lang.IllegalArgumentException: Unable to canonicalize address
> zookeeper-01/:2181 because it's not resolvable
> at
> org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
> at
> org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
> 2022-07-18 13:26:11,116 WARN [ReadOnlyZKClient-zookeeper-01:2181@0x44281bff]
> zookeeper.ReadOnlyZKClient: 0x44281bff to zookeeper-01:2181 failed for get of
> /hbase_01/hbaseid, code = CONNECTIONLOSS, retries = 48
> 2022-07-18 13:26:11,119 WARN [regionserver/ip1:16020.logRoller]
> regionserver.ReplicationSource: peerId=test, WAL group
> ip1%2C16020%2C1658118295598.ip1%2C16020%2C1658118295598.regiongroup-2 queue
> size: 11 exceeds value of replication.source.log.queue.warn 2
> 2022-07-18 13:26:12,055 INFO [MemStoreFlusher.1] regionserver.HRegion:
> Flushing 31bbfb9b76b6795e5d44fabd113174c0 1/2 column families,
> dataSize=245.67 MB heapSize=257.48 MB; f1={dataSize=245.67 MB,
> heapSize=257.48 MB, offHeapSize=0 B}
> 2022-07-18 13:26:12,116 ERROR
> [ReadOnlyZKClient-zookeeper-01:2181@0x44281bff-SendThread(zookeeper-01:2181)]
> client.StaticHostProvider: Unable to resolve address:
> zookeeper-01/:2181
> java.net.UnknownHostException: zookeeper-01
> at
> java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:800)
> at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1507)
> at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1366)
> at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1300)
> at
> org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
> at
> org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
> at
> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
> 2022-07-18 13:26:30,270 INFO [RS_REFRESH_PEER-regionserver/ip1:16020-1]
> regionserver.RefreshPeerCallable: Received a peer change event, peerId=test,
> type=REMOVE_PEER
> 2022-07-18 13:26:30,270 INFO [RS_REFRESH_PEER-regionserver/ip1:16020-1]
> regionserver.ReplicationSourceManager: Number of deleted recovered sources
> for test: 0
> 2022-07-18 13:26:30,270 INFO [RS_REFRESH_PEER-regionserver/ip1:16020-1]
> regionserver.ReplicationSource: peerId=test, Closing source test
[jira] [Work started] (HBASE-27249) Remove invalid peer RegionServer crash
[
https://issues.apache.org/jira/browse/HBASE-27249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-27249 started by zhengsicheng.
> Remove invalid peer RegionServer crash
> --
>
> Key: HBASE-27249
> URL: https://issues.apache.org/jira/browse/HBASE-27249
> Project: HBase
> Issue Type: Bug
>Reporter: zhengsicheng
>Assignee: zhengsicheng
>Priority: Major
>
> add_peer 'test', CLUSTER_KEY => "zookeeper-01:2181:/hbase_01"
> remove_peer 'test'
> find add peer wrong, remove peer but regionserver crash
> The log information is as follows:
> 2022-07-18 13:26:11,016 ERROR
> [ReadOnlyZKClient-zookeeper-01:2181@0x44281bff-SendThread(zookeeper-01:2181)]
> client.StaticHostProvider: Unable to resolve address:
> zookeeper-01/:2181
> java.net.UnknownHostException: zookeeper-01
> at
> java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:800)
> at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1507)
> at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1366)
> at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1300)
> at
> org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
> at
> org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
> at
> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
> 2022-07-18 13:26:11,016 WARN
> [ReadOnlyZKClient-zookeeper-01:2181@0x44281bff-SendThread(zookeeper-01:2181)]
> zookeeper.ClientCnxn: Session 0x0 for server zookeeper-01/:2181,
> unexpected error, closing socket connection and attempting reconnect
> java.lang.IllegalArgumentException: Unable to canonicalize address
> zookeeper-01/:2181 because it's not resolvable
> at
> org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
> at
> org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
> 2022-07-18 13:26:11,116 WARN [ReadOnlyZKClient-zookeeper-01:2181@0x44281bff]
> zookeeper.ReadOnlyZKClient: 0x44281bff to zookeeper-01:2181 failed for get of
> /hbase_01/hbaseid, code = CONNECTIONLOSS, retries = 48
> 2022-07-18 13:26:11,119 WARN [regionserver/ip1:16020.logRoller]
> regionserver.ReplicationSource: peerId=test, WAL group
> ip1%2C16020%2C1658118295598.ip1%2C16020%2C1658118295598.regiongroup-2 queue
> size: 11 exceeds value of replication.source.log.queue.warn 2
> 2022-07-18 13:26:12,055 INFO [MemStoreFlusher.1] regionserver.HRegion:
> Flushing 31bbfb9b76b6795e5d44fabd113174c0 1/2 column families,
> dataSize=245.67 MB heapSize=257.48 MB; f1={dataSize=245.67 MB,
> heapSize=257.48 MB, offHeapSize=0 B}
> 2022-07-18 13:26:12,116 ERROR
> [ReadOnlyZKClient-zookeeper-01:2181@0x44281bff-SendThread(zookeeper-01:2181)]
> client.StaticHostProvider: Unable to resolve address:
> zookeeper-01/:2181
> java.net.UnknownHostException: zookeeper-01
> at
> java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:800)
> at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1507)
> at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1366)
> at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1300)
> at
> org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
> at
> org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
> at
> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
> 2022-07-18 13:26:30,270 INFO [RS_REFRESH_PEER-regionserver/ip1:16020-1]
> regionserver.RefreshPeerCallable: Received a peer change event, peerId=test,
> type=REMOVE_PEER
> 2022-07-18 13:26:30,270 INFO [RS_REFRESH_PEER-regionserver/ip1:16020-1]
> regionserver.ReplicationSourceManager: Number of deleted recovered sources
> for test: 0
> 2022-07-18 13:26:30,270 INFO [RS_REFRESH_PEER-regionserver/ip1:16020-1]
> regionserver.ReplicationSource: peerId=test, Closing source test because:
> Replication stream was removed by a user
> 2022-07-18 13:26:30,271 WARN
> [RS_REFRESH_PEER-regionserver/ip1:16020-0.replicationSource,test]
> client.ConnectionImplementation: Retrieve cluster id failed
> java.lang.InterruptedException
> at
> java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:385)
>
[jira] [Updated] (HBASE-27301) Add Delete addFamilyVersion timestamp verify
[ https://issues.apache.org/jira/browse/HBASE-27301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27301: - Summary: Add Delete addFamilyVersion timestamp verify (was: Add "org.apache.hadoop.hbase.client.Delete#addFamilyVersion" timestamp verify) > Add Delete addFamilyVersion timestamp verify > - > > Key: HBASE-27301 > URL: https://issues.apache.org/jira/browse/HBASE-27301 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HBASE-27301) Add "org.apache.hadoop.hbase.client.Delete#addFamilyVersion" timestamp verify
[ https://issues.apache.org/jira/browse/HBASE-27301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-27301 started by zhengsicheng. > Add "org.apache.hadoop.hbase.client.Delete#addFamilyVersion" timestamp verify > - > > Key: HBASE-27301 > URL: https://issues.apache.org/jira/browse/HBASE-27301 > Project: HBase > Issue Type: Sub-task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27301) Add "org.apache.hadoop.hbase.client.Delete#addFamilyVersion" timestamp verify
zhengsicheng created HBASE-27301: Summary: Add "org.apache.hadoop.hbase.client.Delete#addFamilyVersion" timestamp verify Key: HBASE-27301 URL: https://issues.apache.org/jira/browse/HBASE-27301 Project: HBase Issue Type: Sub-task Reporter: zhengsicheng Assignee: zhengsicheng -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27300) HBASE client add cache used to store non-existent tables that are high frequency accessed
[ https://issues.apache.org/jira/browse/HBASE-27300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27300: - Description: Table already drop. but the task of operating hbase has not stopped. A large number of concurrent accesses to non-existing tables cause the CPU of the machine where the meta table is located to be too high and cannot process other requests. > HBASE client add cache used to store non-existent tables that are high > frequency accessed > -- > > Key: HBASE-27300 > URL: https://issues.apache.org/jira/browse/HBASE-27300 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > > Table already drop. but the task of operating hbase has not stopped. > A large number of concurrent accesses to non-existing tables cause the CPU of > the machine where the meta table is located to be too high and cannot process > other requests. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27300) HBASE client add cache used to store non-existent tables that are high frequency accessed
zhengsicheng created HBASE-27300: Summary: HBASE client add cache used to store non-existent tables that are high frequency accessed Key: HBASE-27300 URL: https://issues.apache.org/jira/browse/HBASE-27300 Project: HBase Issue Type: Sub-task Components: Client Reporter: zhengsicheng Assignee: zhengsicheng -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (HBASE-27267) Delete causes timestamp to be negative
[
https://issues.apache.org/jira/browse/HBASE-27267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578336#comment-17578336
]
zhengsicheng edited comment on HBASE-27267 at 8/11/22 9:18 AM:
---
Add timestamp judgment in "org.apache.hadoop.hbase.KeyValueUtil#oswrite", the
code is as follows.
!screenshot-1.png!
Deployed to an online HBASE cluster and ran for a while and reproduced
"Timestamp cannot be negative"
!image-2022-08-11-17-10-00-389.png!
The jstack information is as follows：
{code:java}
java.io.IOException: Timestamp cannot be negative
at org.apache.hadoop.hbase.KeyValueUtil.oswrite(KeyValueUtil.java:807)
at
org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$EnsureKvEncoder.write(WALCellCodec.java:363)
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.append(ProtobufLogWriter.java:60)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.doAppend(FSHLog.java:320)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.doAppend(FSHLog.java:72)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.appendEntry(AbstractFSWAL.java:1009)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1151)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1040)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:949)
at
com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
at java.lang.Thread.run(Thread.java:748)
{code}
>From the above positioning information, the problem lies in writing wal
>[~zhangduo] Thanks for following this issue.
Client use 1.1.6 and hbase server is 2.3.4
was (Author: zhengsicheng):
Add timestamp judgment in "org.apache.hadoop.hbase.KeyValueUtil#oswrite", the
code is as follows.
!screenshot-1.png!
Deployed to an online HBASE cluster and ran for a while and reproduced
"Timestamp cannot be negative"
!image-2022-08-11-17-10-00-389.png!
The stack information is as follows：
{code:java}
java.io.IOException: Timestamp cannot be negative
at org.apache.hadoop.hbase.KeyValueUtil.oswrite(KeyValueUtil.java:807)
at
org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$EnsureKvEncoder.write(WALCellCodec.java:363)
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.append(ProtobufLogWriter.java:60)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.doAppend(FSHLog.java:320)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.doAppend(FSHLog.java:72)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.appendEntry(AbstractFSWAL.java:1009)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1151)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1040)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:949)
at
com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
at java.lang.Thread.run(Thread.java:748)
{code}
> Delete causes timestamp to be negative
> --
>
> Key: HBASE-27267
> URL: https://issues.apache.org/jira/browse/HBASE-27267
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.3.4
>Reporter: zhengsicheng
>Assignee: zhengsicheng
>Priority: Major
> Attachments: image-2022-08-11-17-10-00-389.png, screenshot-1.png
>
>
> When client-1.1.6 and server-2.3.4 there is a case where the batch delete
> timestamp is negative
> # 1. RegionServer log message:
> {code:java}
> 2022-07-19 12:13:29,324 WARN
> [RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
> hbase.KeyValueUtil: Timestamp cannot be negative, ts=-4323977095312258207,
> KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
> 2022-07-19 12:13:29,324 WARN
> [RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
> wal.ProtobufLogReader: Encountered a malformed edit, seeking back to last
> good position in file, from 1099261 to 1078224
> java.io.EOFException: EOF while reading 660 WAL KVs; started reading at
> 1078317 and read up to 1099261
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:403)
> at
>
[jira] [Comment Edited] (HBASE-27267) Delete causes timestamp to be negative
[
https://issues.apache.org/jira/browse/HBASE-27267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578336#comment-17578336
]
zhengsicheng edited comment on HBASE-27267 at 8/11/22 9:12 AM:
---
Add timestamp judgment in "org.apache.hadoop.hbase.KeyValueUtil#oswrite", the
code is as follows.
!screenshot-1.png!
Deployed to an online HBASE cluster and ran for a while and reproduced
"Timestamp cannot be negative"
!image-2022-08-11-17-10-00-389.png!
The stack information is as follows：
{code:java}
java.io.IOException: Timestamp cannot be negative
at org.apache.hadoop.hbase.KeyValueUtil.oswrite(KeyValueUtil.java:807)
at
org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$EnsureKvEncoder.write(WALCellCodec.java:363)
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.append(ProtobufLogWriter.java:60)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.doAppend(FSHLog.java:320)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.doAppend(FSHLog.java:72)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.appendEntry(AbstractFSWAL.java:1009)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1151)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1040)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:949)
at
com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
at java.lang.Thread.run(Thread.java:748)
{code}
was (Author: zhengsicheng):
Add timestamp judgment in "org.apache.hadoop.hbase.KeyValueUtil#oswrite", the
code is as follows.
!screenshot-1.png!
Deployed to an online HBASE cluster and ran for a while and reproduced
"Timestamp cannot be negative"
!image-2022-08-11-17-10-00-389.png!
> Delete causes timestamp to be negative
> --
>
> Key: HBASE-27267
> URL: https://issues.apache.org/jira/browse/HBASE-27267
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.3.4
>Reporter: zhengsicheng
>Assignee: zhengsicheng
>Priority: Major
> Attachments: image-2022-08-11-17-10-00-389.png, screenshot-1.png
>
>
> When client-1.1.6 and server-2.3.4 there is a case where the batch delete
> timestamp is negative
> # 1. RegionServer log message:
> {code:java}
> 2022-07-19 12:13:29,324 WARN
> [RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
> hbase.KeyValueUtil: Timestamp cannot be negative, ts=-4323977095312258207,
> KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
> 2022-07-19 12:13:29,324 WARN
> [RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
> wal.ProtobufLogReader: Encountered a malformed edit, seeking back to last
> good position in file, from 1099261 to 1078224
> java.io.EOFException: EOF while reading 660 WAL KVs; started reading at
> 1078317 and read up to 1099261
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:403)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:97)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:85)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:264)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:178)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:103)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:230)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:145)
> Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative,
> ts=-4323977095312258207, KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
> at
> org.apache.hadoop.hbase.KeyValueUtil.checkKeyValueBytes(KeyValueUtil.java:612)
> at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:346)
> at
> org.apache.hadoop.hbase.KeyValueUtil.createKeyValueFromInputStream(KeyValueUtil.java:717)
> at
> org.apache.hadoop.hbase.codec.KeyValueCodecWithTags$KeyValueDecoder.parseCell(KeyValueCodecWithTags.java:81)
> at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:68)
>
[jira] [Comment Edited] (HBASE-27267) Delete causes timestamp to be negative
[
https://issues.apache.org/jira/browse/HBASE-27267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578336#comment-17578336
]
zhengsicheng edited comment on HBASE-27267 at 8/11/22 9:10 AM:
---
Add timestamp judgment in "org.apache.hadoop.hbase.KeyValueUtil#oswrite", the
code is as follows.
!screenshot-1.png!
Deployed to an online HBASE cluster and ran for a while and reproduced
"Timestamp cannot be negative"
!image-2022-08-11-17-10-00-389.png!
was (Author: zhengsicheng):
Add timestamp judgment in "org.apache.hadoop.hbase.KeyValueUtil#oswrite", the
code is as follows.
!screenshot-1.png!
Deployed to an online HBASE cluster and ran for a while and reproduced
"Timestamp cannot be negative"
> Delete causes timestamp to be negative
> --
>
> Key: HBASE-27267
> URL: https://issues.apache.org/jira/browse/HBASE-27267
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.3.4
>Reporter: zhengsicheng
>Assignee: zhengsicheng
>Priority: Major
> Attachments: image-2022-08-11-17-10-00-389.png, screenshot-1.png
>
>
> When client-1.1.6 and server-2.3.4 there is a case where the batch delete
> timestamp is negative
> # 1. RegionServer log message:
> {code:java}
> 2022-07-19 12:13:29,324 WARN
> [RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
> hbase.KeyValueUtil: Timestamp cannot be negative, ts=-4323977095312258207,
> KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
> 2022-07-19 12:13:29,324 WARN
> [RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
> wal.ProtobufLogReader: Encountered a malformed edit, seeking back to last
> good position in file, from 1099261 to 1078224
> java.io.EOFException: EOF while reading 660 WAL KVs; started reading at
> 1078317 and read up to 1099261
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:403)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:97)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:85)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:264)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:178)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:103)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:230)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:145)
> Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative,
> ts=-4323977095312258207, KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
> at
> org.apache.hadoop.hbase.KeyValueUtil.checkKeyValueBytes(KeyValueUtil.java:612)
> at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:346)
> at
> org.apache.hadoop.hbase.KeyValueUtil.createKeyValueFromInputStream(KeyValueUtil.java:717)
> at
> org.apache.hadoop.hbase.codec.KeyValueCodecWithTags$KeyValueDecoder.parseCell(KeyValueCodecWithTags.java:81)
> at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:68)
> at org.apache.hadoop.hbase.wal.WALEdit.readFromCells(WALEdit.java:276)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:387)
> ... 7 more
> {code}
> # 2. Debug WAL file ,found that the delete operation is caused
> {code:java}
> Sequence=365693989, table=tableA, region=148cedb7b8ca3145690800fd650e084d, at
> write timestamp=Sat Jul 16 00:50:01 CST 2022
> 2022-07-22 22:09:43,244 ERROR [main] wal.WALPrettyPrinter: Timestamp is
> negative row=rowkey1, column=d:act, timestamp=-4323977095312258207,
> type=Delete
> {code}
> # 3. User use spark read/write hbase
> batchsize is 1
> {code:scala}
> def dataDeleteFromHbase(rdd: RDD[(String, String)], hbase_table: String,
> hbase_instance: String, hbase_accesskey: String, accumulator:
> LongAccumulator, buffersize: String, batchsize: Int): Unit = {
> rdd.foreachPartition(iterator => {
> val partitionId = TaskContext.getPartitionId()
> val conf = HBaseConfiguration.create()
> val connection = SparkHbaseUtils.getconnection(conf)
> val table = connection.getTable(TableName.valueOf(hbase_table))
> var deleteList = new util.LinkedList[Delete]()
> var count = 0
[jira] [Commented] (HBASE-27267) Delete causes timestamp to be negative
[
https://issues.apache.org/jira/browse/HBASE-27267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578336#comment-17578336
]
zhengsicheng commented on HBASE-27267:
--
Add timestamp judgment in "org.apache.hadoop.hbase.KeyValueUtil#oswrite", the
code is as follows.
!screenshot-1.png!
Deployed to an online HBASE cluster and ran for a while and reproduced
"Timestamp cannot be negative"
> Delete causes timestamp to be negative
> --
>
> Key: HBASE-27267
> URL: https://issues.apache.org/jira/browse/HBASE-27267
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.3.4
>Reporter: zhengsicheng
>Assignee: zhengsicheng
>Priority: Major
> Attachments: screenshot-1.png
>
>
> When client-1.1.6 and server-2.3.4 there is a case where the batch delete
> timestamp is negative
> # 1. RegionServer log message:
> {code:java}
> 2022-07-19 12:13:29,324 WARN
> [RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
> hbase.KeyValueUtil: Timestamp cannot be negative, ts=-4323977095312258207,
> KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
> 2022-07-19 12:13:29,324 WARN
> [RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
> wal.ProtobufLogReader: Encountered a malformed edit, seeking back to last
> good position in file, from 1099261 to 1078224
> java.io.EOFException: EOF while reading 660 WAL KVs; started reading at
> 1078317 and read up to 1099261
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:403)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:97)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:85)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:264)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:178)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:103)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:230)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:145)
> Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative,
> ts=-4323977095312258207, KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
> at
> org.apache.hadoop.hbase.KeyValueUtil.checkKeyValueBytes(KeyValueUtil.java:612)
> at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:346)
> at
> org.apache.hadoop.hbase.KeyValueUtil.createKeyValueFromInputStream(KeyValueUtil.java:717)
> at
> org.apache.hadoop.hbase.codec.KeyValueCodecWithTags$KeyValueDecoder.parseCell(KeyValueCodecWithTags.java:81)
> at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:68)
> at org.apache.hadoop.hbase.wal.WALEdit.readFromCells(WALEdit.java:276)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:387)
> ... 7 more
> {code}
> # 2. Debug WAL file ,found that the delete operation is caused
> {code:java}
> Sequence=365693989, table=tableA, region=148cedb7b8ca3145690800fd650e084d, at
> write timestamp=Sat Jul 16 00:50:01 CST 2022
> 2022-07-22 22:09:43,244 ERROR [main] wal.WALPrettyPrinter: Timestamp is
> negative row=rowkey1, column=d:act, timestamp=-4323977095312258207,
> type=Delete
> {code}
> # 3. User use spark read/write hbase
> batchsize is 1
> {code:scala}
> def dataDeleteFromHbase(rdd: RDD[(String, String)], hbase_table: String,
> hbase_instance: String, hbase_accesskey: String, accumulator:
> LongAccumulator, buffersize: String, batchsize: Int): Unit = {
> rdd.foreachPartition(iterator => {
> val partitionId = TaskContext.getPartitionId()
> val conf = HBaseConfiguration.create()
> val connection = SparkHbaseUtils.getconnection(conf)
> val table = connection.getTable(TableName.valueOf(hbase_table))
> var deleteList = new util.LinkedList[Delete]()
> var count = 0
> var batchCount = 0
> while (iterator.hasNext) {
> val element = iterator.next
> val crc32 = new CRC32()
> crc32.update(s"${element._1}_${element._2}".getBytes())
> val crcArr = convertLow4bit2SmallEndan(crc32.getValue)
> val key = concat(DigestUtils.md5(s"${element._1}_${element._2}"),
> crcArr)
> val delete =
[jira] [Updated] (HBASE-27267) Delete causes timestamp to be negative
[
https://issues.apache.org/jira/browse/HBASE-27267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhengsicheng updated HBASE-27267:
-
Attachment: screenshot-1.png
> Delete causes timestamp to be negative
> --
>
> Key: HBASE-27267
> URL: https://issues.apache.org/jira/browse/HBASE-27267
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.3.4
>Reporter: zhengsicheng
>Assignee: zhengsicheng
>Priority: Major
> Attachments: screenshot-1.png
>
>
> When client-1.1.6 and server-2.3.4 there is a case where the batch delete
> timestamp is negative
> # 1. RegionServer log message:
> {code:java}
> 2022-07-19 12:13:29,324 WARN
> [RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
> hbase.KeyValueUtil: Timestamp cannot be negative, ts=-4323977095312258207,
> KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
> 2022-07-19 12:13:29,324 WARN
> [RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
> wal.ProtobufLogReader: Encountered a malformed edit, seeking back to last
> good position in file, from 1099261 to 1078224
> java.io.EOFException: EOF while reading 660 WAL KVs; started reading at
> 1078317 and read up to 1099261
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:403)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:97)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:85)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:264)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:178)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:103)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:230)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:145)
> Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative,
> ts=-4323977095312258207, KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
> at
> org.apache.hadoop.hbase.KeyValueUtil.checkKeyValueBytes(KeyValueUtil.java:612)
> at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:346)
> at
> org.apache.hadoop.hbase.KeyValueUtil.createKeyValueFromInputStream(KeyValueUtil.java:717)
> at
> org.apache.hadoop.hbase.codec.KeyValueCodecWithTags$KeyValueDecoder.parseCell(KeyValueCodecWithTags.java:81)
> at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:68)
> at org.apache.hadoop.hbase.wal.WALEdit.readFromCells(WALEdit.java:276)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:387)
> ... 7 more
> {code}
> # 2. Debug WAL file ,found that the delete operation is caused
> {code:java}
> Sequence=365693989, table=tableA, region=148cedb7b8ca3145690800fd650e084d, at
> write timestamp=Sat Jul 16 00:50:01 CST 2022
> 2022-07-22 22:09:43,244 ERROR [main] wal.WALPrettyPrinter: Timestamp is
> negative row=rowkey1, column=d:act, timestamp=-4323977095312258207,
> type=Delete
> {code}
> # 3. User use spark read/write hbase
> batchsize is 1
> {code:scala}
> def dataDeleteFromHbase(rdd: RDD[(String, String)], hbase_table: String,
> hbase_instance: String, hbase_accesskey: String, accumulator:
> LongAccumulator, buffersize: String, batchsize: Int): Unit = {
> rdd.foreachPartition(iterator => {
> val partitionId = TaskContext.getPartitionId()
> val conf = HBaseConfiguration.create()
> val connection = SparkHbaseUtils.getconnection(conf)
> val table = connection.getTable(TableName.valueOf(hbase_table))
> var deleteList = new util.LinkedList[Delete]()
> var count = 0
> var batchCount = 0
> while (iterator.hasNext) {
> val element = iterator.next
> val crc32 = new CRC32()
> crc32.update(s"${element._1}_${element._2}".getBytes())
> val crcArr = convertLow4bit2SmallEndan(crc32.getValue)
> val key = concat(DigestUtils.md5(s"${element._1}_${element._2}"),
> crcArr)
> val delete = new Delete(key)
> deleteList.add(delete)
> count += 1
> if (count % batchsize.toInt == 0) {
> batchCount = batchCount + 1
> try {
> table.delete(deleteList)
> } catch
[jira] [Work started] (HBASE-27267) Delete causes timestamp to be negative
[
https://issues.apache.org/jira/browse/HBASE-27267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-27267 started by zhengsicheng.
> Delete causes timestamp to be negative
> --
>
> Key: HBASE-27267
> URL: https://issues.apache.org/jira/browse/HBASE-27267
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.3.4
>Reporter: zhengsicheng
>Assignee: zhengsicheng
>Priority: Major
>
> When client-1.1.6 and server-2.3.4 there is a case where the batch delete
> timestamp is negative
> # 1. RegionServer log message:
> {code:java}
> 2022-07-19 12:13:29,324 WARN
> [RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
> hbase.KeyValueUtil: Timestamp cannot be negative, ts=-4323977095312258207,
> KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
> 2022-07-19 12:13:29,324 WARN
> [RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
> wal.ProtobufLogReader: Encountered a malformed edit, seeking back to last
> good position in file, from 1099261 to 1078224
> java.io.EOFException: EOF while reading 660 WAL KVs; started reading at
> 1078317 and read up to 1099261
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:403)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:97)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:85)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:264)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:178)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:103)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:230)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:145)
> Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative,
> ts=-4323977095312258207, KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
> at
> org.apache.hadoop.hbase.KeyValueUtil.checkKeyValueBytes(KeyValueUtil.java:612)
> at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:346)
> at
> org.apache.hadoop.hbase.KeyValueUtil.createKeyValueFromInputStream(KeyValueUtil.java:717)
> at
> org.apache.hadoop.hbase.codec.KeyValueCodecWithTags$KeyValueDecoder.parseCell(KeyValueCodecWithTags.java:81)
> at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:68)
> at org.apache.hadoop.hbase.wal.WALEdit.readFromCells(WALEdit.java:276)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:387)
> ... 7 more
> {code}
> # 2. Debug WAL file ,found that the delete operation is caused
> {code:java}
> Sequence=365693989, table=tableA, region=148cedb7b8ca3145690800fd650e084d, at
> write timestamp=Sat Jul 16 00:50:01 CST 2022
> 2022-07-22 22:09:43,244 ERROR [main] wal.WALPrettyPrinter: Timestamp is
> negative row=rowkey1, column=d:act, timestamp=-4323977095312258207,
> type=Delete
> {code}
> # 3. User use spark read/write hbase
> batchsize is 1
> {code:scala}
> def dataDeleteFromHbase(rdd: RDD[(String, String)], hbase_table: String,
> hbase_instance: String, hbase_accesskey: String, accumulator:
> LongAccumulator, buffersize: String, batchsize: Int): Unit = {
> rdd.foreachPartition(iterator => {
> val partitionId = TaskContext.getPartitionId()
> val conf = HBaseConfiguration.create()
> val connection = SparkHbaseUtils.getconnection(conf)
> val table = connection.getTable(TableName.valueOf(hbase_table))
> var deleteList = new util.LinkedList[Delete]()
> var count = 0
> var batchCount = 0
> while (iterator.hasNext) {
> val element = iterator.next
> val crc32 = new CRC32()
> crc32.update(s"${element._1}_${element._2}".getBytes())
> val crcArr = convertLow4bit2SmallEndan(crc32.getValue)
> val key = concat(DigestUtils.md5(s"${element._1}_${element._2}"),
> crcArr)
> val delete = new Delete(key)
> deleteList.add(delete)
> count += 1
> if (count % batchsize.toInt == 0) {
> batchCount = batchCount + 1
> try {
> table.delete(deleteList)
> } catch {
> case _:
[jira] [Assigned] (HBASE-27267) Delete causes timestamp to be negative
[
https://issues.apache.org/jira/browse/HBASE-27267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhengsicheng reassigned HBASE-27267:
Assignee: zhengsicheng
> Delete causes timestamp to be negative
> --
>
> Key: HBASE-27267
> URL: https://issues.apache.org/jira/browse/HBASE-27267
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.3.4
>Reporter: zhengsicheng
>Assignee: zhengsicheng
>Priority: Major
>
> When client-1.1.6 and server-2.3.4 there is a case where the batch delete
> timestamp is negative
> # 1. RegionServer log message:
> {code:java}
> 2022-07-19 12:13:29,324 WARN
> [RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
> hbase.KeyValueUtil: Timestamp cannot be negative, ts=-4323977095312258207,
> KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
> 2022-07-19 12:13:29,324 WARN
> [RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
> wal.ProtobufLogReader: Encountered a malformed edit, seeking back to last
> good position in file, from 1099261 to 1078224
> java.io.EOFException: EOF while reading 660 WAL KVs; started reading at
> 1078317 and read up to 1099261
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:403)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:97)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:85)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:264)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:178)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:103)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:230)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:145)
> Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative,
> ts=-4323977095312258207, KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
> at
> org.apache.hadoop.hbase.KeyValueUtil.checkKeyValueBytes(KeyValueUtil.java:612)
> at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:346)
> at
> org.apache.hadoop.hbase.KeyValueUtil.createKeyValueFromInputStream(KeyValueUtil.java:717)
> at
> org.apache.hadoop.hbase.codec.KeyValueCodecWithTags$KeyValueDecoder.parseCell(KeyValueCodecWithTags.java:81)
> at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:68)
> at org.apache.hadoop.hbase.wal.WALEdit.readFromCells(WALEdit.java:276)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:387)
> ... 7 more
> {code}
> # 2. Debug WAL file ,found that the delete operation is caused
> {code:java}
> Sequence=365693989, table=tableA, region=148cedb7b8ca3145690800fd650e084d, at
> write timestamp=Sat Jul 16 00:50:01 CST 2022
> 2022-07-22 22:09:43,244 ERROR [main] wal.WALPrettyPrinter: Timestamp is
> negative row=rowkey1, column=d:act, timestamp=-4323977095312258207,
> type=Delete
> {code}
> # 3. User use spark read/write hbase
> batchsize is 1
> {code:scala}
> def dataDeleteFromHbase(rdd: RDD[(String, String)], hbase_table: String,
> hbase_instance: String, hbase_accesskey: String, accumulator:
> LongAccumulator, buffersize: String, batchsize: Int): Unit = {
> rdd.foreachPartition(iterator => {
> val partitionId = TaskContext.getPartitionId()
> val conf = HBaseConfiguration.create()
> val connection = SparkHbaseUtils.getconnection(conf)
> val table = connection.getTable(TableName.valueOf(hbase_table))
> var deleteList = new util.LinkedList[Delete]()
> var count = 0
> var batchCount = 0
> while (iterator.hasNext) {
> val element = iterator.next
> val crc32 = new CRC32()
> crc32.update(s"${element._1}_${element._2}".getBytes())
> val crcArr = convertLow4bit2SmallEndan(crc32.getValue)
> val key = concat(DigestUtils.md5(s"${element._1}_${element._2}"),
> crcArr)
> val delete = new Delete(key)
> deleteList.add(delete)
> count += 1
> if (count % batchsize.toInt == 0) {
> batchCount = batchCount + 1
> try {
> table.delete(deleteList)
> } catch {
> case _:
[jira] [Updated] (HBASE-27270) Add bulkloadExecutor.resizeQueues
[ https://issues.apache.org/jira/browse/HBASE-27270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27270: - Fix Version/s: 2.6.0 3.0.0-alpha-4 > Add bulkloadExecutor.resizeQueues > - > > Key: HBASE-27270 > URL: https://issues.apache.org/jira/browse/HBASE-27270 > Project: HBase > Issue Type: Sub-task > Components: regionserver, rpc >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Fix For: 2.6.0, 3.0.0-alpha-4 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27270) Add bulkloadExecutor.resizeQueues
[ https://issues.apache.org/jira/browse/HBASE-27270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng resolved HBASE-27270. -- Resolution: Resolved > Add bulkloadExecutor.resizeQueues > - > > Key: HBASE-27270 > URL: https://issues.apache.org/jira/browse/HBASE-27270 > Project: HBase > Issue Type: Sub-task > Components: regionserver, rpc >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27144) Add special rpc handlers for bulkload operations
[ https://issues.apache.org/jira/browse/HBASE-27144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27144: - Release Note: Bulkload will consume a lot of resources in the cluster. We try to reduce the impact of bulkload on online services and do simple resource isolation for bulkload. The bulkload RpcExecutor is disabled by default. Enable by setting "hbase.regionserver.bulkload.handler.count" greater than 0. Online HBASE cluster "hbase.regionserver.bulkload.handler.count" is recommended to be set to 1. Offline HBASE clusters can be appropriately increased. "hbase.regionserver.bulkload.handler.count" works with "hbase.ipc.server.bulkload.max.callqueue.length". > Add special rpc handlers for bulkload operations > > > Key: HBASE-27144 > URL: https://issues.apache.org/jira/browse/HBASE-27144 > Project: HBase > Issue Type: Improvement > Components: regionserver, rpc >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Fix For: 2.6.0, 3.0.0-alpha-4 > > Attachments: image-2022-06-22-11-47-26-963.png > > > Bulkload will consume a lot of resources in the cluster. We try to reduce the > impact of bulkload on online services and do simple resource isolation for > bulkload. > !image-2022-06-22-11-47-26-963.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27144) Add special rpc handlers for bulkload operations
[ https://issues.apache.org/jira/browse/HBASE-27144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575030#comment-17575030 ] zhengsicheng commented on HBASE-27144: -- [~zhangduo] add RpcExecutor.resizeQueues [PR|https://github.com/apache/hbase/pull/4677/files] > Add special rpc handlers for bulkload operations > > > Key: HBASE-27144 > URL: https://issues.apache.org/jira/browse/HBASE-27144 > Project: HBase > Issue Type: Improvement > Components: regionserver, rpc >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Fix For: 2.6.0, 3.0.0-alpha-4 > > Attachments: image-2022-06-22-11-47-26-963.png > > > Bulkload will consume a lot of resources in the cluster. We try to reduce the > impact of bulkload on online services and do simple resource isolation for > bulkload. > !image-2022-06-22-11-47-26-963.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27270) Add bulkloadExecutor.resizeQueues
zhengsicheng created HBASE-27270: Summary: Add bulkloadExecutor.resizeQueues Key: HBASE-27270 URL: https://issues.apache.org/jira/browse/HBASE-27270 Project: HBase Issue Type: Sub-task Components: regionserver, rpc Reporter: zhengsicheng Assignee: zhengsicheng -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27267) Delete causes timestamp to be negative
[
https://issues.apache.org/jira/browse/HBASE-27267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17574717#comment-17574717
]
zhengsicheng commented on HBASE-27267:
--
[~zhangduo] We have a problem, when client-1.1.6 and server-2.3.4 there is a
case where the batch delete timestamp is negative
> Delete causes timestamp to be negative
> --
>
> Key: HBASE-27267
> URL: https://issues.apache.org/jira/browse/HBASE-27267
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.3.4
>Reporter: zhengsicheng
>Priority: Major
>
> When client-1.1.6 and server-2.3.4 there is a case where the batch delete
> timestamp is negative
> # 1. RegionServer log message:
> {code:java}
> 2022-07-19 12:13:29,324 WARN
> [RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
> hbase.KeyValueUtil: Timestamp cannot be negative, ts=-4323977095312258207,
> KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
> 2022-07-19 12:13:29,324 WARN
> [RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
> wal.ProtobufLogReader: Encountered a malformed edit, seeking back to last
> good position in file, from 1099261 to 1078224
> java.io.EOFException: EOF while reading 660 WAL KVs; started reading at
> 1078317 and read up to 1099261
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:403)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:97)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:85)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:264)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:178)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:103)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:230)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:145)
> Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative,
> ts=-4323977095312258207, KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
> at
> org.apache.hadoop.hbase.KeyValueUtil.checkKeyValueBytes(KeyValueUtil.java:612)
> at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:346)
> at
> org.apache.hadoop.hbase.KeyValueUtil.createKeyValueFromInputStream(KeyValueUtil.java:717)
> at
> org.apache.hadoop.hbase.codec.KeyValueCodecWithTags$KeyValueDecoder.parseCell(KeyValueCodecWithTags.java:81)
> at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:68)
> at org.apache.hadoop.hbase.wal.WALEdit.readFromCells(WALEdit.java:276)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:387)
> ... 7 more
> {code}
> # 2. Debug WAL file ,found that the delete operation is caused
> {code:java}
> Sequence=365693989, table=tableA, region=148cedb7b8ca3145690800fd650e084d, at
> write timestamp=Sat Jul 16 00:50:01 CST 2022
> 2022-07-22 22:09:43,244 ERROR [main] wal.WALPrettyPrinter: Timestamp is
> negative row=rowkey1, column=d:act, timestamp=-4323977095312258207,
> type=Delete
> {code}
> # 3. User use spark read/write hbase
> batchsize is 1
> {code:scala}
> def dataDeleteFromHbase(rdd: RDD[(String, String)], hbase_table: String,
> hbase_instance: String, hbase_accesskey: String, accumulator:
> LongAccumulator, buffersize: String, batchsize: Int): Unit = {
> rdd.foreachPartition(iterator => {
> val partitionId = TaskContext.getPartitionId()
> val conf = HBaseConfiguration.create()
> val connection = SparkHbaseUtils.getconnection(conf)
> val table = connection.getTable(TableName.valueOf(hbase_table))
> var deleteList = new util.LinkedList[Delete]()
> var count = 0
> var batchCount = 0
> while (iterator.hasNext) {
> val element = iterator.next
> val crc32 = new CRC32()
> crc32.update(s"${element._1}_${element._2}".getBytes())
> val crcArr = convertLow4bit2SmallEndan(crc32.getValue)
> val key = concat(DigestUtils.md5(s"${element._1}_${element._2}"),
> crcArr)
> val delete = new Delete(key)
> deleteList.add(delete)
> count += 1
> if (count % batchsize.toInt == 0) {
> batchCount = batchCount + 1
> try
[jira] [Updated] (HBASE-27267) Delete causes timestamp to be negative
[
https://issues.apache.org/jira/browse/HBASE-27267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhengsicheng updated HBASE-27267:
-
Description:
When client-1.1.6 and server-2.3.4 there is a case where the batch delete
timestamp is negative
# 1. RegionServer log message:
{code:java}
2022-07-19 12:13:29,324 WARN
[RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
hbase.KeyValueUtil: Timestamp cannot be negative, ts=-4323977095312258207,
KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
2022-07-19 12:13:29,324 WARN
[RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
wal.ProtobufLogReader: Encountered a malformed edit, seeking back to last good
position in file, from 1099261 to 1078224
java.io.EOFException: EOF while reading 660 WAL KVs; started reading at
1078317 and read up to 1099261
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:403)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:97)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:85)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:264)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:178)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:103)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:230)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:145)
Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative,
ts=-4323977095312258207, KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
at
org.apache.hadoop.hbase.KeyValueUtil.checkKeyValueBytes(KeyValueUtil.java:612)
at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:346)
at
org.apache.hadoop.hbase.KeyValueUtil.createKeyValueFromInputStream(KeyValueUtil.java:717)
at
org.apache.hadoop.hbase.codec.KeyValueCodecWithTags$KeyValueDecoder.parseCell(KeyValueCodecWithTags.java:81)
at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:68)
at org.apache.hadoop.hbase.wal.WALEdit.readFromCells(WALEdit.java:276)
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:387)
... 7 more
{code}
# 2. Debug WAL file ,found that the delete operation is caused
{code:java}
Sequence=365693989, table=tableA, region=148cedb7b8ca3145690800fd650e084d, at
write timestamp=Sat Jul 16 00:50:01 CST 2022
2022-07-22 22:09:43,244 ERROR [main] wal.WALPrettyPrinter: Timestamp is
negative row=rowkey1, column=d:act, timestamp=-4323977095312258207, type=Delete
{code}
# 3. User use spark read/write hbase
batchsize is 1
{code:scala}
def dataDeleteFromHbase(rdd: RDD[(String, String)], hbase_table: String,
hbase_instance: String, hbase_accesskey: String, accumulator: LongAccumulator,
buffersize: String, batchsize: Int): Unit = {
rdd.foreachPartition(iterator => {
val partitionId = TaskContext.getPartitionId()
val conf = HBaseConfiguration.create()
val connection = SparkHbaseUtils.getconnection(conf)
val table = connection.getTable(TableName.valueOf(hbase_table))
var deleteList = new util.LinkedList[Delete]()
var count = 0
var batchCount = 0
while (iterator.hasNext) {
val element = iterator.next
val crc32 = new CRC32()
crc32.update(s"${element._1}_${element._2}".getBytes())
val crcArr = convertLow4bit2SmallEndan(crc32.getValue)
val key = concat(DigestUtils.md5(s"${element._1}_${element._2}"),
crcArr)
val delete = new Delete(key)
deleteList.add(delete)
count += 1
if (count % batchsize.toInt == 0) {
batchCount = batchCount + 1
try {
table.delete(deleteList)
} catch {
case _: RetriesExhaustedWithDetailsException => {
LOGGER.warn(s"==partitionId: ${partitionId}===batchCount:
${batchCount}===Wait 1000 ms, retry..")
Thread.sleep(1000)
processDelThrottlingException(table, deleteList, partitionId,
batchCount)
}
case _: ThrottlingException => {
LOGGER.warn(s"==partitionId: ${partitionId}===batchCount:
${batchCount}===Wait 1000 ms, retry..")
Thread.sleep(1000)
processDelThrottlingException(table,
[jira] [Updated] (HBASE-27267) Delete causes timestamp to be negative
[
https://issues.apache.org/jira/browse/HBASE-27267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhengsicheng updated HBASE-27267:
-
Description:
When client-1.1.6 and server-2.3.4 there is a case where the batch delete
timestamp is negative
RegionServer log message:
{code:java}
2022-07-19 12:13:29,324 WARN
[RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
hbase.KeyValueUtil: Timestamp cannot be negative, ts=-4323977095312258207,
KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
2022-07-19 12:13:29,324 WARN
[RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
wal.ProtobufLogReader: Encountered a malformed edit, seeking back to last good
position in file, from 1099261 to 1078224
java.io.EOFException: EOF while reading 660 WAL KVs; started reading at
1078317 and read up to 1099261
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:403)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:97)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:85)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:264)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:178)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:103)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:230)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:145)
Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative,
ts=-4323977095312258207, KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
at
org.apache.hadoop.hbase.KeyValueUtil.checkKeyValueBytes(KeyValueUtil.java:612)
at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:346)
at
org.apache.hadoop.hbase.KeyValueUtil.createKeyValueFromInputStream(KeyValueUtil.java:717)
at
org.apache.hadoop.hbase.codec.KeyValueCodecWithTags$KeyValueDecoder.parseCell(KeyValueCodecWithTags.java:81)
at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:68)
at org.apache.hadoop.hbase.wal.WALEdit.readFromCells(WALEdit.java:276)
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:387)
... 7 more
{code}
Debug WAL file ,found that the delete operation is caused
{code:java}
Sequence=365693989, table=tableA, region=148cedb7b8ca3145690800fd650e084d, at
write timestamp=Sat Jul 16 00:50:01 CST 2022
2022-07-22 22:09:43,244 ERROR [main] wal.WALPrettyPrinter: Timestamp is
negative row=rowkey1, column=d:act, timestamp=-4323977095312258207, type=Delete
{code}
was:
RegionServer log message:
{code:java}
2022-07-19 12:13:29,324 WARN
[RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
hbase.KeyValueUtil: Timestamp cannot be negative, ts=-4323977095312258207,
KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
2022-07-19 12:13:29,324 WARN
[RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
wal.ProtobufLogReader: Encountered a malformed edit, seeking back to last good
position in file, from 1099261 to 1078224
java.io.EOFException: EOF while reading 660 WAL KVs; started reading at
1078317 and read up to 1099261
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:403)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:97)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:85)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:264)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:178)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:103)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:230)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:145)
Caused by: java.lang.IllegalArgumentException:
[jira] [Updated] (HBASE-27267) Delete causes timestamp to be negative
[
https://issues.apache.org/jira/browse/HBASE-27267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhengsicheng updated HBASE-27267:
-
Description:
RegionServer log message:
{code:java}
2022-07-19 12:13:29,324 WARN
[RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
hbase.KeyValueUtil: Timestamp cannot be negative, ts=-4323977095312258207,
KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
2022-07-19 12:13:29,324 WARN
[RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
wal.ProtobufLogReader: Encountered a malformed edit, seeking back to last good
position in file, from 1099261 to 1078224
java.io.EOFException: EOF while reading 660 WAL KVs; started reading at
1078317 and read up to 1099261
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:403)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:97)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:85)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:264)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:178)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:103)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:230)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:145)
Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative,
ts=-4323977095312258207, KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
at
org.apache.hadoop.hbase.KeyValueUtil.checkKeyValueBytes(KeyValueUtil.java:612)
at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:346)
at
org.apache.hadoop.hbase.KeyValueUtil.createKeyValueFromInputStream(KeyValueUtil.java:717)
at
org.apache.hadoop.hbase.codec.KeyValueCodecWithTags$KeyValueDecoder.parseCell(KeyValueCodecWithTags.java:81)
at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:68)
at org.apache.hadoop.hbase.wal.WALEdit.readFromCells(WALEdit.java:276)
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:387)
... 7 more
{code}
Debug WAL file ,found that the delete operation is caused
Sequence=365693989, table=tableA, region=148cedb7b8ca3145690800fd650e084d, at
write timestamp=Sat Jul 16 00:50:01 CST 2022
2022-07-22 22:09:43,244 ERROR [main] wal.WALPrettyPrinter: Timestamp is
negative row=rowkey1, column=d:act, timestamp=-4323977095312258207, type=Delete
{code}
was:
RegionServer log message:
{code:java}
2022-07-19 12:13:29,324 WARN
[RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
hbase.KeyValueUtil: Timestamp cannot be negative, ts=-4323977095312258207,
KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
2022-07-19 12:13:29,324 WARN
[RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
wal.ProtobufLogReader: Encountered a malformed edit, seeking back to last good
position in file, from 1099261 to 1078224
java.io.EOFException: EOF while reading 660 WAL KVs; started reading at
1078317 and read up to 1099261
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:403)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:97)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:85)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:264)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:178)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:103)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:230)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:145)
Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative,
ts=-4323977095312258207, KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
[jira] [Updated] (HBASE-27267) Delete causes timestamp to be negative
[
https://issues.apache.org/jira/browse/HBASE-27267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhengsicheng updated HBASE-27267:
-
Description:
RegionServer log message:
{code:java}
2022-07-19 12:13:29,324 WARN
[RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
hbase.KeyValueUtil: Timestamp cannot be negative, ts=-4323977095312258207,
KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
2022-07-19 12:13:29,324 WARN
[RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
wal.ProtobufLogReader: Encountered a malformed edit, seeking back to last good
position in file, from 1099261 to 1078224
java.io.EOFException: EOF while reading 660 WAL KVs; started reading at
1078317 and read up to 1099261
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:403)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:97)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:85)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:264)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:178)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:103)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:230)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:145)
Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative,
ts=-4323977095312258207, KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
at
org.apache.hadoop.hbase.KeyValueUtil.checkKeyValueBytes(KeyValueUtil.java:612)
at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:346)
at
org.apache.hadoop.hbase.KeyValueUtil.createKeyValueFromInputStream(KeyValueUtil.java:717)
at
org.apache.hadoop.hbase.codec.KeyValueCodecWithTags$KeyValueDecoder.parseCell(KeyValueCodecWithTags.java:81)
at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:68)
at org.apache.hadoop.hbase.wal.WALEdit.readFromCells(WALEdit.java:276)
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:387)
... 7 more
{code}
debug WAL file ，found that the delete operation is caused
Sequence=365693989, table=tableA, region=148cedb7b8ca3145690800fd650e084d, at
write timestamp=Sat Jul 16 00:50:01 CST 2022
2022-07-22 22:09:43,244 ERROR [main] wal.WALPrettyPrinter: Timestamp is
negative row=rowkey1, column=d:act, timestamp=-4323977095312258207, type=Delete
was:
RegionServer log message:
{code:java}
2022-07-19 12:13:29,324 WARN
[RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
hbase.KeyValueUtil: Timestamp cannot be negative, ts=-4323977095312258207,
KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
2022-07-19 12:13:29,324 WARN
[RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
wal.ProtobufLogReader: Encountered a malformed edit, seeking back to last good
position in file, from 1099261 to 1078224
java.io.EOFException: EOF while reading 660 WAL KVs; started reading at
1078317 and read up to 1099261
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:403)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:97)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:85)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:264)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:178)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:103)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:230)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:145)
Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative,
ts=-4323977095312258207, KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
at
[jira] [Updated] (HBASE-27267) Delete causes timestamp to be negative
[
https://issues.apache.org/jira/browse/HBASE-27267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhengsicheng updated HBASE-27267:
-
Description:
RegionServer log message:
{code:java}
2022-07-19 12:13:29,324 WARN
[RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
hbase.KeyValueUtil: Timestamp cannot be negative, ts=-4323977095312258207,
KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
2022-07-19 12:13:29,324 WARN
[RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
wal.ProtobufLogReader: Encountered a malformed edit, seeking back to last good
position in file, from 1099261 to 1078224
java.io.EOFException: EOF while reading 660 WAL KVs; started reading at
1078317 and read up to 1099261
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:403)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:97)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:85)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:264)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:178)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:103)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:230)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:145)
Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative,
ts=-4323977095312258207, KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
at
org.apache.hadoop.hbase.KeyValueUtil.checkKeyValueBytes(KeyValueUtil.java:612)
at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:346)
at
org.apache.hadoop.hbase.KeyValueUtil.createKeyValueFromInputStream(KeyValueUtil.java:717)
at
org.apache.hadoop.hbase.codec.KeyValueCodecWithTags$KeyValueDecoder.parseCell(KeyValueCodecWithTags.java:81)
at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:68)
at org.apache.hadoop.hbase.wal.WALEdit.readFromCells(WALEdit.java:276)
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:387)
... 7 more
{code}
Sequence=365693989, table=tableA, region=148cedb7b8ca3145690800fd650e084d, at
write timestamp=Sat Jul 16 00:50:01 CST 2022
2022-07-22 22:09:43,244 ERROR [main] wal.WALPrettyPrinter: Timestamp is
negative row=rowkey1, column=d:act, timestamp=-4323977095312258207, type=Delete
{code}
was:
RegionServer log message:
{code:java}
2022-07-19 12:13:29,324 WARN
[RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
hbase.KeyValueUtil: Timestamp cannot be negative, ts=-4323977095312258207,
KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
2022-07-19 12:13:29,324 WARN
[RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
wal.ProtobufLogReader: Encountered a malformed edit, seeking back to last good
position in file, from 1099261 to 1078224
java.io.EOFException: EOF while reading 660 WAL KVs; started reading at
1078317 and read up to 1099261
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:403)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:97)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:85)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:264)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:178)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:103)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:230)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:145)
Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative,
ts=-4323977095312258207, KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
at
[jira] [Updated] (HBASE-27267) Delete causes timestamp to be negative
[
https://issues.apache.org/jira/browse/HBASE-27267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhengsicheng updated HBASE-27267:
-
Description:
RegionServer log message:
{code:java}
2022-07-19 12:13:29,324 WARN
[RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
hbase.KeyValueUtil: Timestamp cannot be negative, ts=-4323977095312258207,
KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
2022-07-19 12:13:29,324 WARN
[RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
wal.ProtobufLogReader: Encountered a malformed edit, seeking back to last good
position in file, from 1099261 to 1078224
java.io.EOFException: EOF while reading 660 WAL KVs; started reading at
1078317 and read up to 1099261
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:403)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:97)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:85)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:264)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:178)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:103)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:230)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:145)
Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative,
ts=-4323977095312258207, KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
at
org.apache.hadoop.hbase.KeyValueUtil.checkKeyValueBytes(KeyValueUtil.java:612)
at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:346)
at
org.apache.hadoop.hbase.KeyValueUtil.createKeyValueFromInputStream(KeyValueUtil.java:717)
at
org.apache.hadoop.hbase.codec.KeyValueCodecWithTags$KeyValueDecoder.parseCell(KeyValueCodecWithTags.java:81)
at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:68)
at org.apache.hadoop.hbase.wal.WALEdit.readFromCells(WALEdit.java:276)
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:387)
... 7 more
{code}
debug WAL file ，found that the operation is caused
Sequence=365693989, table=tableA, region=148cedb7b8ca3145690800fd650e084d, at
write timestamp=Sat Jul 16 00:50:01 CST 2022
2022-07-22 22:09:43,244 ERROR [main] wal.WALPrettyPrinter: Timestamp is
negative row=rowkey1, column=d:act, timestamp=-4323977095312258207, type=Delete
was:
RegionServer log message:
{code:java}
2022-07-19 12:13:29,324 WARN
[RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
hbase.KeyValueUtil: Timestamp cannot be negative, ts=-4323977095312258207,
KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
2022-07-19 12:13:29,324 WARN
[RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
wal.ProtobufLogReader: Encountered a malformed edit, seeking back to last good
position in file, from 1099261 to 1078224
java.io.EOFException: EOF while reading 660 WAL KVs; started reading at
1078317 and read up to 1099261
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:403)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:97)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:85)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:264)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:178)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:103)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:230)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:145)
Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative,
ts=-4323977095312258207, KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
at
[jira] [Updated] (HBASE-27267) Delete causes timestamp to be negative
[
https://issues.apache.org/jira/browse/HBASE-27267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhengsicheng updated HBASE-27267:
-
Description:
RegionServer log message:
{code:java}
2022-07-19 12:13:29,324 WARN
[RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
hbase.KeyValueUtil: Timestamp cannot be negative, ts=-4323977095312258207,
KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
2022-07-19 12:13:29,324 WARN
[RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
wal.ProtobufLogReader: Encountered a malformed edit, seeking back to last good
position in file, from 1099261 to 1078224
java.io.EOFException: EOF while reading 660 WAL KVs; started reading at
1078317 and read up to 1099261
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:403)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:97)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:85)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:264)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:178)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:103)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:230)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:145)
Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative,
ts=-4323977095312258207, KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
at
org.apache.hadoop.hbase.KeyValueUtil.checkKeyValueBytes(KeyValueUtil.java:612)
at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:346)
at
org.apache.hadoop.hbase.KeyValueUtil.createKeyValueFromInputStream(KeyValueUtil.java:717)
at
org.apache.hadoop.hbase.codec.KeyValueCodecWithTags$KeyValueDecoder.parseCell(KeyValueCodecWithTags.java:81)
at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:68)
at org.apache.hadoop.hbase.wal.WALEdit.readFromCells(WALEdit.java:276)
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:387)
... 7 more
{code}
was:
RegionServer log message:
2022-07-19 12:13:29,324 WARN
[RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
hbase.KeyValueUtil: Timestamp cannot be negative, ts=-4323977095312258207,
KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
2022-07-19 12:13:29,324 WARN
[RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB]
wal.ProtobufLogReader: Encountered a malformed edit, seeking back to last good
position in file, from 1099261 to 1078224
java.io.EOFException: EOF while reading 660 WAL KVs; started reading at
1078317 and read up to 1099261
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:403)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:97)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:85)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:264)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:178)
at
org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:103)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:230)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:145)
Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative,
ts=-4323977095312258207, KeyValueBytesHex=\x00\x00\x00, offset=0, length=40
at
org.apache.hadoop.hbase.KeyValueUtil.checkKeyValueBytes(KeyValueUtil.java:612)
at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:346)
at
org.apache.hadoop.hbase.KeyValueUtil.createKeyValueFromInputStream(KeyValueUtil.java:717)
at
[jira] [Updated] (HBASE-27267) Delete causes timestamp to be negative
[ https://issues.apache.org/jira/browse/HBASE-27267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27267: - Description: RegionServer log message: 2022-07-19 12:13:29,324 WARN [RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB] hbase.KeyValueUtil: Timestamp cannot be negative, ts=-4323977095312258207, KeyValueBytesHex=\x00\x00\x00, offset=0, length=40 2022-07-19 12:13:29,324 WARN [RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB] wal.ProtobufLogReader: Encountered a malformed edit, seeking back to last good position in file, from 1099261 to 1078224 java.io.EOFException: EOF while reading 660 WAL KVs; started reading at 1078317 and read up to 1099261 at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:403) at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:97) at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:85) at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:264) at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:178) at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:103) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:230) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:145) Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative, ts=-4323977095312258207, KeyValueBytesHex=\x00\x00\x00, offset=0, length=40 at org.apache.hadoop.hbase.KeyValueUtil.checkKeyValueBytes(KeyValueUtil.java:612) at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:346) at org.apache.hadoop.hbase.KeyValueUtil.createKeyValueFromInputStream(KeyValueUtil.java:717) at org.apache.hadoop.hbase.codec.KeyValueCodecWithTags$KeyValueDecoder.parseCell(KeyValueCodecWithTags.java:81) at org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:68) at org.apache.hadoop.hbase.wal.WALEdit.readFromCells(WALEdit.java:276) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:387) ... 7 more > Delete causes timestamp to be negative > -- > > Key: HBASE-27267 > URL: https://issues.apache.org/jira/browse/HBASE-27267 > Project: HBase > Issue Type: Bug >Affects Versions: 2.3.4 >Reporter: zhengsicheng >Priority: Major > > RegionServer log message: > 2022-07-19 12:13:29,324 WARN > [RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB] > hbase.KeyValueUtil: Timestamp cannot be negative, ts=-4323977095312258207, > KeyValueBytesHex=\x00\x00\x00, offset=0, length=40 > 2022-07-19 12:13:29,324 WARN > [RS_OPEN_REGION-regionserver/HBASE-HOSTNAME1:16020-1.replicationSource.wal-reader.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.HBASE-HOSTNAME1.local%2C16020%2C1657184880284.regiongroup-2,clusterB] > wal.ProtobufLogReader: Encountered a malformed edit, seeking back to last > good position in file, from 1099261 to 1078224 > java.io.EOFException: EOF while reading 660 WAL KVs; started reading at > 1078317 and read up to 1099261 > at > org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:403) > at > org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:97) > at > org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:85) > at > org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:264) > at > org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:178) > at > org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:103) > at > org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:230) > at > org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:145) > Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative, > ts=-4323977095312258207, KeyValueBytesHex=\x00\x00\x00,
[jira] [Created] (HBASE-27267) Delete causes timestamp to be negative
zhengsicheng created HBASE-27267: Summary: Delete causes timestamp to be negative Key: HBASE-27267 URL: https://issues.apache.org/jira/browse/HBASE-27267 Project: HBase Issue Type: Bug Affects Versions: 2.3.4 Reporter: zhengsicheng -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27144) Add special rpc handlers for bulkload operations
[ https://issues.apache.org/jira/browse/HBASE-27144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17572207#comment-17572207 ] zhengsicheng commented on HBASE-27144: -- [~zhangduo] I as soon as possible provide a PR to fix > Add special rpc handlers for bulkload operations > > > Key: HBASE-27144 > URL: https://issues.apache.org/jira/browse/HBASE-27144 > Project: HBase > Issue Type: Improvement > Components: regionserver, rpc >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Fix For: 2.6.0, 3.0.0-alpha-4 > > Attachments: image-2022-06-22-11-47-26-963.png > > > Bulkload will consume a lot of resources in the cluster. We try to reduce the > impact of bulkload on online services and do simple resource isolation for > bulkload. > !image-2022-06-22-11-47-26-963.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27249) Remove invalid peer RegionServer crash
[
https://issues.apache.org/jira/browse/HBASE-27249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17572199#comment-17572199
]
zhengsicheng commented on HBASE-27249:
--
[~zhangduo] When add peer cluster, but peer cluster zookeeper shuotdown or
invalid cause source cluster RS abort.
> Remove invalid peer RegionServer crash
> --
>
> Key: HBASE-27249
> URL: https://issues.apache.org/jira/browse/HBASE-27249
> Project: HBase
> Issue Type: Bug
>Reporter: zhengsicheng
>Assignee: zhengsicheng
>Priority: Major
>
> add_peer 'test', CLUSTER_KEY => "zookeeper-01:2181:/hbase_01"
> remove_peer 'test'
> find add peer wrong, remove peer but regionserver crash
> The log information is as follows:
> 2022-07-18 13:26:11,016 ERROR
> [ReadOnlyZKClient-zookeeper-01:2181@0x44281bff-SendThread(zookeeper-01:2181)]
> client.StaticHostProvider: Unable to resolve address:
> zookeeper-01/:2181
> java.net.UnknownHostException: zookeeper-01
> at
> java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:800)
> at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1507)
> at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1366)
> at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1300)
> at
> org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
> at
> org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
> at
> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
> 2022-07-18 13:26:11,016 WARN
> [ReadOnlyZKClient-zookeeper-01:2181@0x44281bff-SendThread(zookeeper-01:2181)]
> zookeeper.ClientCnxn: Session 0x0 for server zookeeper-01/:2181,
> unexpected error, closing socket connection and attempting reconnect
> java.lang.IllegalArgumentException: Unable to canonicalize address
> zookeeper-01/:2181 because it's not resolvable
> at
> org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
> at
> org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
> 2022-07-18 13:26:11,116 WARN [ReadOnlyZKClient-zookeeper-01:2181@0x44281bff]
> zookeeper.ReadOnlyZKClient: 0x44281bff to zookeeper-01:2181 failed for get of
> /hbase_01/hbaseid, code = CONNECTIONLOSS, retries = 48
> 2022-07-18 13:26:11,119 WARN [regionserver/ip1:16020.logRoller]
> regionserver.ReplicationSource: peerId=test, WAL group
> ip1%2C16020%2C1658118295598.ip1%2C16020%2C1658118295598.regiongroup-2 queue
> size: 11 exceeds value of replication.source.log.queue.warn 2
> 2022-07-18 13:26:12,055 INFO [MemStoreFlusher.1] regionserver.HRegion:
> Flushing 31bbfb9b76b6795e5d44fabd113174c0 1/2 column families,
> dataSize=245.67 MB heapSize=257.48 MB; f1={dataSize=245.67 MB,
> heapSize=257.48 MB, offHeapSize=0 B}
> 2022-07-18 13:26:12,116 ERROR
> [ReadOnlyZKClient-zookeeper-01:2181@0x44281bff-SendThread(zookeeper-01:2181)]
> client.StaticHostProvider: Unable to resolve address:
> zookeeper-01/:2181
> java.net.UnknownHostException: zookeeper-01
> at
> java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:800)
> at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1507)
> at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1366)
> at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1300)
> at
> org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
> at
> org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
> at
> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
> 2022-07-18 13:26:30,270 INFO [RS_REFRESH_PEER-regionserver/ip1:16020-1]
> regionserver.RefreshPeerCallable: Received a peer change event, peerId=test,
> type=REMOVE_PEER
> 2022-07-18 13:26:30,270 INFO [RS_REFRESH_PEER-regionserver/ip1:16020-1]
> regionserver.ReplicationSourceManager: Number of deleted recovered sources
> for test: 0
> 2022-07-18 13:26:30,270 INFO [RS_REFRESH_PEER-regionserver/ip1:16020-1]
> regionserver.ReplicationSource: peerId=test, Closing source test because:
> Replication stream was removed by a user
> 2022-07-18 13:26:30,271 WARN
> [RS_REFRESH_PEER-regionserver/ip1:16020-0.replicationSource,test]
> client.ConnectionImplementation: Retrieve cluster id failed
[jira] [Updated] (HBASE-27249) Remove invalid peer RegionServer crash
[
https://issues.apache.org/jira/browse/HBASE-27249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhengsicheng updated HBASE-27249:
-
Component/s: (was: Replication)
> Remove invalid peer RegionServer crash
> --
>
> Key: HBASE-27249
> URL: https://issues.apache.org/jira/browse/HBASE-27249
> Project: HBase
> Issue Type: Bug
>Reporter: zhengsicheng
>Assignee: zhengsicheng
>Priority: Major
>
> add_peer 'test', CLUSTER_KEY => "zookeeper-01:2181:/hbase_01"
> remove_peer 'test'
> find add peer wrong, remove peer but regionserver crash
> The log information is as follows:
> 2022-07-18 13:26:11,016 ERROR
> [ReadOnlyZKClient-zookeeper-01:2181@0x44281bff-SendThread(zookeeper-01:2181)]
> client.StaticHostProvider: Unable to resolve address:
> zookeeper-01/:2181
> java.net.UnknownHostException: zookeeper-01
> at
> java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:800)
> at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1507)
> at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1366)
> at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1300)
> at
> org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
> at
> org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
> at
> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
> 2022-07-18 13:26:11,016 WARN
> [ReadOnlyZKClient-zookeeper-01:2181@0x44281bff-SendThread(zookeeper-01:2181)]
> zookeeper.ClientCnxn: Session 0x0 for server zookeeper-01/:2181,
> unexpected error, closing socket connection and attempting reconnect
> java.lang.IllegalArgumentException: Unable to canonicalize address
> zookeeper-01/:2181 because it's not resolvable
> at
> org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
> at
> org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
> 2022-07-18 13:26:11,116 WARN [ReadOnlyZKClient-zookeeper-01:2181@0x44281bff]
> zookeeper.ReadOnlyZKClient: 0x44281bff to zookeeper-01:2181 failed for get of
> /hbase_01/hbaseid, code = CONNECTIONLOSS, retries = 48
> 2022-07-18 13:26:11,119 WARN [regionserver/ip1:16020.logRoller]
> regionserver.ReplicationSource: peerId=test, WAL group
> ip1%2C16020%2C1658118295598.ip1%2C16020%2C1658118295598.regiongroup-2 queue
> size: 11 exceeds value of replication.source.log.queue.warn 2
> 2022-07-18 13:26:12,055 INFO [MemStoreFlusher.1] regionserver.HRegion:
> Flushing 31bbfb9b76b6795e5d44fabd113174c0 1/2 column families,
> dataSize=245.67 MB heapSize=257.48 MB; f1={dataSize=245.67 MB,
> heapSize=257.48 MB, offHeapSize=0 B}
> 2022-07-18 13:26:12,116 ERROR
> [ReadOnlyZKClient-zookeeper-01:2181@0x44281bff-SendThread(zookeeper-01:2181)]
> client.StaticHostProvider: Unable to resolve address:
> zookeeper-01/:2181
> java.net.UnknownHostException: zookeeper-01
> at
> java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:800)
> at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1507)
> at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1366)
> at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1300)
> at
> org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
> at
> org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
> at
> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
> 2022-07-18 13:26:30,270 INFO [RS_REFRESH_PEER-regionserver/ip1:16020-1]
> regionserver.RefreshPeerCallable: Received a peer change event, peerId=test,
> type=REMOVE_PEER
> 2022-07-18 13:26:30,270 INFO [RS_REFRESH_PEER-regionserver/ip1:16020-1]
> regionserver.ReplicationSourceManager: Number of deleted recovered sources
> for test: 0
> 2022-07-18 13:26:30,270 INFO [RS_REFRESH_PEER-regionserver/ip1:16020-1]
> regionserver.ReplicationSource: peerId=test, Closing source test because:
> Replication stream was removed by a user
> 2022-07-18 13:26:30,271 WARN
> [RS_REFRESH_PEER-regionserver/ip1:16020-0.replicationSource,test]
> client.ConnectionImplementation: Retrieve cluster id failed
> java.lang.InterruptedException
> at
>
[jira] [Updated] (HBASE-27249) Remove invalid peer RegionServer crash
[
https://issues.apache.org/jira/browse/HBASE-27249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhengsicheng updated HBASE-27249:
-
Component/s: Replication
> Remove invalid peer RegionServer crash
> --
>
> Key: HBASE-27249
> URL: https://issues.apache.org/jira/browse/HBASE-27249
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: zhengsicheng
>Assignee: zhengsicheng
>Priority: Major
>
> add_peer 'test', CLUSTER_KEY => "zookeeper-01:2181:/hbase_01"
> remove_peer 'test'
> find add peer wrong, remove peer but regionserver crash
> The log information is as follows:
> 2022-07-18 13:26:11,016 ERROR
> [ReadOnlyZKClient-zookeeper-01:2181@0x44281bff-SendThread(zookeeper-01:2181)]
> client.StaticHostProvider: Unable to resolve address:
> zookeeper-01/:2181
> java.net.UnknownHostException: zookeeper-01
> at
> java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:800)
> at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1507)
> at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1366)
> at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1300)
> at
> org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
> at
> org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
> at
> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
> 2022-07-18 13:26:11,016 WARN
> [ReadOnlyZKClient-zookeeper-01:2181@0x44281bff-SendThread(zookeeper-01:2181)]
> zookeeper.ClientCnxn: Session 0x0 for server zookeeper-01/:2181,
> unexpected error, closing socket connection and attempting reconnect
> java.lang.IllegalArgumentException: Unable to canonicalize address
> zookeeper-01/:2181 because it's not resolvable
> at
> org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
> at
> org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
> 2022-07-18 13:26:11,116 WARN [ReadOnlyZKClient-zookeeper-01:2181@0x44281bff]
> zookeeper.ReadOnlyZKClient: 0x44281bff to zookeeper-01:2181 failed for get of
> /hbase_01/hbaseid, code = CONNECTIONLOSS, retries = 48
> 2022-07-18 13:26:11,119 WARN [regionserver/ip1:16020.logRoller]
> regionserver.ReplicationSource: peerId=test, WAL group
> ip1%2C16020%2C1658118295598.ip1%2C16020%2C1658118295598.regiongroup-2 queue
> size: 11 exceeds value of replication.source.log.queue.warn 2
> 2022-07-18 13:26:12,055 INFO [MemStoreFlusher.1] regionserver.HRegion:
> Flushing 31bbfb9b76b6795e5d44fabd113174c0 1/2 column families,
> dataSize=245.67 MB heapSize=257.48 MB; f1={dataSize=245.67 MB,
> heapSize=257.48 MB, offHeapSize=0 B}
> 2022-07-18 13:26:12,116 ERROR
> [ReadOnlyZKClient-zookeeper-01:2181@0x44281bff-SendThread(zookeeper-01:2181)]
> client.StaticHostProvider: Unable to resolve address:
> zookeeper-01/:2181
> java.net.UnknownHostException: zookeeper-01
> at
> java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:800)
> at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1507)
> at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1366)
> at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1300)
> at
> org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
> at
> org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
> at
> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
> 2022-07-18 13:26:30,270 INFO [RS_REFRESH_PEER-regionserver/ip1:16020-1]
> regionserver.RefreshPeerCallable: Received a peer change event, peerId=test,
> type=REMOVE_PEER
> 2022-07-18 13:26:30,270 INFO [RS_REFRESH_PEER-regionserver/ip1:16020-1]
> regionserver.ReplicationSourceManager: Number of deleted recovered sources
> for test: 0
> 2022-07-18 13:26:30,270 INFO [RS_REFRESH_PEER-regionserver/ip1:16020-1]
> regionserver.ReplicationSource: peerId=test, Closing source test because:
> Replication stream was removed by a user
> 2022-07-18 13:26:30,271 WARN
> [RS_REFRESH_PEER-regionserver/ip1:16020-0.replicationSource,test]
> client.ConnectionImplementation: Retrieve cluster id failed
> java.lang.InterruptedException
> at
>
[jira] [Updated] (HBASE-27249) Remove invalid peer RegionServer crash
[
https://issues.apache.org/jira/browse/HBASE-27249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhengsicheng updated HBASE-27249:
-
Description:
add_peer 'test', CLUSTER_KEY => "zookeeper-01:2181:/hbase_01"
remove_peer 'test'
find add peer wrong, remove peer but regionserver crash
The log information is as follows:
2022-07-18 13:26:11,016 ERROR
[ReadOnlyZKClient-zookeeper-01:2181@0x44281bff-SendThread(zookeeper-01:2181)]
client.StaticHostProvider: Unable to resolve address:
zookeeper-01/:2181
java.net.UnknownHostException: zookeeper-01
at
java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:800)
at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1507)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1366)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1300)
at
org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
at
org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
at
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
2022-07-18 13:26:11,016 WARN
[ReadOnlyZKClient-zookeeper-01:2181@0x44281bff-SendThread(zookeeper-01:2181)]
zookeeper.ClientCnxn: Session 0x0 for server zookeeper-01/:2181,
unexpected error, closing socket connection and attempting reconnect
java.lang.IllegalArgumentException: Unable to canonicalize address
zookeeper-01/:2181 because it's not resolvable
at
org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
at
org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
at
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
2022-07-18 13:26:11,116 WARN [ReadOnlyZKClient-zookeeper-01:2181@0x44281bff]
zookeeper.ReadOnlyZKClient: 0x44281bff to zookeeper-01:2181 failed for get of
/hbase_01/hbaseid, code = CONNECTIONLOSS, retries = 48
2022-07-18 13:26:11,119 WARN [regionserver/ip1:16020.logRoller]
regionserver.ReplicationSource: peerId=test, WAL group
ip1%2C16020%2C1658118295598.ip1%2C16020%2C1658118295598.regiongroup-2 queue
size: 11 exceeds value of replication.source.log.queue.warn 2
2022-07-18 13:26:12,055 INFO [MemStoreFlusher.1] regionserver.HRegion:
Flushing 31bbfb9b76b6795e5d44fabd113174c0 1/2 column families, dataSize=245.67
MB heapSize=257.48 MB; f1={dataSize=245.67 MB, heapSize=257.48 MB,
offHeapSize=0 B}
2022-07-18 13:26:12,116 ERROR
[ReadOnlyZKClient-zookeeper-01:2181@0x44281bff-SendThread(zookeeper-01:2181)]
client.StaticHostProvider: Unable to resolve address:
zookeeper-01/:2181
java.net.UnknownHostException: zookeeper-01
at
java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:800)
at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1507)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1366)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1300)
at
org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
at
org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
at
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
2022-07-18 13:26:30,270 INFO [RS_REFRESH_PEER-regionserver/ip1:16020-1]
regionserver.RefreshPeerCallable: Received a peer change event, peerId=test,
type=REMOVE_PEER
2022-07-18 13:26:30,270 INFO [RS_REFRESH_PEER-regionserver/ip1:16020-1]
regionserver.ReplicationSourceManager: Number of deleted recovered sources for
test: 0
2022-07-18 13:26:30,270 INFO [RS_REFRESH_PEER-regionserver/ip1:16020-1]
regionserver.ReplicationSource: peerId=test, Closing source test because:
Replication stream was removed by a user
2022-07-18 13:26:30,271 WARN
[RS_REFRESH_PEER-regionserver/ip1:16020-0.replicationSource,test]
client.ConnectionImplementation: Retrieve cluster id failed
java.lang.InterruptedException
at
java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:385)
at
java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2063)
at
org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:583)
at
org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:316)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
[jira] [Created] (HBASE-27249) Remove invalid peer RegionServer crash
zhengsicheng created HBASE-27249: Summary: Remove invalid peer RegionServer crash Key: HBASE-27249 URL: https://issues.apache.org/jira/browse/HBASE-27249 Project: HBase Issue Type: Bug Reporter: zhengsicheng Assignee: zhengsicheng -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27248) WALPrettyPrinter add print timestamp
[ https://issues.apache.org/jira/browse/HBASE-27248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-27248: - Description: hbase wal -wal -p, print include timestamp At first: row=rowA, column=f:c1, type=Delete value: cell total size sum: 88 New: row=rowA, column=f:c1, timestamp=1657851212679, type=Delete value: cell total size sum: 88 > WALPrettyPrinter add print timestamp > > > Key: HBASE-27248 > URL: https://issues.apache.org/jira/browse/HBASE-27248 > Project: HBase > Issue Type: Task >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > > hbase wal -wal -p, print include timestamp > At first: > row=rowA, column=f:c1, type=Delete > value: > cell total size sum: 88 > > New: > row=rowA, column=f:c1, timestamp=1657851212679, type=Delete > value: > cell total size sum: 88 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27248) WALPrettyPrinter add print timestamp
zhengsicheng created HBASE-27248: Summary: WALPrettyPrinter add print timestamp Key: HBASE-27248 URL: https://issues.apache.org/jira/browse/HBASE-27248 Project: HBase Issue Type: Task Reporter: zhengsicheng Assignee: zhengsicheng -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26981) The CPU usage of the regionserver node where the meta table is located is too high
[ https://issues.apache.org/jira/browse/HBASE-26981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17559154#comment-17559154 ] zhengsicheng commented on HBASE-26981: -- PR HBASE-27146 Prepare to merge to online version and run for a while > The CPU usage of the regionserver node where the meta table is located is too > high > -- > > Key: HBASE-26981 > URL: https://issues.apache.org/jira/browse/HBASE-26981 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.3.4 >Reporter: zhengsicheng >Priority: Major > Attachments: 20220624-144119.html, image-2022-04-27-20-24-33-252.png, > image-2022-04-27-20-45-09-227.png, image-2022-04-28-09-59-40-567.png, > image-2022-06-24-14-48-26-451.png, image-2022-06-24-14-57-10-424.png, jstack, > rs_jstack_202206241458.log > > > When the read and write pressure is high, the CPU usage of the meta table > node is too high > !image-2022-04-28-09-59-40-567.png! > > !image-2022-04-27-20-24-33-252.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Work started] (HBASE-26208) Supports revoke @ns specified permission
[ https://issues.apache.org/jira/browse/HBASE-26208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-26208 started by zhengsicheng. > Supports revoke @ns specified permission > - > > Key: HBASE-26208 > URL: https://issues.apache.org/jira/browse/HBASE-26208 > Project: HBase > Issue Type: Improvement > Components: shell >Affects Versions: 2.3.4 >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: 1.jpeg > > > The current version does not support reclaiming the specified > permissions，Only all permissions of @NS can be reclaimed。 > Supports revoke @ns specified permission :revoke 'bobsmith', '@ns1', 'WRCA' > !1.jpeg! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Work started] (HBASE-27132) Log archiving fail cause regionserver abort
[ https://issues.apache.org/jira/browse/HBASE-27132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-27132 started by zhengsicheng. > Log archiving fail cause regionserver abort > --- > > Key: HBASE-27132 > URL: https://issues.apache.org/jira/browse/HBASE-27132 > Project: HBase > Issue Type: Bug >Affects Versions: 2.3.4 >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Major > > RegionServer log > java.io.IOException: Unable to rename > hdfs://clusterA/hbase/WALs/hostmame1,16020,1647853003113/hostmame1%2C16020%2C1647853003113.default.1655296917964 > to > hdfs://clusterA/hbase/oldWALs/hostmame1,16020,1647853003113/hostmame1%2C16020%2C1647853003113.default.1655296917964 > NameNode log > [2022-06-15T20:51:49.277+08:00] [INFO] [IPC Server handler 0 on 8021] : > Starting log segment at 25945020756 > [2022-06-15T20:51:58.050+08:00] [WARN] [IPC Server handler 180 on 8020] : > DIR* FSDirectory.unprotectedRenameTo: failed to rename /h > base/WALs/hostmame1,16020,1647853003113/hostmame1%2C16020%2C16478530031 > 13.default.1655296917964 to > /hbase/oldWALs/hostmame1,16020,1647853003113/HTYD-clusterA-11-13-112-165.hado > op.jd.local%2C16020%2C1647853003113.default.1655296917964 because > destination's parent does not exist -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Work started] (HBASE-27144) Improvement bulkload rpc
[ https://issues.apache.org/jira/browse/HBASE-27144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-27144 started by zhengsicheng. > Improvement bulkload rpc > > > Key: HBASE-27144 > URL: https://issues.apache.org/jira/browse/HBASE-27144 > Project: HBase > Issue Type: Improvement >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Attachments: image-2022-06-22-11-47-26-963.png > > > Bulkload will consume a lot of resources in the cluster. We try to reduce the > impact of bulkload on online services and do simple resource isolation for > bulkload. > !image-2022-06-22-11-47-26-963.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HBASE-26981) The CPU usage of the regionserver node where the meta table is located is too high
[ https://issues.apache.org/jira/browse/HBASE-26981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558373#comment-17558373 ] zhengsicheng commented on HBASE-26981: -- [~zhangduo] The PR for HBASE-27146 does reduce the MetaCellComparator.compare PR is available > The CPU usage of the regionserver node where the meta table is located is too > high > -- > > Key: HBASE-26981 > URL: https://issues.apache.org/jira/browse/HBASE-26981 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.3.4 >Reporter: zhengsicheng >Priority: Major > Attachments: 20220624-144119.html, image-2022-04-27-20-24-33-252.png, > image-2022-04-27-20-45-09-227.png, image-2022-04-28-09-59-40-567.png, > image-2022-06-24-14-48-26-451.png, image-2022-06-24-14-57-10-424.png, jstack, > rs_jstack_202206241458.log > > > When the read and write pressure is high, the CPU usage of the meta table > node is too high > !image-2022-04-28-09-59-40-567.png! > > !image-2022-04-27-20-24-33-252.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
