date:20220609

[jira] [Comment Edited] (HBASE-25709) Close region may stuck when region is compacting and skipped most cells read

2022-06-09 Thread Xiaolin Ha (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-25709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552539#comment-17552539
 ] 

Xiaolin Ha edited comment on HBASE-25709 at 6/10/22 4:41 AM:
-

Hi, [~vjasani] , I looked at PHOENIX-6702 and your patch, do you doubt that 
this issue breaks large rows results? I added a UT 
TestHRegion#testTTLsUsingSmallHeartBeatCells to verify the scan can return the 
whole row when row cells count is larger than 
StoreScanner.HBASE_CELLS_SCANNED_PER_HEARTBEAT_CHECK in just once next(), 
please take a look. Thanks.


was (Author: xiaolin ha):
Hi, [~vjasani] , I looked at PHOENIX-6702 and your patch, do you doubt that 
this issue breaks large rows results? I added a UT 
TestHRegion#testTTLsUsingSmallHeartBeatCells to test verify the scan can return 
the whole row when row cells count is larger than 
StoreScanner.HBASE_CELLS_SCANNED_PER_HEARTBEAT_CHECK, please take a look. 
Thanks.

> Close region may stuck when region is compacting and skipped most cells read
> 
>
> Key: HBASE-25709
> URL: https://issues.apache.org/jira/browse/HBASE-25709
> Project: HBase
>  Issue Type: Bug
>  Components: Compaction
>Affects Versions: 1.7.1, 3.0.0-alpha-2, 2.4.10
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.11
>
> Attachments: Master-UI-RIT.png, RS-region-state.png
>
>
> We found in our cluster about stop region stuck. The region is compacting, 
> and its store files has many TTL expired cells. Close region state 
> marker(HRegion#writestate.writesEnabled) is not checked in compaction, 
> because most cells were skipped. 
> !RS-region-state.png|width=698,height=310!
>  
> !Master-UI-RIT.png|width=693,height=157!
>  
> HBASE-23968 has encountered similar problem, but the solution in it is outer 
> the method
> InternalScanner#next(List result, ScannerContext scannerContext), which 
> will not return if there are many skipped cells, for current compaction 
> scanner context. As a result, we need to return in time in the next method, 
> and then check the stop marker.
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Commented] (HBASE-25709) Close region may stuck when region is compacting and skipped most cells read

2022-06-09 Thread Xiaolin Ha (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-25709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552539#comment-17552539
 ] 

Xiaolin Ha commented on HBASE-25709:


Hi, [~vjasani] , I looked at PHOENIX-6702 and your patch, do you doubt that 
this issue breaks large rows results? I added a UT 
TestHRegion#testTTLsUsingSmallHeartBeatCells to test verify the scan can return 
the whole row when row cells count is larger than 
StoreScanner.HBASE_CELLS_SCANNED_PER_HEARTBEAT_CHECK, please take a look. 
Thanks.

> Close region may stuck when region is compacting and skipped most cells read
> 
>
> Key: HBASE-25709
> URL: https://issues.apache.org/jira/browse/HBASE-25709
> Project: HBase
>  Issue Type: Bug
>  Components: Compaction
>Affects Versions: 1.7.1, 3.0.0-alpha-2, 2.4.10
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.11
>
> Attachments: Master-UI-RIT.png, RS-region-state.png
>
>
> We found in our cluster about stop region stuck. The region is compacting, 
> and its store files has many TTL expired cells. Close region state 
> marker(HRegion#writestate.writesEnabled) is not checked in compaction, 
> because most cells were skipped. 
> !RS-region-state.png|width=698,height=310!
>  
> !Master-UI-RIT.png|width=693,height=157!
>  
> HBASE-23968 has encountered similar problem, but the solution in it is outer 
> the method
> InternalScanner#next(List result, ScannerContext scannerContext), which 
> will not return if there are many skipped cells, for current compaction 
> scanner context. As a result, we need to return in time in the next method, 
> and then check the stop marker.
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Commented] (HBASE-27097) SimpleRpcServer is broken

2022-06-09 Thread Lijin Bin (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-27097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552529#comment-17552529
 ] 

Lijin Bin commented on HBASE-27097:
---

[~vjasani] No，do not have ut to reproduce it。

> SimpleRpcServer is broken
> -
>
> Key: HBASE-27097
> URL: https://issues.apache.org/jira/browse/HBASE-27097
> Project: HBase
>  Issue Type: Bug
>  Components: rpc
>Affects Versions: 2.5.0
>Reporter: Andrew Kyle Purtell
>Priority: Blocker
> Fix For: 2.5.0, 3.0.0-alpha-3
>
> Attachments: MultiByteBuff.patch
>
>
> Concerns about SimpleRpcServer are not new, and not new to 2.5.  @chenxu 
> noticed a problem on HBASE-23917 back in 2020. After some simple evaluations 
> it seems quite broken. 
> When I run an async version of ITLCC against a 2.5.0 cluster configured with 
> hbase.rpc.server.impl=SimpleRpcServer, the client almost immediately stalls 
> because there are too many in flight requests. The logic to pause with too 
> many in flight requests is my own. That's not important. Looking at the 
> server logs it is apparent that SimpleRpcServer is quite broken. Handlers 
> suffer frequent protobuf parse errors and do not properly return responses to 
> the client. This is what stalls my test client. Rather quickly all available 
> request slots are full of requests that will have to time out on the client 
> side. 
> Exceptions have three patterns but they all have in common 
> SimpleServerRpcConnection#process. It seems likely the root cause is 
> mismatched expectations or bugs in connection buffer handling in 
> SimpleRpcServer/SimpleServerRpcConnection versus downstream classes that 
> process and parse the buffers. It also seems likely that changes were made to 
> downstream classes like ServerRpcConnection expecting NettyRpcServer's 
> particulars without updating SimpleServerRpcConnection and/or 
> SimpleRpcServer. That said, this is just a superficial analysis.
> 1) "Protocol message end-group tag did not match expected tag"
> {noformat}
>  2022-06-07T16:44:04,625 WARN  
> [Reader=5,bindAddress=buildbox.localdomain,port=8120] ipc.RpcServer: 
> /127.0.1.1:8120 is unable to read call parameter from client 127.0.0.1
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException:
>  Protocol message end-group tag did not match expected tag.
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:129)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.checkLastTagWas(CodedInputStream.java:4034)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4275)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue.(ClientProtos.java:10520)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue.(ClientProtos.java:10464)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$1.parsePartialFrom(ClientProtos.java:12251)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$1.parsePartialFrom(ClientProtos.java:12245)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4274)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto.(ClientProtos.java:9981)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto.(ClientProtos.java:9910)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$1.parsePartialFrom(ClientProtos.java:14097)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$1.parsePartialFrom(ClientProtos.java:14091)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4274)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
>

[jira] [Commented] (HBASE-27097) SimpleRpcServer is broken

2022-06-09 Thread Viraj Jasani (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-27097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552517#comment-17552517
 ] 

Viraj Jasani commented on HBASE-27097:
--

Thanks [~binlijin] for the patch! I was wondering if you have any UT that can 
break RPC calls with SimpleRpcServer impl without this patch. I tried writing 
few tests with minicluster and they were passing with SimpleRpcServer, hence I 
was wondering how come we see broken SimpleRpcServer only while testing on 
distributed cluster.

> SimpleRpcServer is broken
> -
>
> Key: HBASE-27097
> URL: https://issues.apache.org/jira/browse/HBASE-27097
> Project: HBase
>  Issue Type: Bug
>  Components: rpc
>Affects Versions: 2.5.0
>Reporter: Andrew Kyle Purtell
>Priority: Blocker
> Fix For: 2.5.0, 3.0.0-alpha-3
>
> Attachments: MultiByteBuff.patch
>
>
> Concerns about SimpleRpcServer are not new, and not new to 2.5.  @chenxu 
> noticed a problem on HBASE-23917 back in 2020. After some simple evaluations 
> it seems quite broken. 
> When I run an async version of ITLCC against a 2.5.0 cluster configured with 
> hbase.rpc.server.impl=SimpleRpcServer, the client almost immediately stalls 
> because there are too many in flight requests. The logic to pause with too 
> many in flight requests is my own. That's not important. Looking at the 
> server logs it is apparent that SimpleRpcServer is quite broken. Handlers 
> suffer frequent protobuf parse errors and do not properly return responses to 
> the client. This is what stalls my test client. Rather quickly all available 
> request slots are full of requests that will have to time out on the client 
> side. 
> Exceptions have three patterns but they all have in common 
> SimpleServerRpcConnection#process. It seems likely the root cause is 
> mismatched expectations or bugs in connection buffer handling in 
> SimpleRpcServer/SimpleServerRpcConnection versus downstream classes that 
> process and parse the buffers. It also seems likely that changes were made to 
> downstream classes like ServerRpcConnection expecting NettyRpcServer's 
> particulars without updating SimpleServerRpcConnection and/or 
> SimpleRpcServer. That said, this is just a superficial analysis.
> 1) "Protocol message end-group tag did not match expected tag"
> {noformat}
>  2022-06-07T16:44:04,625 WARN  
> [Reader=5,bindAddress=buildbox.localdomain,port=8120] ipc.RpcServer: 
> /127.0.1.1:8120 is unable to read call parameter from client 127.0.0.1
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException:
>  Protocol message end-group tag did not match expected tag.
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:129)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.checkLastTagWas(CodedInputStream.java:4034)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4275)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue.(ClientProtos.java:10520)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue.(ClientProtos.java:10464)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$1.parsePartialFrom(ClientProtos.java:12251)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$1.parsePartialFrom(ClientProtos.java:12245)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4274)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto.(ClientProtos.java:9981)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto.(ClientProtos.java:9910)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$1.parsePartialFrom(ClientProtos.java:14097)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
>

[jira] [Commented] (HBASE-26708) Netty "leak detected" and OutOfDirectMemoryError due to direct memory buffering

2022-06-09 Thread Duo Zhang (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-26708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552505#comment-17552505
 ] 

Duo Zhang commented on HBASE-26708:
---

Actually, when auth-int or auth-conf is used, we will copy the bytes from 
netty's BB to on heap byte array, wrap or unwrap it, and then just 
Unpool.wrappedBuffer to pass the on heap byte array to later handlers. In this 
way, actually we can release netty's native byte buf earlier...

https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/security/SaslWrapHandler.java
https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/security/SaslUnwrapHandler.java

> Netty "leak detected" and OutOfDirectMemoryError due to direct memory 
> buffering
> ---
>
> Key: HBASE-26708
> URL: https://issues.apache.org/jira/browse/HBASE-26708
> Project: HBase
>  Issue Type: Bug
>  Components: rpc
>Affects Versions: 2.5.0, 2.4.6
>Reporter: Viraj Jasani
>Priority: Critical
>
> Under constant data ingestion, using default Netty based RpcServer and 
> RpcClient implementation results in OutOfDirectMemoryError, supposedly caused 
> by leaks detected by Netty's LeakDetector.
> {code:java}
> 2022-01-25 17:03:10,084 ERROR [S-EventLoopGroup-1-3] 
> util.ResourceLeakDetector - java:115)
>   
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.expandCumulation(ByteToMessageDecoder.java:538)
>   
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder$1.cumulate(ByteToMessageDecoder.java:97)
>   
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:274)
>   
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
>   
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
>   
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
>   
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>   
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
>   
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
>   
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>   
> org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
>   
> org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
>   
> org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
>   
> org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
>   
> org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>   
> org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>   java.lang.Thread.run(Thread.java:748)
>  {code}
> {code:java}
> 2022-01-25 17:03:14,014 ERROR [S-EventLoopGroup-1-3] 
> util.ResourceLeakDetector - 
> apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
>   
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
>   
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
>   
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
>   
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
>   
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
>   
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>   
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
>   
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
>   
>

[jira] [Commented] (HBASE-27097) SimpleRpcServer is broken

2022-06-09 Thread Lijin Bin (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-27097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552502#comment-17552502
 ] 

Lijin Bin commented on HBASE-27097:
---

[~apurtell] what about apply the patch “MultiByteBuff.patch” ， which fix a bug？

> SimpleRpcServer is broken
> -
>
> Key: HBASE-27097
> URL: https://issues.apache.org/jira/browse/HBASE-27097
> Project: HBase
>  Issue Type: Bug
>  Components: rpc
>Affects Versions: 2.5.0
>Reporter: Andrew Kyle Purtell
>Priority: Blocker
> Fix For: 2.5.0, 3.0.0-alpha-3
>
> Attachments: MultiByteBuff.patch
>
>
> Concerns about SimpleRpcServer are not new, and not new to 2.5.  @chenxu 
> noticed a problem on HBASE-23917 back in 2020. After some simple evaluations 
> it seems quite broken. 
> When I run an async version of ITLCC against a 2.5.0 cluster configured with 
> hbase.rpc.server.impl=SimpleRpcServer, the client almost immediately stalls 
> because there are too many in flight requests. The logic to pause with too 
> many in flight requests is my own. That's not important. Looking at the 
> server logs it is apparent that SimpleRpcServer is quite broken. Handlers 
> suffer frequent protobuf parse errors and do not properly return responses to 
> the client. This is what stalls my test client. Rather quickly all available 
> request slots are full of requests that will have to time out on the client 
> side. 
> Exceptions have three patterns but they all have in common 
> SimpleServerRpcConnection#process. It seems likely the root cause is 
> mismatched expectations or bugs in connection buffer handling in 
> SimpleRpcServer/SimpleServerRpcConnection versus downstream classes that 
> process and parse the buffers. It also seems likely that changes were made to 
> downstream classes like ServerRpcConnection expecting NettyRpcServer's 
> particulars without updating SimpleServerRpcConnection and/or 
> SimpleRpcServer. That said, this is just a superficial analysis.
> 1) "Protocol message end-group tag did not match expected tag"
> {noformat}
>  2022-06-07T16:44:04,625 WARN  
> [Reader=5,bindAddress=buildbox.localdomain,port=8120] ipc.RpcServer: 
> /127.0.1.1:8120 is unable to read call parameter from client 127.0.0.1
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException:
>  Protocol message end-group tag did not match expected tag.
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:129)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.checkLastTagWas(CodedInputStream.java:4034)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4275)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue.(ClientProtos.java:10520)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue.(ClientProtos.java:10464)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$1.parsePartialFrom(ClientProtos.java:12251)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$1.parsePartialFrom(ClientProtos.java:12245)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4274)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto.(ClientProtos.java:9981)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto.(ClientProtos.java:9910)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$1.parsePartialFrom(ClientProtos.java:14097)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$1.parsePartialFrom(ClientProtos.java:14091)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4274)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
>

[jira] [Updated] (HBASE-27097) SimpleRpcServer is broken

2022-06-09 Thread Lijin Bin (Jira)



 [ 
https://issues.apache.org/jira/browse/HBASE-27097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lijin Bin updated HBASE-27097:
--
Attachment: MultiByteBuff.patch

> SimpleRpcServer is broken
> -
>
> Key: HBASE-27097
> URL: https://issues.apache.org/jira/browse/HBASE-27097
> Project: HBase
>  Issue Type: Bug
>  Components: rpc
>Affects Versions: 2.5.0
>Reporter: Andrew Kyle Purtell
>Priority: Blocker
> Fix For: 2.5.0, 3.0.0-alpha-3
>
> Attachments: MultiByteBuff.patch
>
>
> Concerns about SimpleRpcServer are not new, and not new to 2.5.  @chenxu 
> noticed a problem on HBASE-23917 back in 2020. After some simple evaluations 
> it seems quite broken. 
> When I run an async version of ITLCC against a 2.5.0 cluster configured with 
> hbase.rpc.server.impl=SimpleRpcServer, the client almost immediately stalls 
> because there are too many in flight requests. The logic to pause with too 
> many in flight requests is my own. That's not important. Looking at the 
> server logs it is apparent that SimpleRpcServer is quite broken. Handlers 
> suffer frequent protobuf parse errors and do not properly return responses to 
> the client. This is what stalls my test client. Rather quickly all available 
> request slots are full of requests that will have to time out on the client 
> side. 
> Exceptions have three patterns but they all have in common 
> SimpleServerRpcConnection#process. It seems likely the root cause is 
> mismatched expectations or bugs in connection buffer handling in 
> SimpleRpcServer/SimpleServerRpcConnection versus downstream classes that 
> process and parse the buffers. It also seems likely that changes were made to 
> downstream classes like ServerRpcConnection expecting NettyRpcServer's 
> particulars without updating SimpleServerRpcConnection and/or 
> SimpleRpcServer. That said, this is just a superficial analysis.
> 1) "Protocol message end-group tag did not match expected tag"
> {noformat}
>  2022-06-07T16:44:04,625 WARN  
> [Reader=5,bindAddress=buildbox.localdomain,port=8120] ipc.RpcServer: 
> /127.0.1.1:8120 is unable to read call parameter from client 127.0.0.1
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException:
>  Protocol message end-group tag did not match expected tag.
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:129)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.checkLastTagWas(CodedInputStream.java:4034)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4275)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue.(ClientProtos.java:10520)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue.(ClientProtos.java:10464)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$1.parsePartialFrom(ClientProtos.java:12251)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$1.parsePartialFrom(ClientProtos.java:12245)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4274)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto.(ClientProtos.java:9981)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto.(ClientProtos.java:9910)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$1.parsePartialFrom(ClientProtos.java:14097)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$1.parsePartialFrom(ClientProtos.java:14091)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4274)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
>

[jira] [Comment Edited] (HBASE-25709) Close region may stuck when region is compacting and skipped most cells read

2022-06-09 Thread Viraj Jasani (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-25709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552495#comment-17552495
 ] 

Viraj Jasani edited comment on HBASE-25709 at 6/10/22 1:32 AM:
---

FYI, it seems that this change has likely caused a regression in Phoenix 
indexing test PHOENIX-6702.


was (Author: vjasani):
FYI, we see one regression in Phoenix indexing test (PHOENIX-6702) after this 
commit.

> Close region may stuck when region is compacting and skipped most cells read
> 
>
> Key: HBASE-25709
> URL: https://issues.apache.org/jira/browse/HBASE-25709
> Project: HBase
>  Issue Type: Bug
>  Components: Compaction
>Affects Versions: 1.7.1, 3.0.0-alpha-2, 2.4.10
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.11
>
> Attachments: Master-UI-RIT.png, RS-region-state.png
>
>
> We found in our cluster about stop region stuck. The region is compacting, 
> and its store files has many TTL expired cells. Close region state 
> marker(HRegion#writestate.writesEnabled) is not checked in compaction, 
> because most cells were skipped. 
> !RS-region-state.png|width=698,height=310!
>  
> !Master-UI-RIT.png|width=693,height=157!
>  
> HBASE-23968 has encountered similar problem, but the solution in it is outer 
> the method
> InternalScanner#next(List result, ScannerContext scannerContext), which 
> will not return if there are many skipped cells, for current compaction 
> scanner context. As a result, we need to return in time in the next method, 
> and then check the stop marker.
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Commented] (HBASE-26708) Netty "leak detected" and OutOfDirectMemoryError due to direct memory buffering

2022-06-09 Thread Viraj Jasani (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-26708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552496#comment-17552496
 ] 

Viraj Jasani commented on HBASE-26708:
--

{quote}[~vjasani] I am curious if you apply my patch and set 
hbase.netty.rpcserver.allocator=unpooled if the direct memory allocation still 
gets up to > 50 GB. My guess is yes, that it is the concurrent demand for 
buffers at load driving the usage, and not excessive cache retention in the 
pooled allocator. Let's see if experimental results confirm the hypothesis. If 
it helps then I am wrong and pooling configuration tweaks – read on below – 
should be considered.
{quote}
Sounds good [~apurtell], this is worth exploring. Will spend some time, thanks!

> Netty "leak detected" and OutOfDirectMemoryError due to direct memory 
> buffering
> ---
>
> Key: HBASE-26708
> URL: https://issues.apache.org/jira/browse/HBASE-26708
> Project: HBase
>  Issue Type: Bug
>  Components: rpc
>Affects Versions: 2.5.0, 2.4.6
>Reporter: Viraj Jasani
>Priority: Critical
>
> Under constant data ingestion, using default Netty based RpcServer and 
> RpcClient implementation results in OutOfDirectMemoryError, supposedly caused 
> by leaks detected by Netty's LeakDetector.
> {code:java}
> 2022-01-25 17:03:10,084 ERROR [S-EventLoopGroup-1-3] 
> util.ResourceLeakDetector - java:115)
>   
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.expandCumulation(ByteToMessageDecoder.java:538)
>   
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder$1.cumulate(ByteToMessageDecoder.java:97)
>   
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:274)
>   
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
>   
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
>   
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
>   
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>   
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
>   
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
>   
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>   
> org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
>   
> org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
>   
> org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
>   
> org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
>   
> org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>   
> org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>   java.lang.Thread.run(Thread.java:748)
>  {code}
> {code:java}
> 2022-01-25 17:03:14,014 ERROR [S-EventLoopGroup-1-3] 
> util.ResourceLeakDetector - 
> apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
>   
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
>   
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
>   
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
>   
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
>   
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
>   
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>   
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
>   
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
>   
>

[jira] [Commented] (HBASE-25709) Close region may stuck when region is compacting and skipped most cells read

2022-06-09 Thread Viraj Jasani (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-25709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552495#comment-17552495
 ] 

Viraj Jasani commented on HBASE-25709:
--

FYI, we see one regression in Phoenix indexing test (PHOENIX-6702) after this 
commit.

> Close region may stuck when region is compacting and skipped most cells read
> 
>
> Key: HBASE-25709
> URL: https://issues.apache.org/jira/browse/HBASE-25709
> Project: HBase
>  Issue Type: Bug
>  Components: Compaction
>Affects Versions: 1.7.1, 3.0.0-alpha-2, 2.4.10
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.11
>
> Attachments: Master-UI-RIT.png, RS-region-state.png
>
>
> We found in our cluster about stop region stuck. The region is compacting, 
> and its store files has many TTL expired cells. Close region state 
> marker(HRegion#writestate.writesEnabled) is not checked in compaction, 
> because most cells were skipped. 
> !RS-region-state.png|width=698,height=310!
>  
> !Master-UI-RIT.png|width=693,height=157!
>  
> HBASE-23968 has encountered similar problem, but the solution in it is outer 
> the method
> InternalScanner#next(List result, ScannerContext scannerContext), which 
> will not return if there are many skipped cells, for current compaction 
> scanner context. As a result, we need to return in time in the next method, 
> and then check the stop marker.
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Commented] (HBASE-27066) The Region Visualizer display failed

2022-06-09 Thread Hudson (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-27066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552488#comment-17552488
 ] 

Hudson commented on HBASE-27066:


Results for branch branch-2
[build #563 on 
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/563/]: 
(/) *{color:green}+1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/563/General_20Nightly_20Build_20Report/]




(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/563/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]


(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/563/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]


(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11 
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/563/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> The Region Visualizer display failed
> 
>
> Key: HBASE-27066
> URL: https://issues.apache.org/jira/browse/HBASE-27066
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.5.0
>Reporter: Tao Li
>Assignee: Tao Li
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-3
>
> Attachments: 27066-xso.jpg, image-2022-05-27-14-22-29-015.png, 
> image-2022-05-27-14-22-44-336.png
>
>
> The `Region Visualizer` display failed. Because the active master hostname is 
> `localhost`.
> Before the change:
> !image-2022-05-27-14-22-44-336.png|width=520,height=162!
> After the change:
> !image-2022-05-27-14-22-29-015.png|width=562,height=229!



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Commented] (HBASE-27066) The Region Visualizer display failed

2022-06-09 Thread Hudson (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-27066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552479#comment-17552479
 ] 

Hudson commented on HBASE-27066:


Results for branch branch-2.5
[build #138 on 
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/138/]:
 (/) *{color:green}+1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/138/General_20Nightly_20Build_20Report/]




(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/138/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]


(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/138/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]


(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11 
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/138/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> The Region Visualizer display failed
> 
>
> Key: HBASE-27066
> URL: https://issues.apache.org/jira/browse/HBASE-27066
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.5.0
>Reporter: Tao Li
>Assignee: Tao Li
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-3
>
> Attachments: 27066-xso.jpg, image-2022-05-27-14-22-29-015.png, 
> image-2022-05-27-14-22-44-336.png
>
>
> The `Region Visualizer` display failed. Because the active master hostname is 
> `localhost`.
> Before the change:
> !image-2022-05-27-14-22-44-336.png|width=520,height=162!
> After the change:
> !image-2022-05-27-14-22-29-015.png|width=562,height=229!



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Updated] (HBASE-27097) SimpleRpcServer is broken

2022-06-09 Thread Andrew Kyle Purtell (Jira)



 [ 
https://issues.apache.org/jira/browse/HBASE-27097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Kyle Purtell updated HBASE-27097:

Fix Version/s: 2.5.0
   3.0.0-alpha-3

> SimpleRpcServer is broken
> -
>
> Key: HBASE-27097
> URL: https://issues.apache.org/jira/browse/HBASE-27097
> Project: HBase
>  Issue Type: Bug
>  Components: rpc
>Affects Versions: 2.5.0
>Reporter: Andrew Kyle Purtell
>Priority: Blocker
> Fix For: 2.5.0, 3.0.0-alpha-3
>
>
> Concerns about SimpleRpcServer are not new, and not new to 2.5.  @chenxu 
> noticed a problem on HBASE-23917 back in 2020. After some simple evaluations 
> it seems quite broken. 
> When I run an async version of ITLCC against a 2.5.0 cluster configured with 
> hbase.rpc.server.impl=SimpleRpcServer, the client almost immediately stalls 
> because there are too many in flight requests. The logic to pause with too 
> many in flight requests is my own. That's not important. Looking at the 
> server logs it is apparent that SimpleRpcServer is quite broken. Handlers 
> suffer frequent protobuf parse errors and do not properly return responses to 
> the client. This is what stalls my test client. Rather quickly all available 
> request slots are full of requests that will have to time out on the client 
> side. 
> Exceptions have three patterns but they all have in common 
> SimpleServerRpcConnection#process. It seems likely the root cause is 
> mismatched expectations or bugs in connection buffer handling in 
> SimpleRpcServer/SimpleServerRpcConnection versus downstream classes that 
> process and parse the buffers. It also seems likely that changes were made to 
> downstream classes like ServerRpcConnection expecting NettyRpcServer's 
> particulars without updating SimpleServerRpcConnection and/or 
> SimpleRpcServer. That said, this is just a superficial analysis.
> 1) "Protocol message end-group tag did not match expected tag"
> {noformat}
>  2022-06-07T16:44:04,625 WARN  
> [Reader=5,bindAddress=buildbox.localdomain,port=8120] ipc.RpcServer: 
> /127.0.1.1:8120 is unable to read call parameter from client 127.0.0.1
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException:
>  Protocol message end-group tag did not match expected tag.
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:129)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.checkLastTagWas(CodedInputStream.java:4034)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4275)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue.(ClientProtos.java:10520)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue.(ClientProtos.java:10464)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$1.parsePartialFrom(ClientProtos.java:12251)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$1.parsePartialFrom(ClientProtos.java:12245)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4274)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto.(ClientProtos.java:9981)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto.(ClientProtos.java:9910)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$1.parsePartialFrom(ClientProtos.java:14097)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$1.parsePartialFrom(ClientProtos.java:14091)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4274)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
>

[jira] [Updated] (HBASE-27097) SimpleRpcServer is broken

2022-06-09 Thread Andrew Kyle Purtell (Jira)



 [ 
https://issues.apache.org/jira/browse/HBASE-27097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Kyle Purtell updated HBASE-27097:

Priority: Blocker  (was: Critical)

> SimpleRpcServer is broken
> -
>
> Key: HBASE-27097
> URL: https://issues.apache.org/jira/browse/HBASE-27097
> Project: HBase
>  Issue Type: Bug
>  Components: rpc
>Affects Versions: 2.5.0
>Reporter: Andrew Kyle Purtell
>Priority: Blocker
>
> Concerns about SimpleRpcServer are not new, and not new to 2.5.  @chenxu 
> noticed a problem on HBASE-23917 back in 2020. After some simple evaluations 
> it seems quite broken. 
> When I run an async version of ITLCC against a 2.5.0 cluster configured with 
> hbase.rpc.server.impl=SimpleRpcServer, the client almost immediately stalls 
> because there are too many in flight requests. The logic to pause with too 
> many in flight requests is my own. That's not important. Looking at the 
> server logs it is apparent that SimpleRpcServer is quite broken. Handlers 
> suffer frequent protobuf parse errors and do not properly return responses to 
> the client. This is what stalls my test client. Rather quickly all available 
> request slots are full of requests that will have to time out on the client 
> side. 
> Exceptions have three patterns but they all have in common 
> SimpleServerRpcConnection#process. It seems likely the root cause is 
> mismatched expectations or bugs in connection buffer handling in 
> SimpleRpcServer/SimpleServerRpcConnection versus downstream classes that 
> process and parse the buffers. It also seems likely that changes were made to 
> downstream classes like ServerRpcConnection expecting NettyRpcServer's 
> particulars without updating SimpleServerRpcConnection and/or 
> SimpleRpcServer. That said, this is just a superficial analysis.
> 1) "Protocol message end-group tag did not match expected tag"
> {noformat}
>  2022-06-07T16:44:04,625 WARN  
> [Reader=5,bindAddress=buildbox.localdomain,port=8120] ipc.RpcServer: 
> /127.0.1.1:8120 is unable to read call parameter from client 127.0.0.1
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException:
>  Protocol message end-group tag did not match expected tag.
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:129)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.checkLastTagWas(CodedInputStream.java:4034)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4275)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue.(ClientProtos.java:10520)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue.(ClientProtos.java:10464)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$1.parsePartialFrom(ClientProtos.java:12251)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$1.parsePartialFrom(ClientProtos.java:12245)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4274)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto.(ClientProtos.java:9981)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto.(ClientProtos.java:9910)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$1.parsePartialFrom(ClientProtos.java:14097)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$1.parsePartialFrom(ClientProtos.java:14091)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4274)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutateRequest.(ClientProtos.java:14251)
>

[jira] [Commented] (HBASE-27097) SimpleRpcServer is broken

2022-06-09 Thread Andrew Kyle Purtell (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-27097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552473#comment-17552473
 ] 

Andrew Kyle Purtell commented on HBASE-27097:
-

I was taking with [~vjasani] offline and he feels SimpleRpcServer deserves 
fixes and stability. 

I think the consensus is keep SimpleRpcServer, but in its current state it is 
not releasable, so I will raise the priority of this issue to Blocker. 

> SimpleRpcServer is broken
> -
>
> Key: HBASE-27097
> URL: https://issues.apache.org/jira/browse/HBASE-27097
> Project: HBase
>  Issue Type: Bug
>  Components: rpc
>Affects Versions: 2.5.0
>Reporter: Andrew Kyle Purtell
>Priority: Critical
>
> Concerns about SimpleRpcServer are not new, and not new to 2.5.  @chenxu 
> noticed a problem on HBASE-23917 back in 2020. After some simple evaluations 
> it seems quite broken. 
> When I run an async version of ITLCC against a 2.5.0 cluster configured with 
> hbase.rpc.server.impl=SimpleRpcServer, the client almost immediately stalls 
> because there are too many in flight requests. The logic to pause with too 
> many in flight requests is my own. That's not important. Looking at the 
> server logs it is apparent that SimpleRpcServer is quite broken. Handlers 
> suffer frequent protobuf parse errors and do not properly return responses to 
> the client. This is what stalls my test client. Rather quickly all available 
> request slots are full of requests that will have to time out on the client 
> side. 
> Exceptions have three patterns but they all have in common 
> SimpleServerRpcConnection#process. It seems likely the root cause is 
> mismatched expectations or bugs in connection buffer handling in 
> SimpleRpcServer/SimpleServerRpcConnection versus downstream classes that 
> process and parse the buffers. It also seems likely that changes were made to 
> downstream classes like ServerRpcConnection expecting NettyRpcServer's 
> particulars without updating SimpleServerRpcConnection and/or 
> SimpleRpcServer. That said, this is just a superficial analysis.
> 1) "Protocol message end-group tag did not match expected tag"
> {noformat}
>  2022-06-07T16:44:04,625 WARN  
> [Reader=5,bindAddress=buildbox.localdomain,port=8120] ipc.RpcServer: 
> /127.0.1.1:8120 is unable to read call parameter from client 127.0.0.1
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException:
>  Protocol message end-group tag did not match expected tag.
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:129)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.checkLastTagWas(CodedInputStream.java:4034)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4275)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue.(ClientProtos.java:10520)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue.(ClientProtos.java:10464)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$1.parsePartialFrom(ClientProtos.java:12251)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$ColumnValue$1.parsePartialFrom(ClientProtos.java:12245)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hbase.thirdparty.com.google.protobuf.CodedInputStream$ByteInputDecoder.readMessage(CodedInputStream.java:4274)
>  ~[hbase-shaded-protobuf-4.1.0.jar:4.1.0]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto.(ClientProtos.java:9981)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto.(ClientProtos.java:9910)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$1.parsePartialFrom(ClientProtos.java:14097)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MutationProto$1.parsePartialFrom(ClientProtos.java:14091)
>  ~[hbase-protocol-shaded-2.5.1-SNAPSHOT.jar:2.5.1-SNAPSHOT]
>     at 
>

[jira] [Commented] (HBASE-27066) The Region Visualizer display failed

2022-06-09 Thread Hudson (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-27066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552471#comment-17552471
 ] 

Hudson commented on HBASE-27066:


Results for branch master
[build #607 on 
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/607/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(x) {color:red}-1 general checks{color}
-- For more information [see general 
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/607/General_20Nightly_20Build_20Report/]






(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/607/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]


(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11 
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/607/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> The Region Visualizer display failed
> 
>
> Key: HBASE-27066
> URL: https://issues.apache.org/jira/browse/HBASE-27066
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.5.0
>Reporter: Tao Li
>Assignee: Tao Li
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-3
>
> Attachments: 27066-xso.jpg, image-2022-05-27-14-22-29-015.png, 
> image-2022-05-27-14-22-44-336.png
>
>
> The `Region Visualizer` display failed. Because the active master hostname is 
> `localhost`.
> Before the change:
> !image-2022-05-27-14-22-44-336.png|width=520,height=162!
> After the change:
> !image-2022-05-27-14-22-29-015.png|width=562,height=229!



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Comment Edited] (HBASE-26708) Netty "leak detected" and OutOfDirectMemoryError due to direct memory buffering

2022-06-09 Thread Andrew Kyle Purtell (Jira)

[
https://issues.apache.org/jira/browse/HBASE-26708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552464#comment-17552464
]

Andrew Kyle Purtell edited comment on HBASE-26708 at 6/9/22 11:15 PM:
--

On the subject of configuration and NettyRpcServer, we leave netty level
resource limits unbounded. The number of threads to use for the event loop is
default 0 (unbounded). The default for io.netty.eventLoop.maxPendingTasks is
INT_MAX. We don't do this for our own RPC handlers. We have a notion of maximum
handler pool size, with a default of 30, typically raised in production by the
user. We constrain the depth of the request queue in multiple ways... limits on
number of queued calls, limits on total size of calls data that can be queued
(to avoid memory usage overrun, just like this case), CoDel conditioning of the
call queues if it is enabled, and so on.

This is going to be somewhat application dependent too. If the application
interacts synchronously with calls and has its own bound, then in flight
requests or their network level handling will be bounded by the aggregate
(client_in_flight_max x number_of_clients). If the application is highly async,
write-mostly, or a load test client – which is typically write-mostly, async,
and configured with large bounds :) – then this can explain the findings
reported here. It may also explain why security makes it worse, because when
security is active we wrap (encrypt) and unwrap (decrypt) up in the call layer,
beyond netty, and that takes additional time there, which would back things up
at the netty layer more than if call handling would complete more quickly
without encryption.

Consider the hbase.netty.eventloop.rpcserver.thread.count default. It is 0
(unbounded). I don't know what it can actually get up to in production, because
we lack the metric, but there are diminishing returns when threads > cores so a
reasonable default here could be Runtime.getRuntime().availableProcessors()
instead of unbounded?

maxPendingTasks probably should not be INT_MAX, but that may matter less.

The goal would be to limit concurrency at the netty layer in such a way that:
1. Performance is still good
2. Under load, we don't balloon resource usage at the netty layer

I could be looking at something that isn't the real issue but it is notable.

was (Author: apurtell):
On the subject of configuration and NettyRpcServer, we leave netty level
resource limits unbounded. The number of threads to use for the event loop is
default 0 (unbounded). The default for io.netty.eventLoop.maxPendingTasks is
INT_MAX. We don't do this for our own RPC handlers. We have a notion of maximum
handler pool size, with a default of 30, typically raised in production by the
user. We constrain the depth of the request queue in multiple ways... limits on
number of queued calls, limits on total size of calls data that can be queued
(to avoid memory usage overrun, just like this case), CoDel conditioning of the
call queues if it is enabled, and so on.

This is going to be somewhat application dependent too. If the application
interacts synchronously with calls and has its own bound, then in flight
requests or their network level handling will be bounded by the aggregate
(client_limit x number_of_clients). If the application is highly async,
write-mostly, or a load test client – which is typically write-mostly, async,
and configured with large bounds :) – then this can explain the findings
reported here. It may also explain why security makes it worse, because when
security is active we wrap (encrypt) and unwrap (decrypt) up in the call layer,
beyond netty, and that takes additional time there, which would back things up
at the netty layer more than if call handling would complete more quickly
without encryption.

Consider the hbase.netty.eventloop.rpcserver.thread.count default.

[jira] [Comment Edited] (HBASE-26708) Netty "leak detected" and OutOfDirectMemoryError due to direct memory buffering

2022-06-09 Thread Andrew Kyle Purtell (Jira)

[
https://issues.apache.org/jira/browse/HBASE-26708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552464#comment-17552464
]

Andrew Kyle Purtell edited comment on HBASE-26708 at 6/9/22 11:14 PM:
--

This is going to be somewhat application dependent too. If the application
interacts synchronously with calls and has its own bound, then in flight
requests or their network level handling will be bounded by the aggregate
(client_limit x number_of_clients). If the application is highly async,
write-mostly, or a load test client – which is typically write-mostly, async,
and configured with large bounds :) – then this can explain the findings
reported here. It may also explain why security makes it worse, because when
security is active we wrap (encrypt) and unwrap (decrypt) up in the call layer,
beyond netty, and that takes additional time there, which would back things up
at the netty layer more than if call handling would complete more quickly
without encryption.

maxPendingTasks probably should not be INT_MAX, but that may matter less.

The goal would be to limit concurrency at the netty layer in such a way that:
1. Performance is still good
2. Under load, we don't balloon resource usage at the netty layer

I could be looking at something that isn't the real issue but it is notable.

This is going to be somewhat application dependent too. If the application
interacts synchronously with calls and has its own bound, then in flight
requests or their network level handling will be bounded by the aggregate
(client_limit x number_of_clients). If the application is highly async,
write-mostly, or a load test client – which is typically write-mostly, async,
and configured with large bounds :) – then this can explain the findings
reported here. It may also explain why security makes it worse, because when
security is active we wrap (encrypt) and unwrap (decrypt) up in the call layer,
beyond netty, and that takes additional time there, which would back things up
at the netty layer more than if call handling would complete more quickly
without encryption.

Consider the hbase.netty.eventloop.rpcserver.thread.count default. It is 0
(unbounded). I don't know what it can actually

[jira] [Comment Edited] (HBASE-26708) Netty "leak detected" and OutOfDirectMemoryError due to direct memory buffering

2022-06-09 Thread Andrew Kyle Purtell (Jira)

[
https://issues.apache.org/jira/browse/HBASE-26708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552464#comment-17552464
]

Andrew Kyle Purtell edited comment on HBASE-26708 at 6/9/22 11:13 PM:
--

This is going to be somewhat application dependent too. If the application
interacts synchronously with calls and has its own bound, then in flight
requests or their network level handling will be bounded by the aggregate
(client_limit x number_of_clients). If the application is highly async,
write-mostly, or a load test client – which is typically write-mostly, async,
and configured with large bounds :) – then this can explain the findings
reported here. It may also explain why security makes it worse, because when
security is active we wrap (encrypt) and unwrap (decrypt) up in the call layer,
beyond netty, and that takes additional time there, which would back things up
at the netty layer more than if call handling would complete more quickly
without encryption.

maxPendingTasks probably should not be INT_MAX, but that may matter less.

The goal would be to limit concurrency at the netty layer in such a way that:
1. Performance is still good
2. Under load, we don't balloon resource usage at the netty layer

I could be looking at something that isn't the real issue but it is notable.

This is going to be somewhat application dependent too. If the application
interacts synchronously with calls and has its own bound, then in flight
requests or their network level handling will be bounded by the aggregate
(client_limit x number_of_clients). If the application is highly async,
write-mostly, or a load test client – which is typically write-mostly, async,
and configured with large bounds :) – then this can explain the findings
reported here. It may also explain why security makes it worse, because when
security is active we wrap (encrypt) and unwrap (decrypt) up in the call layer,
beyond netty, and that takes additional time there, which would back things up
at the netty layer more than if call handling would complete more quickly
without encryption.

Consider the hbase.netty.eventloop.rpcserver.thread.count default. It is 0
(unbounded). I don't know what it can actually get up to in production, because

[jira] [Comment Edited] (HBASE-26708) Netty "leak detected" and OutOfDirectMemoryError due to direct memory buffering

2022-06-09 Thread Andrew Kyle Purtell (Jira)

[
https://issues.apache.org/jira/browse/HBASE-26708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552464#comment-17552464
]

Andrew Kyle Purtell edited comment on HBASE-26708 at 6/9/22 11:12 PM:
--

This is going to be somewhat application dependent too. If the application
interacts synchronously with calls and has its own bound, then in flight
requests or their network level handling will be bounded by the aggregate
(client_limit x number_of_clients). If the application is highly async,
write-mostly, or a load test client – which is typically write-mostly, async,
and configured with large bounds :) – then this can explain the findings
reported here. It may also explain why security makes it worse, because when
security is active we wrap (encrypt) and unwrap (decrypt) up in the call layer,
beyond netty, and that takes additional time there, which would back things up
at the netty layer more than if call handling would complete more quickly
without encryption.

maxPendingTasks probably should not be INT_MAX, but that may matter less.

The goal would be to limit concurrency at the netty layer in such a way that:
1. Performance is still good
2. Under load, we don't balloon resource usage at the netty layer

I could be looking at something that isn't the real issue but it is notable.

This is going to be somewhat application dependent too. If the application
interacts synchronously with calls and has its own bound, then in flight
requests or their network level handling will be bounded by the aggregate
(client_limit x number_of_clients). If the application is highly async,
write-mostly, or a load test client – which is typically write-mostly, async,
and configured with large bounds :) – then this can explain the findings
reported here. It may also explain why security makes it worse, because when
security is active we wrap (encrypt) and unwrap (decrypt) up in the call layer,
beyond netty, and that takes additional time there, which would back things up
at the netty layer more than if call handling would complete more quickly
without encryption.

Consider the hbase.netty.eventloop.rpcserver.thread.count default. It is 0
(unbounded). I don't know what it can actually get up to in

[jira] [Comment Edited] (HBASE-26708) Netty "leak detected" and OutOfDirectMemoryError due to direct memory buffering

2022-06-09 Thread Andrew Kyle Purtell (Jira)

[
https://issues.apache.org/jira/browse/HBASE-26708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552464#comment-17552464
]

Andrew Kyle Purtell edited comment on HBASE-26708 at 6/9/22 11:09 PM:
--

Under load can we pile up a excess of pending request state, such as direct
buffers containing request bytes, at the netty layer because of downstream
resource limits? Those limits will act as a bottleneck, as intended, and before
would have also applied backpressure through RPC too, *because SimpleRpcServer
had thread limits (hbase.ipc.server.read.threadpool.size", default 10) and was
not async, but now netty is able to queue up a lot of work asynchronously*.
This is going to be somewhat application dependent too. If the application
interacts synchronously with calls and has its own bound, then in flight
requests or their network level handling will be bounded by the aggregate
(client_limit x number_of_clients). If the application is highly async,
write-mostly, or a load test client – which is typically write-mostly, async,
and configured with large bounds :) – then this can explain the findings
reported here.

And this may also explain why security makes it worse, because when security is
active we wrap (encrypt) and unwrap (decrypt) up in the call layer, beyond
netty, and that takes additional time there, which would back things up at the
netty layer more than if call handling would complete more quickly without
encryption.

maxPendingTasks probably should not be INT_MAX, but that may matter less.

The goal would be to limit concurrency at the netty layer in such a way that:
1. Performance is still good
2. Under load, we don't balloon resource usage at the netty layer

Under load can we pile up a excess of pending request state, such as direct
buffers containing request bytes, at the netty layer because of downstream
resource limits? Those limits will act as a bottleneck, as intended, and before
would have also applied backpressure through RPC too, but now netty is able to
queue up a lot of work asynchronously. This is going to be somewhat application
dependent too. If the application interacts synchronously with calls and has
its own bound, then in flight requests or their network level handling will be
bounded by the aggregate (client_limit x number_of_clients). If the application
is highly async, write-mostly, or a load test client – which is typically
write-mostly, async, and configured with large bounds :) – then this can
explain the findings reported here.

[jira] [Comment Edited] (HBASE-26708) Netty "leak detected" and OutOfDirectMemoryError due to direct memory buffering

2022-06-09 Thread Andrew Kyle Purtell (Jira)

[
https://issues.apache.org/jira/browse/HBASE-26708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552464#comment-17552464
]

Andrew Kyle Purtell edited comment on HBASE-26708 at 6/9/22 11:09 PM:
--

This is going to be somewhat application dependent too. If the application
interacts synchronously with calls and has its own bound, then in flight
requests or their network level handling will be bounded by the aggregate
(client_limit x number_of_clients). If the application is highly async,
write-mostly, or a load test client – which is typically write-mostly, async,
and configured with large bounds :) – then this can explain the findings
reported here. It may also explain why security makes it worse, because when
security is active we wrap (encrypt) and unwrap (decrypt) up in the call layer,
beyond netty, and that takes additional time there, which would back things up
at the netty layer more than if call handling would complete more quickly
without encryption.

maxPendingTasks probably should not be INT_MAX, but that may matter less.

The goal would be to limit concurrency at the netty layer in such a way that:
1. Performance is still good
2. Under load, we don't balloon resource usage at the netty layer

[jira] [Comment Edited] (HBASE-26708) Netty "leak detected" and OutOfDirectMemoryError due to direct memory buffering

2022-06-09 Thread Andrew Kyle Purtell (Jira)

[
https://issues.apache.org/jira/browse/HBASE-26708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552464#comment-17552464
]

Andrew Kyle Purtell edited comment on HBASE-26708 at 6/9/22 11:06 PM:
--

Under load can we pile up a excess of pending request state, such as direct
buffers containing request bytes, at the netty layer because of downstream
resource limits? Those limits will act as a bottleneck, as intended, and before
would have also applied backpressure through RPC too, but now netty is able to
queue up a lot of work asynchronously. This is going to be somewhat application
dependent too. If the application interacts synchronously with calls and has
its own bound, then in flight requests or their network level handling will be
bounded by the aggregate (client_limit x number_of_clients). If the application
is highly async, write-mostly, or a load test client – which is typically
write-mostly, async, and configured with large bounds :) – then this can
explain the findings reported here.

maxPendingTasks probably should not be INT_MAX, but that may matter less.

The goal would be to limit concurrency at the netty layer in such a way that:
1. Performance is still good
2. Under load, we don't balloon resource usage at the netty layer

Under load can we pile up a excess of pending request state, such as direct
buffers containing request bytes, at the netty layer because of downstream
resource limits? Those limits will act as a bottleneck, as intended, and before
would have also applied backpressure through RPC too, but now netty is able to
queue up a lot of work asynchronously. This is going to be somewhat application
dependent too. If the application interacts synchronously with calls and has
its own bound, then in flight requests or their network level handling will be
bounded by the aggregate (client_limit x number_of_clients). If the application
is highly async, write-mostly, or a load test client – which is typically
write-mostly, async, and configured with large bounds :) – then this can
explain the findings reported here.

maxPendingTasks should not be INT_MAX, that's not a sane default.

> Netty "leak detected" and

[jira] [Comment Edited] (HBASE-26708) Netty "leak detected" and OutOfDirectMemoryError due to direct memory buffering

2022-06-09 Thread Andrew Kyle Purtell (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-26708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552464#comment-17552464
 ] 

Andrew Kyle Purtell edited comment on HBASE-26708 at 6/9/22 11:04 PM:
--

On the subject of configuration and NettyRpcServer, we leave netty level 
resource limits unbounded. The number of threads to use for the event loop is 
default 0 (unbounded). The default for io.netty.eventLoop.maxPendingTasks is 
INT_MAX. We don't do this for our own RPC handlers. We have a notion of maximum 
handler pool size, with a default of 30, typically raised in production by the 
user. We constrain the depth of the request queue in multiple ways... limits on 
number of queued calls, limits on total size of calls data that can be queued 
(to avoid memory usage overrun, just like this case), CoDel conditioning of the 
call queues if it is enabled, and so on.

Under load can we pile up a excess of pending request state, such as direct 
buffers containing request bytes, at the netty layer because of downstream 
resource limits? Those limits will act as a bottleneck, as intended, and before 
would have also applied backpressure through RPC too, but now netty is able to 
queue up a lot of work asynchronously. This is going to be somewhat application 
dependent too. If the application interacts synchronously with calls and has 
its own bound, then in flight requests or their network level handling will be 
bounded by the aggregate (client_limit x number_of_clients). If the application 
is highly async, write-mostly, or a load test client – which is typically 
write-mostly, async, and configured with large bounds :) – then this can 
explain the findings reported here.

And this may also explain why security makes it worse, because when security is 
active we wrap (encrypt) and unwrap (decrypt) up in the call layer, beyond 
netty, and that takes additional time there, which would back things up at the 
netty layer more than if call handling would complete more quickly without 
encryption.

Consider the hbase.netty.eventloop.rpcserver.thread.count default. It is 0 
(unbounded). I don't know what it can actually get up to in production, because 
we lack the metric, but there are diminishing returns when threads > cores so a 
reasonable default here could be Runtime.getRuntime().availableProcessors() 
instead of unbounded?

maxPendingTasks should not be INT_MAX, that's not a sane default.


was (Author: apurtell):
On the subject of configuration and NettyRpcServer, we leave netty level 
resource limits unbounded. The number of threads to use for the event loop is 
default 0 (unbounded). The default for io.netty.eventLoop.maxPendingTasks is 
INT_MAX. We don't do this for our own RPC handlers. We have a notion of maximum 
handler pool size, with a default of 30, typically raised in production by the 
user. We constrain the depth of the request queue in multiple ways... limits on 
number of queued calls, limits on total size of calls data that can be queued 
(to avoid memory usage overrun, just like this case), CoDel conditioning of the 
call queues if it is enabled, and so on.

Under load can we pile up a excess of pending request state, such as direct 
buffers containing request bytes, at the netty layer because of downstream 
resource limits? Those limits will act as a bottleneck, as intended, and before 
would have also applied backpressure through RPC too, but now netty is able to 
queue up a lot of work asynchronously. This is going to be somewhat application 
dependent too. If the application interacts synchronously with calls and has 
its own bound, then in flight requests or their network level handling will be 
bounded by the aggregate (client_limit x number_of_clients). If the application 
is highly async, write-mostly, or a load test client – which is typically 
write-mostly, async, and configured with large bounds :) – then this can 
explain the findings reported here.

Consider the hbase.netty.eventloop.rpcserver.thread.count default. It is 0 
(unbounded). I don't know what it can actually get up to in production, because 
we lack the metric, but there are diminishing returns when threads > cores so a 
reasonable default here could be Runtime.getRuntime().availableProcessors() 
instead of unbounded?

maxPendingTasks should not be INT_MAX, that's not a sane default.

> Netty "leak detected" and OutOfDirectMemoryError due to direct memory 
> buffering
> ---
>
> Key: HBASE-26708
> URL: https://issues.apache.org/jira/browse/HBASE-26708
> Project: HBase
>  Issue Type: Bug
>  Components: rpc
>Affects Versions: 2.5.0, 2.4.6
>Reporter: Viraj Jasani
>Priority: Critical
>
> Under constant data ingestion, using default Netty based RpcServer and 
>

[jira] [Comment Edited] (HBASE-26708) Netty "leak detected" and OutOfDirectMemoryError due to direct memory buffering

2022-06-09 Thread Andrew Kyle Purtell (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-26708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552464#comment-17552464
 ] 

Andrew Kyle Purtell edited comment on HBASE-26708 at 6/9/22 11:01 PM:
--

On the subject of configuration and NettyRpcServer, we leave netty level 
resource limits unbounded. The number of threads to use for the event loop is 
default 0 (unbounded). The default for io.netty.eventLoop.maxPendingTasks is 
INT_MAX. We don't do this for our own RPC handlers. We have a notion of maximum 
handler pool size, with a default of 30, typically raised in production by the 
user. We constrain the depth of the request queue in multiple ways... limits on 
number of queued calls, limits on total size of calls data that can be queued 
(to avoid memory usage overrun, just like this case), CoDel conditioning of the 
call queues if it is enabled, and so on.

Under load can we pile up a excess of pending request state, such as direct 
buffers containing request bytes, at the netty layer because of downstream 
resource limits? Those limits will act as a bottleneck, as intended, and before 
would have also applied backpressure through RPC too, but now netty is able to 
queue up a lot of work asynchronously. This is going to be somewhat application 
dependent too. If the application interacts synchronously with calls and has 
its own bound, then in flight requests or their network level handling will be 
bounded by the aggregate (client_limit x number_of_clients). If the application 
is highly async, write-mostly, or a load test client – which is typically 
write-mostly, async, and configured with large bounds :) – then this can 
explain the findings reported here.

Consider the hbase.netty.eventloop.rpcserver.thread.count default. It is 0 
(unbounded). I don't know what it can actually get up to in production, because 
we lack the metric, but there are diminishing returns when threads > cores so a 
reasonable default here could be Runtime.getRuntime().availableProcessors() 
instead of unbounded?

maxPendingTasks should not be INT_MAX, that's not a sane default.


was (Author: apurtell):
On the subject of configuration and NettyRpcServer, we leave netty level 
resource limits unbounded. The number of threads to use for the event loop is 
default 0 (unbounded). The default for io.netty.eventLoop.maxPendingTasks is 
INT_MAX. We don't do this for our own RPC handlers. We have a notion of maximum 
handler pool size, with a default of 30, typically raised in production by the 
user. We constrain the depth of the request queue in multiple ways... limits on 
number of queued calls, limits on total size of calls data that can be queued 
(to avoid memory usage overrun, just like this case), CoDel conditioning of the 
call queues if it is enabled, and so on.

Under load can we pile up a excess of pending request state, such as direct 
buffers containing request bytes, at the netty layer because of downstream 
resource limits? Those limits will act as a bottleneck, as intended, and before 
would have also applied backpressure through RPC too, but now netty is able to 
queue up a lot of work asynchronously. 

Consider the hbase.netty.eventloop.rpcserver.thread.count default. It is 0 
(unbounded). I don't know what it can actually get up to in production, because 
we lack the metric, but there are diminishing returns when threads > cores so a 
reasonable default here could be Runtime.getRuntime().availableProcessors() 
instead of unbounded?

maxPendingTasks should not be INT_MAX, that's not a sane default.

> Netty "leak detected" and OutOfDirectMemoryError due to direct memory 
> buffering
> ---
>
> Key: HBASE-26708
> URL: https://issues.apache.org/jira/browse/HBASE-26708
> Project: HBase
>  Issue Type: Bug
>  Components: rpc
>Affects Versions: 2.5.0, 2.4.6
>Reporter: Viraj Jasani
>Priority: Critical
>
> Under constant data ingestion, using default Netty based RpcServer and 
> RpcClient implementation results in OutOfDirectMemoryError, supposedly caused 
> by leaks detected by Netty's LeakDetector.
> {code:java}
> 2022-01-25 17:03:10,084 ERROR [S-EventLoopGroup-1-3] 
> util.ResourceLeakDetector - java:115)
>   
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.expandCumulation(ByteToMessageDecoder.java:538)
>   
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder$1.cumulate(ByteToMessageDecoder.java:97)
>   
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:274)
>   
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
>   
>

1 2 >

1 - 100 of 156 matches

Mail list logo