[
https://issues.apache.org/jira/browse/HBASE-30092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18097514#comment-18097514
]
mazhengxuan commented on HBASE-30092:
-------------------------------------
I would like to investigate this issue. I will check the
MonitoredRPCHandlerImpl.generateCallInfoMap path and try to prepare a bounded
fix to avoid excessive memory usage. If nobody is working on it, could I try to
fix it?
> MonitoredRPCHandlerImpl.generateCallInfoMap() can OOM the RegionServer
> heartbeat path when an in-flight RPC request protobuf is large
> -------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-30092
> URL: https://issues.apache.org/jira/browse/HBASE-30092
> Project: HBase
> Issue Type: Bug
> Components: monitoring, regionserver
> Affects Versions: 2.6.5, 2.5.14
> Reporter: Nick Dimiduk
> Priority: Major
>
> A burst of large {{MultiRequest}} / {{ScanRequest}} RPCs on a RegionServer
> with a tight heap crashed the JVM with {{OutOfMemoryError}} from the periodic
> RegionServer-to-HMaster heartbeat path. The offending allocation is inside
> {{MonitoredRPCHandlerImpl.generateCallInfoMap()}}, which calls
> {{AbstractMessage.toString()}} on the live in-flight RPC request protobuf.
> {{TextFormat$Printer}} recursively appends to a {{StringBuilder}} that has to
> grow past available heap.
> {noformat}
> java.lang.OutOfMemoryError
> at java.util.Arrays.copyOf
> at java.lang.AbstractStringBuilder.ensureCapacityInternal
> at java.lang.StringBuilder.append (CharSequence)
> at
> org.apache.hbase.thirdparty.com.google.protobuf.TextFormat$TextGenerator.eol
> at
> org.apache.hbase.thirdparty.com.google.protobuf.TextFormat$Printer.printSingleField
> at
> org.apache.hbase.thirdparty.com.google.protobuf.TextFormat$Printer.printField
> at
> org.apache.hbase.thirdparty.com.google.protobuf.TextFormat$Printer.printMessage
> ... [recursive printMessage / printField for nested protobuf fields] ...
> at org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessage.toString
> (AbstractMessage.java:91)
> at
> org.apache.hadoop.hbase.monitoring.MonitoredRPCHandlerImpl.generateCallInfoMap
> (MonitoredRPCHandlerImpl.java:253)
> at org.apache.hadoop.hbase.monitoring.MonitoredRPCHandlerImpl.clone
> (MonitoredRPCHandlerImpl.java:57)
> at org.apache.hadoop.hbase.monitoring.TaskMonitor.processTasks
> (TaskMonitor.java:229)
> at org.apache.hadoop.hbase.monitoring.TaskMonitor.getTasks
> (TaskMonitor.java:183)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.buildServerLoad
> (HRegionServer.java:1566)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport
> at org.apache.hadoop.hbase.regionserver.HRegionServer.run
> {noformat}
> {{tryRegionServerReport}} runs on the main RegionServer thread at
> {{hbase.regionserver.msginterval}} (default 3 s). An OOM there terminates the
> JVM under {{-XX:OnOutOfMemoryError}}, or aborts the heartbeat otherwise. On a
> heap sized close to the working set, a brief write slowdown that causes
> handlers to back up is enough to grow per-handler scanner retention and push
> {{generateCallInfoMap()}} over the edge. Auto-replacement by an external pool
> manager doesn't break the loop — each replacement RS inherits the same client
> traffic and crashes again within 15–60 seconds of registering.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
