[
https://issues.apache.org/jira/browse/HBASE-22169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16912589#comment-16912589
]
Nick Dimiduk commented on HBASE-22169:
--------------------------------------
[~luffy123], [~stack] {{TestHRegion#testOpenRegionFailedMemoryLeak}} fails
consistently (2/3 times) for me on 2.1.6RC0 with the following stacktrace. I
think you have a race between the {{Time-Limited test}} timeout and the amount
of time the test waits for metrics to propagate. It'll timeout if the amount of
time spent by other test methods before this test is run eat up the overall
class timeout quota. For this reason, I suggest extracting this method out into
it's own dedicated class.
FYI [~Apache9] .
{noformat}
2019-08-21 11:15:34,239 ERROR [Time-limited test]
coprocessor.CoprocessorHost(398): The coprocessor test threw
java.io.IOException: Failed on local exception:
java.nio.channels.ClosedByInterruptException; Host Details : local host is:
"myson.local/192.168.111.14"; destination host is: "test":8020;
java.io.IOException: Failed on local exception:
java.nio.channels.ClosedByInterruptException; Host Details : local host is:
"myson.local/192.168.111.14"; destination host is: "test":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773)
at org.apache.hadoop.ipc.Client.call(Client.java:1480)
at org.apache.hadoop.ipc.Client.call(Client.java:1413)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy38.getFileInfo(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:776)
at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy39.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2117)
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
at org.apache.hadoop.fs.FileSystem.isFile(FileSystem.java:1451)
at
org.apache.hadoop.hbase.util.CoprocessorClassLoader.getClassLoader(CoprocessorClassLoader.java:250)
at
org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:211)
at
org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:185)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:388)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:275)
at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:827)
at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:716)
at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:7008)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7218)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7191)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7149)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7100)
at
org.apache.hadoop.hbase.regionserver.TestHRegion.testOpenRegionFailedMemoryLeak(TestHRegion.java:6623)
{noformat}
> Open region failed cause memory leak
> ------------------------------------
>
> Key: HBASE-22169
> URL: https://issues.apache.org/jira/browse/HBASE-22169
> Project: HBase
> Issue Type: Bug
> Affects Versions: 2.0.5, 2.1.4
> Reporter: Bing Xiao
> Assignee: Bing Xiao
> Priority: Critical
> Fix For: 3.0.0, 2.2.1, 2.1.6
>
> Attachments:
> 0001-HBASE-22169-Open-region-failed-cause-memory-leak.patch,
> HBASE-22169-branch-2-v1.patch, HBASE-22169-branch-2.1-v1.patch,
> HBASE-22169-branch-2.2-v1.patch, HBASE-22169-master-v1.patch,
> HBASE-22169-master-v2.patch, HBASE-22169-master-v3.patch
>
>
> In some cases (for example, coprocessor path is wrong) region open failed,
> MetricsRegionWrapperImpl is already init and not close, cause memory leak;
> {code:java}
> 2019-02-21 15:41:32,929 ERROR
> [RS_OPEN_REGION-hb-2zedsc3fxjn12dl6u-005:16020-7]
> regionserver.RegionCoprocessorHost(362): Failed to load coprocessor
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService
> java.lang.IllegalArgumentException: java.net.UnknownHostException: emr-cluster
> at
> org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:378)
> at
> org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:310)
> at
> org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
> at
> org.apache.hadoop.hbase.util.CoprocessorClassLoader.init(CoprocessorClassLoader.java:165)
> at
> org.apache.hadoop.hbase.util.CoprocessorClassLoader.getClassLoader(CoprocessorClassLoader.java:250)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:194)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:352)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:240)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:749)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:657)
> at sun.reflect.GeneratedConstructorAccessor13.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6727)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7037)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7009)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6965)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6916)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:362)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129){code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
