[ https://issues.apache.org/jira/browse/HBASE-14819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014929#comment-15014929 ]
stack commented on HBASE-14819: ------------------------------- Hmm... doesn't seem to have helped adding the permgen flag. hbase-it 1.2 build of ITBLL just did below when it was passing the last bunch of times. Let me back out the -XX:+CMSClassUnloadingEnabled flag. 2015-11-20 00:36:05,795 ERROR [B.defaultRpcServer.handler=1,queue=0,port=46374] ipc.RpcServer(2212): Unexpected throwable object java.lang.OutOfMemoryError: PermGen space at org.codehaus.jackson.map.introspect.AnnotatedClass.construct(AnnotatedClass.java:132) at org.codehaus.jackson.map.introspect.BasicClassIntrospector.classWithCreators(BasicClassIntrospector.java:184) at org.codehaus.jackson.map.introspect.BasicClassIntrospector.collectProperties(BasicClassIntrospector.java:157) at org.codehaus.jackson.map.introspect.BasicClassIntrospector.forSerialization(BasicClassIntrospector.java:96) at org.codehaus.jackson.map.introspect.BasicClassIntrospector.forSerialization(BasicClassIntrospector.java:16) at org.codehaus.jackson.map.SerializationConfig.introspect(SerializationConfig.java:973) at org.codehaus.jackson.map.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:251) at org.codehaus.jackson.map.ser.StdSerializerProvider._createUntypedSerializer(StdSerializerProvider.java:782) at org.codehaus.jackson.map.ser.StdSerializerProvider._createAndCacheUntypedSerializer(StdSerializerProvider.java:735) at org.codehaus.jackson.map.ser.StdSerializerProvider.findValueSerializer(StdSerializerProvider.java:344) at org.codehaus.jackson.map.ser.StdSerializerProvider.findTypedValueSerializer(StdSerializerProvider.java:420) at org.codehaus.jackson.map.ser.StdSerializerProvider._serializeValue(StdSerializerProvider.java:601) at org.codehaus.jackson.map.ser.StdSerializerProvider.serializeValue(StdSerializerProvider.java:256) at org.codehaus.jackson.map.ObjectMapper._configAndWriteValue(ObjectMapper.java:2575) at org.codehaus.jackson.map.ObjectMapper.writeValueAsString(ObjectMapper.java:2097) at org.apache.hadoop.hbase.ipc.RpcServer.logResponse(RpcServer.java:2268) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2194) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) at java.lang.Thread.run(Thread.java:745) > hbase-it tests failing with OOME; permgen > ----------------------------------------- > > Key: HBASE-14819 > URL: https://issues.apache.org/jira/browse/HBASE-14819 > Project: HBase > Issue Type: Sub-task > Components: test > Reporter: stack > Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 14819v2.txt, Screen Shot 2015-11-16 at 11.37.41 PM.png, > itag.txt > > > Let me up the heap used when failsafe forks. > Here is example OOME doing ITBLL: > {code} > 2015-11-16 03:09:15,073 INFO [Thread-694] actions.BatchRestartRsAction(69): > Starting region server:asf905.gq1.ygridcore.net > 2015-11-16 03:09:15,099 INFO [Thread-694] client.ConnectionUtils(104): > regionserver/asf905.gq1.ygridcore.net/67.195.81.149:0 server-side HConnection > retries=350 > 2015-11-16 03:09:15,099 INFO [Thread-694] ipc.SimpleRpcScheduler(128): Using > deadline as user call queue, count=1 > 2015-11-16 03:09:15,101 INFO [Thread-694] ipc.RpcServer$Listener(607): > regionserver/asf905.gq1.ygridcore.net/67.195.81.149:0: started 3 reader(s) > listening on port=36114 > 2015-11-16 03:09:15,103 INFO [Thread-694] fs.HFileSystem(252): Added > intercepting call to namenode#getBlockLocations so can do block reordering > using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks > 2015-11-16 03:09:15,104 INFO [Thread-694] > zookeeper.RecoverableZooKeeper(120): Process identifier=regionserver:36114 > connecting to ZooKeeper ensemble=localhost:50139 > 2015-11-16 03:09:15,117 DEBUG [Thread-694-EventThread] > zookeeper.ZooKeeperWatcher(554): regionserver:361140x0, > quorum=localhost:50139, baseZNode=/hbase Received ZooKeeper Event, type=None, > state=SyncConnected, path=null > 2015-11-16 03:09:15,118 DEBUG [Thread-694] zookeeper.ZKUtil(492): > regionserver:361140x0, quorum=localhost:50139, baseZNode=/hbase Set watcher > on existing znode=/hbase/master > 2015-11-16 03:09:15,119 DEBUG [Thread-694] zookeeper.ZKUtil(492): > regionserver:361140x0, quorum=localhost:50139, baseZNode=/hbase Set watcher > on existing znode=/hbase/running > 2015-11-16 03:09:15,119 DEBUG [Thread-694-EventThread] > zookeeper.ZooKeeperWatcher(638): regionserver:36114-0x1510e2c6f1d0029 > connected > 2015-11-16 03:09:15,120 INFO [RpcServer.responder] > ipc.RpcServer$Responder(926): RpcServer.responder: starting > 2015-11-16 03:09:15,121 INFO [RpcServer.listener,port=36114] > ipc.RpcServer$Listener(738): RpcServer.listener,port=36114: starting > 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default > Start Handler index=0 queue=0 > 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default > Start Handler index=1 queue=0 > 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default > Start Handler index=2 queue=0 > 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default > Start Handler index=3 queue=0 > 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default > Start Handler index=4 queue=0 > 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority > Start Handler index=0 queue=0 > 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority > Start Handler index=1 queue=1 > 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority > Start Handler index=2 queue=0 > 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority > Start Handler index=3 queue=1 > 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority > Start Handler index=4 queue=0 > 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication > Start Handler index=0 queue=0 > 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication > Start Handler index=1 queue=0 > 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication > Start Handler index=2 queue=0 > 2015-11-16 03:09:15,761 DEBUG [RS:0;asf905:36114] > client.ConnectionManager$HConnectionImplementation(715): connection > construction failed > java.io.IOException: java.lang.OutOfMemoryError: PermGen space > at > org.apache.hadoop.hbase.client.RegistryFactory.getRegistry(RegistryFactory.java:43) > at > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.setupRegistry(ConnectionManager.java:886) > at > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:692) > at > org.apache.hadoop.hbase.client.ConnectionUtils$2.<init>(ConnectionUtils.java:154) > at > org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:154) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:689) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:720) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:733) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:889) > at > org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:156) > at > org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:108) > at > org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:140) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:356) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) > at > org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:334) > at > org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:138) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.OutOfMemoryError: PermGen space > at sun.misc.Unsafe.defineClass(Native Method) > at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:63) > at > sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:399) > at > sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:396) > at java.security.AccessController.doPrivileged(Native Method) > at > sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:395) > at > sun.reflect.MethodAccessorGenerator.generateConstructor(MethodAccessorGenerator.java:94) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:48) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at java.lang.Class.newInstance(Class.java:383) > at > org.apache.hadoop.hbase.client.RegistryFactory.getRegistry(RegistryFactory.java:41) > ... 17 more > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)