On Fri, May 3, 2013 at 11:43 AM, Michael Giordano < [email protected]> wrote:
> Sorry for resurrecting this ‘old’ thread but I need to open a ticket for > this. > https://issues.apache.org/jira/browse/ACCUMULO > **** > > ** ** > > How do I do that ?**** > > ** ** > > Thanks,**** > > Mike G.**** > > ** ** > > *From:* Keith Turner [mailto:[email protected]] > *Sent:* Wednesday, April 24, 2013 12:23 PM > > *To:* [email protected] > *Subject:* Re: PermGen leak in JBoss**** > > ** ** > > ** ** > > ** ** > > On Wed, Apr 24, 2013 at 11:21 AM, Heath Abelson < > [email protected]> wrote:**** > > Hi, I am a colleague of Michael's, I can provide more info > > The following are the 5 threads that are not being cleaned up on an > undeploy. It's hard to get more stack info than this. > > Daemon Thread [FileWatchdog] (Running) > Daemon Thread [MSC service thread 1-5-SendThread(cbdbtestbox:2181)] > (Running) > Daemon Thread [MSC service thread 1-5-EventThread] (Running) > Daemon Thread [Thrift Connection Pool Checker] (Running) > Daemon Thread [FileWatchdog] (Running) > > Using Plumbr, the report states that the SendThread and EventThread above > both contain an instance of org.apache.zookeeper.ClientCnxn**** > > ** ** > > ** ** > > I tried this on 1.4 and then ran jstack using following program. **** > > ** ** > > ** ** > > public class Test {**** > > public static void main(String[] args) throws Exception {**** > > ZooKeeperInstance instance = new ZooKeeperInstance("test14", > "localhost");**** > > Connector connector = instance.getConnector("root", "secret");**** > > **** > > UtilWaitThread.sleep(600000);**** > > ** ** > > }**** > > }**** > > ** ** > > I see the following threads after filtering out java threads and the > thread calling main. I am running this code in a different way. I used > the accumulo script to execute it, which uses Accumulo start, which > introduces the AccumuloFilesystemAlterationMonitor thread. I am seeing > three of the threads you see, I am not seeing the FileWatchDog threads.*** > * > > ** ** > > "Thrift Connection Pool Checker" daemon prio=10 tid=0x00007f861839f800 > nid=0x61f0 waiting on condition [0x00007f8652a62000]**** > > java.lang.Thread.State: TIMED_WAITING (sleeping)**** > > at java.lang.Thread.sleep(Native Method)**** > > at > org.apache.accumulo.core.client.impl.ThriftTransportPool$Closer.run(ThriftTransportPool.java:129) > **** > > at java.lang.Thread.run(Thread.java:662)**** > > ** ** > > "Test-EventThread" daemon prio=10 tid=0x00007f8618119000 nid=0x61ee > waiting on condition [0x00007f8652d02000]**** > > java.lang.Thread.State: WAITING (parking)**** > > at sun.misc.Unsafe.park(Native Method)**** > > - parking to wait for <0x00000000bb402270> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)**** > > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)**** > > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > **** > > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > **** > > at > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)**** > > ** ** > > "Test-SendThread(localhost:2181)" daemon prio=10 tid=0x00007f8618103000 > nid=0x61ed runnable [0x00007f8652e03000]**** > > java.lang.Thread.State: RUNNABLE**** > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)**** > > at > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)**** > > at > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)**** > > at > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)**** > > - locked <0x00000000bb3e0e88> (a sun.nio.ch.Util$2)**** > > - locked <0x00000000bb3e0e78> (a > java.util.Collections$UnmodifiableSet)**** > > - locked <0x00000000bb3e0aa8> (a sun.nio.ch.EPollSelectorImpl) > **** > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)**** > > at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1131)**** > > ** ** > > "org.apache.accumulo.start.classloader.AccumuloFilesystemAlterationMonitor" > daemon prio=10 tid=0x00007f86a4387000 nid=0x61e9 waiting on condition > [0x00007f865367c000]**** > > java.lang.Thread.State: TIMED_WAITING (sleeping)**** > > at java.lang.Thread.sleep(Native Method)**** > > at > org.apache.accumulo.start.classloader.AccumuloFilesystemAlterationMonitor.run(AccumuloFilesystemAlterationMonitor.java:127) > **** > > at java.lang.Thread.run(Thread.java:662)**** > > ** ** > > **** > > Should probably open a ticket about this issue. We could probably do > something about the Thrift Connection Pool Checker threads. I am not sure > if the zookeeper threads are an issue w/ zookeeper or an issue w/ how > accumulo is using zookeeper.**** > > ** ** > > > -Heath**** > > > -----Original Message----- > From: Michael Giordano > Sent: Wednesday, April 24, 2013 9:48 AM > To: [email protected] > Subject: PermGen leak in JBoss > > Under version 1.3.7 we are using the following code to initialize a > cloudbase connection during initialization of our web app: > > ZooKeeperInstance instance = new > ZooKeeperInstance(instanceName, zooKeepers); > connector = instance.getConnector(userId, > password.getBytes()); > > The problem is that under the hood, this call creates several (5) threads > that are not cleaned up when the app is undeployed in JBoss. This is > occurring without performing any scans or interacting with cloudbase in any > other way. After relatively few redeploys of the app, the PermGen Space is > OOM. > > I can't find any reference in the cloudbase API akin to a close() method > for the Connector object. This is a classloader leak effecting any webapp > that is accessing cloudbase directly. The result of this leak is not simply > orphaned threads, but thousands of classes not gc'd because the classloader > itself can't be gc'd. This is what is filling up PermGen. > > Has anyone discovered this particular issue? Has anyone discovered a > solution? > > Thanks, > Mike G. > > NetCentric Technology, Inc. > 3349 Route 138, Building A > Wall, NJ 07719 > Phone: 732-544-0888 > > **** > > ** ** >
