I inspected the heap dumps already, 2 things are interesting: - The heap dump file is quite small (only approx. 50 MB), so the permgen issues start when not even -Xmx is used completely. - There are lots of (250) reflection class loaders, but no leaks. All classloaders exist one time, just the reflection-based sun.relect.DelegatingClassLoader with the stubs for speeding up reflection shows up, but only with one class/loader. Which is perfectly fine. - ~2500 classes loaded, which is not much.
How do I get the number of interned Strings from jvisualvm? ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: [email protected] > -----Original Message----- > From: [email protected] [mailto:[email protected]] On Behalf > Of Dawid Weiss > Sent: Tuesday, July 09, 2013 9:13 AM > To: [email protected] > Subject: Re: Several builds hanging pecause of permgen > > What's the core of those permgen leaks, guys? Is it the number of classes > (via different classloaders)? Number of interned strings? > Maybe something is going on that we should reflect on before we raise the > default permgen limit? Our typical permgen headaches were due to > repeated classloader leaks -- is Hadoop using something like that? > > Dawid > > On Tue, Jul 9, 2013 at 8:25 AM, Uwe Schindler <[email protected]> wrote: > > Hi Mark, > > > > The problem with raising permgen is: > > - It's Hotspot specific only, so does not work with other JVMs > > - Its no longer available in Java 8 > > > > I would really prefer to maybe tune the tests and maybe not create so > many nodes in the cloud tests. It looks like the bug happens more often with > higher test multiplier (-Dtests.multiplier=3), so maybe we can really tune > that. > > If we want to raise permgen, we have to do it in a similar way like we > > do enable the heap dumps - with lots of <condition/> tasks in ANT... > > :( > > > > Uwe > > > > ----- > > Uwe Schindler > > H.-H.-Meier-Allee 63, D-28213 Bremen > > http://www.thetaphi.de > > eMail: [email protected] > > > > > >> -----Original Message----- > >> From: Mark Miller [mailto:[email protected]] > >> Sent: Tuesday, July 09, 2013 5:20 AM > >> To: [email protected] > >> Subject: Re: Several builds hanging pecause of permgen > >> > >> Looks like we currently don't set the max perm gen for tests, so you > >> get the default - I think we want to change that regardless - we > >> don't want it to vary IMO - it should work like Xmx. > >> > >> I think we should just set it to 128 mb, and these tests should have > >> plenty of room to run. > >> > >> - Mark > >> > >> On Jul 8, 2013, at 11:06 PM, Mark Miller <[email protected]> wrote: > >> > >> > > >> > On Jul 8, 2013, at 2:26 PM, Uwe Schindler <[email protected]> wrote: > >> > > >> >> Next one: > >> >> http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6395/console > >> >> > >> >> I will vote all releases of 4.4 with -1 until this is fixed! > >> > > >> > You can't veto a release, so it's kind of a hollow threat ;) > >> > > >> >> It hangs on my local computer, too! Tests pass only ½ of the time, > >> >> the > >> remaining time it hangs with permgen errors. > >> > > >> > Depending on the host OS and java version, I have had to raise the > >> > max > >> perm gem a bit higher for heavy SolrCloud tests that also start up > >> hdfs. I think I remember raising it from 96 MB to 128 MB. It's simply > >> a test resource issue - those tests are very heavy, which is why most > >> are set to run nightly - I've seen the code run heavily without any > >> perm gen issues though - and on some of my machines, I don't have to > raise the perm gen at all. > >> > > >> > - Mark > >> > > >> >> > >> >> Uwe > >> >> > >> >> ----- > >> >> Uwe Schindler > >> >> H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de > >> >> eMail: [email protected] > >> >> > >> >> From: [email protected] [mailto:[email protected]] On > >> Behalf > >> >> Of Dawid Weiss > >> >> Sent: Monday, July 08, 2013 2:16 PM > >> >> To: [email protected] > >> >> Subject: Re: Several builds hanging pecause of permgen > >> >> > >> >> > >> >> Not much I can do from my side about permgen errors. There is > >> >> really no > >> way to deal with these from within Java (the same process) -- you > >> cannot effectively handle anything because your own classes may not > load at all. > >> >> > >> >> Dawid > >> >> > >> >> On Mon, Jul 8, 2013 at 1:35 PM, Uwe Schindler <[email protected]> > >> wrote: > >> >> Another one, this time on OSX: > >> >> http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/617/ > >> >> > >> >> ----- > >> >> Uwe Schindler > >> >> H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de > >> >> eMail: [email protected] > >> >> > >> >> From: Robert Muir [mailto:[email protected]] > >> >> Sent: Sunday, July 07, 2013 11:15 PM > >> >> To: [email protected] > >> >> Subject: Re: Several builds hanging pecause of permgen > >> >> > >> >> When there were leaks from static classes, we added a checker to > >> LuceneTestCase that looks for RAM > N and fails with debugging > information. > >> >> > >> >> I wonder if some similar check is possible for this case (to make > >> >> it easier > >> than going thru heapdumps, and to find issues before crash-time)... > >> >> > >> >> On Sun, Jul 7, 2013 at 4:10 PM, Uwe Schindler <[email protected]> > wrote: > >> >> Another one: > >> >> http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6385/testRepo > >> >> rt/ > >> >> > >> > junit/junit.framework/TestSuite/org_apache_solr_request_SimpleFacetsT > >> >> est/ > >> >> > >> >> Had to be killed with kill -9 > >> >> > >> >> ----- > >> >> Uwe Schindler > >> >> H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de > >> >> eMail: [email protected] > >> >> > >> >> > >> >>> -----Original Message----- > >> >>> From: Uwe Schindler [mailto:[email protected]] > >> >>> Sent: Saturday, July 06, 2013 10:16 PM > >> >>> To: [email protected] > >> >>> Subject: RE: Several builds hanging pecause of permgen > >> >>> > >> >>> Another one: > >> >>> http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6375/console > >> >>> > >> >>> I was only able to kill the JVM with kill -9 I am sure, it's horrible > >> slowdoop! > >> >>> > >> >>> ----- > >> >>> Uwe Schindler > >> >>> H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de > >> >>> eMail: [email protected] > >> >>> > >> >>> > >> >>>> -----Original Message----- > >> >>>> From: Uwe Schindler [mailto:[email protected]] > >> >>>> Sent: Friday, July 05, 2013 3:59 PM > >> >>>> To: [email protected] > >> >>>> Subject: Several builds hanging pecause of permgen > >> >>>> > >> >>>> Several Jenkins builds now hang because of permgen. The runner > JVM > >> >>>> is dead (can only be killed by -9), last example: > >> >>>> > >> >>>> http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6360/console > >> >>>> > >> >>>> ----- > >> >>>> Uwe Schindler > >> >>>> H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de > >> >>>> eMail: [email protected] > >> >>>> > >> >>>> > >> >>>> > >> >>>> > >> >>>> ------------------------------------------------------------------- > >> >>>> -- To unsubscribe, e-mail: [email protected] For > >> >>>> additional commands, e-mail: [email protected] > >> >>> > >> >>> > >> >>> -------------------------------------------------------------------- > >> >>> - To unsubscribe, e-mail: [email protected] For > >> >>> additional commands, e-mail: [email protected] > >> >> > >> >> > >> >> --------------------------------------------------------------------- > >> >> To unsubscribe, e-mail: [email protected] For > >> >> additional commands, e-mail: [email protected] > >> >> > >> >> > >> > > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] For > additional > >> commands, e-mail: [email protected] > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
