Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
A couple of other findings: - Tomcat 5.5.28 out-of-the-box with its default webapps has no problem starting with the default (parallel) gc - As soon as I remove the default webapps, add ours, and modify the context.xml file to provide it a DataSource, Tomcat hangs. We're definitely loading a lot more into memory and inducing garbage collection with Hibernate and Spring in the mix. I think I'm going to try a Hibernate or Spring sample webapp to see if it has similar problems. On Thu, Sep 10, 2009 at 4:43 PM, Caldarale, Charles R < chuck.caldar...@unisys.com> wrote: > > From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com] > > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate app > > in Tomcat 5.5 on Ubuntu 7 > > > > The *only option* that did not cause the startup process to > > halt indefinitely on startup was the Serial GC option. > > This is really weird. Might be time to file a bug report with Sun. > > > - Does this indicate some kind of threading problem in this Java env? > > Hard to tell without a snapshot of what's going on inside the GC code > (which is all C++, not Java). Could also be a problem with Ubuntu. > > > - Does the option "DoEscapeAnalysis" still use a parallel gc? > > Should be independent of the GC algorithm; it's extra logic in the JIT that > determines whether or not an object even needs to be allocated, or if it can > simply reside on the method's stack. > > > - Is the experimental more similar to the parallel or the serial? > > My understanding is that G1GC is both parallel and concurrent. > > - Chuck > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is thus for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all computers. > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
RE: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
> From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com] > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate app > in Tomcat 5.5 on Ubuntu 7 > > The *only option* that did not cause the startup process to > halt indefinitely on startup was the Serial GC option. This is really weird. Might be time to file a bug report with Sun. > - Does this indicate some kind of threading problem in this Java env? Hard to tell without a snapshot of what's going on inside the GC code (which is all C++, not Java). Could also be a problem with Ubuntu. > - Does the option "DoEscapeAnalysis" still use a parallel gc? Should be independent of the GC algorithm; it's extra logic in the JIT that determines whether or not an object even needs to be allocated, or if it can simply reside on the method's stack. > - Is the experimental more similar to the parallel or the serial? My understanding is that G1GC is both parallel and concurrent. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
I've tried each of the aforementioned GC options individually with the following other parameters: JAVA_OPTS=" -Xms368m -Xmx512m -XX:MaxPermSize=128m -XX:+PrintGC -XX:+PrintGCDetails -Djava.awt.headless=true -Dfile.encoding=UTF-8" The *only option* that did not cause the startup process to halt indefinitely on startup was the Serial GC option. I tried this one on 4 or 5 successive attempts and the server started up in under 30s. The others halted indefinitely (i.e. at least 5-10m before I finally killed the process) during GC as an incomplete GC log statement was the last thing in the catalina.out file. - Does this indicate some kind of threading problem in this Java env? - Does the option "DoEscapeAnalysis" still use a parallel gc? - Is the experimental more similar to the parallel or the serial? Thanks, Bradley On Thu, Sep 3, 2009 at 4:24 PM, Caldarale, Charles R < chuck.caldar...@unisys.com> wrote: > > From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com] > > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate app > > in Tomcat 5.5 on Ubuntu 7 > > > > And again in a new place on subsequent attempt. > > There's nothing unusual that I can see in the traces, so it looks like you > will have to try random GC settings until something works (as Mark suggested > some time ago). Try turning these tags on, one at a time: > > -XX:+UseParallelGC > -XX:+UseParallelOldGC > -XX:+UseSerialGC > -XX:+DoEscapeAnalysis > -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC > > The last one is somewhat experimental, as the parameter implies. > > - Chuck > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is thus for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all computers. > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
Hi, I have the same problem using tomcat 6 and jdk 6, do you find a workaround ? Regards! Mark Thomas wrote: > > Bradley Wagner wrote: >> Ok, I'll try that next. Two things: >> - Did some searching, but couldn't find how to determine the default GC >> method for my Java version > > http://blogs.sun.com/watt/resource/jvm-options-list.html is a useful > reference > >> - Would you recommend I move this problem elsewhere, given that this is >> likely not related to Tomcat at all. I don't want to spam this list. > > No need. GC freezes are close to the most common issue I see with > unstable production Tomcat systems so I think it is very relevant. > > Mark > > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > -- View this message in context: http://www.nabble.com/100--usage-and-hanging-on-startup-of-Spring-Hibernate-app-in-Tomcat--5.5-on-Ubuntu-7-tp25194606p25331069.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
Bradley Wagner wrote: > Ok, I'll try that next. Two things: > - Did some searching, but couldn't find how to determine the default GC > method for my Java version http://blogs.sun.com/watt/resource/jvm-options-list.html is a useful reference > - Would you recommend I move this problem elsewhere, given that this is > likely not related to Tomcat at all. I don't want to spam this list. No need. GC freezes are close to the most common issue I see with unstable production Tomcat systems so I think it is very relevant. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
> From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com] > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate app > in Tomcat 5.5 on Ubuntu 7 > > - Did some searching, but couldn't find how to determine the default GC > method for my Java version You appear to be running the parallel GC; that's one of the things JConsole would tell you. There are probably other ways to determine the setting, but unfortunately I've forgotten them. Lambda Probe does display the heap types, but you have to have Tomcat running to use it... > - Would you recommend I move this problem elsewhere, given that this is > likely not related to Tomcat at all. I don't want to spam this list. That's why people have delete keys. Just add [OT] to the subject line, if you think it's appropriate. Whatever is eventually discovered may well help other Tomcat users. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
Ok, I'll try that next. Two things: - Did some searching, but couldn't find how to determine the default GC method for my Java version - Would you recommend I move this problem elsewhere, given that this is likely not related to Tomcat at all. I don't want to spam this list. Thanks, Bradley On Thu, Sep 3, 2009 at 4:24 PM, Caldarale, Charles R < chuck.caldar...@unisys.com> wrote: > > From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com] > > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate app > > in Tomcat 5.5 on Ubuntu 7 > > > > And again in a new place on subsequent attempt. > > There's nothing unusual that I can see in the traces, so it looks like you > will have to try random GC settings until something works (as Mark suggested > some time ago). Try turning these tags on, one at a time: > > -XX:+UseParallelGC > -XX:+UseParallelOldGC > -XX:+UseSerialGC > -XX:+DoEscapeAnalysis > -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC > > The last one is somewhat experimental, as the parameter implies. > > - Chuck > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is thus for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all computers. > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
RE: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
> From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com] > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate app > in Tomcat 5.5 on Ubuntu 7 > > And again in a new place on subsequent attempt. There's nothing unusual that I can see in the traces, so it looks like you will have to try random GC settings until something works (as Mark suggested some time ago). Try turning these tags on, one at a time: -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:+UseSerialGC -XX:+DoEscapeAnalysis -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC The last one is somewhat experimental, as the parameter implies. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
> From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com] > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate app > in Tomcat 5.5 on Ubuntu 7 > > What's odd is that the PermGen space appears to be totally used > up at 26MB and not expanding, unless I'm reading it wrong. You're reading it wrong. That's just the amount used of the current PermGen size; it's nowhere near your max, so that's not the problem. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
This is odd. After setting permgen to 128m with "-XX:MaxPermSize=128m" I'm
seeing it hang again. What's odd is that the PermGen space appears to be
totally used up at 26MB and not expanding, unless I'm reading it wrong.
Heap after GC invocations=10 (full 0):
PSYoungGen total 47296K, used 2831K [0xb087, 0xb415,
0xb415)
eden space 36992K, 0% used [0xb087,0xb087,0xb2c9)
from space 10304K, 27% used [0xb374,0xb3a03e30,0xb415)
to space 10624K, 0% used [0xb2c9,0xb2c9,0xb36f)
PSOldGentotal 466048K, used 16623K [0x9415, 0xb087,
0xb087)
object space 466048K, 3% used [0x9415,0x9518bea8,0xb087)
PSPermGen total 26240K, used 26114K [0x8c15, 0x8daf,
0x9415)
object space 26240K, 99% used [0x8c15,0x8dad0a50,0x8daf)
}
VM-Thread 9737526 9770673 9771284
GC-Thread 0 entries: 8
[ old-to-young-roots-task 9750971 9766007 ]
[ thread-roots-task 9766013 9766016 ]
[ scavenge-roots-task 9766017 9766024 ]
[ scavenge-roots-task 9766024 9766028 ]
[ scavenge-roots-task 9766029 9766032 ]
[ scavenge-roots-task 9766033 9766504 ]
[ scavenge-roots-task 9766505 9766506 ]
[ steal-task 9766507 9770612 ]
GC-Thread 1 entries: 19
[ old-to-young-roots-task 9737771 9756587 ]
[ serial-old-to-young-roots-task 9756590 9762460 ]
[ scavenge-roots-task 9762462 9762465 ]
[ scavenge-roots-task 9762466 9762481 ]
[ thread-roots-task 9762482 9762500 ]
[ thread-roots-task 9762506 9762516 ]
[ thread-roots-task 9762516 9762534 ]
[ thread-roots-task 9762535 9762552 ]
[ thread-roots-task 9762553 9762558 ]
[ thread-roots-task 9762558 9762565 ]
[ thread-roots-task 9762565 9762567 ]
[ thread-roots-task 9762567 9762569 ]
[ thread-roots-task 9762570 9762572 ]
[ thread-roots-task 9762572 9762574 ]
[ thread-roots-task 9762574 9762580 ]
[ thread-roots-task 9762580 9762584 ]
[ thread-roots-task 9762585 9770048 ]
[ steal-task 9770050 9770591 ]
[ waitfor-barrier-task 9770592 9770632 ]
{Heap before GC invocations=11 (full 0):
PSYoungGen total 47296K, used 39823K [0xb087, 0xb415,
0xb415)
eden space 36992K, 100% used [0xb087,0xb2c9,0xb2c9)
from space 10304K, 27% used [0xb374,0xb3a03e30,0xb415)
to space 10624K, 0% used [0xb2c9,0xb2c9,0xb36f)
PSOldGentotal 466048K, used 16623K [0x9415, 0xb087,
0xb087)
object space 466048K, 3% used [0x9415,0x9518bea8,0xb087)
PSPermGen total 26624K, used 26570K [0x8c15, 0x8db5,
0x9415)
object space 26624K, 99% used [0x8c15,0x8db42ab0,0x8db5)
10.307: [GC
On Thu, Sep 3, 2009 at 3:58 PM, Caldarale, Charles R <
chuck.caldar...@unisys.com> wrote:
> > From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com]
> > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate app
> > in Tomcat 5.5 on Ubuntu 7
> >
> > Hmm, I'm going to try again with more explicit PermGen space as it
> > seems to be out of it.
>
> Try it, but I'd be surprised if it had any effect. The values shown are
> for the current size of PermGen, not the maximum to which it may expand.
> That's usually 64m, unless you've changed it.
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is thus for use only by the intended recipient. If you received
> this in error, please contact the sender and delete the e-mail and its
> attachments from all computers.
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
RE: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
> From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com] > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate app > in Tomcat 5.5 on Ubuntu 7 > > Hmm, I'm going to try again with more explicit PermGen space as it > seems to be out of it. Try it, but I'd be surprised if it had any effect. The values shown are for the current size of PermGen, not the maximum to which it may expand. That's usually 64m, unless you've changed it. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
Hmm, I'm going to try again with more explicit PermGen space as it seems to be out of it. - Bradley On Thu, Sep 3, 2009 at 3:51 PM, Bradley Wagner < bradley.wag...@hannonhill.com> wrote: > And again in a new place on subsequent attempt. I've attached the logging > this time because the amount of logging between the last Tomcat line and the > GC line was significant! > - Bradley > > > On Thu, Sep 3, 2009 at 1:49 PM, Caldarale, Charles R < > chuck.caldar...@unisys.com> wrote: > >> > From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com] >> > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate app >> > in Tomcat 5.5 on Ubuntu 7 >> > >> > Maybe it can't connect when it's hung. >> >> Quite possible; GC stops all threads, so the JMX agent may well be >> suspended. >> >> You might want to try turning on pretty much all of the GC-related trace >> flags: >> >> -XX:+PrintGC >> -XX:+PrintGCDetails >> -XX:+PrintGCTimeStamps >> -XX:+PrintGCTaskTimeStamps >> -XX:+PrintReferenceGC >> -XX:+TraceGen0Time >> -XX:+TraceGen1Time >> -XX:+PrintTenuringDistribution >> -XX:+PrintHeapAtGC >> -XX:+TraceParallelOldGCTasks >> -XX:+PrintParallelOldGCPhaseTimes >> >> (All on one line, of course.) This will produce a *lot* of output. >> >> - Chuck >> >> >> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY >> MATERIAL and is thus for use only by the intended recipient. If you received >> this in error, please contact the sender and delete the e-mail and its >> attachments from all computers. >> >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >>
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
And again in a new place on subsequent attempt. I've attached the logging
this time because the amount of logging between the last Tomcat line and the
GC line was significant!
- Bradley
On Thu, Sep 3, 2009 at 1:49 PM, Caldarale, Charles R <
chuck.caldar...@unisys.com> wrote:
> > From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com]
> > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate app
> > in Tomcat 5.5 on Ubuntu 7
> >
> > Maybe it can't connect when it's hung.
>
> Quite possible; GC stops all threads, so the JMX agent may well be
> suspended.
>
> You might want to try turning on pretty much all of the GC-related trace
> flags:
>
> -XX:+PrintGC
> -XX:+PrintGCDetails
> -XX:+PrintGCTimeStamps
> -XX:+PrintGCTaskTimeStamps
> -XX:+PrintReferenceGC
> -XX:+TraceGen0Time
> -XX:+TraceGen1Time
> -XX:+PrintTenuringDistribution
> -XX:+PrintHeapAtGC
> -XX:+TraceParallelOldGCTasks
> -XX:+PrintParallelOldGCPhaseTimes
>
> (All on one line, of course.) This will produce a *lot* of output.
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is thus for use only by the intended recipient. If you received
> this in error, please contact the sender and delete the e-mail and its
> attachments from all computers.
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
2009-09-03 15:47:31,314 INFO [TableMetadata] : columns: [xmldata, id, version]
{Heap before GC invocations=16 (full 0):
PSYoungGen total 47168K, used 39534K [0xb087, 0xb415, 0xb415)
eden space 36352K, 100% used [0xb087,0xb2bf,0xb2bf)
from space 10816K, 29% used [0xb2bf,0xb2f0b9a8,0xb368)
to space 10496K, 0% used [0xb371,0xb371,0xb415)
PSOldGentotal 466048K, used 27270K [0x9415, 0xb087, 0xb087)
object space 466048K, 5% used [0x9415,0x95bf1b80,0xb087)
PSPermGen total 28032K, used 27963K [0x9015, 0x91cb, 0x9415)
object space 28032K, 99% used [0x9015,0x91c9eed0,0x91cb)
30.488: [GC30.496: [SoftReference, 0 refs, 0.100 secs]30.496:
[WeakReference, 60 refs, 0.240 secs]30.496: [FinalReference, 0 refs,
0.060 secs]30.496: [PhantomReference, 0 refs, 0.060 secs]30.496: [JNI
Weak Reference, 0.020 secs]
Desired survivor size 10747904 bytes, new threshold 1 (max 15)
[PSYoungGen: 39534K->1648K(47744K)] 66805K->32097K(513792K), 0.0086770 secs]
[Times: user=0.00 sys=0.01, real=0.01 secs]
Heap after GC invocations=16 (full 0):
PSYoungGen total 47744K, used 1648K [0xb087, 0xb415, 0xb415)
eden space 37248K, 0% used [0xb087,0xb087,0xb2cd)
from space 10496K, 15% used [0xb371,0xb38ac140,0xb415)
to space 10496K, 0% used [0xb2cd,0xb2cd,0xb371)
PSOldGentotal 466048K, used 30449K [0x9415, 0xb087, 0xb087)
object space 466048K, 6% used [0x9415,0x95f0c480,0xb087)
PSPermGen total 28032K, used 27963K [0x9015, 0x91cb, 0x9415)
object space 28032K, 99% used [0x9015,0x91c9eed0,0x91cb)
}
VM-Thread 30487334 30495913 30496387
GC-Thread 0 entries: 24
[ old-to-young-roots-task 30487618 30491453 ]
[ serial-old-to-young-roots-task 30491456 30491653 ]
[ scavenge-roots-task 30491654 30491657 ]
[ scavenge-roots-task 30491657 30491677 ]
[ thread-roots-task 30491678 30491698 ]
[ thread-roots-task 30491699 30491704 ]
[ thread-roots-task 30491705 30491712 ]
[ thread-roots-task 30491713 30491732 ]
[ thread-roots-task 30491732 30491741 ]
[ thread-roots-task 30491742 30491748 ]
[ thread-roots-task 30491748 30491750 ]
[ thread-roots-task 30491751 30491752 ]
[ thread-roots-task 30491754 30491755 ]
[ thread-roots-task 30491756 30491757 ]
[ thread-roots-task 30491758 30491764 ]
[ thread-roots-task 30491764 30491770 ]
[ thread-roots-task 30491770 30492076 ]
[ thread-roots-task 30492077 30492079 ]
[ scavenge-roots-task 30492080 30492089 ]
[ scavenge-roots-task 30492089 30492091 ]
[ scavenge-roots-task 30492091 30492094 ]
[ scavenge-roots-task 30492095 30492512 ]
[ scavenge-roots-task 30492512 30492514 ]
[ steal-task 30492515 30495882 ]
GC-Thread 1 entries: 3
[ old-to-young-roots-task 30487602 30494947 ]
[ steal-task 30494952 30495881 ]
[ waitfor-barrier-task 30495882 30495897 ]
{Heap before GC invocations=17 (full 0):
PSYoungGen total 47744K, used 38896K [0xb087, 0xb415, 0xb415)
eden space 37248K, 100% used [0xb087,0xb2cd,0xb2cd)
f
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
Ok, hung in a different place, but hung nonetheless with all of that logging
enabled:
Here is the result:
2009-09-03 15:47:11,328 INFO [SettingsFactory] : Default entity-mode: pojo
2009-09-03 15:47:11,328 INFO [SettingsFactory] : Named query checking :
enabled
2009-09-03 15:47:11,355 INFO [SessionFactoryImpl] : building session
factory
{Heap before GC invocations=10 (full 0):
PSYoungGen total 47808K, used 43957K [0xb087, 0xb415,
0xb415)
eden space 37504K, 100% used [0xb087,0xb2d1,0xb2d1)
from space 10304K, 62% used [0xb2d1,0xb335d6f0,0xb372)
to space 9984K, 0% used [0xb379,0xb379,0xb415)
PSOldGentotal 466048K, used 11257K [0x9415, 0xb087,
0xb087)
object space 466048K, 2% used [0x9415,0x94c4e4f0,0xb087)
PSPermGen total 26240K, used 26158K [0x9015, 0x91af,
0x9415)
object space 26240K, 99% used [0x9015,0x91adb890,0x91af)
10.247: [GC10.277: [SoftReference, 0 refs, 0.100 secs]10.277:
[WeakReference, 210 refs, 0.530 secs]10.277: [FinalReference, 124 refs,
0.580 secs]10.277: [PhantomReference, 0 refs, 0.070 secs]10.277:
[JNI Weak Reference, 0.010 secs]
Desired survivor size 10616832 bytes, new threshold 1 (max 15)
[PSYoungGen: 43957K->2894K(47488K)] 55214K->19098K(513536K), 0.0304750
secs] [Times: user=0.01 sys=0.05, real=0.03 secs]
Heap after GC invocations=10 (full 0):
PSYoungGen total 47488K, used 2894K [0xb087, 0xb415,
0xb415)
eden space 37504K, 0% used [0xb087,0xb087,0xb2d1)
from space 9984K, 28% used [0xb379,0xb3a639d0,0xb415)
to space 10368K, 0% used [0xb2d1,0xb2d1,0xb373)
PSOldGentotal 466048K, used 16203K [0x9415, 0xb087,
0xb087)
object space 466048K, 3% used [0x9415,0x95122ed8,0xb087)
PSPermGen total 26240K, used 26158K [0x9015, 0x91af,
0x9415)
object space 26240K, 99% used [0x9015,0x91adb890,0x91af)
}
VM-Thread 10246673 10276996 10277531
GC-Thread 0 entries: 17
[ old-to-young-roots-task 10246926 10272563 ]
[ scavenge-roots-task 10272571 10272574 ]
[ scavenge-roots-task 10272574 10272590 ]
[ thread-roots-task 10272591 10272612 ]
[ thread-roots-task 10272613 10272618 ]
[ thread-roots-task 10272623 10272631 ]
[ thread-roots-task 10272631 10272654 ]
[ thread-roots-task 10272655 10272660 ]
[ thread-roots-task 10272660 10272667 ]
[ thread-roots-task 10272667 10272669 ]
[ thread-roots-task 10272669 10272672 ]
[ thread-roots-task 10272672 10272677 ]
[ thread-roots-task 10272678 10272679 ]
[ thread-roots-task 10272680 10272686 ]
[ thread-roots-task 10272687 10272690 ]
[ thread-roots-task 10272691 10276295 ]
[ steal-task 10276297 10276969 ]
GC-Thread 1 entries: 10
[ old-to-young-roots-task 10246939 10269250 ]
[ serial-old-to-young-roots-task 10269253 10274454 ]
[ thread-roots-task 10274457 10274461 ]
[ scavenge-roots-task 10274462 10274473 ]
[ scavenge-roots-task 10274474 10274478 ]
[ scavenge-roots-task 10274478 10274483 ]
[ scavenge-roots-task 10274483 10274919 ]
[ scavenge-roots-task 10274920 10274921 ]
[ steal-task 10274922 10276966 ]
[ waitfor-barrier-task 10276967 10276991 ]
{Heap before GC invocations=11 (full 0):
PSYoungGen total 47488K, used 40398K [0xb087, 0xb415,
0xb415)
eden space 37504K, 100% used [0xb087,0xb2d1,0xb2d1)
from space 9984K, 28% used [0xb379,0xb3a639d0,0xb415)
to space 10368K, 0% used [0xb2d1,0xb2d1,0xb373)
PSOldGentotal 466048K, used 16203K [0x9415, 0xb087,
0xb087)
object space 466048K, 3% used [0x9415,0x95122ed8,0xb087)
PSPermGen total 26752K, used 26626K [0x9015, 0x91b7,
0x9415)
object space 26752K, 99% used [0x9015,0x91b50878,0x91b7)
10.754: [GC
Once again, it terminates in the middle of GC logging.
- Bradley
On Thu, Sep 3, 2009 at 1:49 PM, Caldarale, Charles R <
chuck.caldar...@unisys.com> wrote:
> > From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com]
> > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate app
> > in Tomcat 5.5 on Ubuntu 7
> >
> > Maybe it can't connect when it's hung.
>
> Quite possible; GC stops all threads, so the JMX agent may well be
> suspended.
>
> You might want to try turning on pretty much all of the GC-related trace
> flags:
>
> -XX:+PrintGC
> -XX:+PrintGCDetails
> -XX:+PrintGCTimeStamps
> -XX:+PrintGCTaskTimeStamps
> -XX:+PrintReferenceGC
> -XX:+TraceGen0Time
> -XX:+TraceGen1Time
> -XX:+PrintTenuringDistribution
> -XX:+PrintHeapAtGC
> -XX:+TraceParallelOldGCTasks
> -XX:+PrintParallelOldGCPhaseTimes
>
> (All on one line, of course.) This will produce a *lot* of output.
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is thus for use only by the intended re
RE: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
> From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com] > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate app > in Tomcat 5.5 on Ubuntu 7 > > Maybe it can't connect when it's hung. Quite possible; GC stops all threads, so the JMX agent may well be suspended. You might want to try turning on pretty much all of the GC-related trace flags: -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCTaskTimeStamps -XX:+PrintReferenceGC -XX:+TraceGen0Time -XX:+TraceGen1Time -XX:+PrintTenuringDistribution -XX:+PrintHeapAtGC -XX:+TraceParallelOldGCTasks -XX:+PrintParallelOldGCPhaseTimes (All on one line, of course.) This will produce a *lot* of output. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
Ok, I've tried to get JConsole working but have been unsuccessful. I've set it up to connect remotely using JMX over port 8081. The program is starting and I've configured the JMX passwords. However, when I try to connect to the hung program using username: controlRole and my password: nothing pops up. I see the following catalina.out: 2009-09-03 12:18:04,295 INFO [XmlWebApplicationContext] : Bean factory for application context [org.springframework.web.context.support.xmlwebapplicationcont...@1f1680f]: org.springframework.beans.factory.support.defaultlistablebeanfact...@45378f [GC [PSYoungGen: 50993K->7222K(51008K)] 51106K->8565K(517056K), 0.0402520 secs] [Times: user=0.03 sys=0.01, real=0.04 secs] [GC However, it is is listing my application name in JConsole's list of windows. Maybe it can't connect when it's hung. - Bradley On Thu, Sep 3, 2009 at 11:17 AM, Caldarale, Charles R < chuck.caldar...@unisys.com> wrote: > > From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com] > > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate app > > in Tomcat 5.5 on Ubuntu 7 > > > > Hmm, so JConsole appears to need a GUI which I don't have in this case. > > You can run it remotely by setting the appropriate command-line options. > > > Should I just start switching through the different GC types and see if > > it makes a difference? > > If you can't get more detailed information about what's going on inside the > heap, then yes. > > - Chuck > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is thus for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all computers. > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
RE: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
> From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com] > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate app > in Tomcat 5.5 on Ubuntu 7 > > Hmm, so JConsole appears to need a GUI which I don't have in this case. You can run it remotely by setting the appropriate command-line options. > Should I just start switching through the different GC types and see if > it makes a difference? If you can't get more detailed information about what's going on inside the heap, then yes. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
Hmm, so JConsole: http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html appears to need a GUI which I don't have in this case. Should I just start switching through the different GC types and see if it makes a difference? On Thu, Sep 3, 2009 at 11:02 AM, Caldarale, Charles R < chuck.caldar...@unisys.com> wrote: > > From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com] > > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate app > > in Tomcat 5.5 on Ubuntu 7 > > > > Probably unrelated, but... when setting JAVA_HOME for Tomcat in Ubuntu > > what's the difference between: > > - /usr/lib/jvm/java-6-sun, and > > - /usr/lib/jvm/java-6-sun/jre > > Every Sun JDK has a JRE inside it; the first directory above is for the > full JDK, the second for just the JRE subset, as you surmised. JConsole > should be located in /usr/lib/jvm/java-6-sun/bin. > > - Chuck > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is thus for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all computers. > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
RE: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
> From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com] > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate app > in Tomcat 5.5 on Ubuntu 7 > > Probably unrelated, but... when setting JAVA_HOME for Tomcat in Ubuntu > what's the difference between: > - /usr/lib/jvm/java-6-sun, and > - /usr/lib/jvm/java-6-sun/jre Every Sun JDK has a JRE inside it; the first directory above is for the full JDK, the second for just the JRE subset, as you surmised. JConsole should be located in /usr/lib/jvm/java-6-sun/bin. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
> From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com] > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate app > in Tomcat 5.5 on Ubuntu 7 > > Is JConsole something that comes bundled with the JRE or JDK or a > separate application. It's part of the JDK, but not the JRE. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
Bradley Wagner wrote: > Probably unrelated, but... when setting JAVA_HOME for Tomcat in Ubuntu > what's the difference between: > - /usr/lib/jvm/java-6-sun, and > - /usr/lib/jvm/java-6-sun/jre > > It doesn't seem to have any effect on my problem, but does it make a > difference? I believe I have the full JDK installed so maybe the second path > only points to the binaries for the JRE? JVM hangs with the garbage collector are a fairly regular occurrence. Have you tried changing the GC settings yet? Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
Probably unrelated, but... when setting JAVA_HOME for Tomcat in Ubuntu what's the difference between: - /usr/lib/jvm/java-6-sun, and - /usr/lib/jvm/java-6-sun/jre It doesn't seem to have any effect on my problem, but does it make a difference? I believe I have the full JDK installed so maybe the second path only points to the binaries for the JRE? - Bradley On Thu, Sep 3, 2009 at 10:31 AM, Ziggy wrote: > Ok my mistake. Thats the official SUN version. > > The linux distribution i use downloads the openjdk version and i had to > manually download the official one to stop the memory errors. > > > > On Thu, Sep 3, 2009 at 3:29 PM, ramzi khlil > wrote: > > > It provides you the version of installed java. > > > > On Thu, Sep 3, 2009 at 10:19 AM, Ziggy wrote: > > > > > What do you get when you run java -version? > > > > > > I think the linux distributions include the open jdk version see > > > http://openjdk.java.net/ > > > > > > > > > > > > > > > On Thu, Sep 3, 2009 at 3:13 PM, Caldarale, Charles R < > > > chuck.caldar...@unisys.com> wrote: > > > > > > > > From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com] > > > > > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate > > app > > > > > in Tomcat 5.5 on Ubuntu 7 > > > > > > > > > > are you saying that the JRE I'm getting when doing "apt-get > > > > > install sun-java6-jdk" is not actually a Sun certified version > > > > > and that I should be downloading it directly from their website > > > > > instead? > > > > > > > > It's certainly something to try. You can find out exactly what you > > have > > > > installed by doing "java -version" (without the quotes). > > > > > > > > - Chuck > > > > > > > > > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE > > PROPRIETARY > > > > MATERIAL and is thus for use only by the intended recipient. If you > > > received > > > > this in error, please contact the sender and delete the e-mail and > its > > > > attachments from all computers. > > > > > > > > > > > > - > > > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > > > > > > > > > > >
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
Is JConsole something that comes bundled with the JRE or JDK or a separate application. I have limited ability to install other things on this machine because it belongs to a client. Thanks, Bradley On Thu, Sep 3, 2009 at 10:32 AM, Caldarale, Charles R < chuck.caldar...@unisys.com> wrote: > > From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com] > > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate app > > in Tomcat 5.5 on Ubuntu 7 > > > > java version "1.6.0_14" > > Java(TM) SE Runtime Environment (build 1.6.0_14-b08) > > Java HotSpot(TM) Server VM (build 14.0-b16, mixed mode) > > That's a real one. This is getting strange. Can you run JConsole against > the Tomcat JVM and see what the state of the various heap parts are when it > gets into the hung state? > > - Chuck > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is thus for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all computers. > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
RE: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
> From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com] > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate app > in Tomcat 5.5 on Ubuntu 7 > > java version "1.6.0_14" > Java(TM) SE Runtime Environment (build 1.6.0_14-b08) > Java HotSpot(TM) Server VM (build 14.0-b16, mixed mode) That's a real one. This is getting strange. Can you run JConsole against the Tomcat JVM and see what the state of the various heap parts are when it gets into the hung state? - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
Ok my mistake. Thats the official SUN version. The linux distribution i use downloads the openjdk version and i had to manually download the official one to stop the memory errors. On Thu, Sep 3, 2009 at 3:29 PM, ramzi khlil wrote: > It provides you the version of installed java. > > On Thu, Sep 3, 2009 at 10:19 AM, Ziggy wrote: > > > What do you get when you run java -version? > > > > I think the linux distributions include the open jdk version see > > http://openjdk.java.net/ > > > > > > > > > > On Thu, Sep 3, 2009 at 3:13 PM, Caldarale, Charles R < > > chuck.caldar...@unisys.com> wrote: > > > > > > From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com] > > > > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate > app > > > > in Tomcat 5.5 on Ubuntu 7 > > > > > > > > are you saying that the JRE I'm getting when doing "apt-get > > > > install sun-java6-jdk" is not actually a Sun certified version > > > > and that I should be downloading it directly from their website > > > > instead? > > > > > > It's certainly something to try. You can find out exactly what you > have > > > installed by doing "java -version" (without the quotes). > > > > > > - Chuck > > > > > > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE > PROPRIETARY > > > MATERIAL and is thus for use only by the intended recipient. If you > > received > > > this in error, please contact the sender and delete the e-mail and its > > > attachments from all computers. > > > > > > > > > - > > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > > > > > >
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
It provides you the version of installed java. On Thu, Sep 3, 2009 at 10:19 AM, Ziggy wrote: > What do you get when you run java -version? > > I think the linux distributions include the open jdk version see > http://openjdk.java.net/ > > > > > On Thu, Sep 3, 2009 at 3:13 PM, Caldarale, Charles R < > chuck.caldar...@unisys.com> wrote: > > > > From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com] > > > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate app > > > in Tomcat 5.5 on Ubuntu 7 > > > > > > are you saying that the JRE I'm getting when doing "apt-get > > > install sun-java6-jdk" is not actually a Sun certified version > > > and that I should be downloading it directly from their website > > > instead? > > > > It's certainly something to try. You can find out exactly what you have > > installed by doing "java -version" (without the quotes). > > > > - Chuck > > > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > > MATERIAL and is thus for use only by the intended recipient. If you > received > > this in error, please contact the sender and delete the e-mail and its > > attachments from all computers. > > > > > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > >
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
Yea, I don't think it's the Open JDK, though that was an option. "java -version" gives: java version "1.6.0_14" Java(TM) SE Runtime Environment (build 1.6.0_14-b08) Java HotSpot(TM) Server VM (build 14.0-b16, mixed mode) - Bradley On Thu, Sep 3, 2009 at 10:19 AM, Ziggy wrote: > What do you get when you run java -version? > > I think the linux distributions include the open jdk version see > http://openjdk.java.net/ > > > > > On Thu, Sep 3, 2009 at 3:13 PM, Caldarale, Charles R < > chuck.caldar...@unisys.com> wrote: > > > > From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com] > > > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate app > > > in Tomcat 5.5 on Ubuntu 7 > > > > > > are you saying that the JRE I'm getting when doing "apt-get > > > install sun-java6-jdk" is not actually a Sun certified version > > > and that I should be downloading it directly from their website > > > instead? > > > > It's certainly something to try. You can find out exactly what you have > > installed by doing "java -version" (without the quotes). > > > > - Chuck > > > > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > > MATERIAL and is thus for use only by the intended recipient. If you > received > > this in error, please contact the sender and delete the e-mail and its > > attachments from all computers. > > > > > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > >
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
What do you get when you run java -version? I think the linux distributions include the open jdk version see http://openjdk.java.net/ On Thu, Sep 3, 2009 at 3:13 PM, Caldarale, Charles R < chuck.caldar...@unisys.com> wrote: > > From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com] > > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate app > > in Tomcat 5.5 on Ubuntu 7 > > > > are you saying that the JRE I'm getting when doing "apt-get > > install sun-java6-jdk" is not actually a Sun certified version > > and that I should be downloading it directly from their website > > instead? > > It's certainly something to try. You can find out exactly what you have > installed by doing "java -version" (without the quotes). > > - Chuck > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is thus for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all computers. > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
RE: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
> From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com] > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate app > in Tomcat 5.5 on Ubuntu 7 > > are you saying that the JRE I'm getting when doing "apt-get > install sun-java6-jdk" is not actually a Sun certified version > and that I should be downloading it directly from their website > instead? It's certainly something to try. You can find out exactly what you have installed by doing "java -version" (without the quotes). - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
Sorry, are you saying that the JRE I'm getting when doing "apt-get install sun-java6-jdk" is not actually a Sun certified version and that I should be downloading it directly from their website instead? On Thu, Sep 3, 2009 at 3:50 AM, Ziggy wrote: > I noticed that you mention that you are using the community version of the > Java runtime environment which i think is what you get when you do an > apt-get. > > I have never honestly been able to use this version of the JVM > successfully. > I had a similar problem recently where i was running an application (That > uses Hibernate) using this version of the JDK and was always getting memory > exceptions. > > What i did was to download sun's version of the JDK and that seem to have > solved the problem. Did you try to use the SUN's version of the JDK from > the > SUN website? It might be usefull to try the SUN's version just to rule out > that the JDK version/release is not the problem. > > > > On Wed, Sep 2, 2009 at 11:26 PM, Mark Thomas wrote: > > > Bradley Wagner wrote: > > > Any thoughts on which GC parameters I should change. > > > > See what the default collector is for your platform and switch to s > > different one. > > > > Mark > > > > > > > > > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > >
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
I noticed that you mention that you are using the community version of the Java runtime environment which i think is what you get when you do an apt-get. I have never honestly been able to use this version of the JVM successfully. I had a similar problem recently where i was running an application (That uses Hibernate) using this version of the JDK and was always getting memory exceptions. What i did was to download sun's version of the JDK and that seem to have solved the problem. Did you try to use the SUN's version of the JDK from the SUN website? It might be usefull to try the SUN's version just to rule out that the JDK version/release is not the problem. On Wed, Sep 2, 2009 at 11:26 PM, Mark Thomas wrote: > Bradley Wagner wrote: > > Any thoughts on which GC parameters I should change. > > See what the default collector is for your platform and switch to s > different one. > > Mark > > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
Bradley Wagner wrote: > Any thoughts on which GC parameters I should change. See what the default collector is for your platform and switch to s different one. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
lumns: [id, firsttriggerid, resultstepid, name, isautomatic, stepid, actiontype, displayname] 2009-09-02 13:58:05,383 INFO [TableMetadata] : table found: hhtest.cxml_workflowdefcontainer 2009-09-02 13:58:05,383 INFO [TableMetadata] : columns: [nextversionid, prevversionid, iscurrentversion, permissionsid, id, isroot, metadataid, versiondate, name, containerid, path, siteid, lockid] 2009-09-02 13:58:05,387 INFO [TableMetadata] : table found: hhtest.cxml_workflowdefinition 2009-09-02 13:58:05,387 INFO [TableMetadata] : columns: [namingvalue, workflowtype, nextversionid, prevversionid, iscurrentversion, permissionsid, id, metadataid, versiondate, name, containerid, path, siteid, xmlid, groups, lockid] 2009-09-02 13:58:05,390 INFO [TableMetadata] : table found: hhtest.cxml_workflowhistory 2009-09-02 13:58:05,390 INFO [TableMetadata] : columns: [id, sourcestepname, prevhistoryid, nexthistoryid, name, deststepname, tstamp, actionname, workflowid, comments, who] 2009-09-02 13:58:05,396 INFO [TableMetadata] : table found: hhtest.cxml_workflowstep 2009-09-02 13:58:05,397 INFO [TableMetadata] : columns: [unorderedstepworkflowid, escalatestepidentifier, prevstepid, startedon, escalateby, workflowid, expirationwarningsent, id, nextstepid, isordered, escalationstepid, maxdurationinseconds, name, finishby, owner, escalationdurationinseconds, ownertype, steptype, displayname] 2009-09-02 13:58:05,399 INFO [TableMetadata] : table found: hhtest.cxml_workflowtrigger 2009-09-02 13:58:05,399 INFO [TableMetadata] : columns: [id, classname, name, nexttriggerid, prevtriggerid] 2009-09-02 13:58:05,401 INFO [TableMetadata] : table found: hhtest.cxml_workflowtriggerparameter 2009-09-02 13:58:05,402 INFO [TableMetadata] : columns: [id, triggerid, name, value] 2009-09-02 13:58:05,403 INFO [TableMetadata] : table found: hhtest.cxml_xml 2009-09-02 13:58:05,404 INFO [TableMetadata] : columns: [xmldata, id, version] [GC [PSYoungGen: 39479K->1589K(47872K)] 66887K->31890K(513920K), 0.0089050 secs] [Times: user=0.01 sys=0.01, real=0.01 secs] [GC [PSYoungGen: 39029K->1094K(47808K)] 69330K->32887K(513856K), 0.0076330 secs] [Times: user=0.01 sys=0.01, real=0.01 secs] [GC [PSYoungGen: 38534K->208K(47744K)] 70327K->32956K(513792K), 0.0051090 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] [GC [PSYoungGen: 37584K->464K(47808K)] 70332K->4K(513856K), 0.0023630 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] [GC [PSYoungGen: 37840K->208K(48128K)] 70710K->33366K(514176K), 0.0029340 secs] [Times: user=0.00 sys=0.01, real=0.01 secs] [GC [PSYoungGen: 38032K->224K(47936K)] 71190K->33578K(513984K), 0.0023790 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] [GC [PSYoungGen: 38048K->320K(48640K)] 71402K->33754K(514688K), 0.0024950 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] [GC [PSYoungGen: 39040K->264K(48320K)] 72474K->33934K(514368K), 0.0024660 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] [GC [PSYoungGen: 38984K->408K(49344K)] 72654K->34318K(515392K), 0.0025370 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] [GC [PSYoungGen: 40472K->560K(48960K)] 74382K->34706K(515008K), 0.0031430 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] [GC [PSYoungGen: 40624K->832K(50176K)] 74770K->35502K(516224K), 0.0061740 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] [GC [PSYoungGen: 42496K->976K(49728K)] 77166K->36432K(515776K), 0.0065300 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] [GC [PSYoungGen: 42640K->1137K(51136K)] 78096K->37540K(517184K), 0.0136230 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] [GC [PSYoungGen: 44721K->1097K(50688K)] 81124K->38556K(516736K), 0.0084640 secs] [Times: user=0.01 sys=0.01, real=0.01 secs] [GC On Wed, Sep 2, 2009 at 4:27 PM, Caldarale, Charles R < chuck.caldar...@unisys.com> wrote: > > From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com] > > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate app > > in Tomcat 5.5 on Ubuntu 7 > > > > I've also attached the catalina.out log which shows the GC > > basically freezing. > > Unfortunately, that log was stripped; can you post it somewhere? > > - Chuck > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is thus for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all computers. > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > -- Hannon Hill - CMS Experience You Can Trust (678) 904-6900 ext 115 http://www.hannonhill.com
RE: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
> From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com] > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate app > in Tomcat 5.5 on Ubuntu 7 > > I've also attached the catalina.out log which shows the GC > basically freezing. Unfortunately, that log was stripped; can you post it somewhere? - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
Thanks for the clarification on the initial heap size. I've set both Xms and Xmx to 512m for the following test. I was able to reproduce the problem immediately with: - Xms and Xmx both set to 512m - Java 1.6 with no JPDA running - XX:+PrintGCDetails enabled I've attached the thread dump which I took on the 3rd or 4th attempt after the process hung. They were all the same on each measure. I've also attached the catalina.out log which shows the GC basically freezing. One thing that it is a little odd is that the Catalina log is basically in the middle of (or possibly JUST finished) some kind of hibernate initialization but the blocked thread (the only thread doing anything but sleeping) doesn't really indicate this. - Bradley On Mon, Aug 31, 2009 at 6:19 PM, Leon Rosenberg < rosenberg.l...@googlemail.com> wrote: > Hello, > This indeed sounds like you have problems with one of the spaces, > please add following option to the JAVA_OPTS (or whatever you are > using) to seubmit jvm parameters: > -XX:+PrintGCDetails > > And, no tomcat's initial heap size don't default to 64Mb. The JVM > allocateds a heap dependent on your computer type, for example the 1.5 > VM would identify a 2 core 2 GB machine as 'server' and allocated more > memory than with 1 core cpu. > > http://72.5.124.55/docs/hotspot/gc5.0/ergo5.html > > regards > Leon > > P.S. I know the link looks strange it come ups from google: > > http://www.google.de/search?q=hotspot+memory+ergonomics&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a > > P.P.S I had a similar situation once, as a client had set space sizes > manually (new and old) and the old space was full. The VM was only > full garbage collecting which led to 100% CPU usage and strange thread > dumps. The above option would help you in this case to identify the > full gc runs previously and during the hangup. > > > > On Mon, Aug 31, 2009 at 11:34 PM, Bradley > Wagner wrote: > >> > >> The other interesting point is that all of those are hung in places > where > > > > they would naturally be allocating an object or have very recently > allocated > > > > an object, and the JVM *might* in theory be growing the heap. Bradley, > are > > > > your initial and maximum heap sizes identical (I suspect not from your > > > > original message)? If not, what happens if you make them so? > > > > > > From my original post, I'm setting the following memory parameters: > > > > -Xmx512M -XX:MaxPermSize=128m > > > > so I'm not setting an initial heap size at all. If I remember correctly, > > Tomcat's initial heap size default's to 64MB or something. > > > > I'll try to set the initial and max to the same to see if that has any > > effect. > > > > Thanks, > > Bradley > > > > On Mon, Aug 31, 2009 at 4:59 PM, Peter Crowther < > peter.crowt...@melandra.com > >> wrote: > > > >> 2009/8/31 Christopher Schultz > >> > >> > -BEGIN PGP SIGNED MESSAGE- > >> > Hash: SHA1 > >> > > >> > Bradley, > >> > > >> > On 8/31/2009 4:02 PM, Bradley Wagner wrote: > >> > > Sorry, I should have mentioned this before, but in all cases I tried > >> the > >> > > thread dumps 2-3 times at least 30s apart and none of the threads > have > >> > > progressed at all. > >> > > >> > I agree with Mark: the three threads you showed were different > >> > (different ids), and showing different call stacks. Are you saying > that > >> > each thread dump you took represents multiple thread dumps where that > >> > particular thread didn't progress /at all/ over your multi-dump > sample? > >> > > >> > You might even want to run strace to see which library function calls > >> > are actually completing. > >> > > >> > >> Agree. > >> > >> The other interesting point is that all of those are hung in places > where > >> they would naturally be allocating an object or have very recently > >> allocated > >> an object, and the JVM *might* in theory be growing the heap. Bradley, > are > >> your initial and maximum heap sizes identical (I suspect not from your > >> original message)? If not, what happens if you make them so? > >> > >> - Peter > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > Attaching to process ID 26202, please wait... Debugger attached successfully. Server compiler detected. JVM version is 14.0-b16 Deadlock Detection: No deadlocks found. Thread 26216: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) - java.util.TimerThread.mainLoop() @bci=28, line=483 (Interpreted frame) - java.util.TimerThread.run() @bci=1, line=462 (Interpreted frame) Thread 26215: (state = BLOCKED) - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame) - com.hannonhill.cascade.model.search.MemoryQueueSearchJobScheduler$ConsumerThread.run() @bci=17, line=56 (Interpreted frame) Thread 26
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Peter, On 9/1/2009 11:18 AM, Peter Crowther wrote: > 2009/9/1 Christopher Schultz > >> Hmm. Do you trust your hardware? >> [...] > > How much physical memory does your machine have? How much swap space? >> Try running badblocks on your swap partition. Try running memtest86+ on >> your hardware (if it's x86... you didn't say). >> > > The OP stated this was in a virtual machine. Some, but not all, of this > advice is appropriate in such an environment :-). Whoops! In that case, run vmemtest86+ ha ha ha :) Seriously, I would check the VM status. There is usually some way to observe which soft/hard limits the VM has exceeded. For instance, in Virtuozzo-based VMs, you can check /proc/user_beancounters. Other techniques must exist for other virtualization technologies (except maybe host/guest types like VMWare player, workstation, etc., since they work differently). - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkqdPx8ACgkQ9CaO5/Lv0PB4MwCdEUmO88r4qZCXCmim1aKwZReV HcAAn1P4kXpBIlSt1jjmgXbjayRGyB2b =vetW -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
2009/9/1 Christopher Schultz > Hmm. Do you trust your hardware? > [...] How much physical memory does your machine have? How much swap space? > Try running badblocks on your swap partition. Try running memtest86+ on > your hardware (if it's x86... you didn't say). > The OP stated this was in a virtual machine. Some, but not all, of this advice is appropriate in such an environment :-). - Peter
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Bradley, On 8/31/2009 5:27 PM, Bradley Wagner wrote: > The three thread dumps were from three different occurrences of the same > problem and three consecutive startup attempts. > > With each occurrence, the startup process hangs in the middle of > initialization. I took 2-3 jstack dumps that were identical each time. After > that, I killed the process (kill -9 ) and started it again. > > So, while it's not hanging in the same place every time, it's definitely > hanging every time. And because it's the main initialization thread that is > hanging, once it hangs, it never gets any further. Hmm. Do you trust your hardware? If you've tried several JVM versions and client vs. server (which are essentially different JVMs) and they are all having problems, then the JVM itself is probably not to blame (although using JPDA gets you off the hook? weird: try running in interpreted mode... run with -Xint just for giggles). How much physical memory does your machine have? How much swap space? Try running badblocks on your swap partition. Try running memtest86+ on your hardware (if it's x86... you didn't say). Are you running the latest kernel version available? We use 2.6.30-gentoo-r4 on one of our development boxes, 2.6.26-2-openvz-686 (from Debian) on another development box (it's an openvz VM), 2.6.18-6-686 (Debian, again) in production, and 2.4.27-2-686-smp (also Debian... 2.4? Yikes?!) in demo. All of these seem to have no problem running our applications for months and months. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkqdNiQACgkQ9CaO5/Lv0PDOMgCgsxCDcoOnQA8eKfVDumfER1I7 eQkAn1JrCr0k7mWJyTusbcnU6Tf4Raj2 =35B0 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
> From: Bradley Wagner [mailto:bradley.wag...@hannonhill.com] > Subject: Re: 100% usage and hanging on startup of Spring/Hibernate app > in Tomcat 5.5 on Ubuntu 7 > > So, while it's not hanging in the same place every time, it's > definitely hanging every time. Try running JConsole against the Tomcat process; this should give you a good idea about what's going on inside the JVM, including heap usage and GC activity. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
Hello, This indeed sounds like you have problems with one of the spaces, please add following option to the JAVA_OPTS (or whatever you are using) to seubmit jvm parameters: -XX:+PrintGCDetails And, no tomcat's initial heap size don't default to 64Mb. The JVM allocateds a heap dependent on your computer type, for example the 1.5 VM would identify a 2 core 2 GB machine as 'server' and allocated more memory than with 1 core cpu. http://72.5.124.55/docs/hotspot/gc5.0/ergo5.html regards Leon P.S. I know the link looks strange it come ups from google: http://www.google.de/search?q=hotspot+memory+ergonomics&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a P.P.S I had a similar situation once, as a client had set space sizes manually (new and old) and the old space was full. The VM was only full garbage collecting which led to 100% CPU usage and strange thread dumps. The above option would help you in this case to identify the full gc runs previously and during the hangup. On Mon, Aug 31, 2009 at 11:34 PM, Bradley Wagner wrote: >> >> The other interesting point is that all of those are hung in places where > > they would naturally be allocating an object or have very recently allocated > > an object, and the JVM *might* in theory be growing the heap. Bradley, are > > your initial and maximum heap sizes identical (I suspect not from your > > original message)? If not, what happens if you make them so? > > > From my original post, I'm setting the following memory parameters: > > -Xmx512M -XX:MaxPermSize=128m > > so I'm not setting an initial heap size at all. If I remember correctly, > Tomcat's initial heap size default's to 64MB or something. > > I'll try to set the initial and max to the same to see if that has any > effect. > > Thanks, > Bradley > > On Mon, Aug 31, 2009 at 4:59 PM, Peter Crowther > wrote: > >> 2009/8/31 Christopher Schultz >> >> > -BEGIN PGP SIGNED MESSAGE- >> > Hash: SHA1 >> > >> > Bradley, >> > >> > On 8/31/2009 4:02 PM, Bradley Wagner wrote: >> > > Sorry, I should have mentioned this before, but in all cases I tried >> the >> > > thread dumps 2-3 times at least 30s apart and none of the threads have >> > > progressed at all. >> > >> > I agree with Mark: the three threads you showed were different >> > (different ids), and showing different call stacks. Are you saying that >> > each thread dump you took represents multiple thread dumps where that >> > particular thread didn't progress /at all/ over your multi-dump sample? >> > >> > You might even want to run strace to see which library function calls >> > are actually completing. >> > >> >> Agree. >> >> The other interesting point is that all of those are hung in places where >> they would naturally be allocating an object or have very recently >> allocated >> an object, and the JVM *might* in theory be growing the heap. Bradley, are >> your initial and maximum heap sizes identical (I suspect not from your >> original message)? If not, what happens if you make them so? >> >> - Peter > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
> > The other interesting point is that all of those are hung in places where they would naturally be allocating an object or have very recently allocated an object, and the JVM *might* in theory be growing the heap. Bradley, are your initial and maximum heap sizes identical (I suspect not from your original message)? If not, what happens if you make them so? >From my original post, I'm setting the following memory parameters: -Xmx512M -XX:MaxPermSize=128m so I'm not setting an initial heap size at all. If I remember correctly, Tomcat's initial heap size default's to 64MB or something. I'll try to set the initial and max to the same to see if that has any effect. Thanks, Bradley On Mon, Aug 31, 2009 at 4:59 PM, Peter Crowther wrote: > 2009/8/31 Christopher Schultz > > > -BEGIN PGP SIGNED MESSAGE- > > Hash: SHA1 > > > > Bradley, > > > > On 8/31/2009 4:02 PM, Bradley Wagner wrote: > > > Sorry, I should have mentioned this before, but in all cases I tried > the > > > thread dumps 2-3 times at least 30s apart and none of the threads have > > > progressed at all. > > > > I agree with Mark: the three threads you showed were different > > (different ids), and showing different call stacks. Are you saying that > > each thread dump you took represents multiple thread dumps where that > > particular thread didn't progress /at all/ over your multi-dump sample? > > > > You might even want to run strace to see which library function calls > > are actually completing. > > > > Agree. > > The other interesting point is that all of those are hung in places where > they would naturally be allocating an object or have very recently > allocated > an object, and the JVM *might* in theory be growing the heap. Bradley, are > your initial and maximum heap sizes identical (I suspect not from your > original message)? If not, what happens if you make them so? > > - Peter
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
> > I agree with Mark: the three threads you showed were different (different ids), and showing different call stacks. Are you saying that each thread dump you took represents multiple thread dumps where that particular thread didn't progress /at all/ over your multi-dump sample? The three thread dumps were from three different occurrences of the same problem and three consecutive startup attempts. With each occurrence, the startup process hangs in the middle of initialization. I took 2-3 jstack dumps that were identical each time. After that, I killed the process (kill -9 ) and started it again. So, while it's not hanging in the same place every time, it's definitely hanging every time. And because it's the main initialization thread that is hanging, once it hangs, it never gets any further. Sorry for the confusion and thanks for the help! On Mon, Aug 31, 2009 at 4:23 PM, Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Bradley, > > On 8/31/2009 4:02 PM, Bradley Wagner wrote: > > Sorry, I should have mentioned this before, but in all cases I tried the > > thread dumps 2-3 times at least 30s apart and none of the threads have > > progressed at all. > > I agree with Mark: the three threads you showed were different > (different ids), and showing different call stacks. Are you saying that > each thread dump you took represents multiple thread dumps where that > particular thread didn't progress /at all/ over your multi-dump sample? > > You might even want to run strace to see which library function calls > are actually completing. > > - -chris > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.9 (MingW32) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAkqcMT0ACgkQ9CaO5/Lv0PBokQCeOJzGRtwS5a9tx6N3KRCnP/L/ > wp4AoLQFItwFOAGWygNNCMizlArc3n66 > =jz8W > -END PGP SIGNATURE- > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
> > - use a 1.6 JVM Tried with Sun 1.6 JVM. Same problems. Not the same exact stack traces but for example: Thread 24427: (state = BLOCKED) - java.lang.String.toCharArray() @bci=4, line=2725 (Compiled frame) - java.beans.Introspector.decapitalize(java.lang.String) @bci=46, line=252 (Compiled frame) - org.hibernate.property.BasicPropertyAccessor.getterMethod(java.lang.Class, java.lang.String) @bci=47, line=325 (Compiled frame) - org.hibernate.property.BasicPropertyAccessor.getGetterOrNull(java.lang.Class, java.lang.String) @bci=33, line=297 (Interpreted frame) - org.hibernate.property.BasicPropertyAccessor.getGetterOrNull(java.lang.Class, java.lang.String) @bci=71, line=304 (Interpreted frame) - org.hibernate.property.BasicPropertyAccessor.setterMethod(java.lang.Class, java.lang.String) @bci=2, line=251 (Interpreted frame) - org.hibernate.property.BasicPropertyAccessor.getSetterOrNull(java.lang.Class, java.lang.String) @bci=33, line=230 (Interpreted frame) - org.hibernate.property.BasicPropertyAccessor.createSetter(java.lang.Class, java.lang.String) @bci=2, line=214 (Interpreted frame) - org.hibernate.property.BasicPropertyAccessor.getSetter(java.lang.Class, java.lang.String) @bci=2, line=209 (Interpreted frame) - org.hibernate.mapping.Property.getSetter(java.lang.Class) @bci=10, line=277 (Interpreted frame) - org.hibernate.tuple.entity.PojoEntityTuplizer.buildPropertySetter(org.hibernate.mapping.Property, - try different JVM settings, particularly GC related ones Any thoughts on which GC parameters I should change. > - switch to/from client from/to server JVM Have tried with "-server" flag. Same problems. I agree with you on JVM problem especially given that all the blocked threads are blocked in calls to the String object. Problem is I'm not doing anything exotic with the JVM. We simply did an "apt-get" to get the latest Sun JVM for Ubuntu. I've been searching their forums as well for any mention of incompatibilities with Sun's JVM. Thanks for your help so far. - Bradley On Mon, Aug 31, 2009 at 4:22 PM, Mark Thomas wrote: > > Bradley Wagner wrote: > > Mark, > > Sorry, I should have mentioned this before, but in all cases I tried the > > thread dumps 2-3 times at least 30s apart and none of the threads have > > progressed at all. > > Hmm. That smells like a JVM issue to me, particularly if it goes away > when you use JPDA. > > Things to try: > - use a 1.6 JVM > - try different JVM settings, particularly GC related ones > - switch to/from client from/to server JVM > > Mark > > > > > > - Bradley > > > > On Mon, Aug 31, 2009 at 3:52 PM, Mark Thomas wrote: > > > >> Bradley Wagner wrote: > >>> Ok, I have some information to add here. I was able to get thread dumps > >> with > >>> "jstack " (very handy!) because "kill -3" wouldn't work on a hung > >>> process. > >>> I've listed 3 consecutive tests that I run and the thread is blocked at > >> the > >>> time the process is hung. > >> That isn't the way to analyse this. It is almost impossible to detect a > >> hung thread from a single thread dump. > >> > >> For a single hang, take three thread dumps ~10s apart and then examine > >> those to see which thread(s) has(have) not progressed between dumps. > >> > >> Mark > >> > >> > >> > >> > >> - > >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > >> > >> > > > > > > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org >
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
2009/8/31 Christopher Schultz > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Bradley, > > On 8/31/2009 4:02 PM, Bradley Wagner wrote: > > Sorry, I should have mentioned this before, but in all cases I tried the > > thread dumps 2-3 times at least 30s apart and none of the threads have > > progressed at all. > > I agree with Mark: the three threads you showed were different > (different ids), and showing different call stacks. Are you saying that > each thread dump you took represents multiple thread dumps where that > particular thread didn't progress /at all/ over your multi-dump sample? > > You might even want to run strace to see which library function calls > are actually completing. > Agree. The other interesting point is that all of those are hung in places where they would naturally be allocating an object or have very recently allocated an object, and the JVM *might* in theory be growing the heap. Bradley, are your initial and maximum heap sizes identical (I suspect not from your original message)? If not, what happens if you make them so? - Peter
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Bradley, On 8/31/2009 4:02 PM, Bradley Wagner wrote: > Sorry, I should have mentioned this before, but in all cases I tried the > thread dumps 2-3 times at least 30s apart and none of the threads have > progressed at all. I agree with Mark: the three threads you showed were different (different ids), and showing different call stacks. Are you saying that each thread dump you took represents multiple thread dumps where that particular thread didn't progress /at all/ over your multi-dump sample? You might even want to run strace to see which library function calls are actually completing. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkqcMT0ACgkQ9CaO5/Lv0PBokQCeOJzGRtwS5a9tx6N3KRCnP/L/ wp4AoLQFItwFOAGWygNNCMizlArc3n66 =jz8W -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
Bradley Wagner wrote: > Mark, > Sorry, I should have mentioned this before, but in all cases I tried the > thread dumps 2-3 times at least 30s apart and none of the threads have > progressed at all. Hmm. That smells like a JVM issue to me, particularly if it goes away when you use JPDA. Things to try: - use a 1.6 JVM - try different JVM settings, particularly GC related ones - switch to/from client from/to server JVM Mark > > - Bradley > > On Mon, Aug 31, 2009 at 3:52 PM, Mark Thomas wrote: > >> Bradley Wagner wrote: >>> Ok, I have some information to add here. I was able to get thread dumps >> with >>> "jstack " (very handy!) because "kill -3" wouldn't work on a hung >>> process. >>> I've listed 3 consecutive tests that I run and the thread is blocked at >> the >>> time the process is hung. >> That isn't the way to analyse this. It is almost impossible to detect a >> hung thread from a single thread dump. >> >> For a single hang, take three thread dumps ~10s apart and then examine >> those to see which thread(s) has(have) not progressed between dumps. >> >> Mark >> >> >> >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
Mark, Sorry, I should have mentioned this before, but in all cases I tried the thread dumps 2-3 times at least 30s apart and none of the threads have progressed at all. - Bradley On Mon, Aug 31, 2009 at 3:52 PM, Mark Thomas wrote: > Bradley Wagner wrote: > > Ok, I have some information to add here. I was able to get thread dumps > with > > "jstack " (very handy!) because "kill -3" wouldn't work on a hung > > process. > > I've listed 3 consecutive tests that I run and the thread is blocked at > the > > time the process is hung. > > That isn't the way to analyse this. It is almost impossible to detect a > hung thread from a single thread dump. > > For a single hang, take three thread dumps ~10s apart and then examine > those to see which thread(s) has(have) not progressed between dumps. > > Mark > > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > -- Hannon Hill - CMS Experience You Can Trust (678) 904-6900 ext 115 http://www.hannonhill.com
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
Bradley Wagner wrote: > Ok, I have some information to add here. I was able to get thread dumps with > "jstack " (very handy!) because "kill -3" wouldn't work on a hung > process. > I've listed 3 consecutive tests that I run and the thread is blocked at the > time the process is hung. That isn't the way to analyse this. It is almost impossible to detect a hung thread from a single thread dump. For a single hang, take three thread dumps ~10s apart and then examine those to see which thread(s) has(have) not progressed between dumps. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
Ok, I have some information to add here. I was able to get thread dumps with "jstack " (very handy!) because "kill -3" wouldn't work on a hung process. I've listed 3 consecutive tests that I run and the thread is blocked at the time the process is hung. Test 1 Thread 21348: (state = BLOCKED) - java.lang.StringCoding$CharsetSE.encode(char[], int, int) @bci=15, line=334 (Compiled frame) - java.lang.StringCoding.encode(java.lang.String, char[], int, int) @bci=123, line=378 (Compiled frame) - java.lang.String.getBytes(java.lang.String) @bci=25, line=812 (Compiled frame) - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 (Compiled frame) - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 (Compiled frame) - java.io.File.exists() @bci=20, line=702 (Compiled frame) This is the main thread and it is hung on startup during the Spring initialization process. Test 2 Thread 24193: (state = BLOCKED) - java.lang.String.(char[], int, int) @bci=50, line=208 (Compiled frame) - java.io.DataInputStream.readUTF() @bci=1, line=522 (Compiled frame) - org.aspectj.apache.bcel.classfile.ClassParser.readConstantPool() @bci=9, line=186 (Interpreted frame) - org.aspectj.apache.bcel.classfile.ClassParser.parse() @bci=9, line=131 (Interpreted frame) - org.aspectj.apache.bcel.util.NonCachingClassLoaderRepository.loadClass(java.lang.String) @bci=100, line=226 (Compiled frame) - org.aspectj.apache.bcel.util.NonCachingClassLoaderRepository.loadClass(java.lang.Class) @bci=5, line=237 (Interpreted frame) - org.aspectj.weaver.reflect.Java15AnnotationFinder.getParameterAnnotationTypes(java.lang.reflect.Member) @bci=21, line=353 (Compiled frame) Once again hung on an I/O operation this time in String's constructor Test 3 Thread 24365: (state = BLOCKED) - java.lang.String.(char[], int, int) @bci=50, line=208 (Compiled frame) - net.sf.cglib.proxy.Enhancer.access$300(net.sf.cglib.proxy.Enhancer, net.sf.cglib.core.CodeEmitter, int) @bci=3, line=60 (Interpreted frame) - net.sf.cglib.proxy.Enhancer$6.emitCallback(net.sf.cglib.core.CodeEmitter, int) @bci=6, line=913 (Interpreted frame) - net.sf.cglib.proxy.InvocationHandlerGenerator.generate(net.sf.cglib.core.ClassEmitter, net.sf.cglib.proxy.CallbackGenerator$Context, java.util.List) @bci=85, line=44 (Compiled frame) - net.sf.cglib.proxy.Enhancer.emitMethods(net.sf.cglib.core.ClassEmitter, java.util.List, java.util.List) @bci=415, line=942 (Interpreted frame) - net.sf.cglib.proxy.Enhancer.generateClass(net.sf.cglib.asm.ClassVisitor) @bci=353, line=498 (Interpreted frame) - net.sf.cglib.core.DefaultGeneratorStrategy.generate(net.sf.cglib.core.ClassGenerator) @bci=11, line=25 (Interpreted frame) - net.sf.cglib.core.AbstractClassGenerator.create(java.lang.Object) @bci=182, line=216 (Interpreted frame) - net.sf.cglib.proxy.Enhancer.createHelper() @bci=105, line=377 (Interpreted frame) - net.sf.cglib.proxy.Enhancer.createClass() @bci=6, line=317 (Interpreted frame) with the last line in the Catalina log reading: 2009-08-31 14:35:46,261 INFO [SettingsFactory] : Named query checking : enabled 2009-08-31 14:35:46,284 INFO [SessionFactoryImpl] : building session factory In all examples, the software appears to be hung on fairly innocuous methods in the java framework itself. Test 1 and 2 both originate from calls in the IO package and the third test from the CGlib framework that we used from Hibernate. In all instances, there is a call to a method/constructor on the String object in the stack. I'm about to try Java 6 to see if it makes a difference. Thanks, Bradley On Sun, Aug 30, 2009 at 2:56 AM, Peter Crowther wrote: > 2009/8/28 Bradley Wagner > > > I have a Spring/Hibernate app running in Tomcat 5.5.20 that we've tested > in > > many environments that is currently faltering when running in a Ubuntu 7 > > VM. > > Basically on startup, frequently, the startup process will halt when > trying > > to read Hibernate's HBM files and in various other places in startup > > process. When it stops both CPUs are pegged at 200% usage by the java > > process (from 'top'). At this point, the only recourse is to kill the app > > with "kill -9 ". Occasionally the app will start all the way up. > Then, > > I can trigger a re-index of the content in the app's database using > Lucene > > and the app will freeze again. > > > > The only thing I've been able to find in common about these operations is > > that they seem to be heavy I/O. > > > One trick would be to take a thread dump when the app locks up. kill -3 > will trigger such a dump. The output will be in one of Tomcat's > logfiles (catalina.out by default, I think). That might allow you to get > debugging information, as you're getting the data without having to enable > JPDA. > > Sometimes it's useful to take several thread dumps, a few seconds apart, > and > analyse them. However, given your suspicions of a race condition, I'm not > sure that's appropriate in this case as perturbing
Re: 100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
2009/8/28 Bradley Wagner > I have a Spring/Hibernate app running in Tomcat 5.5.20 that we've tested in > many environments that is currently faltering when running in a Ubuntu 7 > VM. > Basically on startup, frequently, the startup process will halt when trying > to read Hibernate's HBM files and in various other places in startup > process. When it stops both CPUs are pegged at 200% usage by the java > process (from 'top'). At this point, the only recourse is to kill the app > with "kill -9 ". Occasionally the app will start all the way up. Then, > I can trigger a re-index of the content in the app's database using Lucene > and the app will freeze again. > > The only thing I've been able to find in common about these operations is > that they seem to be heavy I/O. One trick would be to take a thread dump when the app locks up. kill -3 will trigger such a dump. The output will be in one of Tomcat's logfiles (catalina.out by default, I think). That might allow you to get debugging information, as you're getting the data without having to enable JPDA. Sometimes it's useful to take several thread dumps, a few seconds apart, and analyse them. However, given your suspicions of a race condition, I'm not sure that's appropriate in this case as perturbing the system during startup might prevent the race condition. A few notable environment variables > - JAVA_OPTS="-Xmx512M -XX:MaxPermSize=128m -Djava.awt.headless=true > -Dfile.encoding=UTF-8" > - running Sun's Java 1.5.0.16 > - running Ubuntu 4.2.3-2ubuntu7 (from "cat /proc/version") Thanks! Far too few posters provide this information without prompting. Are you stuck on Java 1.5? 1.6 is generally faster, though you may prefer to stay with a version that you've tested with your app. > Strangely, I tried to debug the application by enabling JPDA: > JAVA_OPTS="-Xmx512M -XX:MaxPermSize=128m -Xdebug > -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n > -Djava.awt.headless=true -Dfile.encoding=UTF-8" > and the application starts up flawlessly. In fact, when running with JPDA > enabled, it's impossible to get it to freeze even when doing the content > re-index. Chuck would know better, but I wonder whether enabling JPDA disables some optimisations in the JVM and/or the compiler. Any debugging next steps or ideas are appreciated. > > Should I look at: > - different ubuntu versions > - different Java versions > - different Tomcat versions My first step would be to see whether a thread dump gives you any useful information. I'd suggest at least going to the latest 5.5 release anyway (5.5.28), as there have been some security fixes since 5.5.20 came out. You might wish to try the latest major version (6.0.20) and see how well it behaves; it depends how much testing you want/need to do! > Is the fact that it works with JPDA indicative of some kind of race > condition possibly? > See above - I suspect JPDA changes several things inside the JVM, but I'm far from being the Java guru on this list. - Peter
100% usage and hanging on startup of Spring/Hibernate app in Tomcat 5.5 on Ubuntu 7
I have a Spring/Hibernate app running in Tomcat 5.5.20 that we've tested in many environments that is currently faltering when running in a Ubuntu 7 VM. Basically on startup, frequently, the startup process will halt when trying to read Hibernate's HBM files and in various other places in startup process. When it stops both CPUs are pegged at 200% usage by the java process (from 'top'). At this point, the only recourse is to kill the app with "kill -9 ". Occasionally the app will start all the way up. Then, I can trigger a re-index of the content in the app's database using Lucene and the app will freeze again. The only thing I've been able to find in common about these operations is that they seem to be heavy I/O. A few notable environment variables - JAVA_OPTS="-Xmx512M -XX:MaxPermSize=128m -Djava.awt.headless=true -Dfile.encoding=UTF-8" - running Sun's Java 1.5.0.16 - running Ubuntu 4.2.3-2ubuntu7 (from "cat /proc/version") Strangely, I tried to debug the application by enabling JPDA: JAVA_OPTS="-Xmx512M -XX:MaxPermSize=128m -Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n -Djava.awt.headless=true -Dfile.encoding=UTF-8" and the application starts up flawlessly. In fact, when running with JPDA enabled, it's impossible to get it to freeze even when doing the content re-index. Any debugging next steps or ideas are appreciated. Should I look at: - different ubuntu versions - different Java versions - different Tomcat versions Is the fact that it works with JPDA indicative of some kind of race condition possibly? Thanks, Bradley
