RE: RE : Tomcat 6.0.35 Crashed again
> From: saumil shah [mailto:saumil...@hotmail.com] > Subject: RE: RE : Tomcat 6.0.35 Crashed again > Is there any concerns over Tomcat bootup and shutdown times , if the > heap size is bumped to 4096m ? Not really. You could probably measure the impact in microseconds, but not any coarser granularity. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: RE : Tomcat 6.0.35 Crashed again
Thanks I forgot to mention that big thing I found from this thread is that I should add -XX:HeapDumpPath= -XX:+HeapDumpOnOutOfMemoryError2. Is there any concerns over Tomcat bootup and shutdown times , if the heap size is bumped to 4096m ? Many thanks again.
RE: RE : Tomcat 6.0.35 Crashed again
Thanks again I forgot to mention that big thing I found from this is that I should add
RE: RE : Tomcat 6.0.35 Crashed again
> From: saumil shah [mailto:saumil...@hotmail.com] > Subject: RE : Tomcat 6.0.35 Crashed again > It would be "prefered" to make Min and Max heap size same Usually that is the case. > is -XX:+UseParallelGC recommended , since I have 8 core machine or would > there be penalty to that ? You very likely should use one of the parallel collector settings. Depending on the JVM version and the platform you're running on, it may already be enabled by default. You might want to review this: http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html Although written for Java 6, it's still applicable to Java 7 and 8. There is a newer parallel and concurrent collector (G1) available in Java 7, but I'm not sure what it's stability is like yet. It is destined to be the default collector one of these days. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: RE : Tomcat 6.0.35 Crashed again
On Thu, Apr 11, 2013 at 8:02 PM, Martin Gainty wrote: > you need to do take a look at the loaded JSF webapps and find outwho is > acquiring a resource and not closing the resource > who is acquiring large amounts of heap and not releasingbe aware any > reference to an any object in another class gives the class the right to be > placed into PermGenHibernate with cglib proxies are notorious memory hogs > awatch your PermGen get pegged when Hibernate and cglib proxies are > loadedStatics are another set of culprits of of heap usage > Remember all long lived heap objects are eventually placed into Permgen > Find the tools to track eden heap, tenured heap and PermGen > > http://www.integratingstuff.com/2011/07/24/understanding-and-avoiding-the-java-permgen-space-error/get > familiar with taking heap dumps with jmap and analyzing with jhathttp:// > javarevisited.blogspot.com/2011/05/java-heap-space-memory-size-jvm.htmlMartin > > Interesting. Thanks! > __ > Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité > > Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene > Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte > Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht > dient lediglich dem Austausch von Informationen und entfaltet keine > rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von > E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen. > Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le > destinataire prévu, nous te demandons avec bonté que pour satisfaire > informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie > de ceci est interdite. Ce message sert à l'information seulement et n'aura > pas n'importe quel effet légalement obligatoire. Étant donné que les email > peuvent facilement être sujets à la manipulation, nous ne pouvons accepter > aucune responsabilité pour le contenu fourni. > > > Date: Thu, 11 Apr 2013 11:40:46 -0400 > > Subject: Re: RE : Tomcat 6.0.35 Crashed again > > From: smithh032...@gmail.com > > To: users@tomcat.apache.org > > > > On Thu, Apr 11, 2013 at 10:41 AM, Mark H. Wood wrote: > > > > > Really, no one else can tell you what settings to use. The best we > > > can hope for is some accepted rules of thumb *as starting points* for > > > further tuning. > > > > > > > > +1 to Dan, Neven, and Mark's responses. Please consider-or-do > 'everything' > > that they mentioned/recommended. > > > > I did want to share my java settings for my > > currently-considered-a-low-scale JSF web app running on Windows Server > 2008 > > R2 64bit server with 32GB RAM. > > > > -XX:HeapDumpPath=D:\apache-tomee-plus-1.6.0-SNAPSHOT\temp > > -XX:+HeapDumpOnOutOfMemoryError > > -Djava.awt.headless=true > > -Dcom.sun.management.jmxremote.port=422 > > -Dcom.sun.management.jmxremote.ssl=false > > -Dcom.sun.management.jmxremote.authenticate=false > > -Xms1024m > > -Xmx1024m > > -XX:MaxPermSize=384m > > -XX:+UseTLAB > > -XX:+UseConcMarkSweepGC > > -XX:+CMSClassUnloadingEnabled > > > > I am very pleased with the GC performance of my app, and I do like to > > monitor the performance of the app via JMX remote connection via Java > > Visual VM. My app runs between 200m to 500m, but I am keeping > Xms/Xmx=1024m > > just to see if I ever get an OOME; so far, so good (never experienced an > > OOME), but recently, I did experience some unexpected/unwanted behavior > > with one of my @Schedule processes which was attempting to sync some data > > from database to/with Google Calendar, and google Calendar service > returned > > google calendar error 503, and I recognized that the memory got up to > 500m, > > and the google calendar error 503 did not resolve itself over an hour > > (@Schedule executes every 2 to 4 minutes, if error occurs, then data is > > appended to the queue for later retry attempt). I never seen that > behavior > > and I don't know if I will see it again; i wish I would have done a 'heap > > dump' instead of a 'stop' tomee/tomcat. Everyday, I listen and read these > > questions/responses on tomcat list, and I can't believe that I forgot to > do > > a 'heap dump'. :( > > > > Also, please note that I occasionally stop-deploy-and-start tomee/tomcat > > almost-on-a-daily-basis to deploy new app-or-configuration-or-library > > updates; also, the app or tomee (or tomcat) seem to accumlate > threadlocals > > over time, and if uptime is > 1 day, then I 'think' I see that memory is > > not released, and I think eventually as uptime increases, then the > > app/tomee/tomcat will result in OOME. :) > > > > At any rate, hope this helps. > > > > Howard > >
RE: RE : Tomcat 6.0.35 Crashed again
you need to do take a look at the loaded JSF webapps and find outwho is acquiring a resource and not closing the resource who is acquiring large amounts of heap and not releasingbe aware any reference to an any object in another class gives the class the right to be placed into PermGenHibernate with cglib proxies are notorious memory hogs awatch your PermGen get pegged when Hibernate and cglib proxies are loadedStatics are another set of culprits of of heap usage Remember all long lived heap objects are eventually placed into Permgen Find the tools to track eden heap, tenured heap and PermGen http://www.integratingstuff.com/2011/07/24/understanding-and-avoiding-the-java-permgen-space-error/ get familiar with taking heap dumps with jmap and analyzing with jhathttp://javarevisited.blogspot.com/2011/05/java-heap-space-memory-size-jvm.html Martin __ Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen. Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni. > Date: Thu, 11 Apr 2013 11:40:46 -0400 > Subject: Re: RE : Tomcat 6.0.35 Crashed again > From: smithh032...@gmail.com > To: users@tomcat.apache.org > > On Thu, Apr 11, 2013 at 10:41 AM, Mark H. Wood wrote: > > > Really, no one else can tell you what settings to use. The best we > > can hope for is some accepted rules of thumb *as starting points* for > > further tuning. > > > > > +1 to Dan, Neven, and Mark's responses. Please consider-or-do 'everything' > that they mentioned/recommended. > > I did want to share my java settings for my > currently-considered-a-low-scale JSF web app running on Windows Server 2008 > R2 64bit server with 32GB RAM. > > -XX:HeapDumpPath=D:\apache-tomee-plus-1.6.0-SNAPSHOT\temp > -XX:+HeapDumpOnOutOfMemoryError > -Djava.awt.headless=true > -Dcom.sun.management.jmxremote.port=422 > -Dcom.sun.management.jmxremote.ssl=false > -Dcom.sun.management.jmxremote.authenticate=false > -Xms1024m > -Xmx1024m > -XX:MaxPermSize=384m > -XX:+UseTLAB > -XX:+UseConcMarkSweepGC > -XX:+CMSClassUnloadingEnabled > > I am very pleased with the GC performance of my app, and I do like to > monitor the performance of the app via JMX remote connection via Java > Visual VM. My app runs between 200m to 500m, but I am keeping Xms/Xmx=1024m > just to see if I ever get an OOME; so far, so good (never experienced an > OOME), but recently, I did experience some unexpected/unwanted behavior > with one of my @Schedule processes which was attempting to sync some data > from database to/with Google Calendar, and google Calendar service returned > google calendar error 503, and I recognized that the memory got up to 500m, > and the google calendar error 503 did not resolve itself over an hour > (@Schedule executes every 2 to 4 minutes, if error occurs, then data is > appended to the queue for later retry attempt). I never seen that behavior > and I don't know if I will see it again; i wish I would have done a 'heap > dump' instead of a 'stop' tomee/tomcat. Everyday, I listen and read these > questions/responses on tomcat list, and I can't believe that I forgot to do > a 'heap dump'. :( > > Also, please note that I occasionally stop-deploy-and-start tomee/tomcat > almost-on-a-daily-basis to deploy new app-or-configuration-or-library > updates; also, the app or tomee (or tomcat) seem to accumlate threadlocals > over time, and if uptime is > 1 day, then I 'think' I see that memory is > not released, and I think eventually as uptime increases, then the > app/tomee/tomcat will result in OOME. :) > > At any rate, hope this helps. > > Howard
Re: RE : Tomcat 6.0.35 Crashed again
On Thu, Apr 11, 2013 at 10:41 AM, Mark H. Wood wrote: > Really, no one else can tell you what settings to use. The best we > can hope for is some accepted rules of thumb *as starting points* for > further tuning. > > +1 to Dan, Neven, and Mark's responses. Please consider-or-do 'everything' that they mentioned/recommended. I did want to share my java settings for my currently-considered-a-low-scale JSF web app running on Windows Server 2008 R2 64bit server with 32GB RAM. -XX:HeapDumpPath=D:\apache-tomee-plus-1.6.0-SNAPSHOT\temp -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true -Dcom.sun.management.jmxremote.port=422 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Xms1024m -Xmx1024m -XX:MaxPermSize=384m -XX:+UseTLAB -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled I am very pleased with the GC performance of my app, and I do like to monitor the performance of the app via JMX remote connection via Java Visual VM. My app runs between 200m to 500m, but I am keeping Xms/Xmx=1024m just to see if I ever get an OOME; so far, so good (never experienced an OOME), but recently, I did experience some unexpected/unwanted behavior with one of my @Schedule processes which was attempting to sync some data from database to/with Google Calendar, and google Calendar service returned google calendar error 503, and I recognized that the memory got up to 500m, and the google calendar error 503 did not resolve itself over an hour (@Schedule executes every 2 to 4 minutes, if error occurs, then data is appended to the queue for later retry attempt). I never seen that behavior and I don't know if I will see it again; i wish I would have done a 'heap dump' instead of a 'stop' tomee/tomcat. Everyday, I listen and read these questions/responses on tomcat list, and I can't believe that I forgot to do a 'heap dump'. :( Also, please note that I occasionally stop-deploy-and-start tomee/tomcat almost-on-a-daily-basis to deploy new app-or-configuration-or-library updates; also, the app or tomee (or tomcat) seem to accumlate threadlocals over time, and if uptime is > 1 day, then I 'think' I see that memory is not released, and I think eventually as uptime increases, then the app/tomee/tomcat will result in OOME. :) At any rate, hope this helps. Howard
Re: RE : Tomcat 6.0.35 Crashed again
Really, no one else can tell you what settings to use. The best we can hope for is some accepted rules of thumb *as starting points* for further tuning. I'd suggest choosing a tool that lets you easily monitor the memory pools, and checking it frequently as you adjust the pool sizes. If your applications are not leaking memory, the pools should each expand to a certain size and then tend to stay there. I would set each pool's initial size slightly larger than its steady-state size, and set some additional headroom on its maximum size to cope with unpredictable demand bursts. (Actually I would leave most of them alone and just tune the ones that seem significantly out-of-tune.) I suggest continuing to monitor memory behavior on a regular basis. Your load probably varies over time, and different versions of code behave somewhat differently. I have a repeating reminder on my calendar to check my Tomcat instances weekly. I use PsiProbe for peeking inside Tomcat, but there are a number of other good tools. As your Tomcat tuning progresses, you'll find what its overall size "ought to be", and can then consider tuning and perhaps resizing the surrounding system. A well-tuned servlet container running in a poorly-tuned OS or undersized hardware will still underperform. The general plan here is the same: start with an educated guess, observe, adjust, monitor -- Mark H. Wood, Lead System Programmer mw...@iupui.edu There's an app for that: your browser pgpH1Gyl2LNzZ.pgp Description: PGP signature
Re: RE : Tomcat 6.0.35 Crashed again
Saumil, On Thu, Apr 11, 2013 at 1:09 AM, saumil shah wrote: > Hello All, > We are using Tomcat 6.0.35 for our production system with 64 bit JVM > (1.6.33) on Windows 2008 R2 SP1. Our physical memory is 24gb. Load is ~ 100 > concurrent sessions. > The Tomcat crashed again with OutOfMemoryError: Java heap space error. We > are using COTS product SAP Business Objects. Upon providing Tomcat logs , > SAP came with recommendation of increasing the heap size . Our current > Java params are : > -Xms512m -Xmx1024m > -XX:MaxPermSize=512m > SAP's recommendation is to make it ... > -Xms1024m -Xmx4096m We have ample of free memory on the box ... > 1a. but the question was heap size as Max 4GB , would it cause application > delay when java GC is due? It's NOT time-sensitive application.1b. Also, > when putting MAx as 4GB , does it mean JVM will make base effort to get 4GB > MAX memory for heap , or is that guaranteed? Based on my reading, it > appeared that only Min, was guaranteed.1c. Is it then recommended to have > Min and Max as 4GB ... so that Heap doesn't have to resize and 4GB is > guaranteed at the time of initialization?1d. I have Tomcat service > installer. I have provided Min and Max heap size values under Tomcat > --> Java tab --> Initial Memory Pool and Maximum memory pool .. I am > assuming that means Min and Max values for Heap.is this correct? > You are correct. When starting JVM you are providing JVM parameters as part of the command line. Usually, we define that in some script (Tomcat's best way is to place these in setenv.bat). -Xms represents initial heap size -Xmx represents the maximum size heap can grow to Now, you might see JVM process takes more memory than Xmx value, because JVM needs to allocate memory not only for heap, but some other things as well, e.g. stack for each thread (-Xss value). If the stack is too small you would get StackOverflowError, if you have lot of threads (and stack is fairly large) you will run out of memory. Now, I don't think that's your case. You have 24G on the machine. Read more about JVM tuning here: http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html "...Large server applications often experience two problems with these defaults. One is slow startup, because the initial heap is small and must be resized over many major collections. A more pressing problem is that the default maximum heap size is unreasonably small for most server applications. The rules of thumb for server applications are: - Unless you have problems with pauses, try granting as much memory as possible to the virtual machine. The default size (64MB) is often too small. - Setting -Xms and -Xmx to the same value increases predictability by removing the most important sizing decision from the virtual machine. However, the virtual machine is then unable to compensate if you make a poor choice. - In general, increase the memory as you increase the number of processors, since allocation can be parallelized." Of course, it really comes down the application memory usage. How is your application written? How does it behave under heavy load? How are your objects being created, how many, how often, how big? How do you setup GC, what heuristics/strategies did you setup, etc... Often times, you would see people set -Xmx and -Xms to the same value (e.g. 4G) this is to reduce the resizing of the initial heap multiple times as your heap consumption grows. However, setting the heap size too big, your GC has much more memory to manage, which makes the GC overhead impact the performance of the application... There is no cookie-cutter solution that fits all. JVM optimization is trial-and-error process. One is sure - you have to have a good testing environment where you can reasonably reproduce loads your application usually gets (heavy load, light load, etc...) (Where are you setting these values Xmx and Xms btw?) 2. When trying to monitor Tomcat , using JConsole and VisualVM . do I > need to add JMX agent and port in Java tab of Tomcat and then restart > , or is it possible to Monitor tomcat , without rebooting it ? Since, it > would not be possible to re-boot tomcat in production for it. > Short answer is no. If you can login into the machine where Tomcat runs, you can run jconsole/jvisualvm locally and there is no need to open JMX management ports. However, if you want to connect to remote JVM processes (ie. JVM running on remote machines, e.g. production JVMs) you need to open up the JMX management ports using appropriate switches (again, these are set setenv.bat) More details here: http://docs.oracle.com/javase/tutorial/jmx/remote/jconsole.html 3. Once JConsole and/or VisualVM is setup ... I can look at Heap/Permgen , > memory , threads etc.. I was wondering how do I see memory leaks and > what is causing it , using these tools? > Memory leak is situation when the objects keep getting stuck in memory unnecessary. You can't really detect these things
