Re: Out of memory exception
Niranjan, On 2/18/21 13:36, Niranjan Rao wrote: First apologies for non tomcat question. I have seen that there is enough expertise here to provide hints and hints are what I am looking for to solve the problem and question is generic enough. I have tried researching problem to best of my abilities. It all happens on Ubuntu 20.04 and JDK 15 We have a java program that regularly throws "java.lang.OutOfMemoryError: Java heap space" exception. Puzzling point is it happens only on one VM. We have a set of two VMs/boxes spawned from same AWS image. Machine class/region is exactly same and since they are from same image, they should be mostly identical except stuff like host name, ip address etc. Number of tasks performed by VMs are comparable and not a significant difference. Yet, one VM never runs of out of memory and other one does. Sometimes it's as soon as half an hour after restarting the process while on the other box process is running for days and no issues. I took memory dumps from both VMs and they look similar. Program is started with -Xmx1g flag and we have taken regular memory dumps. In many cases eclipse MAT reports total memory usage was less than 100MB when program crashed with out of memory exception. Has anyone seen anything similar to this? Identical bits of code behaving differently? What else should I be looking for? What is the load profile of each application/server? You said you aren't running Tomcat, but is load on each of the applications balanced in any way similar to how a web-application load-balancer would work? Sometimes, the answer is simply that one server is doing more work than the other. We have two application servers which are "identical" except that only one of them handles our email queue. Maybe though the "types" of tasks are the same for each server, one of them is getting unlucky and is handling a "big" task that fails each time? Do you have any logging which would indicate which task, ro what kinds of tasks are failing? Do you have a stack trace of the OOME? Do you have a bunch of them (from many separate events)? Do they all look the same? The AWS images are the same, have you upgraded the OS on either one after initial launch, or do you always start fresh with the same image and no "apt-get update" on them. Same JVM and everything on each of them? If you start with -Xmx1G then you should consider also using -Xms1G. IF you have a long-running process which you expect may take up 1G of heap space, go ahead and allocate it all at once instead of wasting time re-sizing the heap a bunch of times on your way up to 1G. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Out of memory exception
On 2/18/21 12:53 PM, Shawn Heisey wrote: On 2/18/2021 12:11 PM, Niranjan Rao wrote: Thank you the response. This is not a web application, but a standalone java program. Hence I said it's not a tomcat question, but a generic JVM question. I have been researching about this a lot and based on many mails on this list, lot of people here know about internal behavior of JVM and specs lot better than I do. Apologies for getting that wrong. Is it a custom app or something that you downloaded and installed? Talk to whoever wrote it. They will hopefully know what information is needed to troubleshoot further. Is Java 15 required for the application to function? If you can successfully use Java 11 or even Java 8, you'll be dealing with a far more stable platform. Major show-stopper bugs in Java are rare, but they do happen. I will warn you that although I do recommend downgrading Java for stability purposes, I do not hold out a lot of hope that it will solve this problem. Which garbage collector are you using? I would recommend one of the really stable collectors, like G1. I wrote this wiki page a long time ago that includes garbage collection information for Solr ... I think it would apply well to any application where latency is important than throughput: https://cwiki.apache.org/confluence/display/SOLR/ShawnHeisey Thanks, Shawn - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org I tried talking to the author, myself - but not much luck. Anyways talking with self does not help with new ideas much ;) We added lot of logging and wrote a simple throw away tool to analyze the logs. Even though task counts are similar, there were some time out errors that could be causing the leaks. Currently a patch is deployed and we are waiting to see if it has made any impact. Interesting point was why is one machine getting brunt of bad things. May be we will drop the box and spawn another VM with the assumption that host could be heavily loaded or something similar not easily visible things going on. Your blog entry is very informative. Thank you. Regards, Niranjan - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Out of memory exception
On 2/18/2021 12:11 PM, Niranjan Rao wrote: Thank you the response. This is not a web application, but a standalone java program. Hence I said it's not a tomcat question, but a generic JVM question. I have been researching about this a lot and based on many mails on this list, lot of people here know about internal behavior of JVM and specs lot better than I do. Apologies for getting that wrong. Is it a custom app or something that you downloaded and installed? Talk to whoever wrote it. They will hopefully know what information is needed to troubleshoot further. Is Java 15 required for the application to function? If you can successfully use Java 11 or even Java 8, you'll be dealing with a far more stable platform. Major show-stopper bugs in Java are rare, but they do happen. I will warn you that although I do recommend downgrading Java for stability purposes, I do not hold out a lot of hope that it will solve this problem. Which garbage collector are you using? I would recommend one of the really stable collectors, like G1. I wrote this wiki page a long time ago that includes garbage collection information for Solr ... I think it would apply well to any application where latency is important than throughput: https://cwiki.apache.org/confluence/display/SOLR/ShawnHeisey Thanks, Shawn - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Out of memory exception
You need to monitor the JVM through something like visual VM or JConsole. Monitor the heap space. Your gonna have to modify your code to help you understand where the memory leak is occurring. The stack trace should give you an idea of where in your code it is trying to allocate memory. On Thu, Feb 18, 2021 at 2:12 PM Niranjan Rao wrote: > Hi Shawn > > Thank you the response. This is not a web application, but a standalone > java program. Hence I said it's not a tomcat question, but a generic JVM > question. I have been researching about this a lot and based on many > mails on this list, lot of people here know about internal behavior of > JVM and specs lot better than I do. > > Both the boxes are spawned from same AWS image, we build the image. > There is no other difference. Both receive tasks over MQ. Tasks could > be slightly different - like for different users, number of entities > user holds etc, but they should not be too different or kind of should > average out in the long run. We have examined the data for the tasks and > nothing unusual has come out so far. > > Regards, > > Niranjan > On 2/18/21 10:59 AM, Shawn Heisey wrote: > > Regards, > > Niranjan > > > On 2/18/2021 11:36 AM, Niranjan Rao wrote: > >> First apologies for non tomcat question. I have seen that there is > >> enough expertise here to provide hints and hints are what I am > >> looking for to solve the problem and question is generic enough. I > >> have tried researching problem to best of my abilities. > > > > I believe you're right to think this isn't a tomcat question. There > > are a lot of things it could be. Tomcat is a *possible* source, > > though I think the chance of that is low. Without a LOT of info that > > I would probably be useless at interpreting or asking for, it's > > impossible to say for sure. > > > > With problems like this, it is normally the application running inside > > Tomcat that has a problem, not Tomcat itself. You're likely to get a > > lot more useful information if you go to the people responsible for > > those applications. > > > >> We have a java program that regularly throws > >> "java.lang.OutOfMemoryError: Java heap space" exception. Puzzling > >> point is it happens only on one VM. We have a set of two VMs/boxes > >> spawned from same AWS image. Machine class/region is exactly same and > >> since they are from same image, they should be mostly identical > >> except stuff like host name, ip address etc. > >> > >> Number of tasks performed by VMs are comparable and not a significant > >> difference. Yet, one VM never runs of out of memory and other one > >> does. Sometimes it's as soon as half an hour after restarting the > >> process while on the other box process is running for days and no > >> issues. > > > > "Comparable" isn't "identical". > > > > Are they running the same apps? Which apps are involved? Is the one > > that's throwing OOME handling substantially similar requests when > > compared to one that doesn't? Is the request rate nearly the same, or > > is the problematic one handling a lot more? Another applicable > > question, also off topic for this mailing list: Are the apps in both > > cases configured identically? > > > >> I took memory dumps from both VMs and they look similar. Program is > >> started with -Xmx1g flag and we have taken regular memory dumps. In > >> many cases eclipse MAT reports total memory usage was less than 100MB > >> when program crashed with out of memory exception. > > > > That's extremely odd, unless the application requested a REALLY big > > chunk of memory such that the 100MB existing plus the new allocation > > would be larger than the max heap size of 1GB. > > > > Do you have enough free memory that you could increase the max heap to > > 2GB or beyond and see what happens? > > > >> Has anyone seen anything similar to this? Identical bits of code > >> behaving differently? What else should I be looking for? > > > > Earlier you said "comparable" and now you're saying "identical". So I > > have to ask ... which is it? Remember that differences in > > configurations, types of requests, and request load can lead to very > > different requirements, even if the apps running inside Tomcat are the > > same. > > > > Most of my experience in the Java world comes from Solr. Apache Solr > > is a servlet application, and ships with Jetty. Tomcat is not usually > > involved. I joined this mailing list because I was responsible for > > Tomcat servers running apps developed in-house, and every once in a > > while, I needed to ask something tomcat-specific. > > > > Thanks, > > Shawn > > > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional
Re: Out of memory exception
Have you tried enabling heap dumps on OOM exceptions ( https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/clopts001.html ; HeapDumpOnOutOfMemoryError) and then looking at the heap dump? It should help you identify where the allocated heap is going to, and give you some ideas of where to look next. On Thu, Feb 18, 2021 at 2:12 PM Niranjan Rao wrote: > Hi Shawn > > Thank you the response. This is not a web application, but a standalone > java program. Hence I said it's not a tomcat question, but a generic JVM > question. I have been researching about this a lot and based on many > mails on this list, lot of people here know about internal behavior of > JVM and specs lot better than I do. > > Both the boxes are spawned from same AWS image, we build the image. > There is no other difference. Both receive tasks over MQ. Tasks could > be slightly different - like for different users, number of entities > user holds etc, but they should not be too different or kind of should > average out in the long run. We have examined the data for the tasks and > nothing unusual has come out so far. > > Regards, > > Niranjan > On 2/18/21 10:59 AM, Shawn Heisey wrote: > > Regards, > > Niranjan > > > On 2/18/2021 11:36 AM, Niranjan Rao wrote: > >> First apologies for non tomcat question. I have seen that there is > >> enough expertise here to provide hints and hints are what I am > >> looking for to solve the problem and question is generic enough. I > >> have tried researching problem to best of my abilities. > > > > I believe you're right to think this isn't a tomcat question. There > > are a lot of things it could be. Tomcat is a *possible* source, > > though I think the chance of that is low. Without a LOT of info that > > I would probably be useless at interpreting or asking for, it's > > impossible to say for sure. > > > > With problems like this, it is normally the application running inside > > Tomcat that has a problem, not Tomcat itself. You're likely to get a > > lot more useful information if you go to the people responsible for > > those applications. > > > >> We have a java program that regularly throws > >> "java.lang.OutOfMemoryError: Java heap space" exception. Puzzling > >> point is it happens only on one VM. We have a set of two VMs/boxes > >> spawned from same AWS image. Machine class/region is exactly same and > >> since they are from same image, they should be mostly identical > >> except stuff like host name, ip address etc. > >> > >> Number of tasks performed by VMs are comparable and not a significant > >> difference. Yet, one VM never runs of out of memory and other one > >> does. Sometimes it's as soon as half an hour after restarting the > >> process while on the other box process is running for days and no > >> issues. > > > > "Comparable" isn't "identical". > > > > Are they running the same apps? Which apps are involved? Is the one > > that's throwing OOME handling substantially similar requests when > > compared to one that doesn't? Is the request rate nearly the same, or > > is the problematic one handling a lot more? Another applicable > > question, also off topic for this mailing list: Are the apps in both > > cases configured identically? > > > >> I took memory dumps from both VMs and they look similar. Program is > >> started with -Xmx1g flag and we have taken regular memory dumps. In > >> many cases eclipse MAT reports total memory usage was less than 100MB > >> when program crashed with out of memory exception. > > > > That's extremely odd, unless the application requested a REALLY big > > chunk of memory such that the 100MB existing plus the new allocation > > would be larger than the max heap size of 1GB. > > > > Do you have enough free memory that you could increase the max heap to > > 2GB or beyond and see what happens? > > > >> Has anyone seen anything similar to this? Identical bits of code > >> behaving differently? What else should I be looking for? > > > > Earlier you said "comparable" and now you're saying "identical". So I > > have to ask ... which is it? Remember that differences in > > configurations, types of requests, and request load can lead to very > > different requirements, even if the apps running inside Tomcat are the > > same. > > > > Most of my experience in the Java world comes from Solr. Apache Solr > > is a servlet application, and ships with Jetty. Tomcat is not usually > > involved. I joined this mailing list because I was responsible for > > Tomcat servers running apps developed in-house, and every once in a > > while, I needed to ask something tomcat-specific. > > > > Thanks, > > Shawn > > > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > > - > To unsubscribe, e-mail:
Re: Out of memory exception
Hi Shawn Thank you the response. This is not a web application, but a standalone java program. Hence I said it's not a tomcat question, but a generic JVM question. I have been researching about this a lot and based on many mails on this list, lot of people here know about internal behavior of JVM and specs lot better than I do. Both the boxes are spawned from same AWS image, we build the image. There is no other difference. Both receive tasks over MQ. Tasks could be slightly different - like for different users, number of entities user holds etc, but they should not be too different or kind of should average out in the long run. We have examined the data for the tasks and nothing unusual has come out so far. Regards, Niranjan On 2/18/21 10:59 AM, Shawn Heisey wrote: Regards, Niranjan On 2/18/2021 11:36 AM, Niranjan Rao wrote: First apologies for non tomcat question. I have seen that there is enough expertise here to provide hints and hints are what I am looking for to solve the problem and question is generic enough. I have tried researching problem to best of my abilities. I believe you're right to think this isn't a tomcat question. There are a lot of things it could be. Tomcat is a *possible* source, though I think the chance of that is low. Without a LOT of info that I would probably be useless at interpreting or asking for, it's impossible to say for sure. With problems like this, it is normally the application running inside Tomcat that has a problem, not Tomcat itself. You're likely to get a lot more useful information if you go to the people responsible for those applications. We have a java program that regularly throws "java.lang.OutOfMemoryError: Java heap space" exception. Puzzling point is it happens only on one VM. We have a set of two VMs/boxes spawned from same AWS image. Machine class/region is exactly same and since they are from same image, they should be mostly identical except stuff like host name, ip address etc. Number of tasks performed by VMs are comparable and not a significant difference. Yet, one VM never runs of out of memory and other one does. Sometimes it's as soon as half an hour after restarting the process while on the other box process is running for days and no issues. "Comparable" isn't "identical". Are they running the same apps? Which apps are involved? Is the one that's throwing OOME handling substantially similar requests when compared to one that doesn't? Is the request rate nearly the same, or is the problematic one handling a lot more? Another applicable question, also off topic for this mailing list: Are the apps in both cases configured identically? I took memory dumps from both VMs and they look similar. Program is started with -Xmx1g flag and we have taken regular memory dumps. In many cases eclipse MAT reports total memory usage was less than 100MB when program crashed with out of memory exception. That's extremely odd, unless the application requested a REALLY big chunk of memory such that the 100MB existing plus the new allocation would be larger than the max heap size of 1GB. Do you have enough free memory that you could increase the max heap to 2GB or beyond and see what happens? Has anyone seen anything similar to this? Identical bits of code behaving differently? What else should I be looking for? Earlier you said "comparable" and now you're saying "identical". So I have to ask ... which is it? Remember that differences in configurations, types of requests, and request load can lead to very different requirements, even if the apps running inside Tomcat are the same. Most of my experience in the Java world comes from Solr. Apache Solr is a servlet application, and ships with Jetty. Tomcat is not usually involved. I joined this mailing list because I was responsible for Tomcat servers running apps developed in-house, and every once in a while, I needed to ask something tomcat-specific. Thanks, Shawn - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Out of memory exception
On 2/18/2021 11:36 AM, Niranjan Rao wrote: First apologies for non tomcat question. I have seen that there is enough expertise here to provide hints and hints are what I am looking for to solve the problem and question is generic enough. I have tried researching problem to best of my abilities. I believe you're right to think this isn't a tomcat question. There are a lot of things it could be. Tomcat is a *possible* source, though I think the chance of that is low. Without a LOT of info that I would probably be useless at interpreting or asking for, it's impossible to say for sure. With problems like this, it is normally the application running inside Tomcat that has a problem, not Tomcat itself. You're likely to get a lot more useful information if you go to the people responsible for those applications. We have a java program that regularly throws "java.lang.OutOfMemoryError: Java heap space" exception. Puzzling point is it happens only on one VM. We have a set of two VMs/boxes spawned from same AWS image. Machine class/region is exactly same and since they are from same image, they should be mostly identical except stuff like host name, ip address etc. Number of tasks performed by VMs are comparable and not a significant difference. Yet, one VM never runs of out of memory and other one does. Sometimes it's as soon as half an hour after restarting the process while on the other box process is running for days and no issues. "Comparable" isn't "identical". Are they running the same apps? Which apps are involved? Is the one that's throwing OOME handling substantially similar requests when compared to one that doesn't? Is the request rate nearly the same, or is the problematic one handling a lot more? Another applicable question, also off topic for this mailing list: Are the apps in both cases configured identically? I took memory dumps from both VMs and they look similar. Program is started with -Xmx1g flag and we have taken regular memory dumps. In many cases eclipse MAT reports total memory usage was less than 100MB when program crashed with out of memory exception. That's extremely odd, unless the application requested a REALLY big chunk of memory such that the 100MB existing plus the new allocation would be larger than the max heap size of 1GB. Do you have enough free memory that you could increase the max heap to 2GB or beyond and see what happens? Has anyone seen anything similar to this? Identical bits of code behaving differently? What else should I be looking for? Earlier you said "comparable" and now you're saying "identical". So I have to ask ... which is it? Remember that differences in configurations, types of requests, and request load can lead to very different requirements, even if the apps running inside Tomcat are the same. Most of my experience in the Java world comes from Solr. Apache Solr is a servlet application, and ships with Jetty. Tomcat is not usually involved. I joined this mailing list because I was responsible for Tomcat servers running apps developed in-house, and every once in a while, I needed to ask something tomcat-specific. Thanks, Shawn - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Out of memory exception
On Jan 23, 2014, at 7:43 AM, Randeep randeep...@gmail.com wrote: Hi, I'm getting out of memory exception errors. Exception in thread Timer-1 java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2882) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572) at java.lang.StringBuffer.append(StringBuffer.java:320) at org.json.JSONArray.toString(JSONArray.java:834) at org.json.JSONObject.valueToString(JSONObject.java:1359) at org.json.JSONObject.toString(JSONObject.java:1233) at com.xxx.xxx.servlet.WServlet.writeProgramJson(WServlet.java:503) at com.xxx.xxx.servlet.WServlet$1.run(WServlet.java:414) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) Is there anyway to see currently set jvm heap size details? Any linux commands. If you run “ps” and “grep” for something like “java”, “catalina”, etc.. you can find your running Tomcat process. On Linux this should show the full command that was run. Alternatively, you could use “jinfo pid”. That will dump a bunch of info including the VM Flags. I have tried creating setenv.sh and putting the new parameters in it. but echo $JAVA_OPTS was not giving any value. How did you install Tomcat? Did you use your distro’s package manager? Did you download from tomcat.apache.org? How are you starting your server? Are you using the included start script? Are you using an init.d (or other) script you created? Are you using commons daemon? So I have put in .bash_profile and run it. Don’t do that. Why would you want these in your bash profile? Then restarted tomcat6. What version exactly? There are quite a few. this is the line I added in .bash_profile. JAVA_OPTS=-Xms256m -Xmx512m -Xss512k -verbose:gc -Xloggc:heap.log” Again, don’t put this in your profile. Also I checked in heap.log where catelina.out can be found. But its not created. Then I touched and restarted. But that also didn't work. What is the right way to do this? If you downloaded from tomcat.apache.org and installed from the zip, you’d put them in bin/setenv.sh and it would work. If you’re using a packaged version of Tomcat from your Linux distro, then it may well be different. If you gave more details, perhaps someone on the list could help. Dan -- Randeep Mob: +919447831699[kerala] Mob: +919880050349[B'lore] I blog here: http://www.randeeppr.me/ Follow me Here: http://twitter.com/Randeeppr Poke me here! http://www.facebook.com/Randeeppr A little Linux Help http://www.linuxhelp.in/ Work profile: http://in.linkedin.com/in/randeeppr - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Out of memory exception
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Randeep, On 1/23/14, 7:43 AM, Randeep wrote: I'm getting out of memory exception errors. :( Exception in thread Timer-1 java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2882) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572) at java.lang.StringBuffer.append(StringBuffer.java:320) at org.json.JSONArray.toString(JSONArray.java:834) at org.json.JSONObject.valueToString(JSONObject.java:1359) at org.json.JSONObject.toString(JSONObject.java:1233) at com.xxx.xxx.servlet.WServlet.writeProgramJson(WServlet.java:503) at com.xxx.xxx.servlet.WServlet$1.run(WServlet.java:414) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) If you are trying to serialize huge JSON objects, this is going to happen. I'm not sure if JSON is as smart as the Java Serialization engine, but if you have a circular-reference in your object graph, I'd be willing to bet that the JSON serializer will run forever and exhaust your heap. It's also possible that you simply have a tiny heap and need more, especially if you aren't currently setting any heap parameters. Is there anyway to see currently set jvm heap size details? Any linux commands. You can attach any profiler to your app. Good luck catching this in the act, though, in production. You can also use jmap -heap which comes with the JDK. Just point it at a process and you'll get a heap summary from a running JVM. Make sure your JVM and jmap version match. I have tried creating setenv.sh and putting the new parameters in it. but echo $JAVA_OPTS was not giving any value. setenv.sh is run during Tomcat startup. Running echo $JAVA_OPTS from the command-line isn't going to reveal anything. What you want to do is modify setenv.sh, re-launch Tomcat, and then look at the process command line... something like this: $ ps aux | grep Bootstrap This should give you the whole command-line for Tomcat, and you can verify that your heap options have been set. So I have put in .bash_profile and run it. Then restarted tomcat6. Don't do that: it will only work if you login using bash, and it won't work for other users you may have to restart Tomcat. this is the line I added in .bash_profile. JAVA_OPTS=-Xms256m -Xmx512m -Xss512k -verbose:gc -Xloggc:heap.log You'll need to export JAVA_OPTS, otherwise it won't do anything for you. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJS4TM+AAoJEBzwKT+lPKRYfB4QAMP4/MORiK92nK4kJ+KD9alc 2QYcriasb8QkPPgvYHGhGaxeLIprb1TE6tZdMSp08bHadDr+HpSbxi5xq9C6A1RJ fabHj9ShrXqdtTFiewPH3P/ulvaaJNPD1LeIFBwoG7/4vAIGKW48gLhaRUYcbe+z lV2NsZ18u38EZ9KRR8An66jX4PkwMXxl4QdQx5HDmuS/Sx8G7AfDVkuSB5PMNXbg TVlSEz5VeJnEGtBwnjgIa6ZSTv+iIv+nsw4WVuMe+4h1R07e1axxNismkgpUsU46 Ra4P7TssgdrqXFXmX4+kPNzd3xbbhPYIOOrQ0PYt6wGF0Wd9N1r/gvvViDbgghca wp39c3VJM8eE2vy9n0XCGYBkdq2DnRNVFV2y2xnfwecWKtDPEWfklTk8ZvbcAyxe hi3hxQnChCk74nxd/hvI1uNfgHcd/NwtmifwXPNM0BQCZ+AJarCbGca+a/BTTBaX txVIuIEf3jt/gpFu28UI8kt7Hx/PgWJ32QFyIPCbH5EYaSiKSz/Q6wCTvJTbUG4s GCK8AB/rGf1Z8efKlUZw2ESr1yX/VBCu9uOEK/60v+AAthiubq+O2ke1TzWKvy/I MxDhNiMo710w98h5hqIl5SthwXCgv3vEaaiUgDUV8zP3CoOtHL61NNZmLrMBTD2X 18aTMDKJ9BTvoYoZcmyV =IL2E -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Out of memory exception
Hi all, Thanks a lot for valuable points. Here the information you guys asked and the changes I made. OS : CentOS release 5.4 (Final) httpd-2.2.3-65.el5.centos(front_end) +(mod_jk)+ apache-tomcat-6.0.37 I installed httpd using yum. Downloaded tomcat from Apache as compressed archive(tar.gz) I'm using following startup script. [root@server ~]# cat /etc/init.d/tomcat6 #!/bin/bash # description: Tomcat Start Stop Restart # processname: tomcat # chkconfig: 234 20 80 JAVA_HOME=/usr/java/default export JAVA_HOME PATH=$JAVA_HOME/bin:$PATH export PATH CATALINA_HOME=/usr/share/apache-tomcat-6.0.37 case $1 in start) sh $CATALINA_HOME/bin/startup.sh ;; stop) sh $CATALINA_HOME/bin/shutdown.sh ;; restart) sh $CATALINA_HOME/bin/shutdown.sh sh $CATALINA_HOME/bin/startup.sh ;; esac exit 0 As you said, I removed the JAVA_OPTS declaration from .bash_profile and I have created setenv.sh and set JAVA_OPTS in it. [root@server bin]# pwd /usr/share/apache-tomcat-6.0.37/bin [root@server bin]# cat setenv.sh export JAVA_OPTS='-Xms256m -Xmx512m -Xss512k -verbose:gc -Xloggc:heap.log' Now it shows in grep. [root@server bin]# ps aux | grep Bootstrap root 1100 33.7 5.4 707052 95240 pts/0Sl 11:29 0:06 /usr/java/default/bin/java -Djava.util.logging.config.file=/usr/share/apache-tomcat-6.0.37/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms256m -Xmx512m -Xss512k -verbose:gc -Xloggc:heap.log -Djava.endorsed.dirs=/usr/share/apache-tomcat-6.0.37/endorsed -classpath /usr/share/apache-tomcat-6.0.37/bin/bootstrap.jar -Dcatalina.base=/usr/share/apache-tomcat-6.0.37 -Dcatalina.home=/usr/share/apache-tomcat-6.0.37 -Djava.io.tmpdir=/usr/share/apache-tomcat-6.0.37/temp org.apache.catalina.startup.Bootstrap start root 1158 0.0 0.0 3932 692 pts/0R+ 11:29 0:00 grep Bootstrap [root@server bin]# But nothing came in heap.log. So I better monitor this and if I get the OOM again I have to increase Max limit! On Thu, Jan 23, 2014 at 8:50 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Randeep, On 1/23/14, 7:43 AM, Randeep wrote: I'm getting out of memory exception errors. :( Exception in thread Timer-1 java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2882) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572) at java.lang.StringBuffer.append(StringBuffer.java:320) at org.json.JSONArray.toString(JSONArray.java:834) at org.json.JSONObject.valueToString(JSONObject.java:1359) at org.json.JSONObject.toString(JSONObject.java:1233) at com.xxx.xxx.servlet.WServlet.writeProgramJson(WServlet.java:503) at com.xxx.xxx.servlet.WServlet$1.run(WServlet.java:414) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) If you are trying to serialize huge JSON objects, this is going to happen. I'm not sure if JSON is as smart as the Java Serialization engine, but if you have a circular-reference in your object graph, I'd be willing to bet that the JSON serializer will run forever and exhaust your heap. It's also possible that you simply have a tiny heap and need more, especially if you aren't currently setting any heap parameters. Is there anyway to see currently set jvm heap size details? Any linux commands. You can attach any profiler to your app. Good luck catching this in the act, though, in production. You can also use jmap -heap which comes with the JDK. Just point it at a process and you'll get a heap summary from a running JVM. Make sure your JVM and jmap version match. I have tried creating setenv.sh and putting the new parameters in it. but echo $JAVA_OPTS was not giving any value. setenv.sh is run during Tomcat startup. Running echo $JAVA_OPTS from the command-line isn't going to reveal anything. What you want to do is modify setenv.sh, re-launch Tomcat, and then look at the process command line... something like this: $ ps aux | grep Bootstrap This should give you the whole command-line for Tomcat, and you can verify that your heap options have been set. So I have put in .bash_profile and run it. Then restarted tomcat6. Don't do that: it will only work if you login using bash, and it won't work for other users you may have to restart Tomcat. this is the line I added in .bash_profile. JAVA_OPTS=-Xms256m -Xmx512m -Xss512k -verbose:gc -Xloggc:heap.log You'll need to export JAVA_OPTS, otherwise it won't do anything for you. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJS4TM+AAoJEBzwKT+lPKRYfB4QAMP4/MORiK92nK4kJ+KD9alc 2QYcriasb8QkPPgvYHGhGaxeLIprb1TE6tZdMSp08bHadDr+HpSbxi5xq9C6A1RJ
Re: Out of memory exception
Randeep wrote: Hi all, Thanks a lot for valuable points. Here the information you guys asked and the changes I made. OS : CentOS release 5.4 (Final) httpd-2.2.3-65.el5.centos(front_end) +(mod_jk)+ apache-tomcat-6.0.37 I installed httpd using yum. Downloaded tomcat from Apache as compressed archive(tar.gz) I'm using following startup script. [root@server ~]# cat /etc/init.d/tomcat6 #!/bin/bash # description: Tomcat Start Stop Restart # processname: tomcat # chkconfig: 234 20 80 JAVA_HOME=/usr/java/default export JAVA_HOME PATH=$JAVA_HOME/bin:$PATH export PATH CATALINA_HOME=/usr/share/apache-tomcat-6.0.37 case $1 in start) sh $CATALINA_HOME/bin/startup.sh ;; stop) sh $CATALINA_HOME/bin/shutdown.sh ;; restart) sh $CATALINA_HOME/bin/shutdown.sh sh $CATALINA_HOME/bin/startup.sh ;; esac exit 0 As you said, I removed the JAVA_OPTS declaration from .bash_profile and I have created setenv.sh and set JAVA_OPTS in it. [root@server bin]# pwd /usr/share/apache-tomcat-6.0.37/bin [root@server bin]# cat setenv.sh export JAVA_OPTS='-Xms256m -Xmx512m -Xss512k -verbose:gc -Xloggc:heap.log' Now it shows in grep. [root@server bin]# ps aux | grep Bootstrap root 1100 33.7 5.4 707052 95240 pts/0Sl 11:29 0:06 /usr/java/default/bin/java -Djava.util.logging.config.file=/usr/share/apache-tomcat-6.0.37/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms256m -Xmx512m -Xss512k -verbose:gc -Xloggc:heap.log -Djava.endorsed.dirs=/usr/share/apache-tomcat-6.0.37/endorsed -classpath /usr/share/apache-tomcat-6.0.37/bin/bootstrap.jar -Dcatalina.base=/usr/share/apache-tomcat-6.0.37 -Dcatalina.home=/usr/share/apache-tomcat-6.0.37 -Djava.io.tmpdir=/usr/share/apache-tomcat-6.0.37/temp org.apache.catalina.startup.Bootstrap start root 1158 0.0 0.0 3932 692 pts/0R+ 11:29 0:00 grep Bootstrap [root@server bin]# But nothing came in heap.log. So I better monitor this and if I get the OOM again I have to increase Max limit! Maybe you should provide a full path for the -Xloggc option ? On Thu, Jan 23, 2014 at 8:50 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Randeep, On 1/23/14, 7:43 AM, Randeep wrote: I'm getting out of memory exception errors. :( Exception in thread Timer-1 java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2882) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572) at java.lang.StringBuffer.append(StringBuffer.java:320) at org.json.JSONArray.toString(JSONArray.java:834) at org.json.JSONObject.valueToString(JSONObject.java:1359) at org.json.JSONObject.toString(JSONObject.java:1233) at com.xxx.xxx.servlet.WServlet.writeProgramJson(WServlet.java:503) at com.xxx.xxx.servlet.WServlet$1.run(WServlet.java:414) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) If you are trying to serialize huge JSON objects, this is going to happen. I'm not sure if JSON is as smart as the Java Serialization engine, but if you have a circular-reference in your object graph, I'd be willing to bet that the JSON serializer will run forever and exhaust your heap. It's also possible that you simply have a tiny heap and need more, especially if you aren't currently setting any heap parameters. Is there anyway to see currently set jvm heap size details? Any linux commands. You can attach any profiler to your app. Good luck catching this in the act, though, in production. You can also use jmap -heap which comes with the JDK. Just point it at a process and you'll get a heap summary from a running JVM. Make sure your JVM and jmap version match. I have tried creating setenv.sh and putting the new parameters in it. but echo $JAVA_OPTS was not giving any value. setenv.sh is run during Tomcat startup. Running echo $JAVA_OPTS from the command-line isn't going to reveal anything. What you want to do is modify setenv.sh, re-launch Tomcat, and then look at the process command line... something like this: $ ps aux | grep Bootstrap This should give you the whole command-line for Tomcat, and you can verify that your heap options have been set. So I have put in .bash_profile and run it. Then restarted tomcat6. Don't do that: it will only work if you login using bash, and it won't work for other users you may have to restart Tomcat. this is the line I added in .bash_profile. JAVA_OPTS=-Xms256m -Xmx512m -Xss512k -verbose:gc -Xloggc:heap.log You'll need to export JAVA_OPTS, otherwise it won't do anything for you. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJS4TM+AAoJEBzwKT+lPKRYfB4QAMP4/MORiK92nK4kJ+KD9alc 2QYcriasb8QkPPgvYHGhGaxeLIprb1TE6tZdMSp08bHadDr+HpSbxi5xq9C6A1RJ
Re: Out of memory exception
On Jan 23, 2014, at 11:38 AM, Randeep randeep...@gmail.com wrote: Hi all, Thanks a lot for valuable points. Here the information you guys asked and the changes I made. OS : CentOS release 5.4 (Final) httpd-2.2.3-65.el5.centos(front_end) +(mod_jk)+ apache-tomcat-6.0.37 I installed httpd using yum. Downloaded tomcat from Apache as compressed archive(tar.gz) I'm using following startup script. [root@server ~]# cat /etc/init.d/tomcat6 #!/bin/bash # description: Tomcat Start Stop Restart # processname: tomcat # chkconfig: 234 20 80 JAVA_HOME=/usr/java/default export JAVA_HOME PATH=$JAVA_HOME/bin:$PATH export PATH CATALINA_HOME=/usr/share/apache-tomcat-6.0.37 case $1 in start) sh $CATALINA_HOME/bin/startup.sh ;; stop) sh $CATALINA_HOME/bin/shutdown.sh ;; restart) sh $CATALINA_HOME/bin/shutdown.sh sh $CATALINA_HOME/bin/startup.sh ;; esac exit 0 As you said, I removed the JAVA_OPTS declaration from .bash_profile and I have created setenv.sh and set JAVA_OPTS in it. [root@server bin]# pwd /usr/share/apache-tomcat-6.0.37/bin [root@server bin]# cat setenv.sh export JAVA_OPTS='-Xms256m -Xmx512m -Xss512k -verbose:gc -Xloggc:heap.log’ Based on the name of the file you’re listing, it makes me think you’re confusing what this option does. It’s a garbage collection log. While that shows some info about the heap, it’s mainly targeted towards knowing what’s happening with GC in your process. I can’t help but think you might be looking for -XX:+HeapDumpOnOutOfMemoryError, which generates a heap dump when you get an OOME. Now it shows in grep. [root@server bin]# ps aux | grep Bootstrap root 1100 33.7 5.4 707052 95240 pts/0Sl 11:29 0:06 /usr/java/default/bin/java -Djava.util.logging.config.file=/usr/share/apache-tomcat-6.0.37/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms256m -Xmx512m -Xss512k -verbose:gc -Xloggc:heap.log -Djava.endorsed.dirs=/usr/share/apache-tomcat-6.0.37/endorsed -classpath /usr/share/apache-tomcat-6.0.37/bin/bootstrap.jar -Dcatalina.base=/usr/share/apache-tomcat-6.0.37 -Dcatalina.home=/usr/share/apache-tomcat-6.0.37 -Djava.io.tmpdir=/usr/share/apache-tomcat-6.0.37/temp org.apache.catalina.startup.Bootstrap start root 1158 0.0 0.0 3932 692 pts/0R+ 11:29 0:00 grep Bootstrap [root@server bin]# But nothing came in heap.log. Try using this option instead: -Xloggc:$CATALINA_BASE/logs/heap.log” That should put the file in your “logs” directory. Dan So I better monitor this and if I get the OOM again I have to increase Max limit! On Thu, Jan 23, 2014 at 8:50 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Randeep, On 1/23/14, 7:43 AM, Randeep wrote: I'm getting out of memory exception errors. :( Exception in thread Timer-1 java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2882) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572) at java.lang.StringBuffer.append(StringBuffer.java:320) at org.json.JSONArray.toString(JSONArray.java:834) at org.json.JSONObject.valueToString(JSONObject.java:1359) at org.json.JSONObject.toString(JSONObject.java:1233) at com.xxx.xxx.servlet.WServlet.writeProgramJson(WServlet.java:503) at com.xxx.xxx.servlet.WServlet$1.run(WServlet.java:414) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) If you are trying to serialize huge JSON objects, this is going to happen. I'm not sure if JSON is as smart as the Java Serialization engine, but if you have a circular-reference in your object graph, I'd be willing to bet that the JSON serializer will run forever and exhaust your heap. It's also possible that you simply have a tiny heap and need more, especially if you aren't currently setting any heap parameters. Is there anyway to see currently set jvm heap size details? Any linux commands. You can attach any profiler to your app. Good luck catching this in the act, though, in production. You can also use jmap -heap which comes with the JDK. Just point it at a process and you'll get a heap summary from a running JVM. Make sure your JVM and jmap version match. I have tried creating setenv.sh and putting the new parameters in it. but echo $JAVA_OPTS was not giving any value. setenv.sh is run during Tomcat startup. Running echo $JAVA_OPTS from the command-line isn't going to reveal anything. What you want to do is modify setenv.sh, re-launch Tomcat, and then look at the process command line... something like this: $ ps aux | grep Bootstrap This should give you the whole command-line for Tomcat, and you can verify that your heap options have been set. So I have put in .bash_profile and run it. Then restarted
Re: Out of memory exception
Thanks. That worked. Anything unusual? Garbage collection is happening too often? [root@server logs]# cat /usr/share/apache-tomcat-6.0.37/logs/heap.log 1.910: [GC 16192K-1923K(260160K), 0.0150080 secs] 2.454: [GC 18115K-3151K(260160K), 0.0137140 secs] 2.760: [GC 19343K-3309K(260160K), 0.0058220 secs] 3.648: [GC 19501K-4446K(260160K), 0.0062870 secs] 4.000: [GC 20638K-5063K(260160K), 0.0080940 secs] 4.305: [GC 21255K-6302K(260160K), 0.0060890 secs] 4.598: [GC 22494K-6323K(260160K), 0.0062020 secs] 5.664: [GC 22515K-7098K(260160K), 0.0055070 secs] 6.980: [GC 23290K-8708K(260160K), 0.0176410 secs] 7.332: [GC 24900K-12045K(260160K), 0.0179400 secs] 7.794: [Full GC 24591K-12864K(260160K), 0.2538960 secs] 8.476: [GC 29184K-14200K(260288K), 0.0080560 secs] 8.939: [GC 30520K-15401K(260288K), 0.0407000 secs] 9.379: [GC 31721K-16897K(260288K), 0.0099850 secs] 9.603: [GC 33217K-16652K(260288K), 0.0649140 secs] 10.498: [GC 32972K-17768K(260288K), 0.0068540 secs] 10.885: [GC 34088K-18538K(260288K), 0.0084330 secs] 11.225: [GC 34858K-19584K(260288K), 0.0071730 secs] 11.466: [GC 35904K-20313K(260288K), 0.0050570 secs] 12.128: [GC 36633K-20565K(260288K), 0.0042210 secs] 12.385: [Full GC 26015K-20204K(260288K), 0.2847340 secs] 13.817: [GC 36524K-22174K(260288K), 0.0155690 secs] 14.127: [GC 38494K-25303K(260288K), 0.0816490 secs] 14.671: [GC 41623K-27150K(260288K), 0.0436660 secs] 15.173: [GC 43470K-28757K(260288K), 0.0140720 secs] 15.414: [Full GC 32030K-28926K(260288K), 0.3071990 secs] 65.004: [GC 45310K-31775K(260352K), 0.0105440 secs] [root@server logs]# On Thu, Jan 23, 2014 at 10:33 PM, André Warnier a...@ice-sa.com wrote: Randeep wrote: Hi all, Thanks a lot for valuable points. Here the information you guys asked and the changes I made. OS : CentOS release 5.4 (Final) httpd-2.2.3-65.el5.centos(front_end) +(mod_jk)+ apache-tomcat-6.0.37 I installed httpd using yum. Downloaded tomcat from Apache as compressed archive(tar.gz) I'm using following startup script. [root@server ~]# cat /etc/init.d/tomcat6 #!/bin/bash # description: Tomcat Start Stop Restart # processname: tomcat # chkconfig: 234 20 80 JAVA_HOME=/usr/java/default export JAVA_HOME PATH=$JAVA_HOME/bin:$PATH export PATH CATALINA_HOME=/usr/share/apache-tomcat-6.0.37 case $1 in start) sh $CATALINA_HOME/bin/startup.sh ;; stop) sh $CATALINA_HOME/bin/shutdown.sh ;; restart) sh $CATALINA_HOME/bin/shutdown.sh sh $CATALINA_HOME/bin/startup.sh ;; esac exit 0 As you said, I removed the JAVA_OPTS declaration from .bash_profile and I have created setenv.sh and set JAVA_OPTS in it. [root@server bin]# pwd /usr/share/apache-tomcat-6.0.37/bin [root@server bin]# cat setenv.sh export JAVA_OPTS='-Xms256m -Xmx512m -Xss512k -verbose:gc -Xloggc:heap.log' Now it shows in grep. [root@server bin]# ps aux | grep Bootstrap root 1100 33.7 5.4 707052 95240 pts/0Sl 11:29 0:06 /usr/java/default/bin/java -Djava.util.logging.config.file=/usr/share/apache-tomcat- 6.0.37/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms256m -Xmx512m -Xss512k -verbose:gc -Xloggc:heap.log -Djava.endorsed.dirs=/usr/share/apache-tomcat-6.0.37/endorsed -classpath /usr/share/apache-tomcat-6.0.37/bin/bootstrap.jar -Dcatalina.base=/usr/share/apache-tomcat-6.0.37 -Dcatalina.home=/usr/share/apache-tomcat-6.0.37 -Djava.io.tmpdir=/usr/share/apache-tomcat-6.0.37/temp org.apache.catalina.startup.Bootstrap start root 1158 0.0 0.0 3932 692 pts/0R+ 11:29 0:00 grep Bootstrap [root@server bin]# But nothing came in heap.log. So I better monitor this and if I get the OOM again I have to increase Max limit! Maybe you should provide a full path for the -Xloggc option ? On Thu, Jan 23, 2014 at 8:50 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Randeep, On 1/23/14, 7:43 AM, Randeep wrote: I'm getting out of memory exception errors. :( Exception in thread Timer-1 java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2882) at java.lang.AbstractStringBuilder.expandCapacity( AbstractStringBuilder.java:100) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572) at java.lang.StringBuffer.append(StringBuffer.java:320) at org.json.JSONArray.toString(JSONArray.java:834) at org.json.JSONObject.valueToString(JSONObject.java:1359) at org.json.JSONObject.toString(JSONObject.java:1233) at com.xxx.xxx.servlet.WServlet.writeProgramJson(WServlet.java:503) at com.xxx.xxx.servlet.WServlet$1.run(WServlet.java:414) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) If you are trying to serialize huge JSON objects, this is going to happen. I'm not sure if JSON is as smart as the Java Serialization engine, but if you have a circular-reference in your object graph, I'd be willing to bet that the JSON serializer
Re: Out of memory exception
I mean i'd to give full path. like this export JAVA_OPTS='-Xms256m -Xmx512m -Xss512k -verbose:gc -Xloggc:/usr/share/apache-tomcat-6.0.37/logs/heap.log' On Thu, Jan 23, 2014 at 10:47 PM, Randeep randeep...@gmail.com wrote: Thanks. That worked. Anything unusual? Garbage collection is happening too often? [root@server logs]# cat /usr/share/apache-tomcat-6.0.37/logs/heap.log 1.910: [GC 16192K-1923K(260160K), 0.0150080 secs] 2.454: [GC 18115K-3151K(260160K), 0.0137140 secs] 2.760: [GC 19343K-3309K(260160K), 0.0058220 secs] 3.648: [GC 19501K-4446K(260160K), 0.0062870 secs] 4.000: [GC 20638K-5063K(260160K), 0.0080940 secs] 4.305: [GC 21255K-6302K(260160K), 0.0060890 secs] 4.598: [GC 22494K-6323K(260160K), 0.0062020 secs] 5.664: [GC 22515K-7098K(260160K), 0.0055070 secs] 6.980: [GC 23290K-8708K(260160K), 0.0176410 secs] 7.332: [GC 24900K-12045K(260160K), 0.0179400 secs] 7.794: [Full GC 24591K-12864K(260160K), 0.2538960 secs] 8.476: [GC 29184K-14200K(260288K), 0.0080560 secs] 8.939: [GC 30520K-15401K(260288K), 0.0407000 secs] 9.379: [GC 31721K-16897K(260288K), 0.0099850 secs] 9.603: [GC 33217K-16652K(260288K), 0.0649140 secs] 10.498: [GC 32972K-17768K(260288K), 0.0068540 secs] 10.885: [GC 34088K-18538K(260288K), 0.0084330 secs] 11.225: [GC 34858K-19584K(260288K), 0.0071730 secs] 11.466: [GC 35904K-20313K(260288K), 0.0050570 secs] 12.128: [GC 36633K-20565K(260288K), 0.0042210 secs] 12.385: [Full GC 26015K-20204K(260288K), 0.2847340 secs] 13.817: [GC 36524K-22174K(260288K), 0.0155690 secs] 14.127: [GC 38494K-25303K(260288K), 0.0816490 secs] 14.671: [GC 41623K-27150K(260288K), 0.0436660 secs] 15.173: [GC 43470K-28757K(260288K), 0.0140720 secs] 15.414: [Full GC 32030K-28926K(260288K), 0.3071990 secs] 65.004: [GC 45310K-31775K(260352K), 0.0105440 secs] [root@server logs]# On Thu, Jan 23, 2014 at 10:33 PM, André Warnier a...@ice-sa.com wrote: Randeep wrote: Hi all, Thanks a lot for valuable points. Here the information you guys asked and the changes I made. OS : CentOS release 5.4 (Final) httpd-2.2.3-65.el5.centos(front_end) +(mod_jk)+ apache-tomcat-6.0.37 I installed httpd using yum. Downloaded tomcat from Apache as compressed archive(tar.gz) I'm using following startup script. [root@server ~]# cat /etc/init.d/tomcat6 #!/bin/bash # description: Tomcat Start Stop Restart # processname: tomcat # chkconfig: 234 20 80 JAVA_HOME=/usr/java/default export JAVA_HOME PATH=$JAVA_HOME/bin:$PATH export PATH CATALINA_HOME=/usr/share/apache-tomcat-6.0.37 case $1 in start) sh $CATALINA_HOME/bin/startup.sh ;; stop) sh $CATALINA_HOME/bin/shutdown.sh ;; restart) sh $CATALINA_HOME/bin/shutdown.sh sh $CATALINA_HOME/bin/startup.sh ;; esac exit 0 As you said, I removed the JAVA_OPTS declaration from .bash_profile and I have created setenv.sh and set JAVA_OPTS in it. [root@server bin]# pwd /usr/share/apache-tomcat-6.0.37/bin [root@server bin]# cat setenv.sh export JAVA_OPTS='-Xms256m -Xmx512m -Xss512k -verbose:gc -Xloggc:heap.log' Now it shows in grep. [root@server bin]# ps aux | grep Bootstrap root 1100 33.7 5.4 707052 95240 pts/0Sl 11:29 0:06 /usr/java/default/bin/java -Djava.util.logging.config.file=/usr/share/apache-tomcat- 6.0.37/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms256m -Xmx512m -Xss512k -verbose:gc -Xloggc:heap.log -Djava.endorsed.dirs=/usr/share/apache-tomcat-6.0.37/endorsed -classpath /usr/share/apache-tomcat-6.0.37/bin/bootstrap.jar -Dcatalina.base=/usr/share/apache-tomcat-6.0.37 -Dcatalina.home=/usr/share/apache-tomcat-6.0.37 -Djava.io.tmpdir=/usr/share/apache-tomcat-6.0.37/temp org.apache.catalina.startup.Bootstrap start root 1158 0.0 0.0 3932 692 pts/0R+ 11:29 0:00 grep Bootstrap [root@server bin]# But nothing came in heap.log. So I better monitor this and if I get the OOM again I have to increase Max limit! Maybe you should provide a full path for the -Xloggc option ? On Thu, Jan 23, 2014 at 8:50 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Randeep, On 1/23/14, 7:43 AM, Randeep wrote: I'm getting out of memory exception errors. :( Exception in thread Timer-1 java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2882) at java.lang.AbstractStringBuilder.expandCapacity( AbstractStringBuilder.java:100) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572) at java.lang.StringBuffer.append(StringBuffer.java:320) at org.json.JSONArray.toString(JSONArray.java:834) at org.json.JSONObject.valueToString(JSONObject.java:1359) at org.json.JSONObject.toString(JSONObject.java:1233) at com.xxx.xxx.servlet.WServlet.writeProgramJson(WServlet.java:503) at com.xxx.xxx.servlet.WServlet$1.run(WServlet.java:414) at java.util.TimerThread.mainLoop(Timer.java:512) at
Re: Out of memory exception
I have added -XX:+HeapDumpOnOutOfMemoryError also. Thanks. On Thu, Jan 23, 2014 at 10:49 PM, Randeep randeep...@gmail.com wrote: I mean i'd to give full path. like this export JAVA_OPTS='-Xms256m -Xmx512m -Xss512k -verbose:gc -Xloggc:/usr/share/apache-tomcat-6.0.37/logs/heap.log' On Thu, Jan 23, 2014 at 10:47 PM, Randeep randeep...@gmail.com wrote: Thanks. That worked. Anything unusual? Garbage collection is happening too often? [root@server logs]# cat /usr/share/apache-tomcat-6.0.37/logs/heap.log 1.910: [GC 16192K-1923K(260160K), 0.0150080 secs] 2.454: [GC 18115K-3151K(260160K), 0.0137140 secs] 2.760: [GC 19343K-3309K(260160K), 0.0058220 secs] 3.648: [GC 19501K-4446K(260160K), 0.0062870 secs] 4.000: [GC 20638K-5063K(260160K), 0.0080940 secs] 4.305: [GC 21255K-6302K(260160K), 0.0060890 secs] 4.598: [GC 22494K-6323K(260160K), 0.0062020 secs] 5.664: [GC 22515K-7098K(260160K), 0.0055070 secs] 6.980: [GC 23290K-8708K(260160K), 0.0176410 secs] 7.332: [GC 24900K-12045K(260160K), 0.0179400 secs] 7.794: [Full GC 24591K-12864K(260160K), 0.2538960 secs] 8.476: [GC 29184K-14200K(260288K), 0.0080560 secs] 8.939: [GC 30520K-15401K(260288K), 0.0407000 secs] 9.379: [GC 31721K-16897K(260288K), 0.0099850 secs] 9.603: [GC 33217K-16652K(260288K), 0.0649140 secs] 10.498: [GC 32972K-17768K(260288K), 0.0068540 secs] 10.885: [GC 34088K-18538K(260288K), 0.0084330 secs] 11.225: [GC 34858K-19584K(260288K), 0.0071730 secs] 11.466: [GC 35904K-20313K(260288K), 0.0050570 secs] 12.128: [GC 36633K-20565K(260288K), 0.0042210 secs] 12.385: [Full GC 26015K-20204K(260288K), 0.2847340 secs] 13.817: [GC 36524K-22174K(260288K), 0.0155690 secs] 14.127: [GC 38494K-25303K(260288K), 0.0816490 secs] 14.671: [GC 41623K-27150K(260288K), 0.0436660 secs] 15.173: [GC 43470K-28757K(260288K), 0.0140720 secs] 15.414: [Full GC 32030K-28926K(260288K), 0.3071990 secs] 65.004: [GC 45310K-31775K(260352K), 0.0105440 secs] [root@server logs]# On Thu, Jan 23, 2014 at 10:33 PM, André Warnier a...@ice-sa.com wrote: Randeep wrote: Hi all, Thanks a lot for valuable points. Here the information you guys asked and the changes I made. OS : CentOS release 5.4 (Final) httpd-2.2.3-65.el5.centos(front_end) +(mod_jk)+ apache-tomcat-6.0.37 I installed httpd using yum. Downloaded tomcat from Apache as compressed archive(tar.gz) I'm using following startup script. [root@server ~]# cat /etc/init.d/tomcat6 #!/bin/bash # description: Tomcat Start Stop Restart # processname: tomcat # chkconfig: 234 20 80 JAVA_HOME=/usr/java/default export JAVA_HOME PATH=$JAVA_HOME/bin:$PATH export PATH CATALINA_HOME=/usr/share/apache-tomcat-6.0.37 case $1 in start) sh $CATALINA_HOME/bin/startup.sh ;; stop) sh $CATALINA_HOME/bin/shutdown.sh ;; restart) sh $CATALINA_HOME/bin/shutdown.sh sh $CATALINA_HOME/bin/startup.sh ;; esac exit 0 As you said, I removed the JAVA_OPTS declaration from .bash_profile and I have created setenv.sh and set JAVA_OPTS in it. [root@server bin]# pwd /usr/share/apache-tomcat-6.0.37/bin [root@server bin]# cat setenv.sh export JAVA_OPTS='-Xms256m -Xmx512m -Xss512k -verbose:gc -Xloggc:heap.log' Now it shows in grep. [root@server bin]# ps aux | grep Bootstrap root 1100 33.7 5.4 707052 95240 pts/0Sl 11:29 0:06 /usr/java/default/bin/java -Djava.util.logging.config.file=/usr/share/apache-tomcat- 6.0.37/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms256m -Xmx512m -Xss512k -verbose:gc -Xloggc:heap.log -Djava.endorsed.dirs=/usr/share/apache-tomcat-6.0.37/endorsed -classpath /usr/share/apache-tomcat-6.0.37/bin/bootstrap.jar -Dcatalina.base=/usr/share/apache-tomcat-6.0.37 -Dcatalina.home=/usr/share/apache-tomcat-6.0.37 -Djava.io.tmpdir=/usr/share/apache-tomcat-6.0.37/temp org.apache.catalina.startup.Bootstrap start root 1158 0.0 0.0 3932 692 pts/0R+ 11:29 0:00 grep Bootstrap [root@server bin]# But nothing came in heap.log. So I better monitor this and if I get the OOM again I have to increase Max limit! Maybe you should provide a full path for the -Xloggc option ? On Thu, Jan 23, 2014 at 8:50 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Randeep, On 1/23/14, 7:43 AM, Randeep wrote: I'm getting out of memory exception errors. :( Exception in thread Timer-1 java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2882) at java.lang.AbstractStringBuilder.expandCapacity( AbstractStringBuilder.java:100) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java: 572) at java.lang.StringBuffer.append(StringBuffer.java:320) at org.json.JSONArray.toString(JSONArray.java:834) at org.json.JSONObject.valueToString(JSONObject.java:1359) at org.json.JSONObject.toString(JSONObject.java:1233) at com.xxx.xxx.servlet.WServlet.writeProgramJson(WServlet.java:503) at
Re: Out of memory exception
On Jan 23, 2014, at 12:22 PM, Randeep randeep...@gmail.com wrote: I have added -XX:+HeapDumpOnOutOfMemoryError also. Thanks. Please stop top posting. Post your comments inline or at the bottom, like everyone else. It makes following the conversation much easier. Dan On Thu, Jan 23, 2014 at 10:49 PM, Randeep randeep...@gmail.com wrote: I mean i'd to give full path. like this export JAVA_OPTS='-Xms256m -Xmx512m -Xss512k -verbose:gc -Xloggc:/usr/share/apache-tomcat-6.0.37/logs/heap.log' On Thu, Jan 23, 2014 at 10:47 PM, Randeep randeep...@gmail.com wrote: Thanks. That worked. Anything unusual? Garbage collection is happening too often? [root@server logs]# cat /usr/share/apache-tomcat-6.0.37/logs/heap.log 1.910: [GC 16192K-1923K(260160K), 0.0150080 secs] 2.454: [GC 18115K-3151K(260160K), 0.0137140 secs] 2.760: [GC 19343K-3309K(260160K), 0.0058220 secs] 3.648: [GC 19501K-4446K(260160K), 0.0062870 secs] 4.000: [GC 20638K-5063K(260160K), 0.0080940 secs] 4.305: [GC 21255K-6302K(260160K), 0.0060890 secs] 4.598: [GC 22494K-6323K(260160K), 0.0062020 secs] 5.664: [GC 22515K-7098K(260160K), 0.0055070 secs] 6.980: [GC 23290K-8708K(260160K), 0.0176410 secs] 7.332: [GC 24900K-12045K(260160K), 0.0179400 secs] 7.794: [Full GC 24591K-12864K(260160K), 0.2538960 secs] 8.476: [GC 29184K-14200K(260288K), 0.0080560 secs] 8.939: [GC 30520K-15401K(260288K), 0.0407000 secs] 9.379: [GC 31721K-16897K(260288K), 0.0099850 secs] 9.603: [GC 33217K-16652K(260288K), 0.0649140 secs] 10.498: [GC 32972K-17768K(260288K), 0.0068540 secs] 10.885: [GC 34088K-18538K(260288K), 0.0084330 secs] 11.225: [GC 34858K-19584K(260288K), 0.0071730 secs] 11.466: [GC 35904K-20313K(260288K), 0.0050570 secs] 12.128: [GC 36633K-20565K(260288K), 0.0042210 secs] 12.385: [Full GC 26015K-20204K(260288K), 0.2847340 secs] 13.817: [GC 36524K-22174K(260288K), 0.0155690 secs] 14.127: [GC 38494K-25303K(260288K), 0.0816490 secs] 14.671: [GC 41623K-27150K(260288K), 0.0436660 secs] 15.173: [GC 43470K-28757K(260288K), 0.0140720 secs] 15.414: [Full GC 32030K-28926K(260288K), 0.3071990 secs] 65.004: [GC 45310K-31775K(260352K), 0.0105440 secs] [root@server logs]# On Thu, Jan 23, 2014 at 10:33 PM, André Warnier a...@ice-sa.com wrote: Randeep wrote: Hi all, Thanks a lot for valuable points. Here the information you guys asked and the changes I made. OS : CentOS release 5.4 (Final) httpd-2.2.3-65.el5.centos(front_end) +(mod_jk)+ apache-tomcat-6.0.37 I installed httpd using yum. Downloaded tomcat from Apache as compressed archive(tar.gz) I'm using following startup script. [root@server ~]# cat /etc/init.d/tomcat6 #!/bin/bash # description: Tomcat Start Stop Restart # processname: tomcat # chkconfig: 234 20 80 JAVA_HOME=/usr/java/default export JAVA_HOME PATH=$JAVA_HOME/bin:$PATH export PATH CATALINA_HOME=/usr/share/apache-tomcat-6.0.37 case $1 in start) sh $CATALINA_HOME/bin/startup.sh ;; stop) sh $CATALINA_HOME/bin/shutdown.sh ;; restart) sh $CATALINA_HOME/bin/shutdown.sh sh $CATALINA_HOME/bin/startup.sh ;; esac exit 0 As you said, I removed the JAVA_OPTS declaration from .bash_profile and I have created setenv.sh and set JAVA_OPTS in it. [root@server bin]# pwd /usr/share/apache-tomcat-6.0.37/bin [root@server bin]# cat setenv.sh export JAVA_OPTS='-Xms256m -Xmx512m -Xss512k -verbose:gc -Xloggc:heap.log' Now it shows in grep. [root@server bin]# ps aux | grep Bootstrap root 1100 33.7 5.4 707052 95240 pts/0Sl 11:29 0:06 /usr/java/default/bin/java -Djava.util.logging.config.file=/usr/share/apache-tomcat- 6.0.37/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms256m -Xmx512m -Xss512k -verbose:gc -Xloggc:heap.log -Djava.endorsed.dirs=/usr/share/apache-tomcat-6.0.37/endorsed -classpath /usr/share/apache-tomcat-6.0.37/bin/bootstrap.jar -Dcatalina.base=/usr/share/apache-tomcat-6.0.37 -Dcatalina.home=/usr/share/apache-tomcat-6.0.37 -Djava.io.tmpdir=/usr/share/apache-tomcat-6.0.37/temp org.apache.catalina.startup.Bootstrap start root 1158 0.0 0.0 3932 692 pts/0R+ 11:29 0:00 grep Bootstrap [root@server bin]# But nothing came in heap.log. So I better monitor this and if I get the OOM again I have to increase Max limit! Maybe you should provide a full path for the -Xloggc option ? On Thu, Jan 23, 2014 at 8:50 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Randeep, On 1/23/14, 7:43 AM, Randeep wrote: I'm getting out of memory exception errors. :( Exception in thread Timer-1 java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2882) at java.lang.AbstractStringBuilder.expandCapacity( AbstractStringBuilder.java:100) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java: 572) at
Re: Out of memory exception
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Randeep, On 1/23/14, 11:38 AM, Randeep wrote: Hi all, Thanks a lot for valuable points. Here the information you guys asked and the changes I made. OS : CentOS release 5.4 (Final) httpd-2.2.3-65.el5.centos(front_end) +(mod_jk)+ apache-tomcat-6.0.37 I installed httpd using yum. Downloaded tomcat from Apache as compressed archive(tar.gz) I'm using following startup script. [root@server ~]# cat /etc/init.d/tomcat6 #!/bin/bash # description: Tomcat Start Stop Restart # processname: tomcat # chkconfig: 234 20 80 JAVA_HOME=/usr/java/default export JAVA_HOME PATH=$JAVA_HOME/bin:$PATH export PATH CATALINA_HOME=/usr/share/apache-tomcat-6.0.37 case $1 in start) sh $CATALINA_HOME/bin/startup.sh ;; stop) sh $CATALINA_HOME/bin/shutdown.sh ;; restart) sh $CATALINA_HOME/bin/shutdown.sh sh $CATALINA_HOME/bin/startup.sh ;; esac exit 0 As you said, I removed the JAVA_OPTS declaration from .bash_profile and I have created setenv.sh and set JAVA_OPTS in it. [root@server bin]# pwd /usr/share/apache-tomcat-6.0.37/bin [root@server bin]# cat setenv.sh export JAVA_OPTS='-Xms256m -Xmx512m -Xss512k -verbose:gc -Xloggc:heap.log' Now it shows in grep. [root@server bin]# ps aux | grep Bootstrap root 1100 33.7 5.4 707052 95240 pts/0Sl 11:29 0:06 /usr/java/default/bin/java -Djava.util.logging.config.file=/usr/share/apache-tomcat-6.0.37/conf/logging.properties - -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms256m -Xmx512m -Xss512k -verbose:gc -Xloggc:heap.log -Djava.endorsed.dirs=/usr/share/apache-tomcat-6.0.37/endorsed -classpath /usr/share/apache-tomcat-6.0.37/bin/bootstrap.jar -Dcatalina.base=/usr/share/apache-tomcat-6.0.37 -Dcatalina.home=/usr/share/apache-tomcat-6.0.37 -Djava.io.tmpdir=/usr/share/apache-tomcat-6.0.37/temp org.apache.catalina.startup.Bootstrap start root 1158 0.0 0.0 3932 692 pts/0R+ 11:29 0:00 grep Bootstrap [root@server bin]# But nothing came in heap.log. You should use a full path in Xloggc:heap so you always know where the file is. As it stands, it will be in the CWD of the JVM process. Not sure where that is, because you can run startup.sh from anywhere. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJS4VJYAAoJEBzwKT+lPKRYhgwP/RZq8grS3/bFNdQL69AV05EF lRTQWinc6rC9Au3Nqw2u0coydpyspNYJemLJaRgTAN0NPlKMU1mZBcqOrFftnt7R nUffJeql5Zx2Ef5uT9NhPlLnJpwuSR3oMHk/hs6qiokZaLs5uqzCk/e1/JcH2qi6 Yjnuhfx5wFUO4S6ld0datFmChLFs6XySfhi+2CilFo2Kw4pkdM2Otnf8hD72FW6k zUGomd9AQw/Jj9/+EojyNYVO/dbcBF5Qa2mnt1PO1tE39x7G7TziQgY4My06MXXV xTrVkkPqYaKerA6JHK89GRHqZFOV6/acOZ8PE1BDQ6cb4EhY+FeyEI8CKo18kUDd XoUttISMIElnCpjbNPoxLDbt66CrUIb1gz8Xe2YxWFMib9V1Seq0xvmMarbUTXRg aeql1spWOEJJf3GR/qpu13AmPt9lTD2t0gDpY+MeB/guShJxm1pU3TEJ/V7FwxBn vJWfNI82RBHCzlUt0aGryYjw19bmgc3Z1st2GMU8WGiDeYFx6nrMUn1QzsWi7QuW k6ZVhkc0mdzXBVyeymWcZE87IKNvlwtOJF3rUpQkvSBxOw3ivLvDVILWLKaDxXCk ncPTFenz4ETNJ0bPQFcf3q45+Bb03YiifqTfNlnE5/+57fvd+J0INYIWlI91rsHP sjppv6K1BzrAe5LPvGTN =WyIX -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Out of memory exception
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Randeep, On 1/23/14, 12:17 PM, Randeep wrote: Thanks. That worked. Anything unusual? Garbage collection is happening too often? [root@server logs]# cat /usr/share/apache-tomcat-6.0.37/logs/heap.log 1.910: [GC 16192K-1923K(260160K), 0.0150080 secs] 2.454: [GC 18115K-3151K(260160K), 0.0137140 secs] 2.760: [GC 19343K-3309K(260160K), 0.0058220 secs] 3.648: [GC 19501K-4446K(260160K), 0.0062870 secs] 4.000: [GC 20638K-5063K(260160K), 0.0080940 secs] 4.305: [GC 21255K-6302K(260160K), 0.0060890 secs] 4.598: [GC 22494K-6323K(260160K), 0.0062020 secs] 5.664: [GC 22515K-7098K(260160K), 0.0055070 secs] 6.980: [GC 23290K-8708K(260160K), 0.0176410 secs] 7.332: [GC 24900K-12045K(260160K), 0.0179400 secs] 7.794: [Full GC 24591K-12864K(260160K), 0.2538960 secs] 8.476: [GC 29184K-14200K(260288K), 0.0080560 secs] 8.939: [GC 30520K-15401K(260288K), 0.0407000 secs] 9.379: [GC 31721K-16897K(260288K), 0.0099850 secs] 9.603: [GC 33217K-16652K(260288K), 0.0649140 secs] 10.498: [GC 32972K-17768K(260288K), 0.0068540 secs] 10.885: [GC 34088K-18538K(260288K), 0.0084330 secs] 11.225: [GC 34858K-19584K(260288K), 0.0071730 secs] 11.466: [GC 35904K-20313K(260288K), 0.0050570 secs] 12.128: [GC 36633K-20565K(260288K), 0.0042210 secs] 12.385: [Full GC 26015K-20204K(260288K), 0.2847340 secs] 13.817: [GC 36524K-22174K(260288K), 0.0155690 secs] 14.127: [GC 38494K-25303K(260288K), 0.0816490 secs] 14.671: [GC 41623K-27150K(260288K), 0.0436660 secs] 15.173: [GC 43470K-28757K(260288K), 0.0140720 secs] 15.414: [Full GC 32030K-28926K(260288K), 0.3071990 secs] 65.004: [GC 45310K-31775K(260352K), 0.0105440 secs] [root@server logs]# Do what I do: write a Perl script to pull-apart the data from above into something useful, like CSV, and then use a spreadsheet to graph the data. Graphing heap data *really* helps. Looking at numbers is ... useless. - -chris On Thu, Jan 23, 2014 at 10:33 PM, André Warnier a...@ice-sa.com wrote: Randeep wrote: Hi all, Thanks a lot for valuable points. Here the information you guys asked and the changes I made. OS : CentOS release 5.4 (Final) httpd-2.2.3-65.el5.centos(front_end) +(mod_jk)+ apache-tomcat-6.0.37 I installed httpd using yum. Downloaded tomcat from Apache as compressed archive(tar.gz) I'm using following startup script. [root@server ~]# cat /etc/init.d/tomcat6 #!/bin/bash # description: Tomcat Start Stop Restart # processname: tomcat # chkconfig: 234 20 80 JAVA_HOME=/usr/java/default export JAVA_HOME PATH=$JAVA_HOME/bin:$PATH export PATH CATALINA_HOME=/usr/share/apache-tomcat-6.0.37 case $1 in start) sh $CATALINA_HOME/bin/startup.sh ;; stop) sh $CATALINA_HOME/bin/shutdown.sh ;; restart) sh $CATALINA_HOME/bin/shutdown.sh sh $CATALINA_HOME/bin/startup.sh ;; esac exit 0 As you said, I removed the JAVA_OPTS declaration from .bash_profile and I have created setenv.sh and set JAVA_OPTS in it. [root@server bin]# pwd /usr/share/apache-tomcat-6.0.37/bin [root@server bin]# cat setenv.sh export JAVA_OPTS='-Xms256m -Xmx512m -Xss512k -verbose:gc -Xloggc:heap.log' Now it shows in grep. [root@server bin]# ps aux | grep Bootstrap root 1100 33.7 5.4 707052 95240 pts/0Sl 11:29 0:06 /usr/java/default/bin/java -Djava.util.logging.config.file=/usr/share/apache-tomcat- 6.0.37/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager - -Xms256m -Xmx512m -Xss512k -verbose:gc -Xloggc:heap.log -Djava.endorsed.dirs=/usr/share/apache-tomcat-6.0.37/endorsed -classpath /usr/share/apache-tomcat-6.0.37/bin/bootstrap.jar -Dcatalina.base=/usr/share/apache-tomcat-6.0.37 -Dcatalina.home=/usr/share/apache-tomcat-6.0.37 -Djava.io.tmpdir=/usr/share/apache-tomcat-6.0.37/temp org.apache.catalina.startup.Bootstrap start root 1158 0.0 0.0 3932 692 pts/0R+ 11:29 0:00 grep Bootstrap [root@server bin]# But nothing came in heap.log. So I better monitor this and if I get the OOM again I have to increase Max limit! Maybe you should provide a full path for the -Xloggc option ? On Thu, Jan 23, 2014 at 8:50 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Randeep, On 1/23/14, 7:43 AM, Randeep wrote: I'm getting out of memory exception errors. :( Exception in thread Timer-1 java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2882) at java.lang.AbstractStringBuilder.expandCapacity( AbstractStringBuilder.java:100) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572) at java.lang.StringBuffer.append(StringBuffer.java:320) at org.json.JSONArray.toString(JSONArray.java:834) at org.json.JSONObject.valueToString(JSONObject.java:1359) at org.json.JSONObject.toString(JSONObject.java:1233) at com.xxx.xxx.servlet.WServlet.writeProgramJson(WServlet.java:503) at
Re: Out of memory exception
Doing anything as root conceals the errors. Tomcat is no exception and changing it to a real user makes your testing complete. On Thursday, January 23, 2014 12:36 PM, Daniel Mikusa dmik...@gopivotal.com wrote: On Jan 23, 2014, at 11:38 AM, Randeep randeep...@gmail.com wrote: Hi all, Thanks a lot for valuable points. Here the information you guys asked and the changes I made. OS : CentOS release 5.4 (Final) httpd-2.2.3-65.el5.centos(front_end) +(mod_jk)+ apache-tomcat-6.0.37 I installed httpd using yum. Downloaded tomcat from Apache as compressed archive(tar.gz) I'm using following startup script. [root@server ~]# cat /etc/init.d/tomcat6 #!/bin/bash # description: Tomcat Start Stop Restart # processname: tomcat # chkconfig: 234 20 80 JAVA_HOME=/usr/java/default export JAVA_HOME PATH=$JAVA_HOME/bin:$PATH export PATH CATALINA_HOME=/usr/share/apache-tomcat-6.0.37 case $1 in start) sh $CATALINA_HOME/bin/startup.sh ;; stop) sh $CATALINA_HOME/bin/shutdown.sh ;; restart) sh $CATALINA_HOME/bin/shutdown.sh sh $CATALINA_HOME/bin/startup.sh ;; esac exit 0 As you said, I removed the JAVA_OPTS declaration from .bash_profile and I have created setenv.sh and set JAVA_OPTS in it. [root@server bin]# pwd /usr/share/apache-tomcat-6.0.37/bin [root@server bin]# cat setenv.sh export JAVA_OPTS='-Xms256m -Xmx512m -Xss512k -verbose:gc -Xloggc:heap.log’ Based on the name of the file you’re listing, it makes me think you’re confusing what this option does. It’s a garbage collection log. While that shows some info about the heap, it’s mainly targeted towards knowing what’s happening with GC in your process. I can’t help but think you might be looking for -XX:+HeapDumpOnOutOfMemoryError, which generates a heap dump when you get an OOME. Now it shows in grep. [root@server bin]# ps aux | grep Bootstrap root 1100 33.7 5.4 707052 95240 pts/0 Sl 11:29 0:06 /usr/java/default/bin/java -Djava.util.logging.config.file=/usr/share/apache-tomcat-6.0.37/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms256m -Xmx512m -Xss512k -verbose:gc -Xloggc:heap.log -Djava.endorsed.dirs=/usr/share/apache-tomcat-6.0.37/endorsed -classpath /usr/share/apache-tomcat-6.0.37/bin/bootstrap.jar -Dcatalina.base=/usr/share/apache-tomcat-6.0.37 -Dcatalina.home=/usr/share/apache-tomcat-6.0.37 -Djava.io.tmpdir=/usr/share/apache-tomcat-6.0.37/temp org.apache.catalina.startup.Bootstrap start root 1158 0.0 0.0 3932 692 pts/0 R+ 11:29 0:00 grep Bootstrap [root@server bin]# But nothing came in heap.log. Try using this option instead: -Xloggc:$CATALINA_BASE/logs/heap.log” That should put the file in your “logs” directory. Dan So I better monitor this and if I get the OOM again I have to increase Max limit! On Thu, Jan 23, 2014 at 8:50 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Randeep, On 1/23/14, 7:43 AM, Randeep wrote: I'm getting out of memory exception errors. :( Exception in thread Timer-1 java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2882) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572) at java.lang.StringBuffer.append(StringBuffer.java:320) at org.json.JSONArray.toString(JSONArray.java:834) at org.json.JSONObject.valueToString(JSONObject.java:1359) at org.json.JSONObject.toString(JSONObject.java:1233) at com.xxx.xxx.servlet.WServlet.writeProgramJson(WServlet.java:503) at com.xxx.xxx.servlet.WServlet$1.run(WServlet.java:414) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) If you are trying to serialize huge JSON objects, this is going to happen. I'm not sure if JSON is as smart as the Java Serialization engine, but if you have a circular-reference in your object graph, I'd be willing to bet that the JSON serializer will run forever and exhaust your heap. It's also possible that you simply have a tiny heap and need more, especially if you aren't currently setting any heap parameters. Is there anyway to see currently set jvm heap size details? Any linux commands. You can attach any profiler to your app. Good luck catching this in the act, though, in production. You can also use jmap -heap which comes with the JDK. Just point it at a process and you'll get a heap summary from a running JVM. Make sure your JVM and jmap version match. I have tried creating setenv.sh and putting the new parameters in it. but echo $JAVA_OPTS was not giving any value. setenv.sh is run during Tomcat startup. Running echo $JAVA_OPTS from the command-line isn't going to reveal anything. What you want to do is modify setenv.sh, re-launch Tomcat, and then look at the process command line... something
Re: Out of Memory exception (hotspot compilation)
On Thu, Jun 3, 2010 at 1:54 AM, Leon Kolchinsky lkolc...@gmail.com wrote: Yep :) You're right Charles. Thanks, Leon From Leon To Leon, an advice from personal experience and the golden rule of the tomcat user list, write it down in glowing red letters on the wall you are looking at most: Charles is ALWAYS right. ;-) regards Leon - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Out of Memory exception (hotspot compilation)
Hello, Just wanted to update on this issue. Atlassian support confirms that this is a known issue with Java 6.0_18 and up. So, the best solution for now is to use 6.0_17 Here - http://jira.atlassian.com/browse/CONF-19049http://jira.atlassian.com/browse/CONF-19049?focusedCommentId=197013#action_197013 Regards, Leon Kolchinsky On Thu, Jun 3, 2010 at 16:14, Leon Rosenberg rosenberg.l...@gmail.comwrote: On Thu, Jun 3, 2010 at 1:54 AM, Leon Kolchinsky lkolc...@gmail.com wrote: Yep :) You're right Charles. Thanks, Leon From Leon To Leon, an advice from personal experience and the golden rule of the tomcat user list, write it down in glowing red letters on the wall you are looking at most: Charles is ALWAYS right. ;-) regards Leon - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Out of Memory exception (hotspot compilation)
From: Leon Kolchinsky [mailto:lkolc...@gmail.com] Subject: Re: Out of Memory exception (hotspot compilation) I'd like to file a bug report on similar issue I had. Can you please tell me where can I do it? Let's see what your original post had in it... Yesterday I've got this error in catalina.out and Tomcat died (I'm attaching hs_err_pid5460.log file): # # A fatal error has been detected by the Java Runtime Environment: # # java.lang.OutOfMemoryError: requested 2147483664 bytes for Chunk::new. Out of swap space? # # Internal Error (allocation.cpp:215), pid=5460, tid=1085901120 # Error: Chunk::new # # JRE version: 6.0_18-b07 # Java VM: Java HotSpot(TM) 64-Bit Server VM (16.0-b13 mixed mode linux-amd64 ) # An error report file with more information is saved as: # /tmp/hs_err_pid5460.log # # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp Does that give you a hint? - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Out of Memory exception (hotspot compilation)
Yep :) You're right Charles. Thanks, Leon On Wed, Jun 2, 2010 at 16:27, Caldarale, Charles R chuck.caldar...@unisys.com wrote: From: Leon Kolchinsky [mailto:lkolc...@gmail.com] Subject: Re: Out of Memory exception (hotspot compilation) I'd like to file a bug report on similar issue I had. Can you please tell me where can I do it? Let's see what your original post had in it... Yesterday I've got this error in catalina.out and Tomcat died (I'm attaching hs_err_pid5460.log file): # # A fatal error has been detected by the Java Runtime Environment: # # java.lang.OutOfMemoryError: requested 2147483664 bytes for Chunk::new. Out of swap space? # # Internal Error (allocation.cpp:215), pid=5460, tid=1085901120 # Error: Chunk::new # # JRE version: 6.0_18-b07 # Java VM: Java HotSpot(TM) 64-Bit Server VM (16.0-b13 mixed mode linux-amd64 ) # An error report file with more information is saved as: # /tmp/hs_err_pid5460.log # # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp Does that give you a hint? - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Out of Memory exception (hotspot compilation)
From: Will Glass-Husain [mailto:wglasshus...@gmail.com] Subject: Out of Memory exception (hotspot compilation) Specifically, the JVM requested 8GB of memory while hotspot was trying to compile the exact same Velocity/Foreach method as Leon's example. I'm running 64 bit JDK 1.6.0_18-b07 Since it's the exact same method, I'm guessing the Velocity method triggers a bug in the hotspot compilation. Note that we've been running this configuration for 5 months Using the same JDK for those 5 months, or did you recently upgrade? - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Out of Memory exception (hotspot compilation)
The JDK has been the same since early February. I don't think there's been any other system upgrades. (it's possible there was a upgrade to the OS I'm not aware of, but definitely not the JDK). WILL On Tue, Jun 1, 2010 at 12:45 PM, Caldarale, Charles R chuck.caldar...@unisys.com wrote: From: Will Glass-Husain [mailto:wglasshus...@gmail.com] Subject: Out of Memory exception (hotspot compilation) Specifically, the JVM requested 8GB of memory while hotspot was trying to compile the exact same Velocity/Foreach method as Leon's example. I'm running 64 bit JDK 1.6.0_18-b07 Since it's the exact same method, I'm guessing the Velocity method triggers a bug in the hotspot compilation. Note that we've been running this configuration for 5 months Using the same JDK for those 5 months, or did you recently upgrade? - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Out of Memory exception (hotspot compilation)
Hmm, I'd like to file a bug report on similar issue I had. Can you please tell me where can I do it? Cheers, Leon On Wed, Jun 2, 2010 at 06:46, Will Glass-Husain wglasshus...@gmail.comwrote: The JDK has been the same since early February. I don't think there's been any other system upgrades. (it's possible there was a upgrade to the OS I'm not aware of, but definitely not the JDK). WILL On Tue, Jun 1, 2010 at 12:45 PM, Caldarale, Charles R chuck.caldar...@unisys.com wrote: From: Will Glass-Husain [mailto:wglasshus...@gmail.com] Subject: Out of Memory exception (hotspot compilation) Specifically, the JVM requested 8GB of memory while hotspot was trying to compile the exact same Velocity/Foreach method as Leon's example. I'm running 64 bit JDK 1.6.0_18-b07 Since it's the exact same method, I'm guessing the Velocity method triggers a bug in the hotspot compilation. Note that we've been running this configuration for 5 months Using the same JDK for those 5 months, or did you recently upgrade? - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
