Re: [OT] OutOfMemoryError: GC overhead limit exceeded
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Leon, On 2/17/2011 2:29 AM, Leon Rosenberg wrote: On Wed, Feb 16, 2011 at 6:16 PM, Christopher Schultz wrote: We're working on some decent instrumentation tools to basically log all kinds of stuff like this on a regular interval... say every minute or five minutes. I could connect to the JVM using JConsole and ask Tomcat for the session count (all users have a session) but the event is long gone so any information I get now would not be directly relevant. reinventing the wheel? :-) http://moskito.anotheria.net among others... I should have been more clear: we're working on instrumenting our app... not necessarily writing the instrumentation ourselves. Does Moskito do simple things like log at intervals? Or do you have to view things as HTML-rendered reports? I want something I can process with command-line tools and ultimately feed into JasperReports. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk1e4AsACgkQ9CaO5/Lv0PAsVgCgjFaEl6BabeVUvnZjBwOJxZaW i9EAnjURNw73h5zyum7sEz2RPdepPNgY =7igZ -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] OutOfMemoryError: GC overhead limit exceeded
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chuck, On 2/15/2011 5:57 PM, Caldarale, Charles R wrote: From: Christopher Schultz [mailto:ch...@christopherschultz.net] Subject: [OT] OutOfMemoryError: GC overhead limit exceeded This error occurred /once/ at 16:39 EDT and it's 17:39 EDT now Is it likely that this was something acute and that my server is okay? PS Old Generation capacity = 134217728 (128.0MB) used = 129355080 (123.36261749267578MB) free = 4862648 (4.637382507324219MB) 96.37704491615295% used The above shows that the problem persists, even though at least some work is getting done. With multiple cores available, you may be able to get away with a saturated heap for a while, but a server restart is in order at your earliest convenience. But before doing that, try running jhat or equivalent to find out what's eating up the space. It seems that jmap won't do anything (like -histo) besides the heap map without using the -F switch. When I use the -F switch, the whole JVM pauses while it does the heap walk. I killed the jmap process and everything continued. I'll have to wait until off-hours to take it's temperature and bounce the server. On the other hand, I've invoked System.gc which ran quickly and freed up a lot of memory. Here's what jmap reports this morning: Heap Configuration: MinHeapFreeRatio = 40 MaxHeapFreeRatio = 70 MaxHeapSize = 201326592 (192.0MB) NewSize = 1048576 (1.0MB) MaxNewSize = 4294901760 (4095.9375MB) OldSize = 4194304 (4.0MB) NewRatio = 2 SurvivorRatio= 8 PermSize = 16777216 (16.0MB) MaxPermSize = 67108864 (64.0MB) Heap Usage: PS Young Generation Eden Space: capacity = 54263808 (51.75MB) used = 35766400 (34.1094970703125MB) free = 18497408 (17.6405029296875MB) 65.9120716334541% used - From Space: capacity = 6094848 (5.8125MB) used = 5233120 (4.990692138671875MB) free = 861728 (0.821807861328125MB) 85.86137012768818% used To Space: capacity = 6422528 (6.125MB) used = 0 (0.0MB) free = 6422528 (6.125MB) 0.0% used PS Old Generation capacity = 134217728 (128.0MB) used = 61911720 (59.043617248535156MB) free = 72306008 (68.95638275146484MB) 46.12782597541809% used PS Perm Generation capacity = 20578304 (19.625MB) used = 20152712 (19.21912384033203MB) free = 425592 (0.40587615966796875MB) 97.93184122462182% used Looks like I've got a lot more wiggle room, now, in the Old generation. Does this look healthy enough to continue, or should I schedule a restart ASAP? I'd prefer to wait until off hours. We do have a couple of operations that foolishly load way too much information into memory all at once and cache it in the session (oops), so I suspect one (or more) of those operations was performed and the objects were reachable for a while (lived in the session, probably) but then eventually became collectible and GC recovered. Quick question: does anyone know if it's safe to truncate catalina.out while Tomcat is running? Something like this: cp catalina.out catalina.out.old catalina.out ?? I'd like to stop getting my the sky is falling emails that are triggered by the presence of the string OutOfMemoryError in catalina.out :) Thanks, - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk1b9BcACgkQ9CaO5/Lv0PAFAACbB0afje/4WLCjgE49UnkbpWG6 OmMAoIMei7iQFVZTzZhIDcnb80W0HI8K =yPoC -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: [OT] OutOfMemoryError: GC overhead limit exceeded
From: Christopher Schultz [mailto:ch...@christopherschultz.net] Subject: Re: [OT] OutOfMemoryError: GC overhead limit exceeded Here's what jmap reports this morning: PS Old Generation capacity = 134217728 (128.0MB) used = 61911720 (59.043617248535156MB) free = 72306008 (68.95638275146484MB) 46.12782597541809% used Interesting. Something must have broken loose overnight. Does this look healthy enough to continue, or should I schedule a restart ASAP? I'd let it run, and see what happens. I suspect one (or more) of those operations was performed and the objects were reachable for a while (lived in the session, probably) but then eventually became collectible and GC recovered. Sounds highly plausible. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
Re: [OT] OutOfMemoryError: GC overhead limit exceeded
On Wed, Feb 16, 2011 at 5:37 PM, Caldarale, Charles R chuck.caldar...@unisys.com wrote: From: Christopher Schultz [mailto:ch...@christopherschultz.net] Subject: Re: [OT] OutOfMemoryError: GC overhead limit exceeded Here's what jmap reports this morning: PS Old Generation capacity = 134217728 (128.0MB) used = 61911720 (59.043617248535156MB) free = 72306008 (68.95638275146484MB) 46.12782597541809% used Interesting. Something must have broken loose overnight. Does this look healthy enough to continue, or should I schedule a restart ASAP? I'd let it run, and see what happens. I suspect one (or more) of those operations was performed and the objects were reachable for a while (lived in the session, probably) but then eventually became collectible and GC recovered. Sounds highly plausible. Do you log number of users/sessions? Would probably be interesting to see this in correlation. Leon - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] OutOfMemoryError: GC overhead limit exceeded
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Leon, On 2/16/2011 11:53 AM, Leon Rosenberg wrote: On Wed, Feb 16, 2011 at 5:37 PM, Caldarale, Charles R chuck.caldar...@unisys.com wrote: From: Christopher Schultz [mailto:ch...@christopherschultz.net] Subject: Re: [OT] OutOfMemoryError: GC overhead limit exceeded Here's what jmap reports this morning: PS Old Generation capacity = 134217728 (128.0MB) used = 61911720 (59.043617248535156MB) free = 72306008 (68.95638275146484MB) 46.12782597541809% used Interesting. Something must have broken loose overnight. Does this look healthy enough to continue, or should I schedule a restart ASAP? I'd let it run, and see what happens. I suspect one (or more) of those operations was performed and the objects were reachable for a while (lived in the session, probably) but then eventually became collectible and GC recovered. Sounds highly plausible. Do you log number of users/sessions? Would probably be interesting to see this in correlation. Not currently. We're working on some decent instrumentation tools to basically log all kinds of stuff like this on a regular interval... say every minute or five minutes. I could connect to the JVM using JConsole and ask Tomcat for the session count (all users have a session) but the event is long gone so any information I get now would not be directly relevant. Thanks, - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk1cBnUACgkQ9CaO5/Lv0PCJRwCgo+ssFInOJ53tJQrO1ZdNhFPk UW0An1sKMuCkckA8Rjwzu23xM0CmxOAY =jBKF -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] OutOfMemoryError: GC overhead limit exceeded
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chuck, On 2/16/2011 11:37 AM, Caldarale, Charles R wrote: From: Christopher Schultz [mailto:ch...@christopherschultz.net] Subject: Re: [OT] OutOfMemoryError: GC overhead limit exceeded Here's what jmap reports this morning: PS Old Generation capacity = 134217728 (128.0MB) used = 61911720 (59.043617248535156MB) free = 72306008 (68.95638275146484MB) 46.12782597541809% used Interesting. Something must have broken loose overnight. Definitely. It may have broken loose much earlier... I just didn't observe it again after my initial post last night. Does this look healthy enough to continue, or should I schedule a restart ASAP? I'd let it run, and see what happens. :) Should I be concerned that neither jstack not jmap will run against the JVM unless I specify the -F switch? Thanks, - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk1cBq8ACgkQ9CaO5/Lv0PAxYQCfTQdvZfmMjdewR51Ma8G99mk2 Y34AoJ1RLeEXytG/3zxC2IVH+GXqcWq/ =JXzi -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] OutOfMemoryError: GC overhead limit exceeded
Op woensdag, 16 februari 2011 18:17 schreef Christopher Schultz ch...@christopherschultz.net: Chuck, On 2/16/2011 11:37 AM, Caldarale, Charles R wrote: From: Christopher Schultz [mailto:ch...@christopherschultz.net] Subject: Re: [OT] OutOfMemoryError: GC overhead limit exceeded Here's what jmap reports this morning: PS Old Generation capacity = 134217728 (128.0MB) used = 61911720 (59.043617248535156MB) free = 72306008 (68.95638275146484MB) 46.12782597541809% used Interesting. Something must have broken loose overnight. Definitely. It may have broken loose much earlier... I just didn't observe it again after my initial post last night. Does this look healthy enough to continue, or should I schedule a restart ASAP? I'd let it run, and see what happens. :) Should I be concerned that neither jstack not jmap will run against the JVM unless I specify the -F switch? Thanks, - -chris I see the same problem with jstack and friends if a process is gc'ing a lot. Ronald.
Re: [OT] OutOfMemoryError: GC overhead limit exceeded
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ronald, On 2/16/2011 12:48 PM, Ronald Klop wrote: I see the same problem with jstack and friends if a process is gc'ing a lot. Interesting. I don't have verbose GC enabled (can you enable that after JVM launch using JConsole or something like that?), but the process doesn't seem to be using a ton of CPU time which I would expect from a very active GC. Do you observe this only when GC is /currently/ active or after there has been a lot of GC activity. I'm wondering if the JVM is in a state where everything is working properly except that one small things has broken and prevents these tools from working without sprinkling the force magic onto them. Thanks, - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk1cFPkACgkQ9CaO5/Lv0PB9fACfblGqULi4gJNnOD+PdvWAKlz+ 2hkAn2kWKXb+FZiVatOM/TaL3BCOGam2 =rECB -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] OutOfMemoryError: GC overhead limit exceeded
Op woensdag, 16 februari 2011 19:18 schreef Christopher Schultz ch...@christopherschultz.net: Ronald, On 2/16/2011 12:48 PM, Ronald Klop wrote: I see the same problem with jstack and friends if a process is gc'ing a lot. Interesting. I don't have verbose GC enabled (can you enable that after JVM launch using JConsole or something like that?), but the process doesn't seem to be using a ton of CPU time which I would expect from a very active GC. Do you observe this only when GC is /currently/ active or after there has been a lot of GC activity. I'm wondering if the JVM is in a state where everything is working properly except that one small things has broken and prevents these tools from working without sprinkling the force magic onto them. Thanks, - -chris Where it is /currently/ active. I don't do much jstack/jmap if there is no problem, so can't really say something about it. Ronald.
Re: [OT] OutOfMemoryError: GC overhead limit exceeded
On Wed, Feb 16, 2011 at 6:16 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Leon, On 2/16/2011 11:53 AM, Leon Rosenberg wrote: On Wed, Feb 16, 2011 at 5:37 PM, Caldarale, Charles R chuck.caldar...@unisys.com wrote: From: Christopher Schultz [mailto:ch...@christopherschultz.net] Subject: Re: [OT] OutOfMemoryError: GC overhead limit exceeded Here's what jmap reports this morning: PS Old Generation capacity = 134217728 (128.0MB) used = 61911720 (59.043617248535156MB) free = 72306008 (68.95638275146484MB) 46.12782597541809% used Interesting. Something must have broken loose overnight. Does this look healthy enough to continue, or should I schedule a restart ASAP? I'd let it run, and see what happens. I suspect one (or more) of those operations was performed and the objects were reachable for a while (lived in the session, probably) but then eventually became collectible and GC recovered. Sounds highly plausible. Do you log number of users/sessions? Would probably be interesting to see this in correlation. Not currently. We're working on some decent instrumentation tools to basically log all kinds of stuff like this on a regular interval... say every minute or five minutes. I could connect to the JVM using JConsole and ask Tomcat for the session count (all users have a session) but the event is long gone so any information I get now would not be directly relevant. reinventing the wheel? :-) http://moskito.anotheria.net among others... best regards Leon Thanks, - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk1cBnUACgkQ9CaO5/Lv0PCJRwCgo+ssFInOJ53tJQrO1ZdNhFPk UW0An1sKMuCkckA8Rjwzu23xM0CmxOAY =jBKF -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
[OT] OutOfMemoryError: GC overhead limit exceeded
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 All, This is not Tomcat-related but I figured I'd ask as there are some smart folks here who might be able to give me some good information. As the title suggests, I have suffered an OOME in production with the message GC overhead limit exceeded. A quick Google search yielded this StackOverflow thread: http://stackoverflow.com/questions/1393486/what-means-the-error-message-java-lang-outofmemoryerror-gc-overhead-limit-excee Evidently, when the GC takes a long time to accomplish nothing, you get this error. This error occurred /once/ at 16:39 EDT and it's 17:39 EDT now and I can see the following heap information from my still-running process: Maximum Memory JVM will use: 190775296 bytes (186304 KiB, 181 MiB) Memory Not yet Allocated to JVM: 0 bytes (0 KiB, 0 MiB) Total Memory Allocated to JVM: 190775296 bytes (186304 KiB, 181 MiB) Free Memory: 21809336 (21298 KiB, 20 MiB) Memory Used by JVM: 168965960 (165005 KiB, 161 MiB) The application appears to be running as expected for now. Is it likely that this was something acute and that my server is okay? Obviously, increasing heap space is probably a good idea in general in this situation as it looks like I simply need more for my user load. Unfortunately, I haven't been keeping GC or other heap records for a while... I'll certainly be restarting that process. The only heap/GC options I've set are: - -Xms192M -Xmx192M All else is the default for this JVM, which is: java version 1.6.0_23 Java(TM) SE Runtime Environment (build 1.6.0_23-b05) Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode) FYI jmap reports: Server compiler detected. JVM version is 19.0-b09 using thread-local object allocation. Parallel GC with 4 thread(s) Heap Configuration: MinHeapFreeRatio = 40 MaxHeapFreeRatio = 70 MaxHeapSize = 201326592 (192.0MB) NewSize = 1048576 (1.0MB) MaxNewSize = 4294901760 (4095.9375MB) OldSize = 4194304 (4.0MB) NewRatio = 2 SurvivorRatio= 8 PermSize = 16777216 (16.0MB) MaxPermSize = 67108864 (64.0MB) Heap Usage: PS Young Generation Eden Space: capacity = 45613056 (43.5MB) used = 15735344 (15.006393432617188MB) free = 29877712 (28.493606567382812MB) 34.49745616693606% used - From Space: capacity = 10551296 (10.0625MB) used = 2172200 (2.0715713500976562MB) free = 8379096 (7.990928649902344MB) 20.587044473020185% used To Space: capacity = 10158080 (9.6875MB) used = 0 (0.0MB) free = 10158080 (9.6875MB) 0.0% used PS Old Generation capacity = 134217728 (128.0MB) used = 129355080 (123.36261749267578MB) free = 4862648 (4.637382507324219MB) 96.37704491615295% used PS Perm Generation capacity = 20316160 (19.375MB) used = 20088520 (19.15790557861328MB) free = 227640 (0.21709442138671875MB) 98.87951266381049% used Any thoughts? Thanks, - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk1bAYEACgkQ9CaO5/Lv0PA9dwCfdmJta/0xqhoHOgFaRlezGl7p 52YAn0QYEK348ZydsLD8TBMpSTnJU/IG =AbM9 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] OutOfMemoryError: GC overhead limit exceeded
Your old is full. Either a memory leak or you simply need more (and old gen of 128Mb is virtually nothing). Increase :-) regards Leon On Tue, Feb 15, 2011 at 11:43 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 All, This is not Tomcat-related but I figured I'd ask as there are some smart folks here who might be able to give me some good information. As the title suggests, I have suffered an OOME in production with the message GC overhead limit exceeded. A quick Google search yielded this StackOverflow thread: http://stackoverflow.com/questions/1393486/what-means-the-error-message-java-lang-outofmemoryerror-gc-overhead-limit-excee Evidently, when the GC takes a long time to accomplish nothing, you get this error. This error occurred /once/ at 16:39 EDT and it's 17:39 EDT now and I can see the following heap information from my still-running process: Maximum Memory JVM will use: 190775296 bytes (186304 KiB, 181 MiB) Memory Not yet Allocated to JVM: 0 bytes (0 KiB, 0 MiB) Total Memory Allocated to JVM: 190775296 bytes (186304 KiB, 181 MiB) Free Memory: 21809336 (21298 KiB, 20 MiB) Memory Used by JVM: 168965960 (165005 KiB, 161 MiB) The application appears to be running as expected for now. Is it likely that this was something acute and that my server is okay? Obviously, increasing heap space is probably a good idea in general in this situation as it looks like I simply need more for my user load. Unfortunately, I haven't been keeping GC or other heap records for a while... I'll certainly be restarting that process. The only heap/GC options I've set are: - -Xms192M -Xmx192M All else is the default for this JVM, which is: java version 1.6.0_23 Java(TM) SE Runtime Environment (build 1.6.0_23-b05) Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode) FYI jmap reports: Server compiler detected. JVM version is 19.0-b09 using thread-local object allocation. Parallel GC with 4 thread(s) Heap Configuration: MinHeapFreeRatio = 40 MaxHeapFreeRatio = 70 MaxHeapSize = 201326592 (192.0MB) NewSize = 1048576 (1.0MB) MaxNewSize = 4294901760 (4095.9375MB) OldSize = 4194304 (4.0MB) NewRatio = 2 SurvivorRatio = 8 PermSize = 16777216 (16.0MB) MaxPermSize = 67108864 (64.0MB) Heap Usage: PS Young Generation Eden Space: capacity = 45613056 (43.5MB) used = 15735344 (15.006393432617188MB) free = 29877712 (28.493606567382812MB) 34.49745616693606% used - From Space: capacity = 10551296 (10.0625MB) used = 2172200 (2.0715713500976562MB) free = 8379096 (7.990928649902344MB) 20.587044473020185% used To Space: capacity = 10158080 (9.6875MB) used = 0 (0.0MB) free = 10158080 (9.6875MB) 0.0% used PS Old Generation capacity = 134217728 (128.0MB) used = 129355080 (123.36261749267578MB) free = 4862648 (4.637382507324219MB) 96.37704491615295% used PS Perm Generation capacity = 20316160 (19.375MB) used = 20088520 (19.15790557861328MB) free = 227640 (0.21709442138671875MB) 98.87951266381049% used Any thoughts? Thanks, - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk1bAYEACgkQ9CaO5/Lv0PA9dwCfdmJta/0xqhoHOgFaRlezGl7p 52YAn0QYEK348ZydsLD8TBMpSTnJU/IG =AbM9 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: [OT] OutOfMemoryError: GC overhead limit exceeded
From: Christopher Schultz [mailto:ch...@christopherschultz.net] Subject: [OT] OutOfMemoryError: GC overhead limit exceeded This error occurred /once/ at 16:39 EDT and it's 17:39 EDT now Is it likely that this was something acute and that my server is okay? PS Old Generation capacity = 134217728 (128.0MB) used = 129355080 (123.36261749267578MB) free = 4862648 (4.637382507324219MB) 96.37704491615295% used The above shows that the problem persists, even though at least some work is getting done. With multiple cores available, you may be able to get away with a saturated heap for a while, but a server restart is in order at your earliest convenience. But before doing that, try running jhat or equivalent to find out what's eating up the space. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
