Re: [OT] Re: jvm exits without trace
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Taylan, On 3/30/2010 5:51 AM, Taylan Develioglu wrote: We're still having some crashes (segfaults), they may or not be related to the previous ones. :( It seems libapr had something to do with it. [snip] # Problematic frame: # C [libapr-1.so.0+0x1c0a0] [snip] Stack: [0x41d86000,0x41da7000], sp=0x41da57d0, free space=7d0018k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) C [libapr-1.so.0+0x1c0a0] Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) j org.apache.tomcat.jni.Socket.accept(J)J+0 So, this happens in APR's Socket.accept method? I suspect that if something were wrong in that method, it would be apparent to roughly 50% of the world's web servers. Do you have large electromagnetic flux sources anywhere near your servers? Your problems just don't seem to make any kind of sense whatsoever. Can you bring a server to your home and hit it with JMeter for a week or so and see if you can get it to crash? Maybe you have toxic mold in your rack and the JVM is trying to commit suicide to put itself out of its misery... - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkuzqPgACgkQ9CaO5/Lv0PD7rgCcDWhOzY93YAxoRcGvoXcx9awM TbYAoIP641cecvi4SVpaW1ZW4nWnP/gS =IV6q -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: [OT] Re: jvm exits without trace
From: Christopher Schultz [mailto:ch...@christopherschultz.net] Subject: Re: [OT] Re: jvm exits without trace So, this happens in APR's Socket.accept method? I suspect that if something were wrong in that method, it would be apparent to roughly 50% of the world's web servers. Depends. Try turning off IPv6 and see what happens. Some versions of glibc on some platforms seem to have problems with it. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
Re: [OT] Re: jvm exits without trace
[snip] # Problematic frame: # C [libapr-1.so.0+0x1c0a0] [snip] Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) j org.apache.tomcat.jni.Socket.accept(J)J+0 So, this happens in APR's Socket.accept method? I suspect that if something were wrong in that method, it would be apparent to roughly 50% of the world's web servers. That is TC-Native's Socket.accept method. If that happens at shutdown, then that is a known issue: https://issues.apache.org/bugzilla/show_bug.cgi?id=48584 It is already fixed in native 1.1.20. Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Re: jvm exits without trace
2010/4/1 Konstantin Kolinko knst.koli...@gmail.com: [snip] # Problematic frame: # C [libapr-1.so.0+0x1c0a0] [snip] Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) j org.apache.tomcat.jni.Socket.accept(J)J+0 So, this happens in APR's Socket.accept method? I suspect that if something were wrong in that method, it would be apparent to roughly 50% of the world's web servers. That is TC-Native's Socket.accept method. Actually yet, it is some APR method, called from TC-Native's Socket.accept(). I was a bit confused that BZ 48584 mentions only tcnative-1.dll, but that is because APR + TC-Native are statically linked into single DLL in the Windows versions of TC-Native. The error should be happening in APR (because another thread already destroyed one of APR pools during shutdown -- see BZ 48584 for details). There are certainly some native call frames between Java class of org.apache.tomcat.jni.Socket and that native frame of libapr-1.so (because Java code calls its native counterpart, and not directly the APR), but those are not shown in the stack dump. Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Re: jvm exits without trace
On Tue, Mar 16, 2010 at 4:46 PM, Carl c...@etrak-plus.com wrote: My approach is to get something (a JVM) that works and then gradually change until it breaks. Then, I know what is causing the problem. To date, I haven't been able to get a JVM that works. I have had a lot of issues finding stable JVMs since I moved our software from 1.5 to 1.6... 1.6 has been a mess for ages under our workload on CentOS. Code that ran fine under 1.5 would segfault in all sorts of random places on 1.6. I even tried the 1.7 openJDK early builds... they were even worse. I was pleased to see that the most recent release http://java.sun.com/javase/6/webnotes/6u18.html contains _tons_ of bug fixes, including lots of crash fixes. Seriously... the last few releases have contained only a handful of fixes... this release has hundreds. I'm testing it now, and so far, it looks promising. This may be the first 1.6 release that I've found that doesn't crash with my workload. Unfortunately, I've never been able to pin down a sequence of events that would cause the crash on demand, so its hard for me to verify that things are finally fixed, other than doing long term load testing and waiting for the crashes to (hopefully) not happen. Dan - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Re: jvm exits without trace
Dan, 6u18 did not work for us, crashed with the same regularity as 6u17. However, 6u7 has been running for two weeks without a failure (the others would fail between 15 minutes and 10 days runtime.) Thanks, Carl - Original Message - From: Dan Armbrust daniel.armbrust.l...@gmail.com To: Tomcat Users List users@tomcat.apache.org Sent: Monday, March 22, 2010 1:17 PM Subject: Re: [OT] Re: jvm exits without trace On Tue, Mar 16, 2010 at 4:46 PM, Carl c...@etrak-plus.com wrote: My approach is to get something (a JVM) that works and then gradually change until it breaks. Then, I know what is causing the problem. To date, I haven't been able to get a JVM that works. I have had a lot of issues finding stable JVMs since I moved our software from 1.5 to 1.6... 1.6 has been a mess for ages under our workload on CentOS. Code that ran fine under 1.5 would segfault in all sorts of random places on 1.6. I even tried the 1.7 openJDK early builds... they were even worse. I was pleased to see that the most recent release http://java.sun.com/javase/6/webnotes/6u18.html contains _tons_ of bug fixes, including lots of crash fixes. Seriously... the last few releases have contained only a handful of fixes... this release has hundreds. I'm testing it now, and so far, it looks promising. This may be the first 1.6 release that I've found that doesn't crash with my workload. Unfortunately, I've never been able to pin down a sequence of events that would cause the crash on demand, so its hard for me to verify that things are finally fixed, other than doing long term load testing and waiting for the crashes to (hopefully) not happen. Dan - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: [OT] Re: jvm exits without trace
-Original Message- From: Dan Armbrust [mailto:daniel.armbrust.l...@gmail.com] Sent: Monday, March 22, 2010 12:17 PM To: Tomcat Users List Subject: Re: [OT] Re: jvm exits without trace On Tue, Mar 16, 2010 at 4:46 PM, Carl c...@etrak-plus.com wrote: My approach is to get something (a JVM) that works and then gradually change until it breaks. Then, I know what is causing the problem. To date, I haven't been able to get a JVM that works. I have had a lot of issues finding stable JVMs since I moved our software from 1.5 to 1.6... 1.6 has been a mess for ages under our workload on CentOS. Code that ran fine under 1.5 would segfault in all sorts of random places on 1.6. I even tried the 1.7 openJDK early builds... they were even worse. I was pleased to see that the most recent release http://java.sun.com/javase/6/webnotes/6u18.html contains _tons_ of bug fixes, including lots of crash fixes. Seriously... the last few releases have contained only a handful of fixes... this release has hundreds. I'm running 1.6.0_18 under OpenSuSE 11.0-11.2 and I've only had one problem. It looks like a GLIBC error related to IPV6. I disabled IPV6 on the machine and it's been rock solid since. I don't use native connectors, or AJP. I do about 900,000 hits with 4GB/Day spread across 3 servers and it's just rock steady. I run the tomcat instances for 2-3 weeks each before re-starting. For each machine, that's about 6 million hits with about 40 GB of transfer. George Sexton MH Software, Inc. 303 438-9585 www.mhsoftware.com - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: [OT] Re: jvm exits without trace
Carl- enable logging FileHandler and set level to finest e.g. org.apache.catalina.level=FINEST http://tomcat.apache.org/tomcat-6.0-doc/logging.html hth Martin Gainty __ Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen. Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni. From: c...@etrak-plus.com To: users@tomcat.apache.org Subject: Re: [OT] Re: jvm exits without trace Date: Mon, 22 Mar 2010 14:33:54 -0500 Dan, 6u18 did not work for us, crashed with the same regularity as 6u17. However, 6u7 has been running for two weeks without a failure (the others would fail between 15 minutes and 10 days runtime.) Thanks, Carl - Original Message - From: Dan Armbrust daniel.armbrust.l...@gmail.com To: Tomcat Users List users@tomcat.apache.org Sent: Monday, March 22, 2010 1:17 PM Subject: Re: [OT] Re: jvm exits without trace On Tue, Mar 16, 2010 at 4:46 PM, Carl c...@etrak-plus.com wrote: My approach is to get something (a JVM) that works and then gradually change until it breaks. Then, I know what is causing the problem. To date, I haven't been able to get a JVM that works. I have had a lot of issues finding stable JVMs since I moved our software from 1.5 to 1.6... 1.6 has been a mess for ages under our workload on CentOS. Code that ran fine under 1.5 would segfault in all sorts of random places on 1.6. I even tried the 1.7 openJDK early builds... they were even worse. I was pleased to see that the most recent release http://java.sun.com/javase/6/webnotes/6u18.html contains _tons_ of bug fixes, including lots of crash fixes. Seriously... the last few releases have contained only a handful of fixes... this release has hundreds. I'm testing it now, and so far, it looks promising. This may be the first 1.6 release that I've found that doesn't crash with my workload. Unfortunately, I've never been able to pin down a sequence of events that would cause the crash on demand, so its hard for me to verify that things are finally fixed, other than doing long term load testing and waiting for the crashes to (hopefully) not happen. Dan - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org _ The New Busy is not the old busy. Search, chat and e-mail from your inbox. http://www.windowslive.com/campaign/thenewbusy?ocid=PID27925::T:WLMTAGL:ON:WL:en-US:WM_HMP:032010_3
Re: [OT] Re: jvm exits without trace
On 22/03/2010 22:12, Martin Gainty wrote: Carl- enable logging FileHandler and set level to finest e.g. org.apache.catalina.level=FINEST Not recommended unless you actually want Tomcat to take over 20 minutes to start up. Added to which, that isn't going to help track down this JVM bug. Carl, I think you are safe to ignore this suggestion. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Re: jvm exits without trace
George, I tried OpenSuSE with 6u17 and the results were about the same as Slackware 64 so the implication is that there is something in my app that is triggering the seg fault. My app is mostly jsp's running against MySQL with some AJAX and Flash (communicating through a servlet.) I do not use any native code nor do I use AJP. I do use SSL for everything. It has also failed on different hardware although the OS was the same. The argument against the problem being in my app is that the app runs fine (has for several years) on 32 bit Slackware and it now seems to be running fine on 6u7 (on Slackware 64.) Now that I appear to have a stable system that I can always go back to, I would like to get to the root cause. My thought was that I could get the source code, compile it on the server and then wait for it to fail with a core file that could give me the answer (maybe.) However, I can only find source code for the 'open' fork which may or may not be the same so I am stymied again. I guess I just have to keep stabbing in the dark with the new JVM's until one happens to work... not very scientific. Thanks, Carl - Original Message - From: George Sexton geor...@mhsoftware.com To: 'Tomcat Users List' users@tomcat.apache.org Sent: Monday, March 22, 2010 2:48 PM Subject: RE: [OT] Re: jvm exits without trace -Original Message- From: Dan Armbrust [mailto:daniel.armbrust.l...@gmail.com] Sent: Monday, March 22, 2010 12:17 PM To: Tomcat Users List Subject: Re: [OT] Re: jvm exits without trace On Tue, Mar 16, 2010 at 4:46 PM, Carl c...@etrak-plus.com wrote: My approach is to get something (a JVM) that works and then gradually change until it breaks. Then, I know what is causing the problem. To date, I haven't been able to get a JVM that works. I have had a lot of issues finding stable JVMs since I moved our software from 1.5 to 1.6... 1.6 has been a mess for ages under our workload on CentOS. Code that ran fine under 1.5 would segfault in all sorts of random places on 1.6. I even tried the 1.7 openJDK early builds... they were even worse. I was pleased to see that the most recent release http://java.sun.com/javase/6/webnotes/6u18.html contains _tons_ of bug fixes, including lots of crash fixes. Seriously... the last few releases have contained only a handful of fixes... this release has hundreds. I'm running 1.6.0_18 under OpenSuSE 11.0-11.2 and I've only had one problem. It looks like a GLIBC error related to IPV6. I disabled IPV6 on the machine and it's been rock solid since. I don't use native connectors, or AJP. I do about 900,000 hits with 4GB/Day spread across 3 servers and it's just rock steady. I run the tomcat instances for 2-3 weeks each before re-starting. For each machine, that's about 6 million hits with about 40 GB of transfer. George Sexton MH Software, Inc. 303 438-9585 www.mhsoftware.com - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Re: jvm exits without trace
Here's a hs_err file after a crash I had yesterday. We turned off some things in our code without restarting and the crashes have virtually stopped but we do still get the off one here and there where the application has not been restarted, could be that the problem lingers and builds up in time, who knows. It's a sigsegv in GCTaskThread. From the occupation in eden it looks like it happened during a scavenge (ParNew). Maybe an expert in some dark cave could shed some more light on it. On Tue, 2010-03-16 at 22:00 +0100, André Warnier wrote: Carl wrote: My approach is to get something (a JVM) that works and then gradually change until it breaks. Then, I know what is causing the problem. To date, I haven't been able to get a JVM that works. I think we understand that, and agree. Our remarks were tongue in cheek, if that is the right expression. At the bottom of things, finding a bug in the most recent JVM would be much more globally important than finding it in your applications, particularly a bug that can cause the JVM to segfault. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Re: jvm exits without trace
On 17 March 2010 10:16, Taylan Develioglu tdevelio...@ebuddy.com wrote: Here's a hs_err file after a crash I had yesterday. The list usually strips attachments; could you paste it? It's a sigsegv in GCTaskThread. From the occupation in eden it looks like it happened during a scavenge (ParNew). Ick. To roughly quote Dan Ingalls (who wrote several of the Smalltalk VMs, including their garbage collectors): studying the behaviour of objects in the garbage collector is like studying frogs using a blender. I've tried modifying Squeak's GC algorithm, and that pretty much sums up my experiences! I'd expect seg faults in a GC thread to occur if, for some reason, pointers were getting corrupted on the stack or in the heap, or if a concurrency issue caused a read or write to the old location of an object after it had been moved. The GC is very sensitive to such things as it traces large portions of object memory, so may well stumble across corruption before any other thread does so. - Peter
Re: [OT] Re: jvm exits without trace
On 17 March 2010 10:22, Taylan Develioglu tdevelio...@ebuddy.com wrote: Ofcourse this works better if I really attach the file. [...] Java Threads: ( = current thread ) 0x7f3d3c174000 JavaThread MSN-6488 daemon [_thread_in_native, id=28966, stack(0x42a4f000,0x42a7)] [...] Are you *absolutely sure* you pin down or copy *everything* you access from all of your native code so that the Java garbage collector cannot move it? - Peter
Re: [OT] Re: jvm exits without trace
Ognjen Blagojevic wrote: Hi, Tomcat dies suddenly thread was exciting almost as Prison Break TV series. I couldn't wait to find out what would be solution to the problem, and I must admit that just downgrading to lower sub-sub-sub version JVM left me a bit disappointed. :) Well, I wouldn't call that a true solution; more of a workaround. At some point, they're going to have to be able to update to a more current version. I'd love to see what the root cause is, if it's ever truly determined... D - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Re: jvm exits without trace
Ognjen Blagojevic wrote: Hi, Tomcat dies suddenly thread was exciting almost as Prison Break TV series. I couldn't wait to find out what would be solution to the problem, and I must admit that just downgrading to lower sub-sub-sub version JVM left me a bit disappointed. :) +1 It sounds quite like the standard tech support solution for Windows problems : de-install, re-install; and if that does not help, push the reset button. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Re: jvm exits without trace
My approach is to get something (a JVM) that works and then gradually change until it breaks. Then, I know what is causing the problem. To date, I haven't been able to get a JVM that works. In my case, it might be something in my application that is causing the crash on 64 bit Slackware as there are many people running 64 bit on Linux without a problem. If it is my application, then any 64 bit JVM I throw at it should crash. On the other hand, if the 1.6.0_7 JVM works, then it is likely a bug in one of the changes that brings the JVM to the current version. Time will tell. Thanks, Carl - Original Message - From: André Warnier a...@ice-sa.com To: Tomcat Users List users@tomcat.apache.org Sent: Tuesday, March 16, 2010 3:20 PM Subject: Re: [OT] Re: jvm exits without trace Ognjen Blagojevic wrote: Hi, Tomcat dies suddenly thread was exciting almost as Prison Break TV series. I couldn't wait to find out what would be solution to the problem, and I must admit that just downgrading to lower sub-sub-sub version JVM left me a bit disappointed. :) +1 It sounds quite like the standard tech support solution for Windows problems : de-install, re-install; and if that does not help, push the reset button. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Re: jvm exits without trace
Carl wrote: My approach is to get something (a JVM) that works and then gradually change until it breaks. Then, I know what is causing the problem. To date, I haven't been able to get a JVM that works. I think we understand that, and agree. Our remarks were tongue in cheek, if that is the right expression. At the bottom of things, finding a bug in the most recent JVM would be much more globally important than finding it in your applications, particularly a bug that can cause the JVM to segfault. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
