Re: tomcat jitters, then hangs - please help
Good to hear, I can confirm setting LD_ASSUME_KERNEL to 2.2.5 also works. The server has been running for the past two weeks without a single outage. Best Regards, Noam - Original Message - From: Ryan Lissack [EMAIL PROTECTED] To: 'Tomcat Users List' [EMAIL PROTECTED] Sent: Friday, October 24, 2003 1:35 PM Subject: RE: tomcat jitters, then hangs - please help Hi all, Just an update on issues we were experiencing when running Tomcat on Red Hat 9. As mentioned previously in this thread, the problem appears to be with a bug in NPTL. In order to revert to standard Linux threads you can set the LD_ASSUME_KERNEL environment variable to 2.2.5 or 2.4.1 We set LD_ASSUME_KERNEL to 2.4.1 on one of our servers one week ago and, since then, have *not* had one outage, our other machine which does not have the environment variable set has had numerous outages during the period. We can confirm that this does appear to resolve the issue. I believe Noam Camiel found that setting LD_ASSUME_KERNEL to 2.2.5 also works. I would like to say a special thank you to Remy for all his help during this period. Best regards, Ryan -Original Message- From: Shapira, Yoav [mailto:[EMAIL PROTECTED] Sent: 15 October 2003 14:07 To: Tomcat Users List Subject: RE: tomcat jitters, then hangs - please help Howdy, Cool, please keep us updated ;) I want these linux questions better documented in the list archives ;) Yoav Shapira Millennium ChemInformatics -Original Message- From: Noam Camiel [mailto:[EMAIL PROTECTED] Sent: Wednesday, October 15, 2003 6:37 AM To: Euan Guttridge; Tomcat Users List Subject: Re: tomcat jitters, then hangs - please help Hi Euan Please note the server is now up for 48 hours strait. The change I've made is I've set LD_ASSUME_KERNEL to 2.2.5 I will update again in another 48 hours or if the server hangs, whichever comes first. Regards, Noam - Original Message - From: Euan Guttridge [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, October 15, 2003 9:42 AM Subject: FW: tomcat jitters, then hangs - please help Hi Noam, I work with Ryan on tomcat issues. I would be grateful if you would let me know if your tomcat installation is *still* up (another 24 hours) since the NPTL change as below. Regards, Euan -Original Message- From: Noam Camiel [mailto:[EMAIL PROTECTED] Sent: 14 October 2003 14:46 To: Tomcat Users List Subject: Re: tomcat jitters, then hangs - please help Hi, On Friday, October 10, 2003 11:09 AM Ryan Lissack [EMAIL PROTECTED] wrote: Remy pointed out to me previously that RH9 uses a new form of threading and based on this we have done some investigation. RH9 uses NTPL (Native POSIX Thread Library) http://www.redhat.com/docs/manuals/linux/RHL-9-Manual/release- notes/x86/ It is possible to revert to the older form of threading, LinuxThreads, by setting an environment variable, namely: LD_ASSUME_KERNEL=kernel-version - 2.4.1 - Linuxthreads with floating stacks - 2.2.5 - Linuxthreads without floating stacks This does not revert back to an older kernel, it just changes the GNU C library used http://www-106.ibm.com/developerworks/java/library/j-jtc/ We will be testing with these settings starting today and, if we continue to have problems, we will be trying another distro. setting the LD_ASSUME_KERNEL variable, the server is now running over 24 hours strait. Still, nothing conclusive as of yet.. Ryan, thanks for the information above. Do you have move information regarding this configuration? do you still encounter problems? On Monday, October 13, 2003 9:59 PM Joe Zendle [EMAIL PROTECTED] wrote: We had the same problem w/ TC 4.1.27, sun jvm 1.4.2 and redhat 9. The thing wouldn't work for more than about 12 hours. IMHO, there are some fundamental problems with tomcat as of late. Hate to say it but we solved the problem by throwing away tomcat and using jetty! We are very pleased so far. Jetty is very fast and about 1/3 the memory footprint. Good luck. Hi Joe, Thanks for sharing the info, I am considering what you suggest as well. As a last try before moving away from Tomcat, I'm trying out the suggestion from Ryan above. Regards, Noam. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed
RE: tomcat jitters, then hangs - please help
Hi all, Just an update on issues we were experiencing when running Tomcat on Red Hat 9. As mentioned previously in this thread, the problem appears to be with a bug in NPTL. In order to revert to standard Linux threads you can set the LD_ASSUME_KERNEL environment variable to 2.2.5 or 2.4.1 We set LD_ASSUME_KERNEL to 2.4.1 on one of our servers one week ago and, since then, have *not* had one outage, our other machine which does not have the environment variable set has had numerous outages during the period. We can confirm that this does appear to resolve the issue. I believe Noam Camiel found that setting LD_ASSUME_KERNEL to 2.2.5 also works. I would like to say a special thank you to Remy for all his help during this period. Best regards, Ryan -Original Message- From: Shapira, Yoav [mailto:[EMAIL PROTECTED] Sent: 15 October 2003 14:07 To: Tomcat Users List Subject: RE: tomcat jitters, then hangs - please help Howdy, Cool, please keep us updated ;) I want these linux questions better documented in the list archives ;) Yoav Shapira Millennium ChemInformatics -Original Message- From: Noam Camiel [mailto:[EMAIL PROTECTED] Sent: Wednesday, October 15, 2003 6:37 AM To: Euan Guttridge; Tomcat Users List Subject: Re: tomcat jitters, then hangs - please help Hi Euan Please note the server is now up for 48 hours strait. The change I've made is I've set LD_ASSUME_KERNEL to 2.2.5 I will update again in another 48 hours or if the server hangs, whichever comes first. Regards, Noam - Original Message - From: Euan Guttridge [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, October 15, 2003 9:42 AM Subject: FW: tomcat jitters, then hangs - please help Hi Noam, I work with Ryan on tomcat issues. I would be grateful if you would let me know if your tomcat installation is *still* up (another 24 hours) since the NPTL change as below. Regards, Euan -Original Message- From: Noam Camiel [mailto:[EMAIL PROTECTED] Sent: 14 October 2003 14:46 To: Tomcat Users List Subject: Re: tomcat jitters, then hangs - please help Hi, On Friday, October 10, 2003 11:09 AM Ryan Lissack [EMAIL PROTECTED] wrote: Remy pointed out to me previously that RH9 uses a new form of threading and based on this we have done some investigation. RH9 uses NTPL (Native POSIX Thread Library) http://www.redhat.com/docs/manuals/linux/RHL-9-Manual/release- notes/x86/ It is possible to revert to the older form of threading, LinuxThreads, by setting an environment variable, namely: LD_ASSUME_KERNEL=kernel-version - 2.4.1 - Linuxthreads with floating stacks - 2.2.5 - Linuxthreads without floating stacks This does not revert back to an older kernel, it just changes the GNU C library used http://www-106.ibm.com/developerworks/java/library/j-jtc/ We will be testing with these settings starting today and, if we continue to have problems, we will be trying another distro. setting the LD_ASSUME_KERNEL variable, the server is now running over 24 hours strait. Still, nothing conclusive as of yet.. Ryan, thanks for the information above. Do you have move information regarding this configuration? do you still encounter problems? On Monday, October 13, 2003 9:59 PM Joe Zendle [EMAIL PROTECTED] wrote: We had the same problem w/ TC 4.1.27, sun jvm 1.4.2 and redhat 9. The thing wouldn't work for more than about 12 hours. IMHO, there are some fundamental problems with tomcat as of late. Hate to say it but we solved the problem by throwing away tomcat and using jetty! We are very pleased so far. Jetty is very fast and about 1/3 the memory footprint. Good luck. Hi Joe, Thanks for sharing the info, I am considering what you suggest as well. As a last try before moving away from Tomcat, I'm trying out the suggestion from Ryan above. Regards, Noam. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e
RE: tomcat jitters, then hangs - please help
Nice, and at least you got the speed to post issue sorted.. -Original Message- From: Ryan Lissack [mailto:[EMAIL PROTECTED] Sent: 24 October 2003 12:35 To: 'Tomcat Users List' Subject: RE: tomcat jitters, then hangs - please help Hi all, Just an update on issues we were experiencing when running Tomcat on Red Hat 9. As mentioned previously in this thread, the problem appears to be with a bug in NPTL. In order to revert to standard Linux threads you can set the LD_ASSUME_KERNEL environment variable to 2.2.5 or 2.4.1 We set LD_ASSUME_KERNEL to 2.4.1 on one of our servers one week ago and, since then, have *not* had one outage, our other machine which does not have the environment variable set has had numerous outages during the period. We can confirm that this does appear to resolve the issue. I believe Noam Camiel found that setting LD_ASSUME_KERNEL to 2.2.5 also works. I would like to say a special thank you to Remy for all his help during this period. Best regards, Ryan -Original Message- From: Shapira, Yoav [mailto:[EMAIL PROTECTED] Sent: 15 October 2003 14:07 To: Tomcat Users List Subject: RE: tomcat jitters, then hangs - please help Howdy, Cool, please keep us updated ;) I want these linux questions better documented in the list archives ;) Yoav Shapira Millennium ChemInformatics -Original Message- From: Noam Camiel [mailto:[EMAIL PROTECTED] Sent: Wednesday, October 15, 2003 6:37 AM To: Euan Guttridge; Tomcat Users List Subject: Re: tomcat jitters, then hangs - please help Hi Euan Please note the server is now up for 48 hours strait. The change I've made is I've set LD_ASSUME_KERNEL to 2.2.5 I will update again in another 48 hours or if the server hangs, whichever comes first. Regards, Noam - Original Message - From: Euan Guttridge [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, October 15, 2003 9:42 AM Subject: FW: tomcat jitters, then hangs - please help Hi Noam, I work with Ryan on tomcat issues. I would be grateful if you would let me know if your tomcat installation is *still* up (another 24 hours) since the NPTL change as below. Regards, Euan -Original Message- From: Noam Camiel [mailto:[EMAIL PROTECTED] Sent: 14 October 2003 14:46 To: Tomcat Users List Subject: Re: tomcat jitters, then hangs - please help Hi, On Friday, October 10, 2003 11:09 AM Ryan Lissack [EMAIL PROTECTED] wrote: Remy pointed out to me previously that RH9 uses a new form of threading and based on this we have done some investigation. RH9 uses NTPL (Native POSIX Thread Library) http://www.redhat.com/docs/manuals/linux/RHL-9-Manual/release- notes/x86/ It is possible to revert to the older form of threading, LinuxThreads, by setting an environment variable, namely: LD_ASSUME_KERNEL=kernel-version - 2.4.1 - Linuxthreads with floating stacks - 2.2.5 - Linuxthreads without floating stacks This does not revert back to an older kernel, it just changes the GNU C library used http://www-106.ibm.com/developerworks/java/library/j-jtc/ We will be testing with these settings starting today and, if we continue to have problems, we will be trying another distro. setting the LD_ASSUME_KERNEL variable, the server is now running over 24 hours strait. Still, nothing conclusive as of yet.. Ryan, thanks for the information above. Do you have move information regarding this configuration? do you still encounter problems? On Monday, October 13, 2003 9:59 PM Joe Zendle [EMAIL PROTECTED] wrote: We had the same problem w/ TC 4.1.27, sun jvm 1.4.2 and redhat 9. The thing wouldn't work for more than about 12 hours. IMHO, there are some fundamental problems with tomcat as of late. Hate to say it but we solved the problem by throwing away tomcat and using jetty! We are very pleased so far. Jetty is very fast and about 1/3 the memory footprint. Good luck. Hi Joe, Thanks for sharing the info, I am considering what you suggest as well. As a last try before moving away from Tomcat, I'm trying out the suggestion from Ryan above. Regards, Noam. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you
RE: tomcat jitters, then hangs - please help
Have you updated the nptl-devel and glibc package for RedHat9? And are you running the stock kernel or have you built a new kernel? Are you using Apache in front of Tomcat? I have a suspicion that the stock redhat kernel does something strange with threading that a built kernel wouldn't do. I built Apache with modjk under the RH stock kernel. Then built my own kernel and Apache wouldn't load the modules anymore, and I think it's because of something weird they do with threads. Oscar On Fri, 24 Oct 2003, Ryan Lissack wrote: Hi all, Just an update on issues we were experiencing when running Tomcat on Red Hat 9. As mentioned previously in this thread, the problem appears to be with a bug in NPTL. In order to revert to standard Linux threads you can set the LD_ASSUME_KERNEL environment variable to 2.2.5 or 2.4.1 We set LD_ASSUME_KERNEL to 2.4.1 on one of our servers one week ago and, since then, have *not* had one outage, our other machine which does not have the environment variable set has had numerous outages during the period. We can confirm that this does appear to resolve the issue. I believe Noam Camiel found that setting LD_ASSUME_KERNEL to 2.2.5 also works. I would like to say a special thank you to Remy for all his help during this period. Best regards, Ryan -Original Message- From: Shapira, Yoav [mailto:[EMAIL PROTECTED] Sent: 15 October 2003 14:07 To: Tomcat Users List Subject: RE: tomcat jitters, then hangs - please help Howdy, Cool, please keep us updated ;) I want these linux questions better documented in the list archives ;) Yoav Shapira Millennium ChemInformatics -Original Message- From: Noam Camiel [mailto:[EMAIL PROTECTED] Sent: Wednesday, October 15, 2003 6:37 AM To: Euan Guttridge; Tomcat Users List Subject: Re: tomcat jitters, then hangs - please help Hi Euan Please note the server is now up for 48 hours strait. The change I've made is I've set LD_ASSUME_KERNEL to 2.2.5 I will update again in another 48 hours or if the server hangs, whichever comes first. Regards, Noam - Original Message - From: Euan Guttridge [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, October 15, 2003 9:42 AM Subject: FW: tomcat jitters, then hangs - please help Hi Noam, I work with Ryan on tomcat issues. I would be grateful if you would let me know if your tomcat installation is *still* up (another 24 hours) since the NPTL change as below. Regards, Euan -Original Message- From: Noam Camiel [mailto:[EMAIL PROTECTED] Sent: 14 October 2003 14:46 To: Tomcat Users List Subject: Re: tomcat jitters, then hangs - please help Hi, On Friday, October 10, 2003 11:09 AM Ryan Lissack [EMAIL PROTECTED] wrote: Remy pointed out to me previously that RH9 uses a new form of threading and based on this we have done some investigation. RH9 uses NTPL (Native POSIX Thread Library) http://www.redhat.com/docs/manuals/linux/RHL-9-Manual/release- notes/x86/ It is possible to revert to the older form of threading, LinuxThreads, by setting an environment variable, namely: LD_ASSUME_KERNEL=kernel-version - 2.4.1 - Linuxthreads with floating stacks - 2.2.5 - Linuxthreads without floating stacks This does not revert back to an older kernel, it just changes the GNU C library used http://www-106.ibm.com/developerworks/java/library/j-jtc/ We will be testing with these settings starting today and, if we continue to have problems, we will be trying another distro. setting the LD_ASSUME_KERNEL variable, the server is now running over 24 hours strait. Still, nothing conclusive as of yet.. Ryan, thanks for the information above. Do you have move information regarding this configuration? do you still encounter problems? On Monday, October 13, 2003 9:59 PM Joe Zendle [EMAIL PROTECTED] wrote: We had the same problem w/ TC 4.1.27, sun jvm 1.4.2 and redhat 9. The thing wouldn't work for more than about 12 hours. IMHO, there are some fundamental problems with tomcat as of late. Hate to say it but we solved the problem by throwing away tomcat and using jetty! We are very pleased so far. Jetty is very fast and about 1/3 the memory footprint. Good luck. Hi Joe, Thanks for sharing the info, I am considering what you suggest as well. As a last try before moving away from Tomcat, I'm trying out the suggestion from Ryan above. Regards, Noam. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] This e-mail, including any attachments
RE: tomcat jitters, then hangs - please help
-Message d'origine- De : Zhao Hui [mailto:[EMAIL PROTECTED] Envoyé : mardi 21 octobre 2003 01:35 À : Tomcat Users List Objet : Re: tomcat jitters, then hangs - please help Hi folks, We encountered the same problem where tomcat on our linux box will hang occasionally after several days, and we have yet to find the cause of the problem. Looks like this might solve our problem :). Here is our configuration : OS : Redhat Tomcat : 4 JVM : IBM JDK1.4 We use the same, but didn't find how to solve all the bugs. So, with such configuration, you will encounter 3 problems : IBM Jvm crunching all available memory Tomcat 4.1.27 is sometimes unable to free some threads (and so you go out of maxProcessors IBM 1.4.1 runs more or less well with the libpthread. You could have tomcat freezing, or JITC doing SIG11, or... Well, nothing really clear about that. I would like to know more about how should we go about configuring the LD_ASSUME_KERNEL. Please correct me if I am wrong. setenv LD_ASSUME_KERNEL 2.2.5 I use bash as a shell, and because our final architecture (not finished) will be of several tomcat instances running on several nodes of a linux virtual server, I use a script like that : #!/bin/bash # # Add your own comments # # Java related stuff # We will ask GC to report problems and try to limit memory usage # JAVA_HOME=/opt/IBMJava2-141/ JAVA_OPTS=-verbose:gc -Xmx256m -Xms256m # # You can add the LD thing here : LD_ASSUME_KERNEL=2.2.5 # # Catalina related stuff : # CATALINA_HOME=/usr/tomcat # # Catalina base is on an NFS server (note that this require a # modification of bin/catalina.sh to fix the hard coded log path # CATALINA_BASE=/usr/tomcat # # PID is usefull to get good javacore when stopping Java # You can the use kill -3 `cat $CATALINA_BASE/logs/pid` to # have javacore in the CATALINA_BASE directory. # CATALINA_PID=$CATALINA_BASE/logs/pid # # Never forget to export ALL of it # export JAVA_HOME JAVA_OPTS LD_ASSUME_KERNEL CATALINA_HOME CATALINA_BASE CATALINA_PID # #run the startup script from Tomcat installation # /usr/tomcat/bin/startup.sh and that will do the trick? Does it depends on any kernel version? This could, even if IBM documentation is not clear about it. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: tomcat jitters, then hangs - please help
Do you see errors in your catalina.out when this happens? I believe that I am having the same problem with IBMJava, tomcat 4.1.27, mod_jk2.0.2 unixSocket connectors and apache 2.0.40... I'm hoping that setting LD_ASSUME_KERNEL will solve the issue. My catalina.out has errors as follows: [Mon Oct 20 23:10:02 2003] (error ) [jk_channel_un.c (422)] channelUn.receive(): error receiving -1 86 Oct 20, 2003 11:10:02 PM org.apache.jk.common.JniHandler nativeDispatch SEVERE: nativeDispatch: error 21000 Oct 20, 2003 11:10:02 PM org.apache.jk.common.ChannelUn receive SEVERE: receive error: 21000 Daniel Gibby -Message d'origine- De : Zhao Hui [mailto:[EMAIL PROTECTED] Envoyé : mardi 21 octobre 2003 01:35 À : Tomcat Users List Objet : Re: tomcat jitters, then hangs - please help Hi folks, We encountered the same problem where tomcat on our linux box will hang occasionally after several days, and we have yet to find the cause of the problem. Looks like this might solve our problem :). Here is our configuration : OS : Redhat Tomcat : 4 JVM : IBM JDK1.4 We use the same, but didn't find how to solve all the bugs. So, with such configuration, you will encounter 3 problems : IBM Jvm crunching all available memory Tomcat 4.1.27 is sometimes unable to free some threads (and so you go out of maxProcessors IBM 1.4.1 runs more or less well with the libpthread. You could have tomcat freezing, or JITC doing SIG11, or... Well, nothing really clear about that. I would like to know more about how should we go about configuring the LD_ASSUME_KERNEL. Please correct me if I am wrong. setenv LD_ASSUME_KERNEL 2.2.5 I use bash as a shell, and because our final architecture (not finished) will be of several tomcat instances running on several nodes of a linux virtual server, I use a script like that : #!/bin/bash # # Add your own comments # # Java related stuff # We will ask GC to report problems and try to limit memory usage # JAVA_HOME=/opt/IBMJava2-141/ JAVA_OPTS=-verbose:gc -Xmx256m -Xms256m # # You can add the LD thing here : LD_ASSUME_KERNEL=2.2.5 # # Catalina related stuff : # CATALINA_HOME=/usr/tomcat # # Catalina base is on an NFS server (note that this require a # modification of bin/catalina.sh to fix the hard coded log path # CATALINA_BASE=/usr/tomcat # # PID is usefull to get good javacore when stopping Java # You can the use kill -3 `cat $CATALINA_BASE/logs/pid` to # have javacore in the CATALINA_BASE directory. # CATALINA_PID=$CATALINA_BASE/logs/pid # # Never forget to export ALL of it # export JAVA_HOME JAVA_OPTS LD_ASSUME_KERNEL CATALINA_HOME CATALINA_BASE CATALINA_PID # #run the startup script from Tomcat installation # /usr/tomcat/bin/startup.sh and that will do the trick? Does it depends on any kernel version? This could, even if IBM documentation is not clear about it. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: tomcat jitters, then hangs - please help
I believe that I am having the same problem with IBMJava, Be careful that, according to IBM documentation, 1.4.0 and 1.4.1 do NOT handle the libpthread so-called optimizations the same way. Please give your version of the Jvm, thanks. François. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: tomcat jitters, then hangs - please help
Excellent news Noam.. Our testing (LD_ASSUME_KERNEL=2.4.1) is in progress but too early to confirm anything. Another option is to stick with the NTPL, using an updated glibc. Check out RH errata ; A bug in pthread_cond_wait() which could cause programs hangs in some cases https://rhn.redhat.com/errata/RHBA-2003-136.html If anyone tests this please drop the list a mail.. Cheers Euan -Original Message- From: Remy Maucherat [mailto:[EMAIL PROTECTED] Sent: 20 October 2003 14:32 To: Tomcat Users List Subject: Re: tomcat jitters, then hangs - please help Noam Camiel wrote: Hi folks, A week's gone by and all seems fine. Server is stable and working properly. Thanks for all who helped out! Ryan, Remi Yoav and Arthur, many thanks. It may be a good idea to document this for others who may stumble over this problem. All the best Noam Camiel Details: Problem: Tomcat Server becomes unresponsive to new requests after several hours, regardless of load. OS RH9; Tomcat 4 or 5; VM Sun or IBM (JDK 1.4) Solve: (tcsh): setenv LD_ASSUME_KERNEL 2.2.5 Thanks :) Very good news :) I think I'll need to add that in the release notes (and putting it in the FAQ would be good as well). Very weird symptoms, though. The only explanation for the bug is if the thread waiting on the socket just vanishes (there's no way it can go out of the blocking accept loop otherwise - I reviewed that quite extensively ;-)). Q: would the assume kernel 2.4 work as well ? I believe it would, but it could be worth confirming it eventually. This is not critical, obviously. -- x Rémy Maucherat Senior Developer Consultant JBoss Group (Europe) SàRL x - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: tomcat jitters, then hangs - please help
Very interesting bug report I will currently stick with my current configuration, but if this confirms to solve the prolem, it could explain some oddities Remy wondered upon Cheers, Noam - Original Message - From: Euan Guttridge [EMAIL PROTECTED] To: 'Tomcat Users List' [EMAIL PROTECTED] Sent: Tuesday, October 21, 2003 10:54 AM Subject: RE: tomcat jitters, then hangs - please help Excellent news Noam.. Our testing (LD_ASSUME_KERNEL=2.4.1) is in progress but too early to confirm anything. Another option is to stick with the NTPL, using an updated glibc. Check out RH errata ; A bug in pthread_cond_wait() which could cause programs hangs in some cases https://rhn.redhat.com/errata/RHBA-2003-136.html If anyone tests this please drop the list a mail.. Cheers Euan -Original Message- From: Remy Maucherat [mailto:[EMAIL PROTECTED] Sent: 20 October 2003 14:32 To: Tomcat Users List Subject: Re: tomcat jitters, then hangs - please help Noam Camiel wrote: Hi folks, A week's gone by and all seems fine. Server is stable and working properly. Thanks for all who helped out! Ryan, Remi Yoav and Arthur, many thanks. It may be a good idea to document this for others who may stumble over this problem. All the best Noam Camiel Details: Problem: Tomcat Server becomes unresponsive to new requests after several hours, regardless of load. OS RH9; Tomcat 4 or 5; VM Sun or IBM (JDK 1.4) Solve: (tcsh): setenv LD_ASSUME_KERNEL 2.2.5 Thanks :) Very good news :) I think I'll need to add that in the release notes (and putting it in the FAQ would be good as well). Very weird symptoms, though. The only explanation for the bug is if the thread waiting on the socket just vanishes (there's no way it can go out of the blocking accept loop otherwise - I reviewed that quite extensively ;-)). Q: would the assume kernel 2.4 work as well ? I believe it would, but it could be worth confirming it eventually. This is not critical, obviously. -- x Rémy Maucherat Senior Developer Consultant JBoss Group (Europe) SàRL x - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: tomcat jitters, then hangs - please help
Hi folks, A week's gone by and all seems fine. Server is stable and working properly. Thanks for all who helped out! Ryan, Remi Yoav and Arthur, many thanks. It may be a good idea to document this for others who may stumble over this problem. All the best Noam Camiel Details: Problem: Tomcat Server becomes unresponsive to new requests after several hours, regardless of load. OS RH9; Tomcat 4 or 5; VM Sun or IBM (JDK 1.4) Solve: (tcsh): setenv LD_ASSUME_KERNEL 2.2.5 - Original Message - From: Shapira, Yoav [EMAIL PROTECTED] To: Tomcat Users List [EMAIL PROTECTED] Sent: Wednesday, October 15, 2003 3:06 PM Subject: RE: tomcat jitters, then hangs - please help Howdy, Cool, please keep us updated ;) I want these linux questions better documented in the list archives ;) Yoav Shapira Millennium ChemInformatics -Original Message- From: Noam Camiel [mailto:[EMAIL PROTECTED] Sent: Wednesday, October 15, 2003 6:37 AM To: Euan Guttridge; Tomcat Users List Subject: Re: tomcat jitters, then hangs - please help Hi Euan Please note the server is now up for 48 hours strait. The change I've made is I've set LD_ASSUME_KERNEL to 2.2.5 I will update again in another 48 hours or if the server hangs, whichever comes first. Regards, Noam - Original Message - From: Euan Guttridge [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, October 15, 2003 9:42 AM Subject: FW: tomcat jitters, then hangs - please help Hi Noam, I work with Ryan on tomcat issues. I would be grateful if you would let me know if your tomcat installation is *still* up (another 24 hours) since the NPTL change as below. Regards, Euan -Original Message- From: Noam Camiel [mailto:[EMAIL PROTECTED] Sent: 14 October 2003 14:46 To: Tomcat Users List Subject: Re: tomcat jitters, then hangs - please help Hi, On Friday, October 10, 2003 11:09 AM Ryan Lissack [EMAIL PROTECTED] wrote: Remy pointed out to me previously that RH9 uses a new form of threading and based on this we have done some investigation. RH9 uses NTPL (Native POSIX Thread Library) http://www.redhat.com/docs/manuals/linux/RHL-9-Manual/release- notes/x86/ It is possible to revert to the older form of threading, LinuxThreads, by setting an environment variable, namely: LD_ASSUME_KERNEL=kernel-version - 2.4.1 - Linuxthreads with floating stacks - 2.2.5 - Linuxthreads without floating stacks This does not revert back to an older kernel, it just changes the GNU C library used http://www-106.ibm.com/developerworks/java/library/j-jtc/ We will be testing with these settings starting today and, if we continue to have problems, we will be trying another distro. setting the LD_ASSUME_KERNEL variable, the server is now running over 24 hours strait. Still, nothing conclusive as of yet.. Ryan, thanks for the information above. Do you have move information regarding this configuration? do you still encounter problems? On Monday, October 13, 2003 9:59 PM Joe Zendle [EMAIL PROTECTED] wrote: We had the same problem w/ TC 4.1.27, sun jvm 1.4.2 and redhat 9. The thing wouldn't work for more than about 12 hours. IMHO, there are some fundamental problems with tomcat as of late. Hate to say it but we solved the problem by throwing away tomcat and using jetty! We are very pleased so far. Jetty is very fast and about 1/3 the memory footprint. Good luck. Hi Joe, Thanks for sharing the info, I am considering what you suggest as well. As a last try before moving away from Tomcat, I'm trying out the suggestion from Ryan above. Regards, Noam. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: tomcat jitters, then hangs - please help
Howdy, Yup, it's good to document, so I'm glad you did. Now if only people would search the archives before asking questions for the zillionth time, we'd be in good shape ;) Glad it's working for you. Yoav Shapira Millennium ChemInformatics -Original Message- From: Noam Camiel [mailto:[EMAIL PROTECTED] Sent: Monday, October 20, 2003 7:37 AM To: Tomcat Users List Subject: Re: tomcat jitters, then hangs - please help Hi folks, A week's gone by and all seems fine. Server is stable and working properly. Thanks for all who helped out! Ryan, Remi Yoav and Arthur, many thanks. It may be a good idea to document this for others who may stumble over this problem. All the best Noam Camiel Details: Problem: Tomcat Server becomes unresponsive to new requests after several hours, regardless of load. OS RH9; Tomcat 4 or 5; VM Sun or IBM (JDK 1.4) Solve: (tcsh): setenv LD_ASSUME_KERNEL 2.2.5 - Original Message - From: Shapira, Yoav [EMAIL PROTECTED] To: Tomcat Users List [EMAIL PROTECTED] Sent: Wednesday, October 15, 2003 3:06 PM Subject: RE: tomcat jitters, then hangs - please help Howdy, Cool, please keep us updated ;) I want these linux questions better documented in the list archives ;) Yoav Shapira Millennium ChemInformatics -Original Message- From: Noam Camiel [mailto:[EMAIL PROTECTED] Sent: Wednesday, October 15, 2003 6:37 AM To: Euan Guttridge; Tomcat Users List Subject: Re: tomcat jitters, then hangs - please help Hi Euan Please note the server is now up for 48 hours strait. The change I've made is I've set LD_ASSUME_KERNEL to 2.2.5 I will update again in another 48 hours or if the server hangs, whichever comes first. Regards, Noam - Original Message - From: Euan Guttridge [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, October 15, 2003 9:42 AM Subject: FW: tomcat jitters, then hangs - please help Hi Noam, I work with Ryan on tomcat issues. I would be grateful if you would let me know if your tomcat installation is *still* up (another 24 hours) since the NPTL change as below. Regards, Euan -Original Message- From: Noam Camiel [mailto:[EMAIL PROTECTED] Sent: 14 October 2003 14:46 To: Tomcat Users List Subject: Re: tomcat jitters, then hangs - please help Hi, On Friday, October 10, 2003 11:09 AM Ryan Lissack [EMAIL PROTECTED] wrote: Remy pointed out to me previously that RH9 uses a new form of threading and based on this we have done some investigation. RH9 uses NTPL (Native POSIX Thread Library) http://www.redhat.com/docs/manuals/linux/RHL-9-Manual/release- notes/x86/ It is possible to revert to the older form of threading, LinuxThreads, by setting an environment variable, namely: LD_ASSUME_KERNEL=kernel-version - 2.4.1 - Linuxthreads with floating stacks - 2.2.5 - Linuxthreads without floating stacks This does not revert back to an older kernel, it just changes the GNU C library used http://www-106.ibm.com/developerworks/java/library/j-jtc/ We will be testing with these settings starting today and, if we continue to have problems, we will be trying another distro. setting the LD_ASSUME_KERNEL variable, the server is now running over 24 hours strait. Still, nothing conclusive as of yet.. Ryan, thanks for the information above. Do you have move information regarding this configuration? do you still encounter problems? On Monday, October 13, 2003 9:59 PM Joe Zendle [EMAIL PROTECTED] wrote: We had the same problem w/ TC 4.1.27, sun jvm 1.4.2 and redhat 9. The thing wouldn't work for more than about 12 hours. IMHO, there are some fundamental problems with tomcat as of late. Hate to say it but we solved the problem by throwing away tomcat and using jetty! We are very pleased so far. Jetty is very fast and about 1/3 the memory footprint. Good luck. Hi Joe, Thanks for sharing the info, I am considering what you suggest as well. As a last try before moving away from Tomcat, I'm trying out the suggestion from Ryan above. Regards, Noam. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands
Re: tomcat jitters, then hangs - please help
Noam Camiel wrote: Hi folks, A week's gone by and all seems fine. Server is stable and working properly. Thanks for all who helped out! Ryan, Remi Yoav and Arthur, many thanks. It may be a good idea to document this for others who may stumble over this problem. All the best Noam Camiel Details: Problem: Tomcat Server becomes unresponsive to new requests after several hours, regardless of load. OS RH9; Tomcat 4 or 5; VM Sun or IBM (JDK 1.4) Solve: (tcsh): setenv LD_ASSUME_KERNEL 2.2.5 Thanks :) Very good news :) I think I'll need to add that in the release notes (and putting it in the FAQ would be good as well). Very weird symptoms, though. The only explanation for the bug is if the thread waiting on the socket just vanishes (there's no way it can go out of the blocking accept loop otherwise - I reviewed that quite extensively ;-)). Q: would the assume kernel 2.4 work as well ? I believe it would, but it could be worth confirming it eventually. This is not critical, obviously. -- x Rémy Maucherat Senior Developer Consultant JBoss Group (Europe) SàRL x - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: tomcat jitters, then hangs - please help
Hi, LD_ASSUME_KERNEL Problem: Tomcat Server becomes unresponsive to new requests after several hours, regardless of load. OS RH9; Tomcat 4 or 5; VM Sun or IBM (JDK 1.4) Solve: (tcsh): setenv LD_ASSUME_KERNEL 2.2.5 [...] Q: would the assume kernel 2.4 work as well ? I believe it would, but it could be worth confirming it eventually. This is not critical, obviously. I understood LD_ASSUME_KERNEL is controlling the way the libpthread interact with both the kernel and the applications. I saw lot of things written about this supposed-to-be improved lib from RedHat but the optimizations doesn't seem to be usable by those popular Jvm. François. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: tomcat jitters, then hangs - please help
Hi folks, We encountered the same problem where tomcat on our linux box will hang occasionally after several days, and we have yet to find the cause of the problem. Looks like this might solve our problem :). Here is our configuration : OS : Redhat Tomcat : 4 JVM : IBM JDK1.4 I would like to know more about how should we go about configuring the LD_ASSUME_KERNEL. Please correct me if I am wrong. setenv LD_ASSUME_KERNEL 2.2.5 and that will do the trick? Does it depends on any kernel version? Besides that, we observed that some of the threads that tomcat started did not die after a few days, here is a snapshot of the tomcat processes (as an example): [say I started the tomcat at 19 October and today is 22 October, but there are still some threads running on 20 October] tomcat4 1554 1 0 Oct19 ? 00:00:19 /opt/IBMJava2-141/bin/java -Djav tomcat4 1587 1554 0 Oct19 ?00:00:00 /opt/IBMJava2-141/bin/java -Djav tomcat4 1588 1587 0 Oct19 ?00:00:00 /opt/IBMJava2-141/bin/java -Djav tomcat4 1589 1587 0 Oct19 ?00:00:00 /opt/IBMJava2-141/bin/java -Djav tomcat4 1590 1587 0 Oct19 ?00:00:00 /opt/IBMJava2-141/bin/java -Djav tomcat4 2071 1587 0 Oct19 ?00:00:00 /opt/IBMJava2-141/bin/java -Djav tomcat4 2072 1587 0 Oct19 ?00:00:00 /opt/IBMJava2-141/bin/java -Djav . tomcat4 2088 1587 0 Oct20 ?00:00:00 /opt/IBMJava2-141/bin/java -Djav tomcat4 2089 1587 0 Oct20 ?00:00:00 /opt/IBMJava2-141/bin/java -Djav I guess this might be the same issue here as well. Any comment would be much appreciated. Thanks. Regards, Hui - Original Message - From: Francois JEANMOUGIN [EMAIL PROTECTED] To: Tomcat Users List [EMAIL PROTECTED] Sent: Tuesday, October 21, 2003 12:45 AM Subject: RE: tomcat jitters, then hangs - please help Hi, LD_ASSUME_KERNEL Problem: Tomcat Server becomes unresponsive to new requests after several hours, regardless of load. OS RH9; Tomcat 4 or 5; VM Sun or IBM (JDK 1.4) Solve: (tcsh): setenv LD_ASSUME_KERNEL 2.2.5 [...] Q: would the assume kernel 2.4 work as well ? I believe it would, but it could be worth confirming it eventually. This is not critical, obviously. I understood LD_ASSUME_KERNEL is controlling the way the libpthread interact with both the kernel and the applications. I saw lot of things written about this supposed-to-be improved lib from RedHat but the optimizations doesn't seem to be usable by those popular Jvm. François. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: tomcat jitters, then hangs - please help
Hi Euan Please note the server is now up for 48 hours strait. The change I've made is I've set LD_ASSUME_KERNEL to 2.2.5 I will update again in another 48 hours or if the server hangs, whichever comes first. Regards, Noam - Original Message - From: Euan Guttridge [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, October 15, 2003 9:42 AM Subject: FW: tomcat jitters, then hangs - please help Hi Noam, I work with Ryan on tomcat issues. I would be grateful if you would let me know if your tomcat installation is *still* up (another 24 hours) since the NPTL change as below. Regards, Euan -Original Message- From: Noam Camiel [mailto:[EMAIL PROTECTED] Sent: 14 October 2003 14:46 To: Tomcat Users List Subject: Re: tomcat jitters, then hangs - please help Hi, On Friday, October 10, 2003 11:09 AM Ryan Lissack [EMAIL PROTECTED] wrote: Remy pointed out to me previously that RH9 uses a new form of threading and based on this we have done some investigation. RH9 uses NTPL (Native POSIX Thread Library) http://www.redhat.com/docs/manuals/linux/RHL-9-Manual/release-notes/x86/ It is possible to revert to the older form of threading, LinuxThreads, by setting an environment variable, namely: LD_ASSUME_KERNEL=kernel-version - 2.4.1 - Linuxthreads with floating stacks - 2.2.5 - Linuxthreads without floating stacks This does not revert back to an older kernel, it just changes the GNU C library used http://www-106.ibm.com/developerworks/java/library/j-jtc/ We will be testing with these settings starting today and, if we continue to have problems, we will be trying another distro. setting the LD_ASSUME_KERNEL variable, the server is now running over 24 hours strait. Still, nothing conclusive as of yet.. Ryan, thanks for the information above. Do you have move information regarding this configuration? do you still encounter problems? On Monday, October 13, 2003 9:59 PM Joe Zendle [EMAIL PROTECTED] wrote: We had the same problem w/ TC 4.1.27, sun jvm 1.4.2 and redhat 9. The thing wouldn't work for more than about 12 hours. IMHO, there are some fundamental problems with tomcat as of late. Hate to say it but we solved the problem by throwing away tomcat and using jetty! We are very pleased so far. Jetty is very fast and about 1/3 the memory footprint. Good luck. Hi Joe, Thanks for sharing the info, I am considering what you suggest as well. As a last try before moving away from Tomcat, I'm trying out the suggestion from Ryan above. Regards, Noam. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: tomcat jitters, then hangs - please help
Howdy, Cool, please keep us updated ;) I want these linux questions better documented in the list archives ;) Yoav Shapira Millennium ChemInformatics -Original Message- From: Noam Camiel [mailto:[EMAIL PROTECTED] Sent: Wednesday, October 15, 2003 6:37 AM To: Euan Guttridge; Tomcat Users List Subject: Re: tomcat jitters, then hangs - please help Hi Euan Please note the server is now up for 48 hours strait. The change I've made is I've set LD_ASSUME_KERNEL to 2.2.5 I will update again in another 48 hours or if the server hangs, whichever comes first. Regards, Noam - Original Message - From: Euan Guttridge [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, October 15, 2003 9:42 AM Subject: FW: tomcat jitters, then hangs - please help Hi Noam, I work with Ryan on tomcat issues. I would be grateful if you would let me know if your tomcat installation is *still* up (another 24 hours) since the NPTL change as below. Regards, Euan -Original Message- From: Noam Camiel [mailto:[EMAIL PROTECTED] Sent: 14 October 2003 14:46 To: Tomcat Users List Subject: Re: tomcat jitters, then hangs - please help Hi, On Friday, October 10, 2003 11:09 AM Ryan Lissack [EMAIL PROTECTED] wrote: Remy pointed out to me previously that RH9 uses a new form of threading and based on this we have done some investigation. RH9 uses NTPL (Native POSIX Thread Library) http://www.redhat.com/docs/manuals/linux/RHL-9-Manual/release- notes/x86/ It is possible to revert to the older form of threading, LinuxThreads, by setting an environment variable, namely: LD_ASSUME_KERNEL=kernel-version - 2.4.1 - Linuxthreads with floating stacks - 2.2.5 - Linuxthreads without floating stacks This does not revert back to an older kernel, it just changes the GNU C library used http://www-106.ibm.com/developerworks/java/library/j-jtc/ We will be testing with these settings starting today and, if we continue to have problems, we will be trying another distro. setting the LD_ASSUME_KERNEL variable, the server is now running over 24 hours strait. Still, nothing conclusive as of yet.. Ryan, thanks for the information above. Do you have move information regarding this configuration? do you still encounter problems? On Monday, October 13, 2003 9:59 PM Joe Zendle [EMAIL PROTECTED] wrote: We had the same problem w/ TC 4.1.27, sun jvm 1.4.2 and redhat 9. The thing wouldn't work for more than about 12 hours. IMHO, there are some fundamental problems with tomcat as of late. Hate to say it but we solved the problem by throwing away tomcat and using jetty! We are very pleased so far. Jetty is very fast and about 1/3 the memory footprint. Good luck. Hi Joe, Thanks for sharing the info, I am considering what you suggest as well. As a last try before moving away from Tomcat, I'm trying out the suggestion from Ryan above. Regards, Noam. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: tomcat jitters, then hangs - please help
Shapira, Yoav wrote: Howdy, Cool, please keep us updated ;) I want these linux questions better documented in the list archives ;) Given that RH 9 / HTTP/1.1 seems a fairly common configuration, this should be added to the known issues and the FAQ if confirmed. Since there are three options, I wonder how the 2.4 compat mode works out. I assume Noam played it safe, and chose 2.2 compat mode for his testing just in case ;-) I assume it would work as well as 2.2 compat, since there weren't any major problems before (again, to be confirmed; since the performance difference is likely not very significant, it's better to get stability in this case :)). -- x Rémy Maucherat Senior Developer Consultant JBoss Group (Europe) SàRL x - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: tomcat jitters, then hangs - please help
Hi, On Friday, October 10, 2003 11:09 AM Ryan Lissack [EMAIL PROTECTED] wrote: Remy pointed out to me previously that RH9 uses a new form of threading and based on this we have done some investigation. RH9 uses NTPL (Native POSIX Thread Library) http://www.redhat.com/docs/manuals/linux/RHL-9-Manual/release-notes/x86/ It is possible to revert to the older form of threading, LinuxThreads, by setting an environment variable, namely: LD_ASSUME_KERNEL=kernel-version - 2.4.1 - Linuxthreads with floating stacks - 2.2.5 - Linuxthreads without floating stacks This does not revert back to an older kernel, it just changes the GNU C library used http://www-106.ibm.com/developerworks/java/library/j-jtc/ We will be testing with these settings starting today and, if we continue to have problems, we will be trying another distro. setting the LD_ASSUME_KERNEL variable, the server is now running over 24 hours strait. Still, nothing conclusive as of yet.. Ryan, thanks for the information above. Do you have move information regarding this configuration? do you still encounter problems? On Monday, October 13, 2003 9:59 PM Joe Zendle [EMAIL PROTECTED] wrote: We had the same problem w/ TC 4.1.27, sun jvm 1.4.2 and redhat 9. The thing wouldn't work for more than about 12 hours. IMHO, there are some fundamental problems with tomcat as of late. Hate to say it but we solved the problem by throwing away tomcat and using jetty! We are very pleased so far. Jetty is very fast and about 1/3 the memory footprint. Good luck. Hi Joe, Thanks for sharing the info, I am considering what you suggest as well. As a last try before moving away from Tomcat, I'm trying out the suggestion from Ryan above. Regards, Noam. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: tomcat jitters, then hangs - please help
We had the same problem w/ TC 4.1.27, sun jvm 1.4.2 and redhat 9. The thing wouldn't work for more than about 12 hours. IMHO, there are some fundamental problems with tomcat as of late. Hate to say it but we solved the problem by throwing away tomcat and using jetty! We are very pleased so far. Jetty is very fast and about 1/3 the memory footprint. Good luck. -Original Message- From: Noam Camiel [mailto:[EMAIL PROTECTED] Sent: Monday, April 07, 2003 5:09 AM To: Tomcat Users List Subject: Re: tomcat jitters, then hangs - please help On Thursday, October 09, 2003 10:39 PM Remy Maucherat [EMAIL PROTECTED] wrote: It is very surprising that the connector will manage to exit the accept loop without logging some kind of error (which would be in catalina.out, rather than in the main logs). Do you see some stack trace originating from the PoolTcpEndpoint or ThreadPool classes ? There were no logs indicating this, and actually, after the server started there are no other logs appearing in catalina.out No idea at this point. Most (all) such reports were for RH 9. All also mention a period of about a day before an incident. Thanks, let me know if there's any news. Regards, Noam - Original Message - From: Remy Maucherat [EMAIL PROTECTED] To: Tomcat Users List [EMAIL PROTECTED] Sent: Thursday, October 09, 2003 10:39 PM Subject: Re: tomcat jitters, then hangs - please help Noam Camiel wrote: On Thursday, October 09, 2003 5:11 PM Ryan Lissack [EMAIL PROTECTED] wrote: Hi Noam, We have experienced similar problems. You will notice in the thread dump that there is no thread from the connectors thread pool waiting on an accept I see your what you mean, I compared the thread dump to a good dump when the server is operational and there indeed I found a thread from the connectors thread pool waiting on accept: 3XMTHREADINFO http80-Processor24 (TID:0x100688E0, sys_thread_t:0x87810C0, state:R, native ID:0x41CCCF40) prio=5 4XESTACKTRACE at java.net.PlainSocketImpl.socketAccept(Native Method) This does not appear to exist in the bad dump. It is very surprising that the connector will manage to exit the accept loop without logging some kind of error (which would be in catalina.out, rather than in the main logs). Do you see some stack trace originating from the PoolTcpEndpoint or ThreadPool classes ? What OS are you on, RH9?? Yes, Im on RH9. Any ideas why this could happen? could it be some system resource limit? Is your application an intensive app? My application is not an intensive one and the hangs occur hours after i start the server (usually a day later). No idea at this point. Most (all) such reports were for RH 9. All also mention a period of about a day before an incident. -- x Re'my Maucherat Senior Developer Consultant JBoss Group (Europe) Sa`RL x - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: tomcat jitters, then hangs - please help
On Thursday, October 09, 2003 10:39 PM Remy Maucherat [EMAIL PROTECTED] wrote: It is very surprising that the connector will manage to exit the accept loop without logging some kind of error (which would be in catalina.out, rather than in the main logs). Do you see some stack trace originating from the PoolTcpEndpoint or ThreadPool classes ? There were no logs indicating this, and actually, after the server started there are no other logs appearing in catalina.out No idea at this point. Most (all) such reports were for RH 9. All also mention a period of about a day before an incident. Thanks, let me know if there's any news. Regards, Noam - Original Message - From: Remy Maucherat [EMAIL PROTECTED] To: Tomcat Users List [EMAIL PROTECTED] Sent: Thursday, October 09, 2003 10:39 PM Subject: Re: tomcat jitters, then hangs - please help Noam Camiel wrote: On Thursday, October 09, 2003 5:11 PM Ryan Lissack [EMAIL PROTECTED] wrote: Hi Noam, We have experienced similar problems. You will notice in the thread dump that there is no thread from the connectors thread pool waiting on an accept I see your what you mean, I compared the thread dump to a good dump when the server is operational and there indeed I found a thread from the connectors thread pool waiting on accept: 3XMTHREADINFO http80-Processor24 (TID:0x100688E0, sys_thread_t:0x87810C0, state:R, native ID:0x41CCCF40) prio=5 4XESTACKTRACE at java.net.PlainSocketImpl.socketAccept(Native Method) This does not appear to exist in the bad dump. It is very surprising that the connector will manage to exit the accept loop without logging some kind of error (which would be in catalina.out, rather than in the main logs). Do you see some stack trace originating from the PoolTcpEndpoint or ThreadPool classes ? What OS are you on, RH9?? Yes, Im on RH9. Any ideas why this could happen? could it be some system resource limit? Is your application an intensive app? My application is not an intensive one and the hangs occur hours after i start the server (usually a day later). No idea at this point. Most (all) such reports were for RH 9. All also mention a period of about a day before an incident. -- x Re'my Maucherat Senior Developer Consultant JBoss Group (Europe) Sa`RL x - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: tomcat jitters, then hangs - please help
Hi Noam, Remy pointed out to me previously that RH9 uses a new form of threading and based on this we have done some investigation. RH9 uses NTPL (Native POSIX Thread Library) http://www.redhat.com/docs/manuals/linux/RHL-9-Manual/release-notes/x86/ It is possible to revert to the older form of threading, LinuxThreads, by setting an environment variable, namely: LD_ASSUME_KERNEL=kernel-version - 2.4.1 - Linuxthreads with floating stacks - 2.2.5 - Linuxthreads without floating stacks This does not revert back to an older kernel, it just changes the GNU C library used http://www-106.ibm.com/developerworks/java/library/j-jtc/ We will be testing with these settings starting today and, if we continue to have problems, we will be trying another distro. Ryan More NTPL info http://www.redhat.com/partners/pdf/POSIX_Linux_Threading.pdf -Original Message- From: Noam Camiel [mailto:[EMAIL PROTECTED] Sent: 04 April 2003 18:36 To: Tomcat Users List Subject: Re: tomcat jitters, then hangs - please help On Thursday, October 09, 2003 5:11 PM Ryan Lissack [EMAIL PROTECTED] wrote: Hi Noam, We have experienced similar problems. You will notice in the thread dump that there is no thread from the connectors thread pool waiting on an accept I see your what you mean, I compared the thread dump to a good dump when the server is operational and there indeed I found a thread from the connectors thread pool waiting on accept: 3XMTHREADINFO http80-Processor24 (TID:0x100688E0, sys_thread_t:0x87810C0, state:R, native ID:0x41CCCF40) prio=5 4XESTACKTRACE at java.net.PlainSocketImpl.socketAccept(Native Method) This does not appear to exist in the bad dump. What OS are you on, RH9?? Yes, Im on RH9. Any ideas why this could happen? could it be some system resource limit? Is your application an intensive app? My application is not an intensive one and the hangs occur hours after i start the server (usually a day later). Thanks for the info, i'd be happy to hear any suggestions. Noam - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: tomcat jitters, then hangs - please help
Howdy, Please also post an update if you DON'T have any problems, so that the few good souls who actually search list archives before posting questions can have a reference in the future ;) Thanks, Yoav Shapira Millennium ChemInformatics -Original Message- From: Ryan Lissack [mailto:[EMAIL PROTECTED] Sent: Friday, October 10, 2003 5:09 AM To: 'Tomcat Users List' Subject: RE: tomcat jitters, then hangs - please help Hi Noam, Remy pointed out to me previously that RH9 uses a new form of threading and based on this we have done some investigation. RH9 uses NTPL (Native POSIX Thread Library) http://www.redhat.com/docs/manuals/linux/RHL-9-Manual/release-notes/x86 / It is possible to revert to the older form of threading, LinuxThreads, by setting an environment variable, namely: LD_ASSUME_KERNEL=kernel-version - 2.4.1 - Linuxthreads with floating stacks - 2.2.5 - Linuxthreads without floating stacks This does not revert back to an older kernel, it just changes the GNU C library used http://www-106.ibm.com/developerworks/java/library/j-jtc/ We will be testing with these settings starting today and, if we continue to have problems, we will be trying another distro. Ryan More NTPL info http://www.redhat.com/partners/pdf/POSIX_Linux_Threading.pdf -Original Message- From: Noam Camiel [mailto:[EMAIL PROTECTED] Sent: 04 April 2003 18:36 To: Tomcat Users List Subject: Re: tomcat jitters, then hangs - please help On Thursday, October 09, 2003 5:11 PM Ryan Lissack [EMAIL PROTECTED] wrote: Hi Noam, We have experienced similar problems. You will notice in the thread dump that there is no thread from the connectors thread pool waiting on an accept I see your what you mean, I compared the thread dump to a good dump when the server is operational and there indeed I found a thread from the connectors thread pool waiting on accept: 3XMTHREADINFO http80-Processor24 (TID:0x100688E0, sys_thread_t:0x87810C0, state:R, native ID:0x41CCCF40) prio=5 4XESTACKTRACE at java.net.PlainSocketImpl.socketAccept(Native Method) This does not appear to exist in the bad dump. What OS are you on, RH9?? Yes, Im on RH9. Any ideas why this could happen? could it be some system resource limit? Is your application an intensive app? My application is not an intensive one and the hangs occur hours after i start the server (usually a day later). Thanks for the info, i'd be happy to hear any suggestions. Noam - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: tomcat jitters, then hangs - please help
Arthur Veinstein wrote: Noam, I suspect that the tomcat doesn't hangs but rather you reach the max TCP connection that your server accepts. The best will be to write a small program you run by calling a servlet which write to the output I'm a live every 10 min. Run the program after you restart the server and wait till your problem occurs, then if you see that the output string keep coming then the tomcat isn't dead just your server rejects tcp connection. I think it could be a good idea to include that kind of servlet as part of the manager webapp. Even if we manage to isolate and fix that bug, it won't solve all the other possible stability issues (VM crash, OOM), so admins could find this useful. -- x Re'my Maucherat Senior Developer Consultant JBoss Group (Europe) Sa`RL x - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: tomcat jitters, then hangs - please help
Noam Camiel wrote: On Thursday, October 09, 2003 5:11 PM Ryan Lissack [EMAIL PROTECTED] wrote: Hi Noam, We have experienced similar problems. You will notice in the thread dump that there is no thread from the connectors thread pool waiting on an accept I see your what you mean, I compared the thread dump to a good dump when the server is operational and there indeed I found a thread from the connectors thread pool waiting on accept: 3XMTHREADINFO http80-Processor24 (TID:0x100688E0, sys_thread_t:0x87810C0, state:R, native ID:0x41CCCF40) prio=5 4XESTACKTRACE at java.net.PlainSocketImpl.socketAccept(Native Method) This does not appear to exist in the bad dump. It is very surprising that the connector will manage to exit the accept loop without logging some kind of error (which would be in catalina.out, rather than in the main logs). Do you see some stack trace originating from the PoolTcpEndpoint or ThreadPool classes ? What OS are you on, RH9?? Yes, Im on RH9. Any ideas why this could happen? could it be some system resource limit? Is your application an intensive app? My application is not an intensive one and the hangs occur hours after i start the server (usually a day later). No idea at this point. Most (all) such reports were for RH 9. All also mention a period of about a day before an incident. -- x Re'my Maucherat Senior Developer Consultant JBoss Group (Europe) Sa`RL x - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: tomcat jitters, then hangs - please help
Howdy, Oh, these problems are invariably a pain to debug ;) My Tomcat installation works fine until sometimes (usually after 24 hours), calling the simplest servlet locks up the Tomcat threads, causing the tomcat engine to be unresponsive. At that time, if you send the VM a SIGQUIT, what are the threads doing? The servlets I'm running work with a mysql JDBC connector connecting to mysql database. Is the behavior the same if all you have in your webapp is the Hello servlet? I.e. you take all other servlets, especially the ones that use JDBC, out? There have been problems reported in the past with DB connection pool threads just hanging around. the simple Hello servlet (code at the bottom). After calling it a few times there was no So the first few times it works? more response from the server and the tomcat manager/status page (which sometimes returns and sometimes not) clearly shows that the Hello servlet is stuck: Stage 1: After calling the Hello servlet StageTimeB SentB RecvClientVHost Request S32671 ms 0KB0KB194.90.90.18 mdrmus Get Actually, this shows your servlet wasn't even called: no bytes sent OR received. Are you sure you're not having a network failure? Also, the fact the manager status page sometimes works and sometimes doesn't is worrisome. I've never had problems with it, and it should be very reliable. Can you can the manager webapp to malfunction even without any of your webapps deployed on the server? If so, open a bugzilla issue with steps to reproduce your behavior. If anyone has any idea how to solve this problem, I would be most grateful. Any errors in the logs? (Also make sure you have adequate disk space to write logs and temp files). Yoav Shapira This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: tomcat jitters, then hangs - please help
Hi Noam,
Sounds like you have some problem with the DB connection.
Can you provide the server.xml configuration ?
Arthur
- Original Message -
From: Noam Camiel [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Sent: Wednesday, October 01, 2003 4:38 PM
Subject: tomcat jitters, then hangs - please help
Hi,
My Tomcat installation works fine until sometimes (usually after 24
hours),
calling the simplest servlet locks up the Tomcat threads,
causing the tomcat engine to be unresponsive.
The servlets I'm running work with a mysql JDBC connector connecting to
mysql database.
In the following case, after tomcat has been up for about a day, I've
called
the simple Hello servlet (code at the bottom). After calling it a few
times there was no
more response from the server and the tomcat manager/status page (which
sometimes returns and sometimes not) clearly shows that the Hello
servlet
is stuck:
Stage 1: After calling the Hello servlet
StageTimeB SentB RecvClientVHost
Request
S32671 ms 0KB0KB194.90.90.18 mdrmusGet
/printserver/servlet/Hello HTTP/1.1
P????
??
S 615 ms0 KB0 KB194.90.91.18mdrmus
Get /manager/status HTTP/1.1
P????
??
R????
??
Stage 2: After trying to view the status/all page several times, each time
not getting any response.
StageTimeB SentB RecvClientVHost
Request
S94124 ms 0KB0KB194.90.90.18 mdrmusGet
/printserver/servlet/Hello HTTP/1.1
P????
??
S54513 ms 0 KB 0 KB 194.90.91.18mdrmusGet
/manager/status/all HTTP/1.1
P????
??
S20402 ms 0 KB 0 KB 194.90.91.18mdrmusGet /
HTTP/1.1
P????
??
S11525 ms 0 KB 0 KB 194.90.91.18mdrmusGet
/manager/status/all HTTP/1.1
S11175 ms 0 KB 0 KB 194.90.91.18mdrmusGet
/manager/status/all HTTP/1.1
S10969 ms 0 KB 0 KB 194.90.91.18mdrmusGet
/manager/status/all HTTP/1.1
S10825 ms 0 KB 0 KB 194.90.91.18mdrmusGet
/manager/status/all HTTP/1.1
S6342 ms 0 KB 0 KB 194.90.91.18mdrmusGet
/manager/status/all HTTP/1.1
S241 ms 0 KB 0 KB 194.90.91.18mdrmusGet
/manager/status/all HTTP/1.1
At this point the server is completely stuck and no calls are returned.
If anyone has any idea how to solve this problem, I would be most
grateful.
Thanks
Noam.
- Additional technical info
follows ---
Setup information:
Tomcat Version: 5.0.12
JVM version 1.4.1_05-b01
JVM Vendor: Sun Microsystems Inc.
OS: Linux
OS Version: 2.4.20-8
OS Architecture: i386
Server is running on port 80 (tomcat started by root)
JVM stage 1:
Free Memory: 2.32 MB, Total Memory: 8.88 MB Max memory: 134.21 MB
JVM stage 2:
Free Memory: 6.17 MB, Total Memory: 15.47 MB Max memory: 134.21 MB
http80 stage 1
Max threads: 150 Min spare threads: 25 Max spare threads: 75 Current
thread
count: 25 Current thread busy: 5
Max processing time: 1914 ms Processing time: 13 s Request count: 70 Error
count: 5 Bytes received: 0.0 MB Bytes sent: 0.21 MB
http80 stage 2
Max threads: 150 Min spare threads: 25 Max spare threads: 75 Current
thread
count: 25 Current thread busy: 13
Max processing time: 1914 ms Processing time: 14 s Request count: 78 Error
count: 10 Bytes received: 0.0 MB Bytes sent: 0.22 MB
Hello Servlet code:
import java.io.*;
import java.util.*;
import javax.servlet.*;
import javax.servlet.http.*;
public class Hello extends HttpServlet
public void doGet (HttpServletRequest request,
HttpServletResponse response)
throws ServletException, IOException
response.setContentType(text/html);
PrintWriter out = response.getWriter();
out.println(Hello there);
out.close();
}
public String getServletInfo() {
return The Hello servlet says hello there;
}
}
Thanks again,
Noam
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: tomcat jitters, then hangs - please help
I would also get a thread dump from the JVM. dwh Arthur Veinstein wrote: Hi Noam, Sounds like you have some problem with the DB connection. Can you provide the server.xml configuration ? - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: tomcat jitters, then hangs - please help
-Processor5 daemon prio=1 tid=0x0x8208cc0 nid=0x72c3 in Object.wait() [4d59e000..4d59f090] at java.lang.Object.wait(Native Method) - waiting on 0x44c5f320 (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Object.wait(Object.java:426) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav a:642) - locked 0x44c5f320 (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Thread.run(Thread.java:536) http80-Processor4 daemon prio=1 tid=0x0x8208110 nid=0x72c3 in Object.wait() [4d51e000..4d51f110] at java.lang.Object.wait(Native Method) - waiting on 0x44c5f3a0 (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Object.wait(Object.java:426) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav a:642) - locked 0x44c5f3a0 (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Thread.run(Thread.java:536) http80-Processor3 daemon prio=1 tid=0x0x8207580 nid=0x72c3 in Object.wait() [4d49f000..4d49f190] at java.lang.Object.wait(Native Method) - waiting on 0x44c5f420 (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Object.wait(Object.java:426) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav a:642) - locked 0x44c5f420 (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Thread.run(Thread.java:536) http80-Processor2 daemon prio=1 tid=0x0x8206bb8 nid=0x72c3 in Object.wait() [4d41f000..4d41f210] at java.lang.Object.wait(Native Method) - waiting on 0x44c5f4a0 (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Object.wait(Object.java:426) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav a:642) - locked 0x44c5f4a0 (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Thread.run(Thread.java:536) http80-Processor1 daemon prio=1 tid=0x0x82068d8 nid=0x72c3 in Object.wait() [4d39f000..4d39f290] at java.lang.Object.wait(Native Method) - waiting on 0x44c5f520 (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Object.wait(Object.java:426) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav a:642) - locked 0x44c5f520 (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Thread.run(Thread.java:536) ContainerBackgroundProcessor[StandardEngine[Catalina]] daemon prio=1 tid=0x0x8203b08 nid=0x72c3 waiting on condition [4d31f000..4d31f310] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Cont ainerBase.java:1636) at java.lang.Thread.run(Thread.java:536) Signal Dispatcher daemon prio=1 tid=0x0x80aa330 nid=0x72c3 waiting on condition [0..0] Finalizer daemon prio=1 tid=0x0x808c100 nid=0x72c3 in Object.wait() [4c4ff000..4c4ff590] at java.lang.Object.wait(Native Method) - waiting on 0x448f08d0 (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111) - locked 0x448f08d0 (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) Reference Handler daemon prio=1 tid=0x0x808b520 nid=0x72c3 in Object.wait() [4c47f000..4c47f610] at java.lang.Object.wait(Native Method) - waiting on 0x448f0938 (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:426) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:113) - locked 0x448f0938 (a java.lang.ref.Reference$Lock) main prio=1 tid=0x0x80521e8 nid=0x72c3 runnable [bfffb000..bfffc23c] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353) - locked 0x44cdf918 (a java.net.PlainSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:448) at java.net.ServerSocket.accept(ServerSocket.java:419) at org.apache.catalina.core.StandardServer.await(StandardServer.java:552) at org.apache.catalina.startup.Catalina.await(Catalina.java:634) at org.apache.catalina.startup.Catalina.start(Catalina.java:596) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39 ) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl .java:25) at java.lang.reflect.Method.invoke(Method.java:324) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:295) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:392) VM Thread prio=1 tid=0x0x808a208 nid=0x72c3 runnable VM Periodic Task Thread prio=1 tid=0x0x80a8ee8 nid=0x72c3 waiting on condition Suspend Checker Thread prio=1 tid=0x0x80a98b0 nid=0x72c3 runnable - Original Message - From: Shapira, Yoav [EMAIL PROTECTED] To: Tomcat Users List [EMAIL PROTECTED] Sent: Wednesday, October 01, 2003 2:52 PM Subject: RE: tomcat jitters
RE: tomcat jitters, then hangs - please help
runnable - Original Message - From: Shapira, Yoav [EMAIL PROTECTED] To: Tomcat Users List [EMAIL PROTECTED] Sent: Wednesday, October 01, 2003 2:52 PM Subject: RE: tomcat jitters, then hangs - please help Howdy, Oh, these problems are invariably a pain to debug ;) My Tomcat installation works fine until sometimes (usually after 24 hours), calling the simplest servlet locks up the Tomcat threads, causing the tomcat engine to be unresponsive. At that time, if you send the VM a SIGQUIT, what are the threads doing? The servlets I'm running work with a mysql JDBC connector connecting to mysql database. Is the behavior the same if all you have in your webapp is the Hello servlet? I.e. you take all other servlets, especially the ones that use JDBC, out? There have been problems reported in the past with DB connection pool threads just hanging around. the simple Hello servlet (code at the bottom). After calling it a few times there was no So the first few times it works? more response from the server and the tomcat manager/status page (which sometimes returns and sometimes not) clearly shows that the Hello servlet is stuck: Stage 1: After calling the Hello servlet StageTimeB SentB RecvClientVHost Request S32671 ms 0KB0KB194.90.90.18 mdrmus Get Actually, this shows your servlet wasn't even called: no bytes sent OR received. Are you sure you're not having a network failure? Also, the fact the manager status page sometimes works and sometimes doesn't is worrisome. I've never had problems with it, and it should be very reliable. Can you can the manager webapp to malfunction even without any of your webapps deployed on the server? If so, open a bugzilla issue with steps to reproduce your behavior. If anyone has any idea how to solve this problem, I would be most grateful. Any errors in the logs? (Also make sure you have adequate disk space to write logs and temp files). Yoav Shapira This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: tomcat jitters, then hangs - please help
Hi Noam :) Check the server.xml for the parameter maxWait if it's configured to -1 (which mean maximum time to wait for a dB connection to become available is indefinitely) then you should try changing it to something like 1 (10 sec). If you have a problem with your DB then the tomcat will get stuck waiting for dB connection to become available Good place to get some more information is http://jakarta.apache.org/tomcat/tomcat-4.1-doc/jndi-datasource-examples-howto.html Arthur - Original Message - From: Noam Camiel [EMAIL PROTECTED] To: Tomcat Users List [EMAIL PROTECTED] Sent: Wednesday, October 01, 2003 7:11 PM Subject: Re: tomcat jitters, then hangs - please help Hi, and thanks for the comments, On Wednesday, October 01, 2003 2:52 PM Shapira, Yoav [EMAIL PROTECTED] wrote: Oh, these problems are invariably a pain to debug ;) yea nothing like a good challenge :) At that time, if you send the VM a SIGQUIT, what are the threads doing? I've got a thread dump at the end of the file. Please have a look and tell me if you see something odd. Is the behavior the same if all you have in your webapp is the Hello servlet? I.e. you take all other servlets, especially the ones that use JDBC, out? There have been problems reported in the past with DB connection pool threads just hanging around. I just set up my other linux machine only with the Hello servlet, lets see if that gets to the same state, so far its OK the simple Hello servlet (code at the bottom). After calling it a few times there was no So the first few times it works? Yes, all works well until a time when the Tomcat reaches a state where is sometimes replies to HTTP requests and sometimes not. I managed to get the manager status page this way. Shortly after it all gets stuck. Also, the fact the manager status page sometimes works and sometimes doesn't is worrisome. I've never had problems with it, and it should be very reliable. Can you can the manager webapp to malfunction even without any of your webapps deployed on the server? If so, open a bugzilla issue with steps to reproduce your behavior. The manager webapp work well until after the unstable state where the Tomcat simply stops answering any HTTP request. Any errors in the logs? (Also make sure you have adequate disk space to write logs and temp files). No errors in the logs and no problems with disk space. On Wednesday, October 01, 2003 3:10 PM Arthur Veinstein [EMAIL PROTECTED] wrote: Hi Arthur ;) Sounds like you have some problem with the DB connection. Can you provide the server.xml configuration ? The server.xml is the same as it was with the installation, apart from the port change from 8080 to 80. Could that lead to a problem if I work with JDBC? Please let me know what you think, Thanks, Noam ---thread dump - Full thread dump Java HotSpot(TM) Client VM (1.4.1_05-b01 mixed mode): TP-Monitor daemon prio=1 tid=0x0x82440a0 nid=0x72c3 in Object.wait() [4e29f000..4e29f390] at java.lang.Object.wait(Native Method) - waiting on 0x44cd9760 (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable) at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.jav a:549) - locked 0x44cd9760 (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable) at java.lang.Thread.run(Thread.java:536) TP-Processor4 daemon prio=1 tid=0x0x8243390 nid=0x72c3 runnable [4e21f000..4e21f410] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353) - locked 0x44cd97e8 (a java.net.PlainSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:448) at java.net.ServerSocket.accept(ServerSocket.java:419) at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:278) at org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:572) at org.apache.jk.common.SocketAcceptor.runIt(ChannelSocket.java:758) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav a:666) at java.lang.Thread.run(Thread.java:536) TP-Processor3 daemon prio=1 tid=0x0x8242818 nid=0x72c3 in Object.wait() [4e19f000..4e19f490] at java.lang.Object.wait(Native Method) - waiting on 0x44cd9958 (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Object.wait(Object.java:426) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav a:642) - locked 0x44cd9958 (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Thread.run(Thread.java:536) TP-Processor2 daemon prio=1 tid=0x0x82779d8 nid=0x72c3 in Object.wait() [4e11f000..4e11f510] at java.lang.Object.wait(Native Method) - waiting on 0x44cd99d8 (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Object.wait(Object.java:426
Re: tomcat jitters, then hangs - please help
Shapira, Yoav wrote: Howdy, At that time, if you send the VM a SIGQUIT, what are the threads doing? I've got a thread dump at the end of the file. Please have a look and tell me if you see something odd. It seems you also have a JK connector setup: let's try to reduce the variables by running tomcat standalone with only one Coyote connector. Disable the JK/JK2/whatever connectors to Apache, especially if you're not using them anyways. Other than that, nothing unusual jumped at me from the stack trace. I have to give him some credit: there's only one connector listening on a server socket. All the threads for the HTTP connector are busy doing nothing. Now, could a better explanation of how to get in this situation be given ? -- x Rémy Maucherat Senior Developer Consultant JBoss Group (Europe) SàRL x - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: tomcat jitters, then hangs - please help
Hiya, On Wednesday, October 01, 2003 5:53 PM Shapira, Yoav [EMAIL PROTECTED] Wrote: It seems you also have a JK connector setup: let's try to reduce the variables by running tomcat standalone with only one Coyote connector. Disable the JK/JK2/whatever connectors to Apache, especially if you're not using them anyways. Other than that, nothing unusual jumped at me from the stack trace. OK, I disabled the connector. On Wednesday, October 01, 2003 6:03 PM Arthur Veinstein [EMAIL PROTECTED] Wrote: Hi Noam :) Check the server.xml for the parameter maxWait if it's configured to -1 (which mean maximum time to wait for a dB connection to become available is indefinitely) then you should try changing it to something like 1 (10 sec). If you have a problem with your DB then the tomcat will get stuck waiting for dB connection to become available I actually didnt find mawWait but did find connectionTimeout which was already set to 2 I also received from Remy Maucherat [EMAIL PROTECTED] an update to the tomcat-util.jar so I'm trying that as well. The Tomcat server is now running and I'll keep you posted tomorow. Thanks for your help so far, Noam. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
