RE: Bug in Tomcat AJP Connector?
On 08/04/2012 23:14, Stefan Mayr ste...@mayr-stefan.de wrote: Am 08.04.2012 18:41, schrieb Ofer Israeli: 2012/4/6 Pidp...@pidster.com: On 05/04/2012 22:17, Ofer Israeli wrote: Y On 5 באפר 2012, at 18:58, Konstantin Kolinkoknst.koli...@gmail.com wrote: 2012/4/5 Ofer Israeliof...@checkpoint.com: Mark Thomas wrote: On 04/04/2012 17:02, Ofer Israeli wrote: Once you have an OOME all bets are off. The JVM needs to be restarted. There is no guarantee of reliable operation after an OOME. Mark Hi Mark, I agree that there in such a situation the JVM should be restarted, but it isn't restarted by Tomcat. On the other hand, Tomcat does take some precautious actions and kills the accepting thread, but in such a case it should also close the socket that thread is listening on otherwise it is leaving garbage around after the thread's death. Do you see any reason as not to close the listening socket? 1. Tomcat does not start JVM thus it cannot restart it. You need some external tool or script or admin to perform monitoring and (re)starts. 2. OOM can happen at a random place. Once it happens, it is likely that other places will also start to fail randomly. It is also likely that your attempts to recover will fail as well. Mark already mentioned it: all bets are off. Best regards, Konstantin Kolinko Hi Konstantin, I agree regarding the OOM bringing TC to a state where it must be restored, but my point remains: if there is code that handles catching this exception and terminating the thread, why not terminate gracefully by closing the listening socket before killing the thread? And your point has been answered. After an OOM the JVM is in an unknown, unsafe state so a restart MUST occur to restore service. Closing a socket gracefully after an OOM is a bit like trying to shut one of the portholes on the Titanic, shortly after hearing a large crashing sound. There's only one place I know of where Tomcat attempts to interact with OOM conditions and this is not one of them, so I don't believe it's safe to say that Tomcat is deliberately handling this exception. NB an OOM is an Error, not an Exception - it is a subclass of VirtualMachineError, which is thrown to indicate that the Java Virtual Machine is broken or has run out of resources necessary for it to continue operating. An Error is a subclass of Throwable that indicates serious problems that a reasonable application should not try to catch. /end-quote If anything, the locations where Tomcat catches a Throwable should be modified so it does *not* catch Errors, rather than continuing to do so and then attempting a trivial tidy-up. p Thanks for your input - you're right regarding the error and the fact that Tomcat is indeed catching a Throwable and not an Exception. I assume that if the Throwable were not caught, then the thread would die in any case. Although stated before that Tomcat could not kill itself in such a situation, I still wonder if it would be possible to do so. Or taking a different perspective on this: if the JVM specification is such that it cannot be trusted to continue running after an OOM, then why does it not kill itself or restart itself? I guess you can do this with some vendor specific JVM arguments as SUNs/Oracles -XX:OnOutOfMemoryError: http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp- 140102.html Different findings like kill -9 %p let me suspect that you can use %p as a variable for your current pid. With that you can either kill your current instance and let your monitoring handle the rest or try to initiate the restart by yourself. Give it a try Stefan Thanks Stefan - will look into this option.
RE: Bug in Tomcat AJP Connector?
2012/4/6 Konstantin Kolinko knst.koli...@gmail.com: 2012/4/6 Ofer Israeli of...@checkpoint.com: Hi Konstantin, I agree regarding the OOM bringing TC to a state where it must be restored, but my point remains: if there is code that handles catching this exception and terminating the thread, why not terminate gracefully by closing the listening socket before killing the thread? 1. In theory one can configure several acceptor threads. Though it is not recommended. So one has to check the count before closing the socket. This sounds like the reason for not closing the socket. Is this a TC configuration? 2. The socket will be closed when Tomcat shuts down. Just closing the socket - what will it improve? Anyway you will need to shutdown this Tomcat instance before you can start the second one, because you'll need to free the shutdown port etc. The point is to do proper cleanup when terminating a thread. One good reason for this is the strange situation the system is left in where you can see that the Tomcat process is listening on port 8009 via netstat, yet it actually isn't listening on that port as the acceptor thread has died. This makes it very difficult to troubleshoot as the system is showing false information. if there is code that handles catching this exception and terminating the thread 3. There is no such special code. The thread terminates by itself, because its run() throws an exception and thus finishes execution. There is ThreadGroup.uncaughtException() method in JRE which by default prints the exception to stdout. I see specific Tomcat code that catches Throwable, prints out this exception and then terminates the thread in ThreadPool: ThreadPool.log.error(sm.getString (threadpool.thread_error, t, toRun.toString())); By the way, this is not relevant only for OOME, but also for any other exception that was not handled and reached this code. If you consider to propose a patch - feel free to file an enhancement request in Bugzilla and attach it. In accordance with your comments on the above, I'll file a patch if it looks in place. Thanks, Ofer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Bug in Tomcat AJP Connector?
2012/4/6 Pid p...@pidster.com: On 05/04/2012 22:17, Ofer Israeli wrote: Y On 5 באפר 2012, at 18:58, Konstantin Kolinko knst.koli...@gmail.com wrote: 2012/4/5 Ofer Israeli of...@checkpoint.com: Mark Thomas wrote: On 04/04/2012 17:02, Ofer Israeli wrote: Once you have an OOME all bets are off. The JVM needs to be restarted. There is no guarantee of reliable operation after an OOME. Mark Hi Mark, I agree that there in such a situation the JVM should be restarted, but it isn't restarted by Tomcat. On the other hand, Tomcat does take some precautious actions and kills the accepting thread, but in such a case it should also close the socket that thread is listening on otherwise it is leaving garbage around after the thread's death. Do you see any reason as not to close the listening socket? 1. Tomcat does not start JVM thus it cannot restart it. You need some external tool or script or admin to perform monitoring and (re)starts. 2. OOM can happen at a random place. Once it happens, it is likely that other places will also start to fail randomly. It is also likely that your attempts to recover will fail as well. Mark already mentioned it: all bets are off. Best regards, Konstantin Kolinko Hi Konstantin, I agree regarding the OOM bringing TC to a state where it must be restored, but my point remains: if there is code that handles catching this exception and terminating the thread, why not terminate gracefully by closing the listening socket before killing the thread? And your point has been answered. After an OOM the JVM is in an unknown, unsafe state so a restart MUST occur to restore service. Closing a socket gracefully after an OOM is a bit like trying to shut one of the portholes on the Titanic, shortly after hearing a large crashing sound. There's only one place I know of where Tomcat attempts to interact with OOM conditions and this is not one of them, so I don't believe it's safe to say that Tomcat is deliberately handling this exception. NB an OOM is an Error, not an Exception - it is a subclass of VirtualMachineError, which is thrown to indicate that the Java Virtual Machine is broken or has run out of resources necessary for it to continue operating. An Error is a subclass of Throwable that indicates serious problems that a reasonable application should not try to catch. /end-quote If anything, the locations where Tomcat catches a Throwable should be modified so it does *not* catch Errors, rather than continuing to do so and then attempting a trivial tidy-up. p Thanks for your input - you're right regarding the error and the fact that Tomcat is indeed catching a Throwable and not an Exception. I assume that if the Throwable were not caught, then the thread would die in any case. Although stated before that Tomcat could not kill itself in such a situation, I still wonder if it would be possible to do so. Or taking a different perspective on this: if the JVM specification is such that it cannot be trusted to continue running after an OOM, then why does it not kill itself or restart itself? Thanks, Ofer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Bug in Tomcat AJP Connector?
Mark Thomas wrote: On 04/04/2012 17:02, Ofer Israeli wrote: Hi all, We have recently witnessed a strange situation. Our Tomcat (6.0.35) listens on 2 ports: 8080 and 8009, handling HTTP and AJP respectively. At some point in time we found that Apache is replying HTTP 503s to all clients and a look at netstat showed that indeed Apache could not communicate with Tomcat - there were many connections in state SYN_SENT, that were never replied. A test of attempting to connect to Tomcat's 8009 port turned out to give the same result, so the issue was obviously in Tomcat and not Apache. But netstat also showed that Tomcat was listening on port 8009. After digging into the logs, we found that there was an Out of Memory exception in the JK's accept() sequence and once this exception was caught there was a message of terminating thread. So although the exception is caught and handled by killing the thread (isn't that a little drastic?), the socket isn't closed beforehand and thus the OS thinks that the port is still fine (in LISTENING state), as although the thread is dead, the process is alive. Has anyone encountered this or knows if this bug is known? Once you have an OOME all bets are off. The JVM needs to be restarted. There is no guarantee of reliable operation after an OOME. Mark Hi Mark, I agree that there in such a situation the JVM should be restarted, but it isn't restarted by Tomcat. On the other hand, Tomcat does take some precautious actions and kills the accepting thread, but in such a case it should also close the socket that thread is listening on otherwise it is leaving garbage around after the thread's death. Do you see any reason as not to close the listening socket? Ofer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Bug in Tomcat AJP Connector?
Y On 5 באפר 2012, at 18:58, Konstantin Kolinko knst.koli...@gmail.com wrote: 2012/4/5 Ofer Israeli of...@checkpoint.com: Mark Thomas wrote: On 04/04/2012 17:02, Ofer Israeli wrote: Once you have an OOME all bets are off. The JVM needs to be restarted. There is no guarantee of reliable operation after an OOME. Mark Hi Mark, I agree that there in such a situation the JVM should be restarted, but it isn't restarted by Tomcat. On the other hand, Tomcat does take some precautious actions and kills the accepting thread, but in such a case it should also close the socket that thread is listening on otherwise it is leaving garbage around after the thread's death. Do you see any reason as not to close the listening socket? 1. Tomcat does not start JVM thus it cannot restart it. You need some external tool or script or admin to perform monitoring and (re)starts. 2. OOM can happen at a random place. Once it happens, it is likely that other places will also start to fail randomly. It is also likely that your attempts to recover will fail as well. Mark already mentioned it: all bets are off. Best regards, Konstantin Kolinko Hi Konstantin, I agree regarding the OOM bringing TC to a state where it must be restored, but my point remains: if there is code that handles catching this exception and terminating the thread, why not terminate gracefully by closing the listening socket before killing the thread? Best regards, Ifer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Bug in Tomcat AJP Connector?
Hi all, We have recently witnessed a strange situation. Our Tomcat (6.0.35) listens on 2 ports: 8080 and 8009, handling HTTP and AJP respectively. At some point in time we found that Apache is replying HTTP 503s to all clients and a look at netstat showed that indeed Apache could not communicate with Tomcat - there were many connections in state SYN_SENT, that were never replied. A test of attempting to connect to Tomcat's 8009 port turned out to give the same result, so the issue was obviously in Tomcat and not Apache. But netstat also showed that Tomcat was listening on port 8009. After digging into the logs, we found that there was an Out of Memory exception in the JK's accept() sequence and once this exception was caught there was a message of terminating thread. So although the exception is caught and handled by killing the thread (isn't that a little drastic?), the socket isn't closed beforehand and thus the OS thinks that the port is still fine (in LISTENING state), as although the thread is dead, the process is alive. Has anyone encountered this or knows if this bug is known? Thanks, Ofer
RE: Tomcat with mod_jk becomes irresponsive after working for awhile
Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ofer, On 2/23/12 1:57 PM, Ofer Israeli wrote: Felix Schumacher wrote: Am 23.02.2012 19:32, schrieb Ofer Israeli: Caldarale, Charles R wrote: From: Caldarale, Charles R Subject: RE: Tomcat with mod_jk becomes irresponsive after working for awhile You might want to take a thread dump of Tomcat and see just what is going on. Also look in the Tomcat logs (there are several) to see if errors are being reported there. - Chuck Which logs are you referring to? I've turned on org.apache logs via log4j and haven't found anything related. Normally they can be found under $CATALINA_BASE/logs. The most important ones are usually catalina.out and localhost*log. Regards Felix I don't see these logs in the server. Could it be that this is due to Tomcat being embedded? Wait, you are running Tomcat as an embedded service? That seems liek it would be reasonable information to include in your original post. Why did it take 6 posts to get this information out of you? If you are embedding Tomcat, you should have control over the logging setup. Where are you redirecting stdout? - -chris Hi Chris, Guess it's my bad for not including this information in previous posts - I didn't know if this is related to the problem at hand or not. On startup we redirect stdout and stderr to the same log file that we log other information. Cheers, Ofer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat with mod_jk becomes irresponsive after working for awhile
Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ofer, On 2/27/12 6:11 AM, Ofer Israeli wrote: Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ofer, On 2/23/12 1:57 PM, Ofer Israeli wrote: Felix Schumacher wrote: Am 23.02.2012 19:32, schrieb Ofer Israeli: Caldarale, Charles R wrote: From: Caldarale, Charles R Subject: RE: Tomcat with mod_jk becomes irresponsive after working for awhile You might want to take a thread dump of Tomcat and see just what is going on. Also look in the Tomcat logs (there are several) to see if errors are being reported there. - Chuck Which logs are you referring to? I've turned on org.apache logs via log4j and haven't found anything related. Normally they can be found under $CATALINA_BASE/logs. The most important ones are usually catalina.out and localhost*log. Regards Felix I don't see these logs in the server. Could it be that this is due to Tomcat being embedded? Wait, you are running Tomcat as an embedded service? That seems liek it would be reasonable information to include in your original post. Why did it take 6 posts to get this information out of you? If you are embedding Tomcat, you should have control over the logging setup. Where are you redirecting stdout? - -chris Hi Chris, Guess it's my bad for not including this information in previous posts - I didn't know if this is related to the problem at hand or not. On startup we redirect stdout and stderr to the same log file that we log other information. How exactly do you do that? - -chris In our code, on startup, we close stdout and stderr via the System class. Then we set the output and error streams to a propietary class which uses Log4J to write out to our file. /Ofer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat with mod_jk becomes irresponsive after working for awhile
Felix Schumacher wrote: Am 23.02.2012 19:50, schrieb Ofer Israeli: Felix Schumacher wrote: Am 23.02.2012 19:32, schrieb Ofer Israeli: Caldarale, Charles R wrote: From: Ofer Israeli [mailto:of...@checkpoint.com] Subject: RE: Tomcat with mod_jk becomes irresponsive after working for awhile I am in the situation where the server is giving 503s for all incoming requests, so I did a capture now to see what the situation is and as mentioned above I see Tomcat FINing the connections right away before mod_jk sends any content You might want to take a thread dump of Tomcat and see just what is going on. If all the threads are busy (or stuck), you may well have an application problem causing the 503. - Chuck Hi Chuck, I've taken a thread dump and haven't found anything that looks suspicious. What baffles me is that TaskManager shows that I have 580 threads in the Tomcat process (which makes sense since I use 500 worker threads for requests), but the thread dump only shows a small portion of these. Below is the dump, I'll be happy if you can give me some insight on this. 2012-02-23 20:08:16 Full thread dump Java HotSpot(TM) Client VM (19.1-b02 mixed mode): Low Memory Detector daemon prio=6 tid=0x16bc7400 nid=0x2b38 runnable [0x] java.lang.Thread.State: RUNNABLE ... main prio=6 tid=0x002a7c00 nid=0x2ecc runnable [0x0093f000] java.lang.Thread.State: RUNNABLE at sun.tools.attach.WindowsVirtualMachine.connectPipe(Native Method)at sun.tools.attach.WindowsVirtualMachine.execute(WindowsVirtualMachine .java:82) at sun.tools.attach.HotSpotVirtualMachine.executeCommand(HotSpotVirtual Machine.java:195) at sun.tools.attach.HotSpotVirtualMachine.remoteDataDump(HotSpotVirtualMachine.java:156) at sun.tools.jstack.JStack.runThreadDump(JStack.java:159) at sun.tools.jstack.JStack.main(JStack.java:94) This is not a thread dump of tomcat, but rather jstack. Look at http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_ my_running_webapp_.3F for more info, to get a thread dump of tomcat. Regards Felix Hi Felix, I have seen that page but actually can't use the //MS// option as Tomcat is already running and in this bad state that I want to catch without restarting the service. Is there some way to gather this information without a restart? From what I understood jstack is supposed to give me all the JVM threads - am I missing something, can you elaborate on what it is I see there and why there are portions missing? Hi Ofer, I haven't used tomcat under windows, so I might be a bad adviser. Maybe you could use a tool like visualvm or jconsole from the jdk to attach to the running tomcat jvm and have a look at the threads. We will probably have to wait for windows users to help you. Regards Felix Hi all, So far I haven't found any way to get a thread dump of Tomcat running as a service on Windows 2003 Server without restarting the service. Does anyone know how this can be done? Thanks, Ofer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat with mod_jk becomes irresponsive after working for awhile
Caldarale, Charles R wrote: From: Ofer Israeli [mailto:of...@checkpoint.com] Subject: RE: Tomcat with mod_jk becomes irresponsive after working for awhile I am in the situation where the server is giving 503s for all incoming requests, so I did a capture now to see what the situation is and as mentioned above I see Tomcat FINing the connections right away before mod_jk sends any content You might want to take a thread dump of Tomcat and see just what is going on. If all the threads are busy (or stuck), you may well have an application problem causing the 503. - Chuck Hi Chuck, I've taken a thread dump and haven't found anything that looks suspicious. What baffles me is that TaskManager shows that I have 580 threads in the Tomcat process (which makes sense since I use 500 worker threads for requests), but the thread dump only shows a small portion of these. Below is the dump, I'll be happy if you can give me some insight on this. 2012-02-23 20:08:16 Full thread dump Java HotSpot(TM) Client VM (19.1-b02 mixed mode): Low Memory Detector daemon prio=6 tid=0x16bc7400 nid=0x2b38 runnable [0x] java.lang.Thread.State: RUNNABLE CompilerThread0 daemon prio=10 tid=0x16bbe400 nid=0x2828 waiting on condition [0x] java.lang.Thread.State: RUNNABLE Attach Listener daemon prio=10 tid=0x16bbcc00 nid=0xc4c runnable [0x] java.lang.Thread.State: RUNNABLE Signal Dispatcher daemon prio=10 tid=0x16bbb800 nid=0x1378 waiting on condition [0x] java.lang.Thread.State: RUNNABLE Finalizer daemon prio=8 tid=0x16baac00 nid=0x2f4c in Object.wait() [0x16d1f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x029f1158 (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) - locked 0x029f1158 (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) Reference Handler daemon prio=10 tid=0x16ba6000 nid=0x2c5c in Object.wait() [0x16ccf000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x029f1058 (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) - locked 0x029f1058 (a java.lang.ref.Reference$Lock) main prio=6 tid=0x002a7c00 nid=0x2ecc runnable [0x0093f000] java.lang.Thread.State: RUNNABLE at sun.tools.attach.WindowsVirtualMachine.connectPipe(Native Method) at sun.tools.attach.WindowsVirtualMachine.execute(WindowsVirtualMachine.java:82) at sun.tools.attach.HotSpotVirtualMachine.executeCommand(HotSpotVirtualMachine.java:195) at sun.tools.attach.HotSpotVirtualMachine.remoteDataDump(HotSpotVirtualMachine.java:156) at sun.tools.jstack.JStack.runThreadDump(JStack.java:159) at sun.tools.jstack.JStack.main(JStack.java:94) VM Thread prio=10 tid=0x16ba2400 nid=0x320 runnable VM Periodic Task Thread prio=10 tid=0x16bcac00 nid=0x2cb4 waiting on condition JNI global references: 913 Heap def new generation total 4928K, used 756K [0x029f, 0x02f4, 0x07f4) eden space 4416K, 17% used [0x029f, 0x02aad0a0, 0x02e4) from space 512K, 0% used [0x02e4, 0x02e4, 0x02ec) to space 512K, 0% used [0x02ec, 0x02ec, 0x02f4) tenured generation total 10944K, used 0K [0x07f4, 0x089f, 0x129f) the space 10944K, 0% used [0x07f4, 0x07f4, 0x07f40200, 0x089f) compacting perm gen total 12288K, used 2379K [0x129f, 0x135f, 0x169f) the space 12288K, 19% used [0x129f, 0x12c42d68, 0x12c42e00, 0x135f) No shared spaces configured. Thanks, Ofer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat with mod_jk becomes irresponsive after working for awhile
Caldarale, Charles R wrote: From: Caldarale, Charles R Subject: RE: Tomcat with mod_jk becomes irresponsive after working for awhile You might want to take a thread dump of Tomcat and see just what is going on. Also look in the Tomcat logs (there are several) to see if errors are being reported there. - Chuck Which logs are you referring to? I've turned on org.apache logs via log4j and haven't found anything related. Thanks, Ofer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat with mod_jk becomes irresponsive after working for awhile
Felix Schumacher wrote: Am 23.02.2012 19:32, schrieb Ofer Israeli: Caldarale, Charles R wrote: From: Ofer Israeli [mailto:of...@checkpoint.com] Subject: RE: Tomcat with mod_jk becomes irresponsive after working for awhile I am in the situation where the server is giving 503s for all incoming requests, so I did a capture now to see what the situation is and as mentioned above I see Tomcat FINing the connections right away before mod_jk sends any content You might want to take a thread dump of Tomcat and see just what is going on. If all the threads are busy (or stuck), you may well have an application problem causing the 503. - Chuck Hi Chuck, I've taken a thread dump and haven't found anything that looks suspicious. What baffles me is that TaskManager shows that I have 580 threads in the Tomcat process (which makes sense since I use 500 worker threads for requests), but the thread dump only shows a small portion of these. Below is the dump, I'll be happy if you can give me some insight on this. 2012-02-23 20:08:16 Full thread dump Java HotSpot(TM) Client VM (19.1-b02 mixed mode): Low Memory Detector daemon prio=6 tid=0x16bc7400 nid=0x2b38 runnable [0x] java.lang.Thread.State: RUNNABLE ... main prio=6 tid=0x002a7c00 nid=0x2ecc runnable [0x0093f000] java.lang.Thread.State: RUNNABLE at sun.tools.attach.WindowsVirtualMachine.connectPipe(Native Method) at sun.tools.attach.WindowsVirtualMachine.execute(WindowsVirtualMachine.java:82) at sun.tools.attach.HotSpotVirtualMachine.executeCommand(HotSpotVirtualMachine.java:195) at sun.tools.attach.HotSpotVirtualMachine.remoteDataDump(HotSpotVirtualMachine.java:156) at sun.tools.jstack.JStack.runThreadDump(JStack.java:159) at sun.tools.jstack.JStack.main(JStack.java:94) This is not a thread dump of tomcat, but rather jstack. Look at http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F for more info, to get a thread dump of tomcat. Regards Felix Hi Felix, I have seen that page but actually can't use the //MS// option as Tomcat is already running and in this bad state that I want to catch without restarting the service. Is there some way to gather this information without a restart? From what I understood jstack is supposed to give me all the JVM threads - am I missing something, can you elaborate on what it is I see there and why there are portions missing? Thanks, OFer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat with mod_jk becomes irresponsive after working for awhile
Felix Schumacher wrote: Am 23.02.2012 19:32, schrieb Ofer Israeli: Caldarale, Charles R wrote: From: Caldarale, Charles R Subject: RE: Tomcat with mod_jk becomes irresponsive after working for awhile You might want to take a thread dump of Tomcat and see just what is going on. Also look in the Tomcat logs (there are several) to see if errors are being reported there. - Chuck Which logs are you referring to? I've turned on org.apache logs via log4j and haven't found anything related. Normally they can be found under $CATALINA_BASE/logs. The most important ones are usually catalina.out and localhost*log. Regards Felix I don't see these logs in the server. Could it be that this is due to Tomcat being embedded? Thanks, Ofer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat with mod_jk becomes irresponsive after working for awhile
All, I have a deployment on Windows 2003 Server using Apache 2.2.16, Tomcat 5.5.30 and mod_jk 1.2.31. Apache and Tomcat reside on the same machine. The communication is fine for days / weeks, but at some point all requests to the server (to Apache) that use mod_jk return HTTP 503. The error that shows up in mod_jk.log is: [Mon Feb 20 12:07:01.464 2012] [7280:6912] [error] ajp_send_request::jk_ajp_common.c (1610): (all) failed sending request on a fresh connection (recoverable), socket -1 (errno=54) [Mon Feb 20 12:07:01.464 2012] [7280:6912] [error] ajp_service::jk_ajp_common.c (2562): (all) connecting to tomcat failed. I've done a packet capture and I see that Apache initiates the 3-way handshake with Tomcat which looks fine and immediately after the connection is established, Tomcat sends a FIN packet to Apache whereby both sides gracefully close the connection. Have any of you encountered this before? Do you have any idea what debug logs I can turn on to get further information without bringing down the server so that we don't loose this situation? Many thanks, Ofer
RE: Tomcat with mod_jk becomes irresponsive after working for awhile
André Warnier wrote: Ofer Israeli wrote: All, I have a deployment on Windows 2003 Server using Apache 2.2.16, Tomcat 5.5.30 and mod_jk 1.2.31. Apache and Tomcat reside on the same machine. The communication is fine for days / weeks, but at some point all requests to the server (to Apache) that use mod_jk return HTTP 503. The error that shows up in mod_jk.log is: [Mon Feb 20 12:07:01.464 2012] [7280:6912] [error] ajp_send_request::jk_ajp_common.c (1610): (all) failed sending request on a fresh connection (recoverable), socket -1 (errno=54) [Mon Feb 20 12:07:01.464 2012] [7280:6912] [error] ajp_service::jk_ajp_common.c (2562): (all) connecting to tomcat failed. I've done a packet capture and I see that Apache initiates the 3-way handshake with Tomcat which looks fine and immediately after the connection is established, Tomcat sends a FIN packet to Apache whereby both sides gracefully close the connection. Have any of you encountered this before? Do you have any idea what debug logs I can turn on to get further information without bringing down the server so that we don't loose this situation? Hi. It sounds unlikely that you would have done a permanent packet capture during days/weeks, just waiting for this to happen. So what made you decide when to start your packet capture, or how do you force the problem to happen when you have started your packet capture ? Hi Andre, I do not have a permenant packet capture, but rather now I am in the situation where the server is giving 503s for all incoming requests, so I did a capture now to see what the situation is and as mentioned above I see Tomcat FINing the connections right away before mod_jk sends any content (putting aside the connection establishment). - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
