The process does respond to a kill -3. The resulting threadump contains
the following entry for every thread from 13 - 67. The data for each of
these threads is identical except for the thread number, 'tid', 'nid',
etc...:
"Thread-13" daemon prio=5 tid=0x6ee600 nid=0x1f waiting on monitor
[ef901000..ef9019c0]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:426)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:509)
- locked <f329d5b0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:536)
This is the rest of the dump:
"Thread-12" daemon prio=5 tid=0x594938 nid=0x1e runnable
[efb81000..efb819c0]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:343)
- locked <f2fe33b0> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:438)
at java.net.ServerSocket.accept(ServerSocket.java:409)
at
org.apache.axis.monitor.SOAPMonitorService$ServerSocketThread.run(SOAPMonitorService.java:224)
at java.lang.Thread.run(Thread.java:536)
"MonitorRunnable" daemon prio=5 tid=0x6077c8 nid=0x1c waiting on monitor
[f0381000..f03819c0]
at java.lang.Object.wait(Native Method)
- waiting on <f2aab3b0> (a
org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
at
org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:420)
- locked <f2aab3b0> (a
org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
at java.lang.Thread.run(Thread.java:536)
"Thread-10" daemon prio=5 tid=0x609c48 nid=0x1b runnable
[f0481000..f04819c0]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:343)
- locked <f2aab420> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:438)
at java.net.ServerSocket.accept(ServerSocket.java:409)
at
org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:268)
at
org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:559)
at
org.apache.jk.common.SocketAcceptor.runIt(ChannelSocket.java:688)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:530)
at java.lang.Thread.run(Thread.java:536)
"Thread-9" daemon prio=5 tid=0x60e3b8 nid=0x1a waiting on monitor
[f0581000..f05819c0]
at java.lang.Object.wait(Native Method)
- waiting on <f2aab5d0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:426)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:509)
- locked <f2aab5d0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:536)
"Thread-8" daemon prio=5 tid=0x60d3a0 nid=0x19 waiting on monitor
[f0681000..f06819c0]
at java.lang.Object.wait(Native Method)
- waiting on <f2aab638> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:426)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:509)
- locked <f2aab638> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:536)
"Thread-7" daemon prio=5 tid=0x60db78 nid=0x18 waiting on monitor
[f0781000..f07819c0]
at java.lang.Object.wait(Native Method)
- waiting on <f2aab6a0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:426)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:509)
- locked <f2aab6a0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:536)
"MonitorRunnable" daemon prio=5 tid=0x60a3b0 nid=0x17 waiting on monitor
[f0881000..f08819c0]
at java.lang.Object.wait(Native Method)
- waiting on <f2aab708> (a
org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
at
org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:420)
- locked <f2aab708> (a
org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
at java.lang.Thread.run(Thread.java:536)
"Thread-5" daemon prio=5 tid=0x604a90 nid=0x16 waiting on monitor
[f0981000..f09819c0]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:426)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:509)
- locked <f2aab8a8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:536)
"Thread-4" daemon prio=5 tid=0x6047f8 nid=0x15 waiting on monitor
[f0a81000..f0a819c0]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:426)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:509)
- locked <f2aab928> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:536)
"Thread-3" daemon prio=5 tid=0x603980 nid=0x14 waiting on monitor
[f0b81000..f0b819c0]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:426)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:509)
- locked <f2aab990> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:536)
"Thread-2" daemon prio=5 tid=0x607bd0 nid=0x13 waiting on monitor
[f0c81000..f0c819c0]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:426)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:509)
"HostConfig[localhost]" daemon prio=5 tid=0x603cb8 nid=0x12 waiting on
monitor [f0d81000..f0d819c0]
at java.lang.Thread.sleep(Native Method)
at
org.apache.catalina.startup.HostConfig.threadSleep(HostConfig.java:985)
at org.apache.catalina.startup.HostConfig.run(HostConfig.java:1009)
at java.lang.Thread.run(Thread.java:536)
"StandardManager[/webdav]" daemon prio=5 tid=0x6a3250 nid=0x11 waiting on
monitor [f0e81000..f0e819c0]
at java.lang.Thread.sleep(Native Method)
at
org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java:780)
at
org.apache.catalina.session.StandardManager.run(StandardManager.java:839)
at java.lang.Thread.run(Thread.java:536)
"StandardManager[]" daemon prio=5 tid=0x5006d8 nid=0x10 waiting on monitor
[f0f81000..f0f819c0]
at java.lang.Thread.sleep(Native Method)
at
org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java:780)
at
org.apache.catalina.session.StandardManager.run(StandardManager.java:839)
at java.lang.Thread.run(Thread.java:536)
"StandardManager[/tomcat-docs]" daemon prio=5 tid=0x3adc70 nid=0xf waiting
on monitor [f1081000..f10819c0]
at java.lang.Thread.sleep(Native Method)
at
org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java:780)
at
org.apache.catalina.session.StandardManager.run(StandardManager.java:839)
at java.lang.Thread.run(Thread.java:536)
"StandardManager[/admin]" daemon prio=5 tid=0xa4a28 nid=0xe waiting on
monitor [f1181000..f11819c0]
at java.lang.Thread.sleep(Native Method)
at
org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java:780)
at
org.apache.catalina.session.StandardManager.run(StandardManager.java:839)
at java.lang.Thread.run(Thread.java:536)
"StandardManager[/manager]" daemon prio=5 tid=0x38b7a0 nid=0xd waiting on
monitor [f1281000..f12819c0]
at java.lang.Thread.sleep(Native Method)
at
org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java:780)
at
org.apache.catalina.session.StandardManager.run(StandardManager.java:839)
at java.lang.Thread.run(Thread.java:536)
"StandardManager[/examples]" daemon prio=5 tid=0x767268 nid=0xc waiting on
monitor [f1381000..f13819c0]
at java.lang.Thread.sleep(Native Method)
at
org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java:780)
at
org.apache.catalina.session.StandardManager.run(StandardManager.java:839)
at java.lang.Thread.run(Thread.java:536)
"WebappLoader[/examples]" daemon prio=5 tid=0x7508c0 nid=0xb waiting on
monitor [f1981000..f19819c0]
at java.lang.Thread.sleep(Native Method)
at
org.apache.catalina.loader.WebappLoader.threadSleep(WebappLoader.java:1221)
at
org.apache.catalina.loader.WebappLoader.run(WebappLoader.java:1341)
at java.lang.Thread.run(Thread.java:536)
"Signal Dispatcher" daemon prio=10 tid=0xae370 nid=0x9 waiting on monitor
[0..0]
"Finalizer" daemon prio=8 tid=0xa9b98 nid=0x6 waiting on monitor
[fa281000..fa2819c0]
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
- locked <f27580c0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0xa8248 nid=0x5 waiting on monitor
[fa381000..fa3819c0]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:426)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:113)
- locked <f2758128> (a java.lang.ref.Reference$Lock)
"main" prio=5 tid=0x2baf0 nid=0x1 runnable [ffbee000..ffbef038]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:343)
- locked <f2aabab8> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:438)
at java.net.ServerSocket.accept(ServerSocket.java:409)
at
org.apache.catalina.core.StandardServer.await(StandardServer.java:526)
at org.apache.catalina.startup.Catalina.start(Catalina.java:521)
at org.apache.catalina.startup.Catalina.execute(Catalina.java:400)
at org.apache.catalina.startup.Catalina.process(Catalina.java:180)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:203)
"VM Thread" prio=5 tid=0xa7498 nid=0x4 runnable
"VM Periodic Task Thread" prio=10 tid=0xad0a0 nid=0x7 waiting on monitor
"Suspend Checker Thread" prio=10 tid=0xada08 nid=0x8 runnable
Let me if anyone wants to see the entire threaddump.
I don't really know how to tell if axis is hanging from this information,
so any help would be appreciated.
Thanks,
Brook M. Lovatt
-----+-----
Software Engineer - AMWeb Development
IBM Software Group - Security
Santa Cruz, CA 95060
Tel: (831)466-5102
T/L: 587-5102
[EMAIL PROTECTED]
|---------+---------------------------->
| | Gene Chuang |
| | <genechuang@yahoo|
| | .com> |
| | |
| | 01/04/2003 01:58 |
| | AM |
| | Please respond to|
| | axis-user |
| | |
|---------+---------------------------->
>------------------------------------------------------------------------------------------------------------------------|
|
|
| To: [EMAIL PROTECTED]
|
| cc:
|
| Subject: Re: Axis hannging under load?????
|
|
|
|
|
>------------------------------------------------------------------------------------------------------------------------|
Will the hung process respond to a kill -3? Getting a threaddump will
confirm or deny your suspicions.
Brook M Lovatt <[EMAIL PROTECTED]> wrote:
To help debug a more complex issue, I have written a very simple Web
Service that looks like this:
public class helloService{
public String echoName(String name){
return name;
}
}
I deployed this service on a Tomcat 4.1.18 App Server using the Axis
servlet. I did so by naming the file above helloService.jws and placing it
in the ${CATALINA_HOME}/webapps/axis directory.
A request like this:
********************************************************
POST /axis/helloService.jws HTTP/1.0
Content-Type: text/xml; charset=utf-8
Accept: application/soap+xml, application/dime, multipart/related, text/*
User-Agent: Axis/1.0
Host: gonzo.cruz.ibm.com
Cache-Control: no-cache
Pragma: no-cache
SOAPAction: ""
Content-Length: 403
<?xml version="1.0" encoding="UTF-8"?>
<?xml:namespace prefix = soapenv />xmlns:xsd="
http://www.w3.org/2001/XMLSchema";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";>
soapenv:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/";>
DATA
**********************************************************
Yields a response like this:
**********************************************************
HTTP/1.1 200 OK
Content-Type: text/xml; charset=utf-8
Date: Fri, 03 Jan 2003 23:19:28 GMT
Server: Apache Tomcat/4.0.6 (HTTP/1.1 Connector)
Connection: close
Set-Cookie: JSESSIONID=92707C09552F5CC3A0D4EB82B2B58353;Path=/axis
<?xml version="1.0" encoding="UTF-8"?>
xmlns:xsd="http://www.w3.org/2001/XMLSchema";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance! ">
soapenv:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/";>
DATA
***********************************************************
However, when I put the Service under load (100 threads making
simultaneous
requests for ~10 minutes) the hits/second rate starts fast (500 hits/sec)
then slows, and eventually just stops.
When the service is hung, there is no way of connecting to Tomcat. Also, a
">netstat -a | grep 8080" yields a list of connections that are in
"CLOSE_WAIT" state, that will not close until you restart Tomcat.
I have tested this on AIX, SOLARIS, and Windows... using IBM and Sun
JREs...with WebSphere and Tomcat, and It ALWAYS eventually hangs. I
believe that this problem is with the Axis servlet, and is the root cause
of the problem I originally! set out to fix.
Has anyone seen this before??? Any help would be appreciated!!!
Thanks,
Brook M. Lovatt
-----+-----
Software Engineer - AMWeb Development
IBM Software Group - Security
Santa Cruz, CA 95060
Tel: (831)466-5102
T/L: 587-5102
[EMAIL PROTECTED]
