More info:
netstat on the client that is making the HttpClient call:
tcp 0 0 ::ffff:192.168.167.1:48990 ::ffff:
192.168.167.133:8081 ESTABLISHED
netstat on the server that is accepting the call:
TCP node-x:8081 192.168.167.1:48990 ESTABLISHED
Complete stack dump of the JVM that is accepting the call (I see no
active socket connection here):
---> SocketListener0-0
java.lang.Object.wait(Native Method)
org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:522)
---> Signal Dispatcher
---> SessionScavenger
java.lang.Thread.sleep(Native Method)
org.mortbay.jetty.servlet.AbstractSessionManager
$SessionScavenger.run(Abstra
ctSessionManager.java:587)
---> Thread-9
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:177)
org.apache.tools.ant.taskdefs.StreamPumper.run(StreamPumper.java:
90)
java.lang.Thread.run(Thread.java:595)
---> Finalizer
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
---> Reference Handler
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:474)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
---> Thread-8
java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
java.io.FilterInputStream.read(FilterInputStream.java:90)
org.apache.tools.ant.taskdefs.StreamPumper.run(StreamPumper.java:
90)
java.lang.Thread.run(Thread.java:595)
---> Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8081]
java.net.PlainSocketImpl.socketAccept(Native Method)
java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
java.net.ServerSocket.implAccept(ServerSocket.java:450)
java.net.ServerSocket.accept(ServerSocket.java:421)
org.mortbay.util.ThreadedServer.acceptSocket(ThreadedServer.java:
432)
org.mortbay.util.ThreadedServer$Acceptor.run(ThreadedServer.java:
634)
---> Thread-7
java.lang.ProcessImpl.waitFor(Native Method)
org.apache.tools.ant.taskdefs.Execute.waitFor(Execute.java:539)
org.apache.tools.ant.taskdefs.Execute.execute(Execute.java:471)
org.apache.tools.ant.taskdefs.Java.fork(Java.java:751)
org.apache.tools.ant.taskdefs.Java.executeJava(Java.java:171)
org.apache.tools.ant.taskdefs.Java.execute(Java.java:84)
org.codehaus.cargo.container.internal.AntContainerExecutorThread.run
(AntCont
ainerExecutorThread.java:50)
---> DestroyJavaVM
---> MultiThreadedHttpConnectionManager cleanup
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager
$ReferenceQu
eueThread.run(Unknown Source)
---> Thread-1
java.lang.Thread.dumpThreads(Native Method)
java.lang.Thread.getAllStackTraces(Thread.java:1434)
com.hostedqa.utils.ThreadUtils.stackDump(ThreadUtils.java:29)
com.hostedqa.client.App$1.run(App.java:43)
java.lang.Thread.run(Thread.java:595)
---> SocketListener0-1
java.lang.Object.wait(Native Method)
org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:522)
And finally, XFire reuses HttpClient objects. When I forced it not
to, everything worked fine. What does that mean?
Patrick
Patrick Lightbody
Autoriginate, Inc.
503-488-5402
http://www.autoriginate.com
[EMAIL PROTECTED]
"Intelligent testing made convenient"
On Aug 8, 2006, at 1:43 AM, Oleg Kalnichevski wrote:
On Mon, 2006-08-07 at 18:10 -0700, Patrick Lightbody wrote:
On Aug 7, 2006, at 3:18 PM, Oleg Kalnichevski wrote:
On Mon, 2006-08-07 at 14:40 -0700, Patrick Lightbody wrote:
Oleg,
Sure, I understand that the issue is due to a native timeout - but
the question is: why?
(1) The only reason for a native socket read to block
indefinitely is
the SO_TIMEOUT value set to zero. Does XFire explicitly set the
socket
timeout value to a positive value?
No, XFire does not. I tried setting it and that didn't produce the
desireable affect, as I have several requests that I want to take up
to several minutes to respond. I think an indefinite time is fine
(which the default, which is what I was using). My issue isn't that
the timeout is set to 0, but that the connection isn't proceeding
like it should be.
The connection is not proceeding because apparently the server is not
sending any data.
Specifically, on the server side, I have done a
stack dump and the only thing related to sockets there is this:
---> Acceptor ServerSocket
[addr=0.0.0.0/0.0.0.0,port=0,localport=8081]
java.net.PlainSocketImpl.socketAccept(Native Method)
java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
java.net.ServerSocket.implAccept(ServerSocket.java:450)
java.net.ServerSocket.accept(ServerSocket.java:421)
org.mortbay.util.ThreadedServer.acceptSocket
(ThreadedServer.java:
432)
org.mortbay.util.ThreadedServer$Acceptor.run
(ThreadedServer.java:
634)
Which clearly is not related to the other open socket.
Are there any other threads not related to sockets? The server may
stop
responding on a connection if the connection thread gets dead-
locked on
a mutex or enters an infinite loop.
So I'm back to my original problem: why is the client stuck in the
native state while the server has no record of it? And why does it
only happen for that one request in those exact steps that I execute.
I should also add that I have done a netstat call at the time, and
got back:
[EMAIL PROTECTED] webapps]$ netstat -a --numeric-port |
grep 8081
tcp 1 0 ::ffff:192.168.167.1:58125 ::ffff:
192.168.167.129:8081 CLOSE_WAIT
Any other ideas? I'm really at a loss here :(
As far as I am concerned there's nothing wrong in how HttpClient is
behaving. It blocks indefinitely in the native socket read because
this
is precisely what it has been instructed to do. In my opinion the
problem is on the server side.
Just in case consider upgrading the JVM you are using both client and
server side to the latest patch level, and consider using a different
version of Jetty or a different servlet engine to see if that makes
any
difference
Oleg
Patrick
(2) org.apache.commons.httpclient.HttpMethodBase writeRequest
100 (continue) read timeout. Resume sending the request
This message is logged when the target server fails to properly
respond
to the "Expect: 100-continue" handshake. When the handshake is
activated
HttpClient sends the request header containing the "Expect:
100-continue" directive prior to sending the request body and
expects
the target server to respond with status code 100 indicating that
it is
okay to proceed with sending the request content. Apparently the
server
has issues with the "Expect: 100-continue" handshake or simply
locks up
while processing the request header.
I turned off this feature and the INFO log goes away, but the lock up
still occurs. So I don't think this is related.
Hope this helps.
Oleg
Like I reported in my initial email, subsequent requests in other
threads (coming from test.jsp) work fine.
Patrick
Patrick Lightbody
Autoriginate, Inc.
503-488-5402
http://www.autoriginate.com
[EMAIL PROTECTED]
"Intelligent testing made convenient"
On Aug 7, 2006, at 2:34 PM, Oleg Kalnichevski wrote:
On Mon, 2006-08-07 at 12:29 -0700, Patrick Lightbody wrote:
I've tried using XFire 1.1.1 and 1.2-RC, combined with
HttpClient 3.0
and 3.1-alpha1. I get the same result, outlined below, which
causes a
complete lockup of a thread. I can't figure out what would cause
this.
When making a call via XFire (ClientService.getAppLog()), the
current
thread locks up just after printing the following out in the
logs:
org.apache.commons.httpclient.HttpMethodBase writeRequest
100 (continue) read timeout. Resume sending the request
I see that this log comes from an InterruptedIOException here:
http://jakarta.apache.org/commons/httpclient/xref/org/apache/
commons/
httpclient/HttpMethodBase.html#2004
The stack dump of the locked thread is:
"Thread-62" daemon prio=1 tid=0x082602c0 nid=0x51ca runnable
[0x79926000..0x79926e30]
Patrick,
As you can see the thread gets blocked in the native socket read
method,
so this is very unlikely to be a threading dead-lock in the
HttpClient
code. Most likely the socket read operation blocks indefinitely
because
socket timeout is not set (SO_TIMEOUT value is set to zero).
Hope this helps
Oleg
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read
(SocketInputStream.java:
129)
at java.io.BufferedInputStream.fill
(BufferedInputStream.java:
218)
at java.io.BufferedInputStream.read
(BufferedInputStream.java:
235)
- locked <0x830328c8> (a java.io.BufferedInputStream)
at org.apache.commons.httpclient.HttpParser.readRawLine
(HttpParser.java:77)
at org.apache.commons.httpclient.HttpParser.readLine
(HttpParser.java:105)
at org.apache.commons.httpclient.HttpConnection.readLine
(HttpConnection.java:1115)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager
$HttpConnectionAdapter.readLine
(MultiThreadedHttpConnectionManager.java:1373)
at
org.apache.commons.httpclient.HttpMethodBase.readStatusLine
(HttpMethodBase.java:1832)
at
org.apache.commons.httpclient.HttpMethodBase.readResponse
(HttpMethodBase.java:1590)
at org.apache.commons.httpclient.HttpMethodBase.execute
(HttpMethodBase.java:995)
at
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry
(HttpMethodDirector.java:397)
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod
(HttpMethodDirector.java:170)
at org.apache.commons.httpclient.HttpClient.executeMethod
(HttpClient.java:396)
at
org.codehaus.xfire.transport.http.CommonsHttpMessageSender.send
(CommonsHttpMessageSender.java:226)
at
org.codehaus.xfire.transport.http.HttpChannel.sendViaClient
(HttpChannel.java:118)
at org.codehaus.xfire.transport.http.HttpChannel.send
(HttpChannel.java:48)
at org.codehaus.xfire.handler.OutMessageSender.invoke
(OutMessageSender.java:26)
at org.codehaus.xfire.handler.HandlerPipeline.invoke
(HandlerPipeline.java:130)
at org.codehaus.xfire.client.Invocation.invoke
(Invocation.java:75)
at org.codehaus.xfire.client.Client.invoke(Client.java:
335)
at org.codehaus.xfire.client.XFireProxy.handleRequest
(XFireProxy.java:77)
at org.codehaus.xfire.client.XFireProxy.invoke
(XFireProxy.java:57)
at $Proxy5.getAppLog(Unknown Source)
at com.hostedqa.model.TestContextImpl.dispose
(TestContextImpl.java:83)
at com.hostedqa.model.Suite.playback(Suite.java:85)
at com.hostedqa.service.PlaybackService.runTest
(PlaybackService.java:83)
at com.hostedqa.service.PlaybackService.playSuite
(PlaybackService.java:48)
at com.hostedqa.action.project.suite.PlayAction$1.run
(PlayAction.java:25)
at java.lang.Thread.run(Thread.java:595)
What's very weird is that I am able to drop a JSP (test.jsp) that
makes the exact same call and it completes just fine. This
tells me
that there is something environmental about _this_ thread that
causes
HttpClient to do this. The call alone is not the issue.
Also, I might add that the XFire call never makes it to the
other end
(ClientServiceImpl), as I have a print line there that never gets
invoked. I ran a stack dump on the other side as well, and
nothing
stood out (though it is possible part of the request made it
through
to XFire's Servlet, and then broke and was no longer in the
active
thread dump by the time I forced the dump).
Finally, this request is running over HTTP. I'd really like to
figure
out:
1) What that log from HttpMethodBase.writeRequest() is all about
2) Why there would be a perpetual "pause" in the native
method, but
no actual visible deadlock.
3) How to fix this :)
Patrick
Patrick Lightbody
Autoriginate, Inc.
503-488-5402
http://www.autoriginate.com
[EMAIL PROTECTED]
"Intelligent testing made convenient"
-----------------------------------------------------------------
--
--
To unsubscribe, e-mail: httpclient-user-
[EMAIL PROTECTED]
For additional commands, e-mail: httpclient-user-
[EMAIL PROTECTED]
------------------------------------------------------------------
--
-
To unsubscribe, e-mail: httpclient-user-
[EMAIL PROTECTED]
For additional commands, e-mail: httpclient-user-
[EMAIL PROTECTED]
-------------------------------------------------------------------
--
To unsubscribe, e-mail: httpclient-user-
[EMAIL PROTECTED]
For additional commands, e-mail: httpclient-user-
[EMAIL PROTECTED]
--------------------------------------------------------------------
-
To unsubscribe, e-mail: httpclient-user-
[EMAIL PROTECTED]
For additional commands, e-mail: httpclient-user-
[EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-
[EMAIL PROTECTED]
For additional commands, e-mail: httpclient-user-
[EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: httpclient-user-
[EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
