We're running into a problem with a brand new production system that has a
single remote EJB. We are using JBoss EJB3, which uses JBoss Remoting to do
the remote communication. We are using the standard socket invoker to handle
the communication. We are using JBoss Serialization instead of Java
Serialization, which, in tests, provided significantly better overall
performance. We've been testing this in a stage environment for several
months, with no problems. Once we got it into production, we started to hit a
wall. We reach a point where the socket connections between the client and the
server just start timing out. A thread dump on the client shows a bunch of
threads waiting for a socket connection to the server. A thread dump on the
server shows several SeverThreads waiting to be used (via
ServerThread.wakeup()). The problem appears to be that the thread in
SocketServerInvoker that is responsible for the calls to serverSocket.accept()
is getting stuck, or at lea!
st severely slowed. In the server side thread dump, here is the stack of that
accept thread:
Thread: SocketServerInvoker#0-3873 : priority:5, demon:false, threadId:23,
threadState:RUNNABLE, threadLockName:null
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
java.io.BufferedInputStream.read(BufferedInputStream.java:317)
java.io.FilterInputStream.read(FilterInputStream.java:90)
org.jboss.serial.io.JBossObjectInputStream.checkSignature(JBossObjectInputStream.java:110)
org.jboss.serial.io.JBossObjectInputStream.(JBossObjectInputStream.java:94)
org.jboss.serial.io.JBossObjectInputStream.(JBossObjectInputStream.java:83)
org.jboss.remoting.serialization.impl.jboss.JBossSerializationManager.createInput(JBossSerializationManager.java:57)
org.jboss.remoting.transport.socket.ServerSocketWrapper.createInputStream(ServerSocketWrapper.java:56)
org.jboss.remoting.transport.socket.ClientSocketWrapper.createStreams(ClientSocketWrapper.java:76)
org.jboss.remoting.transport.socket.ClientSocketWrapper.(ClientSocketWrapper.java:54)
org.jboss.remoting.transport.socket.ServerSocketWrapper.(ServerSocketWrapper.java:50)
sun.reflect.GeneratedConstructorAccessor13.newInstance(Unknown Source)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
org.jboss.remoting.transport.socket.ServerThread.createServerSocket(ServerThread.java:185)
org.jboss.remoting.transport.socket.ServerThread.(ServerThread.java:87)
org.jboss.remoting.transport.socket.SocketServerInvoker.processInvocation(SocketServerInvoker.java:429)
org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:391)
java.lang.Thread.run(Thread.java:619)
It appears that this thread, which ought to be quickly finding a ServerThread
to handle the incoming request, calling ServerThread.start(), and getting back
to the call to serverSocket.accept() as quickly as possible, is instead doing
some socket I/O. ServerSocketWrapper.java has a method called
createInputStream() that creates a new ObjectInputStream. The stack trace
shows that the JBossObjectInputStream constructor is reading some bytes from
the input stream, but I don't think that this problem is specific to JBoss
Serialization, as java.io.ObjectInputStream's constructor also reads bytes from
the stream.
It seems to me that this could be the cause of the problem that we are seeing.
Any time that you're doing a blocking socket read like this, you can
potentially just sit for a while until the socket times out. When you've got a
thread calling serverSocket.accept(), you can't normally afford to do something
like this on that thread.
Does this analysis make sense? Any thoughts/suggestions would be greatly
appreciated.
Thanks.
--Tim
View the original post :
http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4057841#4057841
Reply to the post :
http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4057841
_______________________________________________
jboss-user mailing list
[email protected]
https://lists.jboss.org/mailman/listinfo/jboss-user
