In a production environment, I've had several cases where my JBossMQ has failed
under load. The problem is described below. I was wondering if someone else has
solved this problem before, or has any ideas?
The setup is that I have a JBoss application server (1) on Windows sending logs
over log4j JMS appender to JBoss application server (2) on Linux.
The server (2) is configured to use the HSQLDB persistence manager, is set to
use 1GB ram for the JVM and has the high/max memory set to 80% and 90% of the
JVM ram.
The source of the logging data server (1) can send very large amounts of logs
over JMS to the logging server (2). This load is fairly high during the day,
and occasionally causes the JBossMQ on server (2) to completely fail.
The first errors are of the format below. In some cases reconnection occurs,
but in other cases the logging server (2) will continuously refuse connections.
When this failure happens, I have to stop the logging server (2), often delete
the HSQLDB data directory and then restart logging server (2).
org.jboss.mq.SpyJMSException: Exiting on IOE; - nested throwable:
(java.net.SocketException: Connection reset)
at
org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:66)
at org.jboss.mq.Connection.asynchFailure(Connection.java:437)
at
org.jboss.mq.il.uil2.UILClientILService.asynchFailure(UILClientILService.java:156)
at
org.jboss.mq.il.uil2.SocketManager$ReadTask.handleStop(SocketManager.java:413)
at
org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:345)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
at
org.jboss.util.stream.NotifyingBufferedInputStream.read(NotifyingBufferedInputStream.java:67)
at
java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2200)
at
java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2380)
at
java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2447)
at
java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2519)
at
java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2668)
at java.io.ObjectInputStream.readByte(ObjectInputStream.java:864)
at
org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:290)
Hardcore failure can give continuous messages:
javax.naming.CommunicationException [Root exception is
java.rmi.UnmarshalException: Error unmarshaling return; nested exception is:
java.io.EOFException]
at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:707)
at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:572)
at javax.naming.InitialContext.lookup(InitialContext.java:351)
at
com.lmco.lits.james.core.cdf.ReconnectingJMSAppender.initializeConnection(ReconnectingJMSAppender.java:166)
at
com.lmco.lits.james.core.cdf.ReconnectingJMSAppender.onException(ReconnectingJMSAppender.java:319)
at
org.jboss.mq.Connection$ExceptionListenerRunnable.run(Connection.java:1366)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.rmi.UnmarshalException: Error unmarshaling return; nested
exception is:
java.io.EOFException
at
sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:217)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:126)
at org.jnp.server.NamingServer_Stub.lookup(Unknown Source)
at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:610)
... 6 more
Caused by: java.io.EOFException
at
java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2502)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1267)
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1628)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1293)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1912)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1836)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1713)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1299)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:339)
at
sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:215)
... 9 more
View the original post :
http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3932439#3932439
Reply to the post :
http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=3932439
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
JBoss-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/jboss-user
