Thanks for the response. We have analyzed a little more on the log files from
hanging issue and find out the hanging issue is always after the thread dump.
Here is the jboss infrastructure from the failing scenario:
There are 3 jboss nodes (node1, node2, and node3). node1 and node2 sit on
solaris box1. node3 sits on solaris box2. node1 is JMS provider and node2 and
node3 are JMS listeners with MDBs. The protocal between JMS client and server
is RMI.
At node startup time, node2 and node3 will shake hands with node1. The hanging
happens 1 out of 10 and always happens on node3. It seems to be some
combination issue of Solaris/Network/Distributed JBoss/JMS issue.
Here is the full output of the thread dump. Could you please check it out and
let us know if it rings any bell of some bug related to this issue?
Thanks a lot,
output of the thead dump:
Full thread dump Java HotSpot(TM) Server VM (1.4.2_08-b03 mixed mode):
"Thread-6" daemon prio=5 tid=0x01710e78 nid=0x1d in Object.wait()
[36aff000..36affc28]
at java.lang.Object.wait(Native Method)
- waiting on <0x46e619b8> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:429)
- locked <0x46e619b8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:382)
"JBossMQ Cache Reference Softner" daemon prio=5 tid=0x0168e900 nid=0x1c in
Object.wait() [36bff000..36bffc28]
at java.lang.Object.wait(Native Method)
- waiting on <0x468c5258> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
- locked <0x468c5258> (a java.lang.ref.ReferenceQueue$Lock)
at org.jboss.mq.server.MessageCache.run(MessageCache.java:136)
at java.lang.Thread.run(Thread.java:534)
"Thread-5" prio=5 tid=0x01709438 nid=0x1b waiting on condition
[36cff000..36cffc28]
at java.lang.Thread.sleep(Native Method)
at org.hsqldb.Log.run(Log.java:119)
at java.lang.Thread.run(Thread.java:534)
"hypersonic-default" prio=5 tid=0x01692d60 nid=0x1a runnable
[36dff000..36dffc28]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0x469cbc00> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at org.hsqldb.Embedded_Server.run(Embedded_Server.java:169)
at org.hsqldb.Embedded_Server.main(Embedded_Server.java:92)
at org.jboss.jdbc.HypersonicDatabase$2.run(HypersonicDatabase.java:186)
"RMI TCP Accept-4444" daemon prio=5 tid=0x0161fe80 nid=0x19 runnable
[36eff000..36effc28]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0x45d2c4f0> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
at java.lang.Thread.run(Thread.java:534)
"JNP Server" prio=5 tid=0x015e9ad0 nid=0x18 runnable [36fff000..36fffc28]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0x45b69b08> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at org.jnp.server.Main.run(Main.java:248)
at java.lang.Thread.run(Thread.java:534)
"GC Daemon" daemon prio=2 tid=0x01404a28 nid=0x17 in Object.wait()
[370ff000..370ffc28]
at java.lang.Object.wait(Native Method)
- waiting on <0x45b4a718> (a sun.misc.GC$LatencyLock)
at sun.misc.GC$Daemon.run(GC.java:100)
- locked <0x45b4a718> (a sun.misc.GC$LatencyLock)
"RMI Reaper" prio=5 tid=0x00fdc260 nid=0x16 in Object.wait()
[371ff000..371ffc28]
at java.lang.Object.wait(Native Method)
- waiting on <0x45b49440> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
- locked <0x45b49440> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:330)
at java.lang.Thread.run(Thread.java:534)
"Thread-4" daemon prio=5 tid=0x00fdcb28 nid=0x15 in Object.wait()
[372ff000..372ffc28]
at java.lang.Object.wait(Native Method)
- waiting on <0x45b49bf8> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:429)
at java.util.TimerThread.mainLoop(Timer.java:403)
- locked <0x45b49bf8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:382)
"RMI TCP Accept-0" daemon prio=5 tid=0x002fc130 nid=0x14 runnable
[373ff000..373ffc28]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0x45b495d0> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
at java.lang.Thread.run(Thread.java:534)
"Thread-3" daemon prio=5 tid=0x015e9d48 nid=0x13 runnable [374ff000..374ffc28]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0x45aec420> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at org.jboss.web.WebServer.run(WebServer.java:248)
at org.jboss.web.ThreadPool$Worker.run(ThreadPool.java:140)
"ScannerThread" daemon prio=5 tid=0x01405428 nid=0x12 in Object.wait()
[375ff000..375ffc28]
at java.lang.Object.wait(Native Method)
- waiting on <0x45422fc0> (a java.lang.Object)
at java.lang.Object.wait(Object.java:429)
at
org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.run(AbstractDeploymentScanner.java:193)
- locked <0x45422fc0> (a java.lang.Object)
"Thread-1" daemon prio=5 tid=0x00d76088 nid=0x11 in Object.wait()
[376ff000..376ffc28]
at java.lang.Object.wait(Native Method)
- waiting on <0x44f01910> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:429)
- locked <0x44f01910> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:382)
"Thread-0" daemon prio=5 tid=0x002799b0 nid=0x10 in Object.wait()
[398ff000..398ffc28]
at java.lang.Object.wait(Native Method)
- waiting on <0x41ac2b28> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
- locked <0x41ac2b28> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
at
org.jboss.mx.logging.LoggerContext$GarbageMonitor.run(LoggerContext.java:451)
"DestroyJavaVM" prio=5 tid=0x00037460 nid=0x1 waiting on condition [0..ffbfe790]
"main" prio=5 tid=0x001a70b8 nid=0xf runnable [3bcfa000..3bcffc28]
at java.util.HashMap.get(HashMap.java:325)
at
org.jboss.mx.loading.UnifiedLoaderRepository2.loadClassFromCache(UnifiedLoaderRepository2.java:208)
at
org.jboss.mx.loading.UnifiedLoaderRepository2.loadClass(UnifiedLoaderRepository2.java:137)
at
org.jboss.mx.loading.UnifiedClassLoader.loadClass(UnifiedClassLoader.java:283)
at java.lang.ClassLoader.loadClass(ClassLoader.java:235)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:302)
- locked <0x43184ad8> (a org.jboss.mx.loading.UnifiedClassLoader)
at org.jconfig.ConfigurationManager.(ConfigurationManager.java:43)
at
org.jconfig.ConfigurationManager.getInstance(ConfigurationManager.java:68)
- locked <0xf1eee248> (a java.lang.Class)
at org.jconfig.mbean.ConfigHandler.(ConfigHandler.java:23)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:274)
at
org.jboss.mx.server.MBeanServerImpl.instantiate(MBeanServerImpl.java:812)
at
org.jboss.mx.server.MBeanServerImpl.instantiate(MBeanServerImpl.java:266)
at
org.jboss.mx.server.MBeanServerImpl.createMBean(MBeanServerImpl.java:293)
at org.jboss.system.ServiceCreator.install(ServiceCreator.java:86)
at
org.jboss.system.ServiceConfigurator.internalInstall(ServiceConfigurator.java:167)
at
org.jboss.system.ServiceConfigurator.install(ServiceConfigurator.java:130)
at org.jboss.system.ServiceController.install(ServiceController.java:225)
- locked <0x41c20bc0> (a org.jboss.system.ServiceController)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at
org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:517)
at org.jboss.util.jmx.MBeanProxy.invoke(MBeanProxy.java:174)
at $Proxy3.install(Unknown Source)
at org.jboss.deployment.SARDeployer.create(SARDeployer.java:209)
at org.jboss.deployment.MainDeployer.create(MainDeployer.java:755)
at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:615)
at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:580)
at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at
org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:517)
at org.jboss.util.jmx.MBeanProxy.invoke(MBeanProxy.java:174)
at $Proxy4.deploy(Unknown Source)
at
org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:427)
at
org.jboss.deployment.scanner.URLDeploymentScanner.scanDirectory(URLDeploymentScanner.java:648)
at
org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:499)
- locked <0x45375480> (a
java.util.Collections$SynchronizedRandomAccessList)
- locked <0x4536de70> (a org.jboss.deployment.scanner.URLDeploymentScanner)
at
org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:261)
at org.jboss.system.ServiceMBeanSupport.start(ServiceMBeanSupport.java:164)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at
org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:517)
at
org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:976)
at $Proxy0.start(Unknown Source)
at org.jboss.system.ServiceController.start(ServiceController.java:397)
- locked <0x41c20bc0> (a org.jboss.system.ServiceController)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at
org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:517)
at org.jboss.util.jmx.MBeanProxy.invoke(MBeanProxy.java:174)
at $Proxy3.start(Unknown Source)
at org.jboss.deployment.SARDeployer.start(SARDeployer.java:249)
at org.jboss.deployment.MainDeployer.start(MainDeployer.java:802)
at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:616)
at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:580)
at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:564)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at
org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:517)
at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:324)
at org.jboss.system.server.ServerImpl.start(ServerImpl.java:221)
at org.jboss.Main.boot(Main.java:148)
at org.jboss.Main$1.run(Main.java:381)
at java.lang.Thread.run(Thread.java:534)
"Signal Dispatcher" daemon prio=10 tid=0x00194118 nid=0xa waiting on condition
[0..0]
"Surrogate Locker Thread (CMS)" daemon prio=5 tid=0x00192d00 nid=0x8 waiting on
condition [0..0]
"Finalizer" daemon prio=8 tid=0x0018e678 nid=0x7 in Object.wait()
[f9b7f000..f9b7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0x41800490> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
- locked <0x41800490> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x0018d518 nid=0x6 in Object.wait()
[fbd7f000..fbd7fc28]
at java.lang.Object.wait(Native Method)
- waiting on <0x41800380> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:429)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:115)
- locked <0x41800380> (a java.lang.ref.Reference$Lock)
"VM Thread" prio=5 tid=0x0018ced0 nid=0x5 runnable
"VM Periodic Task Thread" prio=10 tid=0x00198f48 nid=0xe waiting on condition
"Suspend Checker Thread" prio=10 tid=0x001937b0 nid=0x9 runnable
View the original post :
http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3940215#3940215
Reply to the post :
http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=3940215
-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
JBoss-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/jboss-user
