Hi, * JBoss 5.0.1.GA * JBoss messaging 1.4.6.GA * JBoss remoting 2.5.2 * Java jdk1.5.0_19
We have a cluster with two nodes using clustered queue, and recently we are getting the following errors with JBoss messaging: 1. The following appears in Node 1: | 2009-11-30 16:21:47,525 [Timer-6] WARN remoting.transport.bisocket.BisocketServerInvoker : org.jboss.remoting.transport.bisocket.bisocketserverinvoker$controlmonitortimert...@10d107: detected failure on control connection Thread[control: Socket[addr=myhost2/172.32.29.21,port=32062,localport=1881],5,] (4sl3l3l-7k73nm-g2mqo1ic-1-g2mqxt8r-jn: requesting new control connection | 2. After a few minutes, on node 2[172.32.29.21]: | 2009-11-30 16:24:45,243 [Timer-3] WARN remoting.transport.bisocket.BisocketServerInvoker : org.jboss.remoting.transport.bisocket.bisocketserverinvoker$controlmonitortimert...@17663eb: detected failure on control connection Thread[control: Socket[addr=myhost2/172.32.29.20,port=18861,localport=1686],5,] (4sl3l3k-n6bz0c-g2mqho1y-1-g2mqqj8h-j6: requesting new control connection | 2009-11-30 16:24:49,886 [Timer-7] WARN remoting.transport.bisocket.BisocketClientInvoker : Unable to send ping: shutting down PingTimerTask | java.net.SocketException: Software caused connection abort: socket write error | at java.net.SocketOutputStream.socketWrite0(Native Method) | at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) | at java.net.SocketOutputStream.write(SocketOutputStream.java:115) | at org.jboss.remoting.transport.bisocket.BisocketClientInvoker$PingTimerTask.run(BisocketClientInvoker.java:723) | at java.util.TimerThread.mainLoop(Timer.java:512) | at java.util.TimerThread.run(Timer.java:462) | 2009-11-30 16:25:45,864 [WorkerThread#0[172.21.129.129:1995]] ERROR jboss.messaging.util.ExceptionUtil : SessionEndpoint[l31-8c8rsm2g-1-y1ohqm2g-c0zb6n-k3l3ls4] createConsumerDelegate [m31-oc8rsm2g-1-y1ohqm2g-c0zb6n-k3l3ls4] | javax.jms.InvalidDestinationException: No such destination: JBossTemporaryQueue[wj-xktxqm2g-1-ci1oqm2g-mn37k7-l3l3ls4] has it been deployed? | at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegateInternal(ServerSessionEndpoint.java:1891) | at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegate(ServerSessionEndpoint.java:262) | at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$createConsumerDelegate$aop(SessionAdvised.java:94) | at org.jboss.jms.server.endpoint.advised.SessionAdvised$createConsumerDelegate_8721389917985689973.invokeTarget(SessionAdvised$createConsumerDelegate_8721389917985689973.java) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111) | at org.jboss.jms.server.container.SecurityAspect.handleCreateConsumerDelegate(SecurityAspect.java:123) | at sun.reflect.GeneratedMethodAccessor3253.invoke(Unknown Source) | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) | at java.lang.reflect.Method.invoke(Method.java:585) | at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) | at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) | at org.jboss.jms.server.endpoint.advised.SessionAdvised.createConsumerDelegate(SessionAdvised.java) | at org.jboss.jms.wireformat.SessionCreateConsumerDelegateRequest.serverInvoke(SessionCreateConsumerDelegateRequest.java:100) | at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:157) | at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:897) | at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:768) | at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:721) | at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:575) | at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234) | 2009-11-30 16:30:55,300 [Timer-3] WARN remoting.transport.bisocket.BisocketServerInvoker : org.jboss.remoting.transport.bisocket.bisocketserverinvoker$controlmonitortimert...@17663eb: detected failure on control connection Thread[control: Socket[addr=myhost1/172.32.29.20,port=18861,localport=1843],5,] (4sl3l3k-n6bz0c-g2mqho1y-1-g2mqqj8h-j6: requesting new control connection | 3. Then on node 1: | 2009-11-30 16:25:45,706 [Thread-75] ERROR jboss.jms.client.FailoverCommandCenter : Failover failed | javax.jms.InvalidDestinationException: No such destination: JBossTemporaryQueue[wj-xktxqm2g-1-ci1oqm2g-mn37k7-l3l3ls4] has it been deployed? | at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegateInternal(ServerSessionEndpoint.java:1891) | at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegate(ServerSessionEndpoint.java:262) | at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$createConsumerDelegate$aop(SessionAdvised.java:94) | at org.jboss.jms.server.endpoint.advised.SessionAdvised$createConsumerDelegate_8721389917985689973.invokeTarget(SessionAdvised$createConsumerDelegate_8721389917985689973.java) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111) | at org.jboss.jms.server.container.SecurityAspect.handleCreateConsumerDelegate(SecurityAspect.java:123) | at sun.reflect.GeneratedMethodAccessor3253.invoke(Unknown Source) | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) | at java.lang.reflect.Method.invoke(Method.java:585) | at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) | at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) | at org.jboss.jms.server.endpoint.advised.SessionAdvised.createConsumerDelegate(SessionAdvised.java) | at org.jboss.jms.wireformat.SessionCreateConsumerDelegateRequest.serverInvoke(SessionCreateConsumerDelegateRequest.java:100) | at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:157) | at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:897) | at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:768) | at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:721) | at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:575) | at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234) | at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:213) | at org.jboss.remoting.Client.invoke(Client.java:1917) | at org.jboss.remoting.Client.invoke(Client.java:768) | at org.jboss.remoting.Client.invoke(Client.java:756) | at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189) | at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160) | at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$createConsumerDelegate$aop(ClientSessionDelegate.java:267) | at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_8721389917985689973.invokeTarget(ClientSessionDelegate$createConsumerDelegate_8721389917985689973.java) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111) | at org.jboss.jms.client.container.StateCreationAspect.handleCreateConsumerDelegate(StateCreationAspect.java:136) | at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect_z_handleCreateConsumerDelegate_13797435.invoke(StateCreationAspect_z_handleCreateConsumerDelegate_13797435.java) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) | at org.jboss.jms.client.container.ConsumerAspect.handleCreateConsumerDelegate(ConsumerAspect.java:76) | at org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect_z_handleCreateConsumerDelegate_13797435.invoke(ConsumerAspect_z_handleCreateConsumerDelegate_13797435.java) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) | at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:114) | at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) | at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170) | at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) | at org.jboss.jms.client.delegate.ClientSessionDelegate.createConsumerDelegate(ClientSessionDelegate.java) | at org.jboss.jms.client.state.SessionState.synchronizeWith(SessionState.java:249) | at org.jboss.jms.client.delegate.ClientSessionDelegate.synchronizeWith(ClientSessionDelegate.java:128) | at org.jboss.jms.client.state.ConnectionState.synchronizeWith(ConnectionState.java:177) | at org.jboss.jms.client.delegate.ClientConnectionDelegate.synchronizeWith(ClientConnectionDelegate.java:129) | at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:145) | at org.jboss.jms.client.container.ConnectionFailureListener.handleConnectionException(ConnectionFailureListener.java:62) | at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:88) | at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:498) | | org.jboss.jms.exception.MessagingJMSException: Failed to invoke | at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:271) | at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:205) | at org.jboss.jms.client.delegate.DelegateSupport.doInvokeOneway(DelegateSupport.java:165) | at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:495) | at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeTarget(ClientSessionDelegate$send_6145266547759487588.java) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111) | at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:661) | at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect_z_handleSend_13797435.invoke(SessionAspect_z_handleSend_13797435.java) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) | at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:114) | at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) | at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170) | at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) | at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java) | at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:276) | at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect_z_handleSend_13797435.invoke(ProducerAspect_z_handleSend_13797435.java) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) | at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170) | at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) | at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java) | at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:165) | at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:208) | at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:146) | ............................... | | at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:260) | at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651) | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676) | at java.lang.Thread.run(Thread.java:595) | Caused by: java.lang.Exception: Can not make remoting client invocation due to not being connected to server. | at org.jboss.remoting.Client.invoke(Client.java:1922) | at org.jboss.remoting.Client.invoke(Client.java:768) | at org.jboss.remoting.Client.invokeOneway(Client.java:817) | at org.jboss.remoting.Client.invokeOneway(Client.java:1003) | at org.jboss.remoting.Client.invokeOneway(Client.java:994) | at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:180) | ... 102 more | 2009-11-30 16:25:45,815 [Thread-75] ERROR jms.client.container.ConnectionFailureListener : Caught exception in handling failure | javax.jms.InvalidDestinationException: No such destination: JBossTemporaryQueue[wj-xktxqm2g-1-ci1oqm2g-mn37k7-l3l3ls4] has it been deployed? | at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegateInternal(ServerSessionEndpoint.java:1891) | at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegate(ServerSessionEndpoint.java:262) | at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$createConsumerDelegate$aop(SessionAdvised.java:94) | at org.jboss.jms.server.endpoint.advised.SessionAdvised$createConsumerDelegate_8721389917985689973.invokeTarget(SessionAdvised$createConsumerDelegate_8721389917985689973.java) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111) | at org.jboss.jms.server.container.SecurityAspect.handleCreateConsumerDelegate(SecurityAspect.java:123) | at sun.reflect.GeneratedMethodAccessor3253.invoke(Unknown Source) | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) | at java.lang.reflect.Method.invoke(Method.java:585) | at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) | at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) | at org.jboss.jms.server.endpoint.advised.SessionAdvised.createConsumerDelegate(SessionAdvised.java) | at org.jboss.jms.wireformat.SessionCreateConsumerDelegateRequest.serverInvoke(SessionCreateConsumerDelegateRequest.java:100) | at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:157) | at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:897) | at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:768) | at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:721) | at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:575) | at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234) | at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:213) | at org.jboss.remoting.Client.invoke(Client.java:1917) | at org.jboss.remoting.Client.invoke(Client.java:768) | at org.jboss.remoting.Client.invoke(Client.java:756) | at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189) | at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160) | at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$createConsumerDelegate$aop(ClientSessionDelegate.java:267) | at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_8721389917985689973.invokeTarget(ClientSessionDelegate$createConsumerDelegate_8721389917985689973.java) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111) | at org.jboss.jms.client.container.StateCreationAspect.handleCreateConsumerDelegate(StateCreationAspect.java:136) | at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect_z_handleCreateConsumerDelegate_13797435.invoke(StateCreationAspect_z_handleCreateConsumerDelegate_13797435.java) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) | at org.jboss.jms.client.container.ConsumerAspect.handleCreateConsumerDelegate(ConsumerAspect.java:76) | at org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect_z_handleCreateConsumerDelegate_13797435.invoke(ConsumerAspect_z_handleCreateConsumerDelegate_13797435.java) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) | at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:114) | at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) | at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170) | at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) | at org.jboss.jms.client.delegate.ClientSessionDelegate.createConsumerDelegate(ClientSessionDelegate.java) | at org.jboss.jms.client.state.SessionState.synchronizeWith(SessionState.java:249) | at org.jboss.jms.client.delegate.ClientSessionDelegate.synchronizeWith(ClientSessionDelegate.java:128) | at org.jboss.jms.client.state.ConnectionState.synchronizeWith(ConnectionState.java:177) | at org.jboss.jms.client.delegate.ClientConnectionDelegate.synchronizeWith(ClientConnectionDelegate.java:129) | at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:145) | at org.jboss.jms.client.container.ConnectionFailureListener.handleConnectionException(ConnectionFailureListener.java:62) | at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:88) | at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:498) | We saw the similar issue with JBoss messaing 1.4.2.SP1, but still got it with JBoss messaging 1.4.6, as a result, the two nodes lost connection with each other, can you please advise what could be the cause? Thanks in advance. View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4268030#4268030 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4268030 _______________________________________________ jboss-user mailing list [email protected] https://lists.jboss.org/mailman/listinfo/jboss-user
