It looks like the broker side reader threads are hung or stopped b/c the tcp buffers are backing up. Can you post a broker thread dump?
Also can you use tcp in place of nio in the transport connector to eliminate nio. If you can reproduce with a 5.10-SNAPSHOT please raise a jira issue with your test case. On 19 Feb 2014 13:46, "Ashwini Kuntamukkala" <[email protected]> wrote: > Torsten Mielke > > All the limits are under control > > Store percent used = 3 > Memory percent used = 0 > Temp percent used = 1 > > Currently all the 100 queues are empty. All the messages are dequeued. > > Producer flow control is turned on. I have had the same result when > producer flow control is turned off as well > > Repeated a test last night with following setting > > Queue policy: > <policyEntry memoryLimit="10mb" maxPageSize="20" > maxBrowsePageSize="200" expireMessagesPeriod="0" > cursorMemoryHighWaterMark="40" producerFlowControl="true" queue=">" > optimizedDispatch="true"> > > Openwire Broker Connector: > <transportConnector name="openwire" > uri="nio:// > 0.0.0.0:61616?maximumConnections=1000&wireformat.maxFrameSize=104857600&useQueueForAccept=false&wireFormat.maxInactivityDuration=0&transport.socketBufferSize=131072&transport.ioBufferSize=65536&transport.soTimeout=10000&transport.soWriteTimeout=15000 > "/> > > > Client Broker URL > broker.url=failover:(tcp://sit-amq-1:61616?soWriteTimeout=15000) > > After 29 threads processed a million message each in 2.5 hours, the > producer > got stuck again > Here is the stack trace this time around > > Name: ActiveMQ Task-3 > State: RUNNABLE > Total blocked: 0 Total waited: 2 > > Stack trace: > java.net.SocketOutputStream.socketWrite0(Native Method) > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) > java.net.SocketOutputStream.write(SocketOutputStream.java:159) > > org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115) > java.io.DataOutputStream.flush(DataOutputStream.java:123) > > org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:176) > > org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:322) > > org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:304) > > org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85) > > org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104) > > org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85) > > org.apache.activemq.transport.WriteTimeoutFilter.oneway(WriteTimeoutFilter.java:73) > > org.apache.activemq.state.ConnectionStateTracker.restoreProducers(ConnectionStateTracker.java:299) > > org.apache.activemq.state.ConnectionStateTracker.restoreSessions(ConnectionStateTracker.java:252) > > org.apache.activemq.state.ConnectionStateTracker.restore(ConnectionStateTracker.java:165) > > org.apache.activemq.transport.failover.FailoverTransport.restoreTransport(FailoverTransport.java:821) > > org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:996) > - locked java.lang.Object@179e7a37 > > org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:143) > - locked java.lang.Object@61efd9d8 > > org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:129) > org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:47) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:744) > > Name: producerTaskExecutor-22 > State: BLOCKED on java.lang.Object@179e7a37 owned by: ActiveMQ Task-3 > Total blocked: 7 Total waited: 127,678 > > Stack trace: > java.lang.Object.wait(Native Method) > > org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:600) > org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68) > > org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60) > > org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1304) > > org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1298) > org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1782) > - locked java.lang.Object@37785c52 > > org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:289) > > org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:224) > > org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:269) > > org.springframework.jms.connection.CachedMessageProducer.send(CachedMessageProducer.java:121) > org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:589) > org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:569) > org.springframework.jms.core.JmsTemplate$4.doInJms(JmsTemplate.java:546) > org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:466) > org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:543) > producer.ProducerTask.run(ProducerTask.java:43) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:744) > > > > > -- > View this message in context: > http://activemq.2283324.n4.nabble.com/Hung-Producer-tp4678060p4678146.html > Sent from the ActiveMQ - Dev mailing list archive at Nabble.com. >
