A Jira already exists for this issue https://issues.apache.org/jira/browse/ UIMA-4225. It has been marked "Won't Fix" because the Job Driver and corresponding Job Process have been re-written for 2.0 version of DUCC (not yet released, but available from trunk).
Are you able to reproduce this error with regularity? I've only seen it once before, which resulted in Jira 4225 and its subsequent closure. It also seems that you may have more serious problems than just one job failing. Based on the message: 22 Mar 2015 08:49:49,723 14 WARN jms.DefaultJmsMessageListenerC ontainer Setup of JMS message listener invoker failed for destination 'ducc.orchestrator.abbreviated.state' - trying to recover. it seems you may have network or broker problems? Lou. On Mon, Mar 23, 2015 at 2:16 AM, Satya Nand Kanodia < [email protected]> wrote: > Hi, > > In DUCC when I submit a job , sometimes it gets hung for infinite time. > Today it happened again. A job started with 2 processes on 22 Mar 2015 > 04:34:38 had been continuously running till today. > > I used this configuration in job file > > process_thread_count=3 > process_failures_limit=50 > process_per_item_time_max=5 > process_deployments_max=2 > attach_console=true > process_jvm_args=-Xms4g -XX\:+UseConcMarkSweepGC > process_memory_size=7 > classpath_order=ducc-before-user > > Finally I had to terminate it from DUCC-MON. Job usually takes 15 minutes > max to process documents. I submitted a batch of 433 documents and 342 > documents had been processed. There were two errors of cas timed out. There > were connection exceptions in both processes , which lasted for 10 seconds. > exception happened at 22 Mar 2015 08:50:42. After that no response from > both processes. > > I got these exceptions from jd.out file > > These exception in jd.out file started at 05:09:12. > > 22 Mar 2015 05:32:12,352 154 ERROR jd.JobDriver waitForProcessStatus > N/A process status error? java.lang.NullPointerException > > At: > java.util.TreeMap.getEntry(TreeMap.java:342) > java.util.TreeMap.get(TreeMap.java:273) > org.apache.uima.ducc.jd.JobDriver.getProcess(JobDriver.java:1081) > org.apache.uima.ducc.jd.JobDriver.waitForProcessStatus( > JobDriver.java:1123) > org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.java:1585) > org.apache.uima.ducc.jd.client.WorkItem.exception( > WorkItem.java:196) > org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.java:160) > java.util.concurrent.Executors$RunnableAdapter. > call(Executors.java:471) > java.util.concurrent.FutureTask.run(FutureTask.java:262) > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > > 22 Mar 2015 05:32:12,353 154 ERROR jd.JobDriver exception 19524 > processing error? java.lang.NullPointerException > > At: > java.util.TreeMap.getEntry(TreeMap.java:342) > java.util.TreeMap.get(TreeMap.java:273) > org.apache.uima.ducc.jd.JobDriver.getProcess(JobDriver.java:1081) > org.apache.uima.ducc.jd.JobDriver.isNodeFailure( > JobDriver.java:1092) > org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.java:1586) > org.apache.uima.ducc.jd.client.WorkItem.exception( > WorkItem.java:196) > org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.java:160) > java.util.concurrent.Executors$RunnableAdapter. > call(Executors.java:471) > java.util.concurrent.FutureTask.run(FutureTask.java:262) > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > > 22 Mar 2015 05:32:12,354 152 ERROR jd.JobDriver waitForProcessStatus > N/A process status error? java.lang.NullPointerException > > At: > java.util.TreeMap.getEntry(TreeMap.java:342) > java.util.TreeMap.get(TreeMap.java:273) > org.apache.uima.ducc.jd.JobDriver.getProcess(JobDriver.java:1081) > org.apache.uima.ducc.jd.JobDriver.waitForProcessStatus( > JobDriver.java:1123) > org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.java:1585) > org.apache.uima.ducc.jd.client.WorkItem.exception( > WorkItem.java:196) > org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.java:160) > java.util.concurrent.Executors$RunnableAdapter. > call(Executors.java:471) > java.util.concurrent.FutureTask.run(FutureTask.java:262) > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > > 22 Mar 2015 05:32:12,354 152 ERROR jd.JobDriver exception 19524 > processing error? java.lang.NullPointerException > > At: > java.util.TreeMap.getEntry(TreeMap.java:342) > java.util.TreeMap.get(TreeMap.java:273) > org.apache.uima.ducc.jd.JobDriver.getProcess(JobDriver.java:1081) > org.apache.uima.ducc.jd.JobDriver.isNodeFailure( > JobDriver.java:1092) > org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.java:1586) > org.apache.uima.ducc.jd.client.WorkItem.exception( > WorkItem.java:196) > org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.java:160) > java.util.concurrent.Executors$RunnableAdapter. > call(Executors.java:471) > java.util.concurrent.FutureTask.run(FutureTask.java:262) > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > > 22 Mar 2015 05:32:12,354 60 ERROR jd.JobDriver waitForProcessStatus > N/A process status error? java.lang.NullPointerException > > At: > java.util.TreeMap.getEntry(TreeMap.java:342) > java.util.TreeMap.get(TreeMap.java:273) > org.apache.uima.ducc.jd.JobDriver.getProcess(JobDriver.java:1081) > org.apache.uima.ducc.jd.JobDriver.waitForProcessStatus( > JobDriver.java:1123) > org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.java:1585) > org.apache.uima.ducc.jd.client.WorkItem.exception( > WorkItem.java:196) > org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.java:160) > java.util.concurrent.Executors$RunnableAdapter. > call(Executors.java:471) > java.util.concurrent.FutureTask.run(FutureTask.java:262) > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > > 22 Mar 2015 05:32:12,354 60 ERROR jd.JobDriver exception 19524 > processing error? java.lang.NullPointerException > > At: > java.util.TreeMap.getEntry(TreeMap.java:342) > java.util.TreeMap.get(TreeMap.java:273) > org.apache.uima.ducc.jd.JobDriver.getProcess(JobDriver.java:1081) > org.apache.uima.ducc.jd.JobDriver.isNodeFailure( > JobDriver.java:1092) > org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.java:1586) > org.apache.uima.ducc.jd.client.WorkItem.exception( > WorkItem.java:196) > org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.java:160) > java.util.concurrent.Executors$RunnableAdapter. > call(Executors.java:471) > java.util.concurrent.FutureTask.run(FutureTask.java:262) > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > > 22 Mar 2015 07:10:59,845 14 WARN jms.DefaultJmsMessageListenerContainer > Setup of JMS message listener invoker failed for destination > 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: Channel > was inactive for too (>30000) long: tcp://192.168.10.144:61617 > 22 Mar 2015 07:13:09,855 14 WARN jms.DefaultJmsMessageListenerContainer > Setup of JMS message listener invoker failed for destination > 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: Channel > was inactive for too (>30000) long: tcp://192.168.10.144:61617 > 22 Mar 2015 07:16:50,007 14 WARN jms.DefaultJmsMessageListenerContainer > Setup of JMS message listener invoker failed for destination > 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: Channel > was inactive for too (>30000) long: tcp://192.168.10.144:61617 > 22 Mar 2015 07:24:35,994 14 WARN jms.EndpointMessageListener Execution of > JMS message listener failed. Caused by: [javax.jms.IllegalStateException > - The Session is closed] > javax.jms.IllegalStateException: The Session is closed > at org.apache.activemq.ActiveMQSession.checkClosed( > ActiveMQSession.java:731) > at org.apache.activemq.ActiveMQSession.getTransacted( > ActiveMQSession.java:521) > at org.apache.activemq.pool.PooledSession.getTransacted( > PooledSession.java:256) > at org.springframework.jms.listener.AbstractMessageListenerContain > er.commitIfNecessary(AbstractMessageListenerContainer.java:571) > at org.springframework.jms.listener.AbstractMessageListenerContain > er.doExecuteListener(AbstractMessageListenerContainer.java:481) > at org.springframework.jms.listener.AbstractPollingMessageListener > Container.doReceiveAndExecute(AbstractPollingMessageListener > Container.java:325) > at org.springframework.jms.listener.AbstractPollingMessageListener > Container.receiveAndExecute(AbstractPollingMessageListener > Container.java:263) > at org.springframework.jms.listener.DefaultMessageListenerContaine > r$AsyncMessageListenerInvoker.invokeListener( > DefaultMessageListenerContainer.java:1058) > at org.springframework.jms.listener.DefaultMessageListenerContaine > r$AsyncMessageListenerInvoker.executeOngoingLoop( > DefaultMessageListenerContainer.java:1050) > at org.springframework.jms.listener.DefaultMessageListenerContaine > r$AsyncMessageListenerInvoker.run(DefaultMessageListenerContaine > r.java:947) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 22 Mar 2015 07:24:36,617 14 WARN jms.DefaultJmsMessageListenerContainer > Setup of JMS message listener invoker failed for destination > 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: The > Session is closed > 22 Mar 2015 07:26:31,822 14 WARN jms.EndpointMessageListener Execution of > JMS message listener failed. Caused by: [javax.jms.IllegalStateException > - The Session is closed] > javax.jms.IllegalStateException: The Session is closed > at org.apache.activemq.ActiveMQSession.checkClosed( > ActiveMQSession.java:731) > at org.apache.activemq.ActiveMQSession.getTransacted( > ActiveMQSession.java:521) > at org.apache.activemq.pool.PooledSession.getTransacted( > PooledSession.java:256) > at org.springframework.jms.listener.AbstractMessageListenerContain > er.commitIfNecessary(AbstractMessageListenerContainer.java:571) > at org.springframework.jms.listener.AbstractMessageListenerContain > er.doExecuteListener(AbstractMessageListenerContainer.java:481) > at org.springframework.jms.listener.AbstractPollingMessageListener > Container.doReceiveAndExecute(AbstractPollingMessageListener > Container.java:325) > at org.springframework.jms.listener.AbstractPollingMessageListener > Container.receiveAndExecute(AbstractPollingMessageListener > Container.java:263) > at org.springframework.jms.listener.DefaultMessageListenerContaine > r$AsyncMessageListenerInvoker.invokeListener( > DefaultMessageListenerContainer.java:1058) > at org.springframework.jms.listener.DefaultMessageListenerContaine > r$AsyncMessageListenerInvoker.executeOngoingLoop( > DefaultMessageListenerContainer.java:1050) > at org.springframework.jms.listener.DefaultMessageListenerContaine > r$AsyncMessageListenerInvoker.run(DefaultMessageListenerContaine > r.java:947) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 22 Mar 2015 07:26:32,067 14 WARN jms.DefaultJmsMessageListenerContainer > Setup of JMS message listener invoker failed for destination > 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: The > Session is closed > 22 Mar 2015 07:28:12,075 14 WARN jms.DefaultJmsMessageListenerContainer > Setup of JMS message listener invoker failed for destination > 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: Channel > was inactive for too (>30000) long: tcp://192.168.10.144:61617 > 22 Mar 2015 07:30:55,001 14 WARN jms.DefaultJmsMessageListenerContainer > Could not refresh JMS Connection for destination > 'ducc.orchestrator.abbreviated.state' > - retrying in 5000 ms. Cause: Channel was inactive for too (>30000) long: > tcp://192.168.10.144:61617 > 22 Mar 2015 08:29:26,669 14 WARN jms.EndpointMessageListener Execution of > JMS message listener failed. Caused by: [javax.jms.IllegalStateException > - The Session is closed] > javax.jms.IllegalStateException: The Session is closed > at org.apache.activemq.ActiveMQSession.checkClosed( > ActiveMQSession.java:731) > at org.apache.activemq.ActiveMQSession.getTransacted( > ActiveMQSession.java:521) > at org.apache.activemq.pool.PooledSession.getTransacted( > PooledSession.java:256) > at org.springframework.jms.listener.AbstractMessageListenerContain > er.commitIfNecessary(AbstractMessageListenerContainer.java:571) > at org.springframework.jms.listener.AbstractMessageListenerContain > er.doExecuteListener(AbstractMessageListenerContainer.java:481) > at org.springframework.jms.listener.AbstractPollingMessageListener > Container.doReceiveAndExecute(AbstractPollingMessageListener > Container.java:325) > at org.springframework.jms.listener.AbstractPollingMessageListener > Container.receiveAndExecute(AbstractPollingMessageListener > Container.java:263) > at org.springframework.jms.listener.DefaultMessageListenerContaine > r$AsyncMessageListenerInvoker.invokeListener( > DefaultMessageListenerContainer.java:1058) > at org.springframework.jms.listener.DefaultMessageListenerContaine > r$AsyncMessageListenerInvoker.executeOngoingLoop( > DefaultMessageListenerContainer.java:1050) > at org.springframework.jms.listener.DefaultMessageListenerContaine > r$AsyncMessageListenerInvoker.run(DefaultMessageListenerContaine > r.java:947) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 22 Mar 2015 08:29:26,671 14 WARN jms.DefaultJmsMessageListenerContainer > Setup of JMS message listener invoker failed for destination > 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: The > Session is closed > 22 Mar 2015 08:33:06,678 14 WARN jms.DefaultJmsMessageListenerContainer > Setup of JMS message listener invoker failed for destination > 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: Channel > was inactive for too (>30000) long: tcp://192.168.10.144:61617 > 22 Mar 2015 08:36:16,841 14 WARN jms.DefaultJmsMessageListenerContainer > Setup of JMS message listener invoker failed for destination > 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: Channel > was inactive for too (>30000) long: tcp://192.168.10.144:61617 > 22 Mar 2015 08:37:27,506 14 WARN jms.DefaultJmsMessageListenerContainer > Setup of JMS message listener invoker failed for destination > 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: Channel > was inactive for too (>30000) long: tcp://192.168.10.144:61617 > 22 Mar 2015 08:39:07,876 14 WARN jms.DefaultJmsMessageListenerContainer > Setup of JMS message listener invoker failed for destination > 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: Channel > was inactive for too (>30000) long: tcp://192.168.10.144:61617 > 22 Mar 2015 08:42:08,029 14 WARN jms.DefaultJmsMessageListenerContainer > Setup of JMS message listener invoker failed for destination > 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: Channel > was inactive for too (>30000) long: tcp://192.168.10.144:61617 > 22 Mar 2015 08:42:39,341 15 ERROR processor.DefaultErrorHandler Failed > delivery for (MessageId: ID-S143-52155-1426979058276-0-2895 on > ExchangeId: ID-S143-52155-1426979058276-0-2896). Exhausted after delivery > attempt: 1 caught: org.springframework.jms.UncategorizedJmsException: > Uncategorized exception occured during JMS processing; nested exception is > javax.jms.JMSException: Channel was inactive for too (>30000) long: tcp:// > 192.168.10.144:61617 > org.springframework.jms.UncategorizedJmsException: Uncategorized > exception occured during JMS processing; nested exception is > javax.jms.JMSException: Channel was inactive for too (>30000) long: tcp:// > 192.168.10.144:61617 > at org.springframework.jms.support.JmsUtils. > convertJmsAccessException(JmsUtils.java:316) > at org.springframework.jms.support.JmsAccessor. > convertJmsAccessException(JmsAccessor.java:168) > at org.springframework.jms.core.JmsTemplate.execute( > JmsTemplate.java:469) > at org.apache.camel.component.jms.JmsConfiguration$ > CamelJmsTemplate.send(JmsConfiguration.java:188) > at org.apache.camel.component.jms.JmsProducer.doSend( > JmsProducer.java:398) > at org.apache.camel.component.jms.JmsProducer.processInOnly( > JmsProducer.java:352) > at org.apache.camel.component.jms.JmsProducer.process( > JmsProducer.java:132) > at org.apache.camel.util.AsyncProcessorHelper.process( > AsyncProcessorHelper.java:73) > at org.apache.camel.processor.SendProcessor$2.doInAsyncProducer( > SendProcessor.java:120) > at org.apache.camel.impl.ProducerCache.doInAsyncProducer( > ProducerCache.java:292) > at org.apache.camel.processor.SendProcessor.process( > SendProcessor.java:115) > at org.apache.camel.util.AsyncProcessorHelper.process( > AsyncProcessorHelper.java:73) > at org.apache.camel.processor.DelegateAsyncProcessor.processNext( > DelegateAsyncProcessor.java:99) > at org.apache.camel.processor.DelegateAsyncProcessor.process( > DelegateAsyncProcessor.java:90) > at org.apache.camel.management.InstrumentationProcessor.process( > InstrumentationProcessor.java:73) > at org.apache.camel.util.AsyncProcessorHelper.process( > AsyncProcessorHelper.java:73) > at org.apache.camel.processor.DelegateAsyncProcessor.processNext( > DelegateAsyncProcessor.java:99) > at org.apache.camel.processor.DelegateAsyncProcessor.process( > DelegateAsyncProcessor.java:90) > at org.apache.camel.processor.interceptor.TraceInterceptor. > process(TraceInterceptor.java:91) > at org.apache.camel.util.AsyncProcessorHelper.process( > AsyncProcessorHelper.java:73) > at org.apache.camel.processor.RedeliveryErrorHandler. > processErrorHandler(RedeliveryErrorHandler.java:330) > at org.apache.camel.processor.RedeliveryErrorHandler.process( > RedeliveryErrorHandler.java:220) > at org.apache.camel.processor.RouteContextProcessor.processNext( > RouteContextProcessor.java:45) > at org.apache.camel.processor.DelegateAsyncProcessor.process( > DelegateAsyncProcessor.java:90) > at org.apache.camel.processor.interceptor.DefaultChannel. > process(DefaultChannel.java:303) > at org.apache.camel.util.AsyncProcessorHelper.process( > AsyncProcessorHelper.java:73) > at org.apache.camel.processor.Pipeline.process(Pipeline.java:117) > at org.apache.camel.processor.Pipeline.process(Pipeline.java:80) > at org.apache.camel.processor.RouteContextProcessor.processNext( > RouteContextProcessor.java:45) > at org.apache.camel.processor.DelegateAsyncProcessor.process( > DelegateAsyncProcessor.java:90) > at org.apache.camel.processor.UnitOfWorkProcessor.processAsync( > UnitOfWorkProcessor.java:150) > at org.apache.camel.processor.UnitOfWorkProcessor.process( > UnitOfWorkProcessor.java:117) > at org.apache.camel.processor.RouteInflightRepositoryProcess > or.processNext(RouteInflightRepositoryProcessor.java:48) > at org.apache.camel.processor.DelegateAsyncProcessor.process( > DelegateAsyncProcessor.java:90) > at org.apache.camel.util.AsyncProcessorHelper.process( > AsyncProcessorHelper.java:73) > at org.apache.camel.processor.DelegateAsyncProcessor.processNext( > DelegateAsyncProcessor.java:99) > at org.apache.camel.processor.DelegateAsyncProcessor.process( > DelegateAsyncProcessor.java:90) > at org.apache.camel.management.InstrumentationProcessor.process( > InstrumentationProcessor.java:73) > at org.apache.camel.util.AsyncProcessorHelper.process( > AsyncProcessorHelper.java:99) > at org.apache.camel.processor.DelegateAsyncProcessor.process( > DelegateAsyncProcessor.java:86) > at org.apache.camel.component.timer.TimerConsumer. > sendTimerExchange(TimerConsumer.java:134) > at org.apache.camel.component.timer.TimerConsumer$1.run( > TimerConsumer.java:63) > at java.util.TimerThread.mainLoop(Timer.java:555) > at java.util.TimerThread.run(Timer.java:505) > Caused by: javax.jms.JMSException: Channel was inactive for too (>30000) > long: tcp://192.168.10.144:61617 > at org.apache.activemq.util.JMSExceptionSupport.create( > JMSExceptionSupport.java:49) > at org.apache.activemq.ActiveMQConnection.syncSendPacket( > ActiveMQConnection.java:1377) > at org.apache.activemq.ActiveMQConnection.syncSendPacket( > ActiveMQConnection.java:1305) > at org.apache.activemq.ActiveMQSession.send( > ActiveMQSession.java:1795) > at org.apache.activemq.ActiveMQMessageProducer.send( > ActiveMQMessageProducer.java:277) > at org.apache.activemq.ActiveMQMessageProducer.send( > ActiveMQMessageProducer.java:212) > at org.apache.activemq.pool.PooledProducer.send( > PooledProducer.java:79) > at org.apache.activemq.pool.PooledProducer.send( > PooledProducer.java:62) > at org.springframework.jms.core.JmsTemplate.doSend( > JmsTemplate.java:592) > at org.apache.camel.component.jms.JmsConfiguration$ > CamelJmsTemplate.doSend(JmsConfiguration.java:276) > at org.apache.camel.component.jms.JmsConfiguration$ > CamelJmsTemplate.doSendToDestination(JmsConfiguration.java:215) > at org.apache.camel.component.jms.JmsConfiguration$ > CamelJmsTemplate.access$100(JmsConfiguration.java:157) > at org.apache.camel.component.jms.JmsConfiguration$ > CamelJmsTemplate$3.doInJms(JmsConfiguration.java:191) > at org.springframework.jms.core.JmsTemplate.execute( > JmsTemplate.java:466) > ... 41 more > Caused by: org.apache.activemq.transport.InactivityIOException: Channel > was inactive for too (>30000) long: tcp://192.168.10.144:61617 > at org.apache.activemq.transport.AbstractInactivityMonitor$4. > run(AbstractInactivityMonitor.java:209) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 22 Mar 2015 08:42:39,546 15 WARN timer.TimerConsumer Error processing > exchange. Exchange[Message: org.apache.uima.ducc.transport.event. > JdStateDuccEvent@db25b23]. Caused by: > [org.springframework.jms.UncategorizedJmsException > - Uncategorized exception occured during JMS processing; nested exception > is javax.jms.JMSException: Channel was inactive for too (>30000) long: > tcp://192.168.10.144:61617] > org.springframework.jms.UncategorizedJmsException: Uncategorized > exception occured during JMS processing; nested exception is > javax.jms.JMSException: Channel was inactive for too (>30000) long: tcp:// > 192.168.10.144:61617 > at org.springframework.jms.support.JmsUtils. > convertJmsAccessException(JmsUtils.java:316) > at org.springframework.jms.support.JmsAccessor. > convertJmsAccessException(JmsAccessor.java:168) > at org.springframework.jms.core.JmsTemplate.execute( > JmsTemplate.java:469) > at org.apache.camel.component.jms.JmsConfiguration$ > CamelJmsTemplate.send(JmsConfiguration.java:188) > at org.apache.camel.component.jms.JmsProducer.doSend( > JmsProducer.java:398) > at org.apache.camel.component.jms.JmsProducer.processInOnly( > JmsProducer.java:352) > at org.apache.camel.component.jms.JmsProducer.process( > JmsProducer.java:132) > at org.apache.camel.util.AsyncProcessorHelper.process( > AsyncProcessorHelper.java:73) > at org.apache.camel.processor.SendProcessor$2.doInAsyncProducer( > SendProcessor.java:120) > at org.apache.camel.impl.ProducerCache.doInAsyncProducer( > ProducerCache.java:292) > at org.apache.camel.processor.SendProcessor.process( > SendProcessor.java:115) > at org.apache.camel.util.AsyncProcessorHelper.process( > AsyncProcessorHelper.java:73) > at org.apache.camel.processor.DelegateAsyncProcessor.processNext( > DelegateAsyncProcessor.java:99) > at org.apache.camel.processor.DelegateAsyncProcessor.process( > DelegateAsyncProcessor.java:90) > at org.apache.camel.management.InstrumentationProcessor.process( > InstrumentationProcessor.java:73) > at org.apache.camel.util.AsyncProcessorHelper.process( > AsyncProcessorHelper.java:73) > at org.apache.camel.processor.DelegateAsyncProcessor.processNext( > DelegateAsyncProcessor.java:99) > at org.apache.camel.processor.DelegateAsyncProcessor.process( > DelegateAsyncProcessor.java:90) > at org.apache.camel.processor.interceptor.TraceInterceptor. > process(TraceInterceptor.java:91) > at org.apache.camel.util.AsyncProcessorHelper.process( > AsyncProcessorHelper.java:73) > at org.apache.camel.processor.RedeliveryErrorHandler. > processErrorHandler(RedeliveryErrorHandler.java:330) > at org.apache.camel.processor.RedeliveryErrorHandler.process( > RedeliveryErrorHandler.java:220) > at org.apache.camel.processor.RouteContextProcessor.processNext( > RouteContextProcessor.java:45) > at org.apache.camel.processor.DelegateAsyncProcessor.process( > DelegateAsyncProcessor.java:90) > at org.apache.camel.processor.interceptor.DefaultChannel. > process(DefaultChannel.java:303) > at org.apache.camel.util.AsyncProcessorHelper.process( > AsyncProcessorHelper.java:73) > at org.apache.camel.processor.Pipeline.process(Pipeline.java:117) > at org.apache.camel.processor.Pipeline.process(Pipeline.java:80) > at org.apache.camel.processor.RouteContextProcessor.processNext( > RouteContextProcessor.java:45) > at org.apache.camel.processor.DelegateAsyncProcessor.process( > DelegateAsyncProcessor.java:90) > at org.apache.camel.processor.UnitOfWorkProcessor.processAsync( > UnitOfWorkProcessor.java:150) > at org.apache.camel.processor.UnitOfWorkProcessor.process( > UnitOfWorkProcessor.java:117) > at org.apache.camel.processor.RouteInflightRepositoryProcess > or.processNext(RouteInflightRepositoryProcessor.java:48) > at org.apache.camel.processor.DelegateAsyncProcessor.process( > DelegateAsyncProcessor.java:90) > at org.apache.camel.util.AsyncProcessorHelper.process( > AsyncProcessorHelper.java:73) > at org.apache.camel.processor.DelegateAsyncProcessor.processNext( > DelegateAsyncProcessor.java:99) > at org.apache.camel.processor.DelegateAsyncProcessor.process( > DelegateAsyncProcessor.java:90) > at org.apache.camel.management.InstrumentationProcessor.process( > InstrumentationProcessor.java:73) > at org.apache.camel.util.AsyncProcessorHelper.process( > AsyncProcessorHelper.java:99) > at org.apache.camel.processor.DelegateAsyncProcessor.process( > DelegateAsyncProcessor.java:86) > at org.apache.camel.component.timer.TimerConsumer. > sendTimerExchange(TimerConsumer.java:134) > at org.apache.camel.component.timer.TimerConsumer$1.run( > TimerConsumer.java:63) > at java.util.TimerThread.mainLoop(Timer.java:555) > at java.util.TimerThread.run(Timer.java:505) > Caused by: javax.jms.JMSException: Channel was inactive for too (>30000) > long: tcp://192.168.10.144:61617 > at org.apache.activemq.util.JMSExceptionSupport.create( > JMSExceptionSupport.java:49) > at org.apache.activemq.ActiveMQConnection.syncSendPacket( > ActiveMQConnection.java:1377) > at org.apache.activemq.ActiveMQConnection.syncSendPacket( > ActiveMQConnection.java:1305) > at org.apache.activemq.ActiveMQSession.send( > ActiveMQSession.java:1795) > at org.apache.activemq.ActiveMQMessageProducer.send( > ActiveMQMessageProducer.java:277) > at org.apache.activemq.ActiveMQMessageProducer.send( > ActiveMQMessageProducer.java:212) > at org.apache.activemq.pool.PooledProducer.send( > PooledProducer.java:79) > at org.apache.activemq.pool.PooledProducer.send( > PooledProducer.java:62) > at org.springframework.jms.core.JmsTemplate.doSend( > JmsTemplate.java:592) > at org.apache.camel.component.jms.JmsConfiguration$ > CamelJmsTemplate.doSend(JmsConfiguration.java:276) > at org.apache.camel.component.jms.JmsConfiguration$ > CamelJmsTemplate.doSendToDestination(JmsConfiguration.java:215) > at org.apache.camel.component.jms.JmsConfiguration$ > CamelJmsTemplate.access$100(JmsConfiguration.java:157) > at org.apache.camel.component.jms.JmsConfiguration$ > CamelJmsTemplate$3.doInJms(JmsConfiguration.java:191) > at org.springframework.jms.core.JmsTemplate.execute( > JmsTemplate.java:466) > ... 41 more > Caused by: org.apache.activemq.transport.InactivityIOException: Channel > was inactive for too (>30000) long: tcp://192.168.10.144:61617 > at org.apache.activemq.transport.AbstractInactivityMonitor$4. > run(AbstractInactivityMonitor.java:209) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 22 Mar 2015 08:43:19,482 14 WARN jms.DefaultJmsMessageListenerContainer > Setup of JMS message listener invoker failed for destination > 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: Channel > was inactive for too (>30000) long: tcp://192.168.10.144:61617 > 22 Mar 2015 08:44:59,642 14 WARN jms.DefaultJmsMessageListenerContainer > Setup of JMS message listener invoker failed for destination > 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: Channel > was inactive for too (>30000) long: tcp://192.168.10.144:61617 > 22 Mar 2015 08:46:10,729 14 WARN jms.DefaultJmsMessageListenerContainer > Setup of JMS message listener invoker failed for destination > 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: Channel > was inactive for too (>30000) long: tcp://192.168.10.144:61617 > 22 Mar 2015 08:48:05,731 14 WARN jms.DefaultJmsMessageListenerContainer > Setup of JMS message listener invoker failed for destination > 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: Channel > was inactive for too (>30000) long: tcp://192.168.10.144:61617 > 22 Mar 2015 08:49:49,723 14 WARN jms.DefaultJmsMessageListenerContainer > Setup of JMS message listener invoker failed for destination > 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: Channel > was inactive for too (>30000) long: tcp://192.168.10.144:61617 > > > > Please help me to find out root cause of this and how can I kill a job or > start a new one when any job gets stuck after a certain time.. > > -- > Thanks and Regards, > Satya Nand Kanodia > >
