I don't think forcing the JD node and the broker on the same host will help much if your network has issues since all parts of DUCC communicate thru the network, some via the broker.
Nevertheless, if you have only two machines then you could first start DUCC only on the machine that you wish to have the JD+broker. After DUCC is up on the single machine, you could start the DUCC agent on the other machine. Lou. On Mon, Mar 30, 2015 at 12:34 AM, Satya Nand Kanodia < [email protected]> wrote: > Hi, > > Actually I have two systems. One is running broker and other one is > running job driver( I could not find any way to force job driver to be on > any specific system, It automatically goes on 2nd system). I think when the > connectivity goes, the problem comes because broker and driver are on two > different systems. Is there any way I can force job driver to be on the > same system, which is running broker. > > > On 03/24/2015 06:00 PM, Lou DeGenaro wrote: > >> By "serious" I simply meant that if your network is having problems then >> applications that are dependent on the network will be having problems, >> DUCC >> being one such application. Granted, DUCC's Job Driver should not take >> NPE, >> but fixing the NPE may not make your application run much better. >> >> I recommend fixing your network, then it won't matter if you run DUCC 1.x >> or DUCC 2.x. >> >> I can make no promises, but my guess is that DUCC 2.0 will be released in >> April or May. >> >> If this error is really a problem for you and you need a fix for DUCC 1.x, >> I can try to do that - but I'd prefer to leave it as Won't Fix in light of >> the re-write that was done for DUCC 2.0. >> >> Lou. >> >> On Tue, Mar 24, 2015 at 12:50 AM, Satya Nand Kanodia < >> [email protected]> wrote: >> >> Dear Lou, >>> >>> It some time happens in weekends. when It happens we also get some >>> exceptions of networking like exception in connection with database etc >>> etc. So upgrading to DUCC 2.0 is a solution or we can do some thing else >>> ? >>> what did you mean by more serious problems? >>> >>> On 03/23/2015 06:26 PM, Lou DeGenaro wrote: >>> >>> A Jira already exists for this issue https://issues.apache.org/ >>>> jira/browse/ >>>> UIMA-4225. It has been marked "Won't Fix" because the Job Driver and >>>> corresponding Job Process have been re-written for 2.0 version of DUCC >>>> (not >>>> yet released, but available from trunk). >>>> >>>> Are you able to reproduce this error with regularity? I've only seen it >>>> once before, which resulted in Jira 4225 and its subsequent closure. It >>>> also seems that you may have more serious problems than just one job >>>> failing. Based on the message: >>>> >>>> 22 Mar 2015 08:49:49,723 14 WARN jms.DefaultJmsMessageListenerC >>>> ontainer Setup of JMS message listener invoker failed for destination >>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. >>>> >>>> it seems you may have network or broker problems? >>>> >>>> Lou. >>>> >>>> >>>> On Mon, Mar 23, 2015 at 2:16 AM, Satya Nand Kanodia < >>>> [email protected]> wrote: >>>> >>>> Hi, >>>> >>>>> In DUCC when I submit a job , sometimes it gets hung for infinite time. >>>>> Today it happened again. A job started with 2 processes on 22 Mar 2015 >>>>> 04:34:38 had been continuously running till today. >>>>> >>>>> I used this configuration in job file >>>>> >>>>> process_thread_count=3 >>>>> process_failures_limit=50 >>>>> process_per_item_time_max=5 >>>>> process_deployments_max=2 >>>>> attach_console=true >>>>> process_jvm_args=-Xms4g -XX\:+UseConcMarkSweepGC >>>>> process_memory_size=7 >>>>> classpath_order=ducc-before-user >>>>> >>>>> Finally I had to terminate it from DUCC-MON. Job usually takes 15 >>>>> minutes >>>>> max to process documents. I submitted a batch of 433 documents and 342 >>>>> documents had been processed. There were two errors of cas timed out. >>>>> There >>>>> were connection exceptions in both processes , which lasted for 10 >>>>> seconds. >>>>> exception happened at 22 Mar 2015 08:50:42. After that no response from >>>>> both processes. >>>>> >>>>> I got these exceptions from jd.out file >>>>> >>>>> These exception in jd.out file started at 05:09:12. >>>>> >>>>> 22 Mar 2015 05:32:12,352 154 ERROR jd.JobDriver waitForProcessStatus >>>>> N/A process status error? java.lang.NullPointerException >>>>> >>>>> At: >>>>> java.util.TreeMap.getEntry(TreeMap.java:342) >>>>> java.util.TreeMap.get(TreeMap.java:273) >>>>> org.apache.uima.ducc.jd.JobDriver.getProcess( >>>>> JobDriver.java:1081) >>>>> org.apache.uima.ducc.jd.JobDriver.waitForProcessStatus( >>>>> JobDriver.java:1123) >>>>> org.apache.uima.ducc.jd.JobDriver.exception(JobDriver. >>>>> java:1585) >>>>> org.apache.uima.ducc.jd.client.WorkItem.exception( >>>>> WorkItem.java:196) >>>>> org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem. >>>>> java:160) >>>>> java.util.concurrent.Executors$RunnableAdapter. >>>>> call(Executors.java:471) >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker( >>>>> ThreadPoolExecutor.java:1145) >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>>> ThreadPoolExecutor.java:615) >>>>> java.lang.Thread.run(Thread.java:745) >>>>> >>>>> 22 Mar 2015 05:32:12,353 154 ERROR jd.JobDriver exception 19524 >>>>> processing error? java.lang.NullPointerException >>>>> >>>>> At: >>>>> java.util.TreeMap.getEntry(TreeMap.java:342) >>>>> java.util.TreeMap.get(TreeMap.java:273) >>>>> org.apache.uima.ducc.jd.JobDriver.getProcess( >>>>> JobDriver.java:1081) >>>>> org.apache.uima.ducc.jd.JobDriver.isNodeFailure( >>>>> JobDriver.java:1092) >>>>> org.apache.uima.ducc.jd.JobDriver.exception(JobDriver. >>>>> java:1586) >>>>> org.apache.uima.ducc.jd.client.WorkItem.exception( >>>>> WorkItem.java:196) >>>>> org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem. >>>>> java:160) >>>>> java.util.concurrent.Executors$RunnableAdapter. >>>>> call(Executors.java:471) >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker( >>>>> ThreadPoolExecutor.java:1145) >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>>> ThreadPoolExecutor.java:615) >>>>> java.lang.Thread.run(Thread.java:745) >>>>> >>>>> 22 Mar 2015 05:32:12,354 152 ERROR jd.JobDriver waitForProcessStatus >>>>> N/A process status error? java.lang.NullPointerException >>>>> >>>>> At: >>>>> java.util.TreeMap.getEntry(TreeMap.java:342) >>>>> java.util.TreeMap.get(TreeMap.java:273) >>>>> org.apache.uima.ducc.jd.JobDriver.getProcess( >>>>> JobDriver.java:1081) >>>>> org.apache.uima.ducc.jd.JobDriver.waitForProcessStatus( >>>>> JobDriver.java:1123) >>>>> org.apache.uima.ducc.jd.JobDriver.exception(JobDriver. >>>>> java:1585) >>>>> org.apache.uima.ducc.jd.client.WorkItem.exception( >>>>> WorkItem.java:196) >>>>> org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem. >>>>> java:160) >>>>> java.util.concurrent.Executors$RunnableAdapter. >>>>> call(Executors.java:471) >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker( >>>>> ThreadPoolExecutor.java:1145) >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>>> ThreadPoolExecutor.java:615) >>>>> java.lang.Thread.run(Thread.java:745) >>>>> >>>>> 22 Mar 2015 05:32:12,354 152 ERROR jd.JobDriver exception 19524 >>>>> processing error? java.lang.NullPointerException >>>>> >>>>> At: >>>>> java.util.TreeMap.getEntry(TreeMap.java:342) >>>>> java.util.TreeMap.get(TreeMap.java:273) >>>>> org.apache.uima.ducc.jd.JobDriver.getProcess( >>>>> JobDriver.java:1081) >>>>> org.apache.uima.ducc.jd.JobDriver.isNodeFailure( >>>>> JobDriver.java:1092) >>>>> org.apache.uima.ducc.jd.JobDriver.exception(JobDriver. >>>>> java:1586) >>>>> org.apache.uima.ducc.jd.client.WorkItem.exception( >>>>> WorkItem.java:196) >>>>> org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem. >>>>> java:160) >>>>> java.util.concurrent.Executors$RunnableAdapter. >>>>> call(Executors.java:471) >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker( >>>>> ThreadPoolExecutor.java:1145) >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>>> ThreadPoolExecutor.java:615) >>>>> java.lang.Thread.run(Thread.java:745) >>>>> >>>>> 22 Mar 2015 05:32:12,354 60 ERROR jd.JobDriver waitForProcessStatus >>>>> N/A process status error? java.lang.NullPointerException >>>>> >>>>> At: >>>>> java.util.TreeMap.getEntry(TreeMap.java:342) >>>>> java.util.TreeMap.get(TreeMap.java:273) >>>>> org.apache.uima.ducc.jd.JobDriver.getProcess( >>>>> JobDriver.java:1081) >>>>> org.apache.uima.ducc.jd.JobDriver.waitForProcessStatus( >>>>> JobDriver.java:1123) >>>>> org.apache.uima.ducc.jd.JobDriver.exception(JobDriver. >>>>> java:1585) >>>>> org.apache.uima.ducc.jd.client.WorkItem.exception( >>>>> WorkItem.java:196) >>>>> org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem. >>>>> java:160) >>>>> java.util.concurrent.Executors$RunnableAdapter. >>>>> call(Executors.java:471) >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker( >>>>> ThreadPoolExecutor.java:1145) >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>>> ThreadPoolExecutor.java:615) >>>>> java.lang.Thread.run(Thread.java:745) >>>>> >>>>> 22 Mar 2015 05:32:12,354 60 ERROR jd.JobDriver exception 19524 >>>>> processing error? java.lang.NullPointerException >>>>> >>>>> At: >>>>> java.util.TreeMap.getEntry(TreeMap.java:342) >>>>> java.util.TreeMap.get(TreeMap.java:273) >>>>> org.apache.uima.ducc.jd.JobDriver.getProcess( >>>>> JobDriver.java:1081) >>>>> org.apache.uima.ducc.jd.JobDriver.isNodeFailure( >>>>> JobDriver.java:1092) >>>>> org.apache.uima.ducc.jd.JobDriver.exception(JobDriver. >>>>> java:1586) >>>>> org.apache.uima.ducc.jd.client.WorkItem.exception( >>>>> WorkItem.java:196) >>>>> org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem. >>>>> java:160) >>>>> java.util.concurrent.Executors$RunnableAdapter. >>>>> call(Executors.java:471) >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker( >>>>> ThreadPoolExecutor.java:1145) >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>>> ThreadPoolExecutor.java:615) >>>>> java.lang.Thread.run(Thread.java:745) >>>>> >>>>> 22 Mar 2015 07:10:59,845 14 WARN jms.DefaultJmsMessageListenerConta >>>>> iner >>>>> Setup of JMS message listener invoker failed for destination >>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: >>>>> Channel >>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617 >>>>> 22 Mar 2015 07:13:09,855 14 WARN jms.DefaultJmsMessageListenerConta >>>>> iner >>>>> Setup of JMS message listener invoker failed for destination >>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: >>>>> Channel >>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617 >>>>> 22 Mar 2015 07:16:50,007 14 WARN jms.DefaultJmsMessageListenerConta >>>>> iner >>>>> Setup of JMS message listener invoker failed for destination >>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: >>>>> Channel >>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617 >>>>> 22 Mar 2015 07:24:35,994 14 WARN jms.EndpointMessageListener Execution >>>>> of >>>>> JMS message listener failed. Caused by: [javax.jms. >>>>> IllegalStateException >>>>> - The Session is closed] >>>>> javax.jms.IllegalStateException: The Session is closed >>>>> at org.apache.activemq.ActiveMQSession.checkClosed( >>>>> ActiveMQSession.java:731) >>>>> at org.apache.activemq.ActiveMQSession.getTransacted( >>>>> ActiveMQSession.java:521) >>>>> at org.apache.activemq.pool.PooledSession.getTransacted( >>>>> PooledSession.java:256) >>>>> at org.springframework.jms.listener. >>>>> AbstractMessageListenerContain >>>>> er.commitIfNecessary(AbstractMessageListenerContainer.java:571) >>>>> at org.springframework.jms.listener. >>>>> AbstractMessageListenerContain >>>>> er.doExecuteListener(AbstractMessageListenerContainer.java:481) >>>>> at org.springframework.jms.listener. >>>>> AbstractPollingMessageListener >>>>> Container.doReceiveAndExecute(AbstractPollingMessageListener >>>>> Container.java:325) >>>>> at org.springframework.jms.listener. >>>>> AbstractPollingMessageListener >>>>> Container.receiveAndExecute(AbstractPollingMessageListener >>>>> Container.java:263) >>>>> at org.springframework.jms.listener. >>>>> DefaultMessageListenerContaine >>>>> r$AsyncMessageListenerInvoker.invokeListener( >>>>> DefaultMessageListenerContainer.java:1058) >>>>> at org.springframework.jms.listener. >>>>> DefaultMessageListenerContaine >>>>> r$AsyncMessageListenerInvoker.executeOngoingLoop( >>>>> DefaultMessageListenerContainer.java:1050) >>>>> at org.springframework.jms.listener. >>>>> DefaultMessageListenerContaine >>>>> r$AsyncMessageListenerInvoker.run(DefaultMessageListenerContaine >>>>> r.java:947) >>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>>>> ThreadPoolExecutor.java:1145) >>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>>> ThreadPoolExecutor.java:615) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> 22 Mar 2015 07:24:36,617 14 WARN jms.DefaultJmsMessageListenerConta >>>>> iner >>>>> Setup of JMS message listener invoker failed for destination >>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: The >>>>> Session is closed >>>>> 22 Mar 2015 07:26:31,822 14 WARN jms.EndpointMessageListener Execution >>>>> of >>>>> JMS message listener failed. Caused by: [javax.jms. >>>>> IllegalStateException >>>>> - The Session is closed] >>>>> javax.jms.IllegalStateException: The Session is closed >>>>> at org.apache.activemq.ActiveMQSession.checkClosed( >>>>> ActiveMQSession.java:731) >>>>> at org.apache.activemq.ActiveMQSession.getTransacted( >>>>> ActiveMQSession.java:521) >>>>> at org.apache.activemq.pool.PooledSession.getTransacted( >>>>> PooledSession.java:256) >>>>> at org.springframework.jms.listener. >>>>> AbstractMessageListenerContain >>>>> er.commitIfNecessary(AbstractMessageListenerContainer.java:571) >>>>> at org.springframework.jms.listener. >>>>> AbstractMessageListenerContain >>>>> er.doExecuteListener(AbstractMessageListenerContainer.java:481) >>>>> at org.springframework.jms.listener. >>>>> AbstractPollingMessageListener >>>>> Container.doReceiveAndExecute(AbstractPollingMessageListener >>>>> Container.java:325) >>>>> at org.springframework.jms.listener. >>>>> AbstractPollingMessageListener >>>>> Container.receiveAndExecute(AbstractPollingMessageListener >>>>> Container.java:263) >>>>> at org.springframework.jms.listener. >>>>> DefaultMessageListenerContaine >>>>> r$AsyncMessageListenerInvoker.invokeListener( >>>>> DefaultMessageListenerContainer.java:1058) >>>>> at org.springframework.jms.listener. >>>>> DefaultMessageListenerContaine >>>>> r$AsyncMessageListenerInvoker.executeOngoingLoop( >>>>> DefaultMessageListenerContainer.java:1050) >>>>> at org.springframework.jms.listener. >>>>> DefaultMessageListenerContaine >>>>> r$AsyncMessageListenerInvoker.run(DefaultMessageListenerContaine >>>>> r.java:947) >>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>>>> ThreadPoolExecutor.java:1145) >>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>>> ThreadPoolExecutor.java:615) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> 22 Mar 2015 07:26:32,067 14 WARN jms.DefaultJmsMessageListenerConta >>>>> iner >>>>> Setup of JMS message listener invoker failed for destination >>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: The >>>>> Session is closed >>>>> 22 Mar 2015 07:28:12,075 14 WARN jms.DefaultJmsMessageListenerConta >>>>> iner >>>>> Setup of JMS message listener invoker failed for destination >>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: >>>>> Channel >>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617 >>>>> 22 Mar 2015 07:30:55,001 14 WARN jms.DefaultJmsMessageListenerConta >>>>> iner >>>>> Could not refresh JMS Connection for destination 'ducc.orchestrator. >>>>> abbreviated.state' >>>>> - retrying in 5000 ms. Cause: Channel was inactive for too (>30000) >>>>> long: >>>>> tcp://192.168.10.144:61617 >>>>> 22 Mar 2015 08:29:26,669 14 WARN jms.EndpointMessageListener Execution >>>>> of >>>>> JMS message listener failed. Caused by: [javax.jms. >>>>> IllegalStateException >>>>> - The Session is closed] >>>>> javax.jms.IllegalStateException: The Session is closed >>>>> at org.apache.activemq.ActiveMQSession.checkClosed( >>>>> ActiveMQSession.java:731) >>>>> at org.apache.activemq.ActiveMQSession.getTransacted( >>>>> ActiveMQSession.java:521) >>>>> at org.apache.activemq.pool.PooledSession.getTransacted( >>>>> PooledSession.java:256) >>>>> at org.springframework.jms.listener. >>>>> AbstractMessageListenerContain >>>>> er.commitIfNecessary(AbstractMessageListenerContainer.java:571) >>>>> at org.springframework.jms.listener. >>>>> AbstractMessageListenerContain >>>>> er.doExecuteListener(AbstractMessageListenerContainer.java:481) >>>>> at org.springframework.jms.listener. >>>>> AbstractPollingMessageListener >>>>> Container.doReceiveAndExecute(AbstractPollingMessageListener >>>>> Container.java:325) >>>>> at org.springframework.jms.listener. >>>>> AbstractPollingMessageListener >>>>> Container.receiveAndExecute(AbstractPollingMessageListener >>>>> Container.java:263) >>>>> at org.springframework.jms.listener. >>>>> DefaultMessageListenerContaine >>>>> r$AsyncMessageListenerInvoker.invokeListener( >>>>> DefaultMessageListenerContainer.java:1058) >>>>> at org.springframework.jms.listener. >>>>> DefaultMessageListenerContaine >>>>> r$AsyncMessageListenerInvoker.executeOngoingLoop( >>>>> DefaultMessageListenerContainer.java:1050) >>>>> at org.springframework.jms.listener. >>>>> DefaultMessageListenerContaine >>>>> r$AsyncMessageListenerInvoker.run(DefaultMessageListenerContaine >>>>> r.java:947) >>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>>>> ThreadPoolExecutor.java:1145) >>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>>> ThreadPoolExecutor.java:615) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> 22 Mar 2015 08:29:26,671 14 WARN jms.DefaultJmsMessageListenerConta >>>>> iner >>>>> Setup of JMS message listener invoker failed for destination >>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: The >>>>> Session is closed >>>>> 22 Mar 2015 08:33:06,678 14 WARN jms.DefaultJmsMessageListenerConta >>>>> iner >>>>> Setup of JMS message listener invoker failed for destination >>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: >>>>> Channel >>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617 >>>>> 22 Mar 2015 08:36:16,841 14 WARN jms.DefaultJmsMessageListenerConta >>>>> iner >>>>> Setup of JMS message listener invoker failed for destination >>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: >>>>> Channel >>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617 >>>>> 22 Mar 2015 08:37:27,506 14 WARN jms.DefaultJmsMessageListenerConta >>>>> iner >>>>> Setup of JMS message listener invoker failed for destination >>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: >>>>> Channel >>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617 >>>>> 22 Mar 2015 08:39:07,876 14 WARN jms.DefaultJmsMessageListenerConta >>>>> iner >>>>> Setup of JMS message listener invoker failed for destination >>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: >>>>> Channel >>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617 >>>>> 22 Mar 2015 08:42:08,029 14 WARN jms.DefaultJmsMessageListenerConta >>>>> iner >>>>> Setup of JMS message listener invoker failed for destination >>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: >>>>> Channel >>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617 >>>>> 22 Mar 2015 08:42:39,341 15 ERROR processor.DefaultErrorHandler Failed >>>>> delivery for (MessageId: ID-S143-52155-1426979058276-0-2895 on >>>>> ExchangeId: ID-S143-52155-1426979058276-0-2896). Exhausted after >>>>> delivery >>>>> attempt: 1 caught: org.springframework.jms.UncategorizedJmsException: >>>>> Uncategorized exception occured during JMS processing; nested exception >>>>> is >>>>> javax.jms.JMSException: Channel was inactive for too (>30000) long: >>>>> tcp:// >>>>> 192.168.10.144:61617 >>>>> org.springframework.jms.UncategorizedJmsException: Uncategorized >>>>> exception occured during JMS processing; nested exception is >>>>> javax.jms.JMSException: Channel was inactive for too (>30000) long: >>>>> tcp:// >>>>> 192.168.10.144:61617 >>>>> at org.springframework.jms.support.JmsUtils. >>>>> convertJmsAccessException(JmsUtils.java:316) >>>>> at org.springframework.jms.support.JmsAccessor. >>>>> convertJmsAccessException(JmsAccessor.java:168) >>>>> at org.springframework.jms.core.JmsTemplate.execute( >>>>> JmsTemplate.java:469) >>>>> at org.apache.camel.component.jms.JmsConfiguration$ >>>>> CamelJmsTemplate.send(JmsConfiguration.java:188) >>>>> at org.apache.camel.component.jms.JmsProducer.doSend( >>>>> JmsProducer.java:398) >>>>> at org.apache.camel.component.jms.JmsProducer.processInOnly( >>>>> JmsProducer.java:352) >>>>> at org.apache.camel.component.jms.JmsProducer.process( >>>>> JmsProducer.java:132) >>>>> at org.apache.camel.util.AsyncProcessorHelper.process( >>>>> AsyncProcessorHelper.java:73) >>>>> at org.apache.camel.processor.SendProcessor$2. >>>>> doInAsyncProducer( >>>>> SendProcessor.java:120) >>>>> at org.apache.camel.impl.ProducerCache.doInAsyncProducer( >>>>> ProducerCache.java:292) >>>>> at org.apache.camel.processor.SendProcessor.process( >>>>> SendProcessor.java:115) >>>>> at org.apache.camel.util.AsyncProcessorHelper.process( >>>>> AsyncProcessorHelper.java:73) >>>>> at org.apache.camel.processor.DelegateAsyncProcessor. >>>>> processNext( >>>>> DelegateAsyncProcessor.java:99) >>>>> at org.apache.camel.processor.DelegateAsyncProcessor. >>>>> process( >>>>> DelegateAsyncProcessor.java:90) >>>>> at org.apache.camel.management.InstrumentationProcessor. >>>>> process( >>>>> InstrumentationProcessor.java:73) >>>>> at org.apache.camel.util.AsyncProcessorHelper.process( >>>>> AsyncProcessorHelper.java:73) >>>>> at org.apache.camel.processor.DelegateAsyncProcessor. >>>>> processNext( >>>>> DelegateAsyncProcessor.java:99) >>>>> at org.apache.camel.processor.DelegateAsyncProcessor. >>>>> process( >>>>> DelegateAsyncProcessor.java:90) >>>>> at org.apache.camel.processor.interceptor.TraceInterceptor. >>>>> process(TraceInterceptor.java:91) >>>>> at org.apache.camel.util.AsyncProcessorHelper.process( >>>>> AsyncProcessorHelper.java:73) >>>>> at org.apache.camel.processor.RedeliveryErrorHandler. >>>>> processErrorHandler(RedeliveryErrorHandler.java:330) >>>>> at org.apache.camel.processor.RedeliveryErrorHandler. >>>>> process( >>>>> RedeliveryErrorHandler.java:220) >>>>> at org.apache.camel.processor.RouteContextProcessor. >>>>> processNext( >>>>> RouteContextProcessor.java:45) >>>>> at org.apache.camel.processor.DelegateAsyncProcessor. >>>>> process( >>>>> DelegateAsyncProcessor.java:90) >>>>> at org.apache.camel.processor.interceptor.DefaultChannel. >>>>> process(DefaultChannel.java:303) >>>>> at org.apache.camel.util.AsyncProcessorHelper.process( >>>>> AsyncProcessorHelper.java:73) >>>>> at org.apache.camel.processor.Pipeline.process(Pipeline. >>>>> java:117) >>>>> at org.apache.camel.processor.Pipeline.process(Pipeline. >>>>> java:80) >>>>> at org.apache.camel.processor.RouteContextProcessor. >>>>> processNext( >>>>> RouteContextProcessor.java:45) >>>>> at org.apache.camel.processor.DelegateAsyncProcessor. >>>>> process( >>>>> DelegateAsyncProcessor.java:90) >>>>> at org.apache.camel.processor.UnitOfWorkProcessor. >>>>> processAsync( >>>>> UnitOfWorkProcessor.java:150) >>>>> at org.apache.camel.processor.UnitOfWorkProcessor.process( >>>>> UnitOfWorkProcessor.java:117) >>>>> at org.apache.camel.processor.RouteInflightRepositoryProcess >>>>> or.processNext(RouteInflightRepositoryProcessor.java:48) >>>>> at org.apache.camel.processor.DelegateAsyncProcessor. >>>>> process( >>>>> DelegateAsyncProcessor.java:90) >>>>> at org.apache.camel.util.AsyncProcessorHelper.process( >>>>> AsyncProcessorHelper.java:73) >>>>> at org.apache.camel.processor.DelegateAsyncProcessor. >>>>> processNext( >>>>> DelegateAsyncProcessor.java:99) >>>>> at org.apache.camel.processor.DelegateAsyncProcessor. >>>>> process( >>>>> DelegateAsyncProcessor.java:90) >>>>> at org.apache.camel.management.InstrumentationProcessor. >>>>> process( >>>>> InstrumentationProcessor.java:73) >>>>> at org.apache.camel.util.AsyncProcessorHelper.process( >>>>> AsyncProcessorHelper.java:99) >>>>> at org.apache.camel.processor.DelegateAsyncProcessor. >>>>> process( >>>>> DelegateAsyncProcessor.java:86) >>>>> at org.apache.camel.component.timer.TimerConsumer. >>>>> sendTimerExchange(TimerConsumer.java:134) >>>>> at org.apache.camel.component.timer.TimerConsumer$1.run( >>>>> TimerConsumer.java:63) >>>>> at java.util.TimerThread.mainLoop(Timer.java:555) >>>>> at java.util.TimerThread.run(Timer.java:505) >>>>> Caused by: javax.jms.JMSException: Channel was inactive for too >>>>> (>30000) >>>>> long: tcp://192.168.10.144:61617 >>>>> at org.apache.activemq.util.JMSExceptionSupport.create( >>>>> JMSExceptionSupport.java:49) >>>>> at org.apache.activemq.ActiveMQConnection.syncSendPacket( >>>>> ActiveMQConnection.java:1377) >>>>> at org.apache.activemq.ActiveMQConnection.syncSendPacket( >>>>> ActiveMQConnection.java:1305) >>>>> at org.apache.activemq.ActiveMQSession.send( >>>>> ActiveMQSession.java:1795) >>>>> at org.apache.activemq.ActiveMQMessageProducer.send( >>>>> ActiveMQMessageProducer.java:277) >>>>> at org.apache.activemq.ActiveMQMessageProducer.send( >>>>> ActiveMQMessageProducer.java:212) >>>>> at org.apache.activemq.pool.PooledProducer.send( >>>>> PooledProducer.java:79) >>>>> at org.apache.activemq.pool.PooledProducer.send( >>>>> PooledProducer.java:62) >>>>> at org.springframework.jms.core.JmsTemplate.doSend( >>>>> JmsTemplate.java:592) >>>>> at org.apache.camel.component.jms.JmsConfiguration$ >>>>> CamelJmsTemplate.doSend(JmsConfiguration.java:276) >>>>> at org.apache.camel.component.jms.JmsConfiguration$ >>>>> CamelJmsTemplate.doSendToDestination(JmsConfiguration.java:215) >>>>> at org.apache.camel.component.jms.JmsConfiguration$ >>>>> CamelJmsTemplate.access$100(JmsConfiguration.java:157) >>>>> at org.apache.camel.component.jms.JmsConfiguration$ >>>>> CamelJmsTemplate$3.doInJms(JmsConfiguration.java:191) >>>>> at org.springframework.jms.core.JmsTemplate.execute( >>>>> JmsTemplate.java:466) >>>>> ... 41 more >>>>> Caused by: org.apache.activemq.transport.InactivityIOException: >>>>> Channel >>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617 >>>>> at org.apache.activemq.transport. >>>>> AbstractInactivityMonitor$4. >>>>> run(AbstractInactivityMonitor.java:209) >>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>>>> ThreadPoolExecutor.java:1145) >>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>>> ThreadPoolExecutor.java:615) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> 22 Mar 2015 08:42:39,546 15 WARN timer.TimerConsumer Error processing >>>>> exchange. Exchange[Message: org.apache.uima.ducc.transport.event. >>>>> JdStateDuccEvent@db25b23]. Caused by: [org.springframework.jms. >>>>> UncategorizedJmsException >>>>> - Uncategorized exception occured during JMS processing; nested >>>>> exception >>>>> is javax.jms.JMSException: Channel was inactive for too (>30000) long: >>>>> tcp://192.168.10.144:61617] >>>>> org.springframework.jms.UncategorizedJmsException: Uncategorized >>>>> exception occured during JMS processing; nested exception is >>>>> javax.jms.JMSException: Channel was inactive for too (>30000) long: >>>>> tcp:// >>>>> 192.168.10.144:61617 >>>>> at org.springframework.jms.support.JmsUtils. >>>>> convertJmsAccessException(JmsUtils.java:316) >>>>> at org.springframework.jms.support.JmsAccessor. >>>>> convertJmsAccessException(JmsAccessor.java:168) >>>>> at org.springframework.jms.core.JmsTemplate.execute( >>>>> JmsTemplate.java:469) >>>>> at org.apache.camel.component.jms.JmsConfiguration$ >>>>> CamelJmsTemplate.send(JmsConfiguration.java:188) >>>>> at org.apache.camel.component.jms.JmsProducer.doSend( >>>>> JmsProducer.java:398) >>>>> at org.apache.camel.component.jms.JmsProducer.processInOnly( >>>>> JmsProducer.java:352) >>>>> at org.apache.camel.component.jms.JmsProducer.process( >>>>> JmsProducer.java:132) >>>>> at org.apache.camel.util.AsyncProcessorHelper.process( >>>>> AsyncProcessorHelper.java:73) >>>>> at org.apache.camel.processor.SendProcessor$2. >>>>> doInAsyncProducer( >>>>> SendProcessor.java:120) >>>>> at org.apache.camel.impl.ProducerCache.doInAsyncProducer( >>>>> ProducerCache.java:292) >>>>> at org.apache.camel.processor.SendProcessor.process( >>>>> SendProcessor.java:115) >>>>> at org.apache.camel.util.AsyncProcessorHelper.process( >>>>> AsyncProcessorHelper.java:73) >>>>> at org.apache.camel.processor.DelegateAsyncProcessor. >>>>> processNext( >>>>> DelegateAsyncProcessor.java:99) >>>>> at org.apache.camel.processor.DelegateAsyncProcessor. >>>>> process( >>>>> DelegateAsyncProcessor.java:90) >>>>> at org.apache.camel.management.InstrumentationProcessor. >>>>> process( >>>>> InstrumentationProcessor.java:73) >>>>> at org.apache.camel.util.AsyncProcessorHelper.process( >>>>> AsyncProcessorHelper.java:73) >>>>> at org.apache.camel.processor.DelegateAsyncProcessor. >>>>> processNext( >>>>> DelegateAsyncProcessor.java:99) >>>>> at org.apache.camel.processor.DelegateAsyncProcessor. >>>>> process( >>>>> DelegateAsyncProcessor.java:90) >>>>> at org.apache.camel.processor.interceptor.TraceInterceptor. >>>>> process(TraceInterceptor.java:91) >>>>> at org.apache.camel.util.AsyncProcessorHelper.process( >>>>> AsyncProcessorHelper.java:73) >>>>> at org.apache.camel.processor.RedeliveryErrorHandler. >>>>> processErrorHandler(RedeliveryErrorHandler.java:330) >>>>> at org.apache.camel.processor.RedeliveryErrorHandler. >>>>> process( >>>>> RedeliveryErrorHandler.java:220) >>>>> at org.apache.camel.processor.RouteContextProcessor. >>>>> processNext( >>>>> RouteContextProcessor.java:45) >>>>> at org.apache.camel.processor.DelegateAsyncProcessor. >>>>> process( >>>>> DelegateAsyncProcessor.java:90) >>>>> at org.apache.camel.processor.interceptor.DefaultChannel. >>>>> process(DefaultChannel.java:303) >>>>> at org.apache.camel.util.AsyncProcessorHelper.process( >>>>> AsyncProcessorHelper.java:73) >>>>> at org.apache.camel.processor.Pipeline.process(Pipeline. >>>>> java:117) >>>>> at org.apache.camel.processor.Pipeline.process(Pipeline. >>>>> java:80) >>>>> at org.apache.camel.processor.RouteContextProcessor. >>>>> processNext( >>>>> RouteContextProcessor.java:45) >>>>> at org.apache.camel.processor.DelegateAsyncProcessor. >>>>> process( >>>>> DelegateAsyncProcessor.java:90) >>>>> at org.apache.camel.processor.UnitOfWorkProcessor. >>>>> processAsync( >>>>> UnitOfWorkProcessor.java:150) >>>>> at org.apache.camel.processor.UnitOfWorkProcessor.process( >>>>> UnitOfWorkProcessor.java:117) >>>>> at org.apache.camel.processor.RouteInflightRepositoryProcess >>>>> or.processNext(RouteInflightRepositoryProcessor.java:48) >>>>> at org.apache.camel.processor.DelegateAsyncProcessor. >>>>> process( >>>>> DelegateAsyncProcessor.java:90) >>>>> at org.apache.camel.util.AsyncProcessorHelper.process( >>>>> AsyncProcessorHelper.java:73) >>>>> at org.apache.camel.processor.DelegateAsyncProcessor. >>>>> processNext( >>>>> DelegateAsyncProcessor.java:99) >>>>> at org.apache.camel.processor.DelegateAsyncProcessor. >>>>> process( >>>>> DelegateAsyncProcessor.java:90) >>>>> at org.apache.camel.management.InstrumentationProcessor. >>>>> process( >>>>> InstrumentationProcessor.java:73) >>>>> at org.apache.camel.util.AsyncProcessorHelper.process( >>>>> AsyncProcessorHelper.java:99) >>>>> at org.apache.camel.processor.DelegateAsyncProcessor. >>>>> process( >>>>> DelegateAsyncProcessor.java:86) >>>>> at org.apache.camel.component.timer.TimerConsumer. >>>>> sendTimerExchange(TimerConsumer.java:134) >>>>> at org.apache.camel.component.timer.TimerConsumer$1.run( >>>>> TimerConsumer.java:63) >>>>> at java.util.TimerThread.mainLoop(Timer.java:555) >>>>> at java.util.TimerThread.run(Timer.java:505) >>>>> Caused by: javax.jms.JMSException: Channel was inactive for too >>>>> (>30000) >>>>> long: tcp://192.168.10.144:61617 >>>>> at org.apache.activemq.util.JMSExceptionSupport.create( >>>>> JMSExceptionSupport.java:49) >>>>> at org.apache.activemq.ActiveMQConnection.syncSendPacket( >>>>> ActiveMQConnection.java:1377) >>>>> at org.apache.activemq.ActiveMQConnection.syncSendPacket( >>>>> ActiveMQConnection.java:1305) >>>>> at org.apache.activemq.ActiveMQSession.send( >>>>> ActiveMQSession.java:1795) >>>>> at org.apache.activemq.ActiveMQMessageProducer.send( >>>>> ActiveMQMessageProducer.java:277) >>>>> at org.apache.activemq.ActiveMQMessageProducer.send( >>>>> ActiveMQMessageProducer.java:212) >>>>> at org.apache.activemq.pool.PooledProducer.send( >>>>> PooledProducer.java:79) >>>>> at org.apache.activemq.pool.PooledProducer.send( >>>>> PooledProducer.java:62) >>>>> at org.springframework.jms.core.JmsTemplate.doSend( >>>>> JmsTemplate.java:592) >>>>> at org.apache.camel.component.jms.JmsConfiguration$ >>>>> CamelJmsTemplate.doSend(JmsConfiguration.java:276) >>>>> at org.apache.camel.component.jms.JmsConfiguration$ >>>>> CamelJmsTemplate.doSendToDestination(JmsConfiguration.java:215) >>>>> at org.apache.camel.component.jms.JmsConfiguration$ >>>>> CamelJmsTemplate.access$100(JmsConfiguration.java:157) >>>>> at org.apache.camel.component.jms.JmsConfiguration$ >>>>> CamelJmsTemplate$3.doInJms(JmsConfiguration.java:191) >>>>> at org.springframework.jms.core.JmsTemplate.execute( >>>>> JmsTemplate.java:466) >>>>> ... 41 more >>>>> Caused by: org.apache.activemq.transport.InactivityIOException: >>>>> Channel >>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617 >>>>> at org.apache.activemq.transport. >>>>> AbstractInactivityMonitor$4. >>>>> run(AbstractInactivityMonitor.java:209) >>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>>>> ThreadPoolExecutor.java:1145) >>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>>> ThreadPoolExecutor.java:615) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> 22 Mar 2015 08:43:19,482 14 WARN jms.DefaultJmsMessageListenerConta >>>>> iner >>>>> Setup of JMS message listener invoker failed for destination >>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: >>>>> Channel >>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617 >>>>> 22 Mar 2015 08:44:59,642 14 WARN jms.DefaultJmsMessageListenerConta >>>>> iner >>>>> Setup of JMS message listener invoker failed for destination >>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: >>>>> Channel >>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617 >>>>> 22 Mar 2015 08:46:10,729 14 WARN jms.DefaultJmsMessageListenerConta >>>>> iner >>>>> Setup of JMS message listener invoker failed for destination >>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: >>>>> Channel >>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617 >>>>> 22 Mar 2015 08:48:05,731 14 WARN jms.DefaultJmsMessageListenerConta >>>>> iner >>>>> Setup of JMS message listener invoker failed for destination >>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: >>>>> Channel >>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617 >>>>> 22 Mar 2015 08:49:49,723 14 WARN jms.DefaultJmsMessageListenerConta >>>>> iner >>>>> Setup of JMS message listener invoker failed for destination >>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause: >>>>> Channel >>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617 >>>>> >>>>> >>>>> >>>>> Please help me to find out root cause of this and how can I kill a job >>>>> or >>>>> start a new one when any job gets stuck after a certain time.. >>>>> >>>>> -- >>>>> Thanks and Regards, >>>>> Satya Nand Kanodia >>>>> >>>>> >>>>> >>>>> -- >>> Thanks and Regards, >>> Satya Nand Kanodia >>> >>> >>> > -- > Thanks and Regards, > Satya Nand Kanodia > >
