I will try to collect a thread dump - and also try to let the Windows Service just continue trying to stop to see what happens - off course assuming I can reproduce this in my test environment :-/
Thanks so far. /Michael Venlig hilsen / Best Regards Michael Martinsen Openminds Fredens Torv 1B, 1. DK-8000 Aarhus C Denmark Cvr; 33647166 M: +45 6169 6779 E: m...@openminds.dk www.openminds.dk On Fri, Jan 29, 2021 at 9:56 AM Claus Ibsen <claus.ib...@gmail.com> wrote: > > Hi > > When your app is stuck then you can take a thread dump, which can help > pin point to which thread and where its stuck. > > Camel uses a separate thread to shutdown routes, while waiting for > that to complete. And after a timeout then Camel will attempt to force > shutdown. > Don't you see any kind of shutdown logging about waiting for routes to > finish and a timeout was hit or something? > > On Fri, Jan 29, 2021 at 9:11 AM Michael Martinsen <m...@openminds.dk> wrote: > > > > Hello Camel users > > > > I have a problem with some Apache Camel applications (Spring Boot) that are > > consuming messages from RabbitMQ. > > I am running these applications as Windows Services (using WinSW). > > > > Sometime when I deploy a new application to the Windows server running the > > applications, the following exception is logged: > > > > 2021-01-28 19:57:24,554 WARN > > org.apache.camel.component.rabbitmq.RabbitMQConsumer [Camel (camel-1) > > thread #13 - ShutdownTask] Error occurred while stopping consumer. This > > exception is ignored > > java.io.IOException: Unknown consumerTag > > at com.rabbitmq.client.impl.ChannelN.basicCancel(ChannelN.java:1476) > > at > > com.rabbitmq.client.impl.recovery.AutorecoveringChannel.basicCancel(AutorecoveringChannel.java:642) > > at > > org.apache.camel.component.rabbitmq.RabbitConsumer.doStop(RabbitConsumer.java:185) > > at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:119) > > at org.apache.camel.util.ServiceHelper.stopService(ServiceHelper.java:142) > > at > > org.apache.camel.util.ServiceHelper.stopAndShutdownService(ServiceHelper.java:205) > > at > > org.apache.camel.component.rabbitmq.RabbitMQConsumer.closeConnectionAndChannel(RabbitMQConsumer.java:146) > > at > > org.apache.camel.component.rabbitmq.RabbitMQConsumer.doSuspend(RabbitMQConsumer.java:161) > > at org.apache.camel.support.ServiceSupport.suspend(ServiceSupport.java:145) > > at > > org.apache.camel.util.ServiceHelper.suspendService(ServiceHelper.java:388) > > at > > org.apache.camel.impl.DefaultShutdownStrategy.suspendNow(DefaultShutdownStrategy.java:401) > > at > > org.apache.camel.impl.DefaultShutdownStrategy$ShutdownTask.run(DefaultShutdownStrategy.java:572) > > at > > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > > at > > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > at > > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > at java.base/java.lang.Thread.run(Thread.java:830) > > 2021-01-28 19:57:24,981 INFO > > org.apache.camel.impl.DefaultShutdownStrategy$ShutdownTask [Camel (camel-1) > > thread #13 - ShutdownTask] Route: XXX complete, was consuming from: > > rabbitmq://?declare=false&networkRecoveryInterval=30000&queue=test.queue > > 2021-01-28 19:57:24,981 INFO > > org.apache.camel.impl.DefaultShutdownStrategy$ShutdownTask [Camel (camel-1) > > thread #13 - ShutdownTask] Route: XXX shutdown complete, was consuming > > from: seda://dlq > > 2021-01-28 19:57:24,981 INFO > > org.apache.camel.impl.DefaultShutdownStrategy$ShutdownTask [Camel (camel-1) > > thread #13 - ShutdownTask] Route: XXX shutdown complete, was consuming > > from: direct://YYY > > 2021-01-28 19:57:24,981 INFO > > org.apache.camel.impl.DefaultShutdownStrategy$ShutdownTask [Camel (camel-1) > > thread #13 - ShutdownTask] Route: XXX shutdown complete, was consuming > > from: file://YYY > > 2021-01-28 19:57:24,981 INFO org.apache.camel.impl.DefaultShutdownStrategy > > [RMI TCP Connection(2)-10.0.3.117] Graceful shutdown of 4 routes completed > > in 0 seconds > > 2021-01-28 19:57:24,996 INFO org.apache.camel.impl.DefaultCamelContext [RMI > > TCP Connection(2)-10.0.3.117] Apache Camel 2.25.2 (CamelContext: camel-1) > > uptime 7 days 17 hours > > 2021-01-28 19:57:24,996 INFO org.apache.camel.impl.DefaultCamelContext [RMI > > TCP Connection(2)-10.0.3.117] Apache Camel 2.25.2 (CamelContext: camel-1) > > is shutdown in 0.442 seconds > > 2021-01-28 19:57:25,076 INFO > > org.springframework.scheduling.concurrent.ExecutorConfigurationSupport [RMI > > TCP Connection(2)-10.0.3.117] Shutting down ExecutorService > > 'applicationTaskExecutor' > > > > And that's it. > > Although the exception is only a warning and it states that it's ignored, > > nothing more happens. > > The Windows Service does not stop and it's stuck in state "Stopping".. > > Either the entire Windows server has to be restarted or some manual actions > > have to be taken to stop the Windows Service. > > > > So I am really not sure where to look for a problem - if this is in Apache > > Camel / Spring Boot / RabbitMQ / WinSW... > > > > I found this: https://github.com/spring-projects/spring-amqp/issues/1113 > > > > But that relates to the Spring AMQP component which I am not using > > (directly). I am using the Spring Boot Camel RabbitMQ Starter. > > > > I noticed someone writing that this happens when the Camel app is stopped > > BEFORE it has read any messages from the RabbitMQ server. > > That's not what I am seeing - I have seen this in production environments > > several times where the Camel apps have been running for months and HAVE > > consumed messages. > > But I am not sure if this is caused by the RabbitMQ server being retarted > > while the Camel apps are running... > > > > > > Any help is most welcome. > > > > More details about my setup: > > > > > > Spring Boot App > > 2.3.5.RELEASE > > > > Apache Camel RabbitMQ Starter: > > org.apache.camel > > camel-rabbitmq-starter > > 2.25.2 > > > > Configured only from applications.properties using: > > camel.component.rabbitmq.hostname=... > > camel.component.rabbitmq.port-number=... > > camel.component.rabbitmq.username=... > > camel.component.rabbitmq.password=... > > > > Using WinWS for Windows Service > > WinSW 2.2.0.0 (I know there is a newer version 2.10.3 available, but I > > doubt this is the problem) > > > > Cheers > > Michael > > > > -- > Claus Ibsen > ----------------- > http://davsclaus.com @davsclaus > Camel in Action 2: https://www.manning.com/ibsen2
