[
https://issues.apache.org/jira/browse/AMQ-8432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17460680#comment-17460680
]
Daniel Mensinger commented on AMQ-8432:
---------------------------------------
I am not sure what could trigger this on our side. We don't connect to our
broker with JMX and there is nothing (at least in theory) that could terminate
our broker otherwise. Is there some way to figure out the exact cause of it
shutting down?
> Broker shuts down randomly after some time
> ------------------------------------------
>
> Key: AMQ-8432
> URL: https://issues.apache.org/jira/browse/AMQ-8432
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.16.3
> Reporter: Daniel Mensinger
> Assignee: Jean-Baptiste Onofré
> Priority: Major
> Attachments: activemq.log
>
>
> Our ActiveMQ broker shuts down after a certain amount of time (days) of
> operation for no apparent reason:
>
> {code:java}
> 2021-12-09 15:36:56,221 | DEBUG | WriteChecker: 10017ms elapsed since last
> write check. | org.apache.activemq.transport.AbstractInactivityMonitor |
> ActiveMQ InactivityMonitor WriteCheckTimer
> 2021-12-09 15:36:56,227 | DEBUG | Running WriteCheck[tcp://127.0.0.1:33638] |
> org.apache.activemq.transport.AbstractInactivityMonitor | ActiveMQ
> InactivityMonitor Worker
> 2021-12-09 15:36:56,216 | DEBUG | Running WriteCheck[tcp://127.0.0.1:56489] |
> org.apache.activemq.transport.AbstractInactivityMonitor | ActiveMQ
> InactivityMonitor Worker
> 2021-12-09 15:36:56,254 | DEBUG | WriteChecker: 10000ms elapsed since last
> write check. | org.apache.activemq.transport.AbstractInactivityMonitor |
> ActiveMQ InactivityMonitor WriteCheckTimer
> 2021-12-09 15:36:56,221 | DEBUG | Running WriteCheck[tcp://127.0.0.1:53948] |
> org.apache.activemq.transport.AbstractInactivityMonitor | ActiveMQ
> InactivityMonitor Worker
> 2021-12-09 15:36:56,264 | DEBUG | WriteChecker: 10004ms elapsed since last
> write check. | org.apache.activemq.transport.AbstractInactivityMonitor |
> ActiveMQ InactivityMonitor WriteCheckTimer
> 2021-12-09 15:36:56,273 | DEBUG | Running WriteCheck[tcp://127.0.0.1:37383] |
> org.apache.activemq.transport.AbstractInactivityMonitor | ActiveMQ
> InactivityMonitor Worker
> 2021-12-09 15:36:56,284 | DEBUG | WriteChecker: 10000ms elapsed since last
> write check. | org.apache.activemq.transport.AbstractInactivityMonitor |
> ActiveMQ InactivityMonitor WriteCheckTimer
> 2021-12-09 15:36:56,299 | DEBUG | Running WriteCheck[tcp://127.0.0.1:45952] |
> org.apache.activemq.transport.AbstractInactivityMonitor | ActiveMQ
> InactivityMonitor Worker
> 2021-12-09 15:36:56,302 | DEBUG | Running WriteCheck[tcp://127.0.0.1:58722] |
> org.apache.activemq.transport.AbstractInactivityMonitor | ActiveMQ
> InactivityMonitor Worker
> 2021-12-09 15:36:56,302 | DEBUG | WriteChecker: 10001ms elapsed since last
> write check. | org.apache.activemq.transport.AbstractInactivityMonitor |
> ActiveMQ InactivityMonitor WriteCheckTimer
> 2021-12-09 15:36:56,316 | DEBUG | Running WriteCheck[tcp://127.0.0.1:35706] |
> org.apache.activemq.transport.AbstractInactivityMonitor | ActiveMQ
> InactivityMonitor Worker
> 2021-12-09 15:36:56,328 | DEBUG | WriteChecker: 10006ms elapsed since last
> write check. | org.apache.activemq.transport.AbstractInactivityMonitor |
> ActiveMQ InactivityMonitor WriteCheckTimer
> 2021-12-09 15:36:56,344 | DEBUG | WriteChecker: 10003ms elapsed since last
> write check. | org.apache.activemq.transport.AbstractInactivityMonitor |
> ActiveMQ InactivityMonitor WriteCheckTimer
> 2021-12-09 15:36:56,351 | DEBUG | Running WriteCheck[tcp://127.0.0.1:44245] |
> org.apache.activemq.transport.AbstractInactivityMonitor | ActiveMQ
> InactivityMonitor Worker
> 2021-12-09 15:36:56,344 | DEBUG | Running WriteCheck[tcp://127.0.0.1:48056] |
> org.apache.activemq.transport.AbstractInactivityMonitor | ActiveMQ
> InactivityMonitor Worker
> 2021-12-09 15:36:57,085 | DEBUG | Checkpoint started. |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> 2021-12-09 15:36:57,088 | DEBUG | Checkpoint done. |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> 2021-12-09 15:36:57,718 | INFO | Apache ActiveMQ 5.16.3 (localhost,
> ID:****************:1) is shutting down |
> org.apache.activemq.broker.BrokerService | ActiveMQ ShutdownHook
> 2021-12-09 15:36:57,722 | DEBUG | Caught exception, must be shutting down.
> This exception is ignored. | org.apache.activemq.broker.BrokerService |
> ActiveMQ ShutdownHook
> java.lang.IllegalStateException: Shutdown in progress
> at
> java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)[:1.8.0_201]
> at java.lang.Runtime.removeShutdownHook(Runtime.java:239)[:1.8.0_201]
> at
> org.apache.activemq.broker.BrokerService.removeShutdownHook(BrokerService.java:2576)[activemq-broker-5.16.3.jar:5.16.3]
> at
> org.apache.activemq.broker.BrokerService.stop(BrokerService.java:847)[activemq-broker-5.16.3.jar:5.16.3]
> at
> org.apache.activemq.xbean.XBeanBrokerService.stop(XBeanBrokerService.java:122)[activemq-spring-5.16.3.jar:5.16.3]
> at
> org.apache.activemq.broker.BrokerService.containerShutdown(BrokerService.java:2599)[activemq-broker-5.16.3.jar:5.16.3]
> at
> org.apache.activemq.broker.BrokerService$7.run(BrokerService.java:2566)[activemq-broker-5.16.3.jar:5.16.3]
> 2021-12-09 15:36:57,728 | DEBUG | Unregistering MBean
> org.apache.activemq:type=Broker,brokerName=localhost,connector=clientConnectors,connectorName=openwire
> | org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ ShutdownHook
> 2021-12-09 15:36:57,746 | DEBUG | Stopping connection: tcp://127.0.0.1:46306
> | org.apache.activemq.broker.TransportConnection | ActiveMQ
> BrokerService[localhost] Task-17115
> 2021-12-09 15:36:57,746 | DEBUG | Unregistering MBean
> org.apache.activemq:type=Broker,brokerName=localhost,connector=clientConnectors,connectorName=openwire,connectionViewType=clientId,connectionName=ID_****************-0_1
> | org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ ShutdownHook
> 2021-12-09 15:36:57,863 | DEBUG | Unregistering MBean
> org.apache.activemq:type=Broker,brokerName=localhost,connector=clientConnectors,connectorName=openwire,connectionViewType=remoteAddress,connectionName=tcp_//127.0.0.1_46306
> | org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ ShutdownHook
> 2021-12-09 15:36:57,868 | DEBUG | Stopping transport
> tcp:///127.0.0.1:46306@61616 | org.apache.activemq.transport.tcp.TcpTransport
> | ActiveMQ BrokerService[localhost] Task-17115
> 2021-12-09 15:36:57,918 | DEBUG | Initialized TaskRunnerFactory[ActiveMQ
> Task] using ExecutorService:
> java.util.concurrent.ThreadPoolExecutor@38deec91[Running, pool size = 0,
> active threads = 0, queued tasks = 0, completed tasks = 0] |
> org.apache.activemq.thread.TaskRunnerFactory | ActiveMQ
> BrokerService[localhost] Task-17115
> 2021-12-09 15:36:57,955 | DEBUG | Closed socket
> Socket[addr=/127.0.0.1,port=46306,localport=61616] |
> org.apache.activemq.transport.tcp.TcpTransport | ActiveMQ Task-1
> 2021-12-09 15:36:57,970 | DEBUG | Forcing shutdown of ExecutorService:
> java.util.concurrent.ThreadPoolExecutor@38deec91[Running, pool size = 1,
> active threads = 0, queued tasks = 0, completed tasks = 1] |
> org.apache.activemq.util.ThreadPoolUtils | ActiveMQ BrokerService[localhost]
> Task-17115
> 2021-12-09 15:36:57,975 | DEBUG | Stopped transport: tcp://127.0.0.1:46306 |
> org.apache.activemq.broker.TransportConnection | ActiveMQ
> BrokerService[localhost] Task-17115
> 2021-12-09 15:36:58,086 | DEBUG | Cleaning up connection resources:
> tcp://127.0.0.1:46306 | org.apache.activemq.broker.TransportConnection |
> ActiveMQ BrokerService[localhost] Task-17115
> 2021-12-09 15:36:58,140 | DEBUG | remove connection id: ID:****************:1
> | org.apache.activemq.broker.TransportConnection | ActiveMQ
> BrokerService[localhost] Task-17115
> ......
> ......
> ......
> 2021-12-09 15:37:16,694 | DEBUG | Unregistering MBean
> org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=tup.lisam.distribution
> | org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ ShutdownHook
> 2021-12-09 15:37:16,698 | DEBUG | Unregistering MBean
> org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Topic,destinationName=ActiveMQ.Advisory.MessageDLQd.Queue.tup.lisam.standort2lisam.normal
> | org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ ShutdownHook
> 2021-12-09 15:37:16,703 | DEBUG | Unregistering MBean
> org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Topic,destinationName=ActiveMQ.Advisory.MasterBroker
> | org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ ShutdownHook
> 2021-12-09 15:37:16,708 | INFO |
> PListStore:[/proj/lisam/tmp/activemq-data/localhost/tmp_storage] stopped |
> org.apache.activemq.store.kahadb.plist.PListStoreImpl | ActiveMQ ShutdownHook
> 2021-12-09 15:37:16,711 | INFO | Stopping async queue tasks |
> org.apache.activemq.store.kahadb.KahaDBStore | ActiveMQ ShutdownHook
> 2021-12-09 15:37:16,715 | INFO | Stopping async topic tasks |
> org.apache.activemq.store.kahadb.KahaDBStore | ActiveMQ ShutdownHook
> 2021-12-09 15:37:16,718 | DEBUG | Forcing shutdown of ExecutorService:
> org.apache.activemq.store.kahadb.KahaDBStore$StoreTaskExecutor@66d10a8b[Running,
> pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 3331]
> | org.apache.activemq.util.ThreadPoolUtils | ActiveMQ ShutdownHook
> 2021-12-09 15:37:16,724 | DEBUG | Forcing shutdown of ExecutorService:
> org.apache.activemq.store.kahadb.KahaDBStore$StoreTaskExecutor@11ab93c9[Running,
> pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] |
> org.apache.activemq.util.ThreadPoolUtils | ActiveMQ ShutdownHook
> 2021-12-09 15:37:17,142 | INFO | Stopped KahaDB |
> org.apache.activemq.store.kahadb.KahaDBStore | ActiveMQ ShutdownHook
> 2021-12-09 15:37:17,215 | DEBUG | Checkpoint started. |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ ShutdownHook
> 2021-12-09 15:37:17,218 | DEBUG | Checkpoint done. |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ ShutdownHook
> 2021-12-09 15:37:17,300 | DEBUG | Shutdown of ExecutorService:
> java.util.concurrent.ScheduledThreadPoolExecutor@4b9a60ef[Terminated, pool
> size = 0, active threads = 0, queued tasks = 0, completed tasks = 29621] is
> shutdown: true and terminated: true took: 0.057 seconds. |
> org.apache.activemq.util.ThreadPoolUtils | ActiveMQ ShutdownHook
> 2021-12-09 15:37:17,308 | DEBUG | Shutdown of ExecutorService:
> java.util.concurrent.Executors$DelegatedScheduledExecutorService@219b3c9f is
> shutdown: true and terminated: true took: 0.001 seconds. |
> org.apache.activemq.util.ThreadPoolUtils | ActiveMQ ShutdownHook
> 2021-12-09 15:37:17,315 | DEBUG | Unregistering MBean
> org.apache.activemq:type=Broker,brokerName=localhost,service=PersistenceAdapter,instanceName=KahaDBPersistenceAdapter[/proj/lisam/tmp/activemq-data/kahadb_Index_/proj/lisam/tmp/activemq-data/kahadb]
> | org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ ShutdownHook
> 2021-12-09 15:37:17,321 | DEBUG | Unregistering MBean
> org.apache.activemq:type=Broker,brokerName=localhost,service=Health |
> org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ ShutdownHook
> 2021-12-09 15:37:17,326 | DEBUG | Unregistering MBean
> org.apache.activemq:type=Broker,brokerName=localhost |
> org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ ShutdownHook
> 2021-12-09 15:37:17,533 | DEBUG | Unregistering MBean
> org.apache.activemq:type=Broker,brokerName=localhost,service=Log4JConfiguration
> | org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ ShutdownHook
> 2021-12-09 15:37:18,026 | DEBUG | Shutdown of ExecutorService:
> java.util.concurrent.ThreadPoolExecutor@7331196b[Terminated, pool size = 0,
> active threads = 0, queued tasks = 0, completed tasks = 2156103] is shutdown:
> true and terminated: false took: 0.001 seconds. |
> org.apache.activemq.util.ThreadPoolUtils | ActiveMQ ShutdownHook
> 2021-12-09 15:37:18,035 | DEBUG | Forcing shutdown of ExecutorService:
> java.util.concurrent.ThreadPoolExecutor@6f7a041c[Running, pool size = 1,
> active threads = 0, queued tasks = 0, completed tasks = 1222] |
> org.apache.activemq.util.ThreadPoolUtils | ActiveMQ ShutdownHook
> 2021-12-09 15:37:18,042 | INFO | Apache ActiveMQ 5.16.3 (localhost,
> ID:****************:1) uptime 10 days 6 hours |
> org.apache.activemq.broker.BrokerService | ActiveMQ ShutdownHook
> 2021-12-09 15:37:18,048 | INFO | Apache ActiveMQ 5.16.3 (localhost,
> ID:****************:1) is shutdown | org.apache.activemq.broker.BrokerService
> | ActiveMQ ShutdownHook
> 2021-12-09 15:37:18,056 | INFO | Closing
> org.apache.activemq.xbean.XBeanBrokerFactory$1@22eeefeb: startup date [Mon
> Nov 29 08:45:17 CET 2021]; root of context hierarchy |
> org.apache.activemq.xbean.XBeanBrokerFactory$1 | ActiveMQ ShutdownHook
> 2021-12-09 15:37:18,696 | DEBUG | JspServlet.destroy() |
> org.apache.jasper.servlet.JspServlet | ActiveMQ ShutdownHook
> 2021-12-09 15:37:19,256 | DEBUG | JspServlet.destroy() |
> org.apache.jasper.servlet.JspServlet | ActiveMQ ShutdownHook{code}
>
> Is there something external from our processes that could trigger a shutdown?
> If yes, how can we figure out what this is?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
