[
https://issues.apache.org/jira/browse/AMQ-8049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17695011#comment-17695011
]
Lukasz Raczko edited comment on AMQ-8049 at 3/1/23 11:23 AM:
-------------------------------------------------------------
We had similar issue when using filtered kahadb with 'catch all' perdestination
like:
{code:java}
<persistenceAdapter>
<mKahaDB directory="${activemq.data}/kahadb">
<filteredPersistenceAdapters>
<filteredKahaDB queue="XXX.>">
<persistenceAdapter>
<kahaDB checkForCorruptJournalFiles="true"
indexCacheSize="10000"
journalMaxFileLength="32mb"
lockKeepAlivePeriod="2000"
preallocationScope="entire_journal"
preallocationStrategy="os_kernel_copy">
<locker>
<shared-file-locker failIfLocked="false"
lockAcquireSleepInterval="1000"/>
</locker>
</kahaDB>
</persistenceAdapter>
</filteredKahaDB>
<filteredKahaDB topic="XXX.>">
<persistenceAdapter>
<kahaDB checkForCorruptJournalFiles="true"
indexCacheSize="1000"
indexWriteBatchSize="10"
journalMaxFileLength="10kb"
lockKeepAlivePeriod="2000"
preallocationScope="entire_journal"
preallocationStrategy="os_kernel_copy"
journalMaxWriteBatchSize="100mb"
journalDiskSyncStrategy="never">
</kahaDB>
</persistenceAdapter>
</filteredKahaDB>
<!-- kahaDB per destinations -->
<filteredKahaDB perDestination="true">
<persistenceAdapter>
<kahaDB checkForCorruptJournalFiles="true"
preallocationScope="entire_journal"
preallocationStrategy="os_kernel_copy"
indexCacheSize="10000"
journalMaxFileLength="32mb"
lockKeepAlivePeriod="2000">
<locker>
<shared-file-locker failIfLocked="false"
lockAcquireSleepInterval="1000"/>
</locker>
</kahaDB>
</persistenceAdapter>
</filteredKahaDB>
</filteredPersistenceAdapters>
</mKahaDB>
</persistenceAdapter>
{code}
In our case it was problem when restarting ActiveMQ. Right now code assumes
that all directories are connected to perDestination filter and starts them. If
there are destinations in metadata (what I belive could be true almost always
for example in case of topics for STOMP). As a result it will start adapter for
topic directory and then try to start another one for configured adapter.
I have created proposal of fix.
was (Author: JIRAUSER299032):
We had similar issue when using filtered kahadb with 'catch all' perdestination
like:
{code:java}
<persistenceAdapter>
<mKahaDB directory="${activemq.data}/kahadb">
<filteredPersistenceAdapters>
<filteredKahaDB queue="XXX.>">
<persistenceAdapter>
<kahaDB checkForCorruptJournalFiles="true"
indexCacheSize="10000"
journalMaxFileLength="32mb"
lockKeepAlivePeriod="2000"
preallocationScope="entire_journal"
preallocationStrategy="os_kernel_copy">
<locker>
<shared-file-locker failIfLocked="false"
lockAcquireSleepInterval="1000"/>
</locker>
</kahaDB>
</persistenceAdapter>
</filteredKahaDB>
<filteredKahaDB topic="XXX.>">
<persistenceAdapter>
<kahaDB checkForCorruptJournalFiles="true"
indexCacheSize="1000"
indexWriteBatchSize="10"
journalMaxFileLength="10kb"
lockKeepAlivePeriod="2000"
preallocationScope="entire_journal"
preallocationStrategy="os_kernel_copy"
journalMaxWriteBatchSize="100mb"
journalDiskSyncStrategy="never">
</kahaDB>
</persistenceAdapter>
</filteredKahaDB>
<!-- kahaDB per destinations -->
<filteredKahaDB perDestination="true">
<persistenceAdapter>
<kahaDB checkForCorruptJournalFiles="true"
preallocationScope="entire_journal"
preallocationStrategy="os_kernel_copy"
indexCacheSize="10000"
journalMaxFileLength="32mb"
lockKeepAlivePeriod="2000">
<locker>
<shared-file-locker failIfLocked="false"
lockAcquireSleepInterval="1000"/>
</locker>
</kahaDB>
</persistenceAdapter>
</filteredKahaDB>
</filteredPersistenceAdapters>
</mKahaDB>
</persistenceAdapter>
{code}
In our case it was problem when restarting ActiveMQ. Right now code assumes
that all directories are connected to perDestination filter and starts them. If
there are destinations in metadata (what could be true almost always for
example in case of topics for STOMP). As a result it will start adapter for
topic directory and then try to start another one for configured adapter.
I have created proposal of fix.
> Failed to start Apache ActiveMQ (mKahaDB / JMX)
> -----------------------------------------------
>
> Key: AMQ-8049
> URL: https://issues.apache.org/jira/browse/AMQ-8049
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.15.12
> Environment: - Ubuntu 18.04.4 LTS (x86_64, server)
> - OpenJDK 11.0.7 (64-Bit Server VM)
> - ActiveMQ 5.15.12 (vanilla; tar.gz)
> Reporter: Guillermo Grandes
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Summary:
> {noformat}
> 2020-10-06 15:02:03,889 | ERROR | Failed to start Apache ActiveMQ (localhost,
> null) | org.apache.activemq.broker.BrokerService | main
> javax.management.InstanceAlreadyExistsException:
> org.apache.activemq:type=Broker,brokerName=localhost,service=PersistenceAdapter,instanceName=KahaDBPersistenceAdapter[/opt/activemq/data/kahadb/queue#3a#2f#2fActiveMQ.#3e_Index_/opt/activemq/data/kahadb/queue#3a#2f#2fActiveMQ.#3e]
> {noformat}
> This is the full-log (start-to-fail-and-shutdown):
> {code:none}
> 2020-10-06 15:01:58,507 | INFO | Refreshing
> org.apache.activemq.xbean.XBeanBrokerFactory$1@56620197: startup date [Tue
> Oct 06 15:01:58 CEST 2020]; root of context hierarchy |
> org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
> 2020-10-06 15:02:02,417 | INFO | Using Persistence Adapter:
> MultiKahaDBPersistenceAdapter[/opt/activemq/data/kahadb][KahaDBPersistenceAdapter[/opt/activemq/data/kahadb/queue#3a#2f#2fActiveMQ.#3e],
>
> KahaDBPersistenceAdapter[/opt/activemq/data/kahadb/topic#3a#2f#2fActiveMQ.#3e]]
> | org.apache.activemq.broker.BrokerService | main
> 2020-10-06 15:02:02,638 | INFO | KahaDB is version 6 |
> org.apache.activemq.store.kahadb.MessageDatabase | main
> 2020-10-06 15:02:02,823 | INFO | KahaDB is version 6 |
> org.apache.activemq.store.kahadb.MessageDatabase | main
> 2020-10-06 15:02:02,978 | INFO | KahaDB is version 6 |
> org.apache.activemq.store.kahadb.MessageDatabase | main
> 2020-10-06 15:02:03,007 | INFO | Stopping async queue tasks |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:03,010 | INFO | Stopping async topic tasks |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:03,095 | INFO | Stopped KahaDB |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:03,257 | INFO | KahaDB is version 6 |
> org.apache.activemq.store.kahadb.MessageDatabase | main
> 2020-10-06 15:02:03,480 | INFO | KahaDB is version 6 |
> org.apache.activemq.store.kahadb.MessageDatabase | main
> 2020-10-06 15:02:03,575 | INFO | KahaDB is version 6 |
> org.apache.activemq.store.kahadb.MessageDatabase | main
> 2020-10-06 15:02:03,681 | INFO | KahaDB is version 6 |
> org.apache.activemq.store.kahadb.MessageDatabase | main
> 2020-10-06 15:02:03,798 | INFO | KahaDB is version 6 |
> org.apache.activemq.store.kahadb.MessageDatabase | main
> 2020-10-06 15:02:03,851 | INFO | KahaDB is version 6 |
> org.apache.activemq.store.kahadb.MessageDatabase | main
> 2020-10-06 15:02:03,859 | INFO | Page File:
> /opt/activemq/data/kahadb/queue#3a#2f#2fActiveMQ.#3e/db.data. Recovering
> pageFile free list due to prior unclean shutdown.. |
> org.apache.activemq.store.kahadb.disk.page.PageFile | KahaDB Index Free Page
> Recovery
> 2020-10-06 15:02:03,889 | ERROR | Failed to start Apache ActiveMQ (localhost,
> null) | org.apache.activemq.broker.BrokerService | main
> javax.management.InstanceAlreadyExistsException:
> org.apache.activemq:type=Broker,brokerName=localhost,service=PersistenceAdapter,instanceName=KahaDBPersistenceAdapter[/opt/activemq/data/kahadb/queue#3a#2f#2fActiveMQ.#3e_Index_/opt/activemq/data/kahadb/queue#3a#2f#2fActiveMQ.#3e]
> at
> java.management/com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:436)[:]
> at
> java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1855)[:]
> at
> java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:955)[:]
> at
> java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:890)[:]
> at
> java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:320)[:]
> at
> java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)[:]
> at
> org.apache.activemq.broker.jmx.ManagementContext.registerMBean(ManagementContext.java:424)[activemq-broker-5.15.12.jar:5.15.12]
> at
> org.apache.activemq.broker.jmx.AnnotatedMBean.registerMBean(AnnotatedMBean.java:93)[activemq-broker-5.15.12.jar:5.15.12]
> at
> org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.doStart(KahaDBPersistenceAdapter.java:251)[activemq-kahadb-store-5.15.12.jar:5.15.12]
> at
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)[activemq-client-5.15.12.jar:5.15.12]
> at
> org.apache.activemq.store.kahadb.MultiKahaDBPersistenceAdapter.doStart(MultiKahaDBPersistenceAdapter.java:381)[activemq-kahadb-store-5.15.12.jar:5.15.12]
> at
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)[activemq-client-5.15.12.jar:5.15.12]
> at
> org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:693)[activemq-broker-5.15.12.jar:5.15.12]
> at
> org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:677)[activemq-broker-5.15.12.jar:5.15.12]
> at
> org.apache.activemq.broker.BrokerService.start(BrokerService.java:641)[activemq-broker-5.15.12.jar:5.15.12]
> at
> org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:73)[activemq-spring-5.15.12.jar:5.15.12]
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)[:]
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:]
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:]
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)[:]
> ...cut...
> 2020-10-06 15:02:03,949 | INFO | Page File:
> /opt/activemq/data/kahadb/queue#3a#2f#2fActiveMQ.#3e/db.data. Recovered
> pageFile free list of size: 0 |
> org.apache.activemq.store.kahadb.disk.page.PageFile | KahaDB Index Free Page
> Recovery
> 2020-10-06 15:02:03,956 | INFO | Apache ActiveMQ 5.15.12 (localhost, null)
> is shutting down | org.apache.activemq.broker.BrokerService | main
> 2020-10-06 15:02:03,958 | INFO | Connector openwire stopped |
> org.apache.activemq.broker.TransportConnector | main
> 2020-10-06 15:02:03,967 | INFO |
> PListStore:[/opt/activemq/data/localhost/tmp_storage] stopped |
> org.apache.activemq.store.kahadb.plist.PListStoreImpl | main
> 2020-10-06 15:02:03,976 | INFO | Stopping async queue tasks |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:03,977 | INFO | Stopping async topic tasks |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:03,978 | INFO | Stopped KahaDB |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:04,034 | INFO | Stopping async queue tasks |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:04,038 | INFO | Stopping async topic tasks |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:04,039 | INFO | Stopped KahaDB |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:04,042 | INFO | Stopping async queue tasks |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:04,043 | INFO | Stopping async topic tasks |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:04,044 | INFO | Stopped KahaDB |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:04,061 | INFO | Stopping async queue tasks |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:04,063 | INFO | Stopping async topic tasks |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:04,065 | INFO | Stopped KahaDB |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:04,084 | INFO | Stopping async queue tasks |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:04,085 | INFO | Stopping async topic tasks |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:04,086 | INFO | Stopped KahaDB |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:04,110 | INFO | Stopping async queue tasks |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:04,113 | INFO | Stopping async topic tasks |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:04,114 | INFO | Stopped KahaDB |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:04,134 | INFO | Stopping async queue tasks |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:04,136 | INFO | Stopping async topic tasks |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:04,137 | INFO | Stopped KahaDB |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:04,163 | INFO | Stopping async queue tasks |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:04,165 | INFO | Stopping async topic tasks |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:04,168 | INFO | Stopped KahaDB |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:04,192 | INFO | Stopping async queue tasks |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:04,193 | INFO | Stopping async topic tasks |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:04,194 | INFO | Stopped KahaDB |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2020-10-06 15:02:04,232 | INFO | Apache ActiveMQ 5.15.12 (localhost, null)
> uptime 1.904 seconds | org.apache.activemq.broker.BrokerService | main
> 2020-10-06 15:02:04,235 | INFO | Apache ActiveMQ 5.15.12 (localhost, null)
> is shutdown | org.apache.activemq.broker.BrokerService | main
> 2020-10-06 15:02:04,238 | INFO | Closing
> org.apache.activemq.xbean.XBeanBrokerFactory$1@56620197: startup date [Tue
> Oct 06 15:01:58 CEST 2020]; root of context hierarchy |
> org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
> 2020-10-06 15:02:04,248 | WARN | Exception encountered during context
> initialization - cancelling refresh attempt:
> org.springframework.beans.factory.BeanCreationException: Error creating bean
> with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class
> path resource [activemq.xml]: Invocation of init method failed; nested
> exception is javax.management.InstanceAlreadyExistsException:
> org.apache.activemq:type=Broker,brokerName=localhost,service=PersistenceAdapter,instanceName=KahaDBPersistenceAdapter[/opt/activemq/data/kahadb/queue#3a#2f#2fActiveMQ.#3e_Index_/opt/activemq/data/kahadb/queue#3a#2f#2fActiveMQ.#3e]
> | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
> {code}
> This is the Config:
> {code:xml}
> <beans
> xmlns="http://www.springframework.org/schema/beans";
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
> xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd
> http://activemq.apache.org/schema/core
> http://activemq.apache.org/schema/core/activemq-core.xsd";>
> <bean
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> <property name="locations">
> <value>file:${activemq.conf}/credentials.properties</value>
> </property>
> </bean>
> <bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
> lazy-init="false" scope="singleton"
> init-method="start" destroy-method="stop">
> </bean>
> <broker xmlns="http://activemq.apache.org/schema/core";
> brokerName="localhost" schedulePeriodForDestinationPurge="10000"
> dataDirectory="${activemq.data}">
> <plugins>
> <discardingDLQBrokerPlugin dropAll="true"
> dropTemporaryQueues="true" dropTemporaryTopics="true"/>
> <statisticsBrokerPlugin/>
> </plugins>
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry gcInactiveDestinations="true"
> inactiveTimoutBeforeGC="600000" topic=">">
> <pendingMessageLimitStrategy>
> <constantPendingMessageLimitStrategy limit="1000"/>
> </pendingMessageLimitStrategy>
> </policyEntry>
> <policyEntry gcInactiveDestinations="true"
> inactiveTimoutBeforeGC="600000" queue=">"/>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
> <managementContext>
> <managementContext createConnector="false"/>
> </managementContext>
> <!-- Use multiple kahaDB stores -->
> <persistenceAdapter>
> <mKahaDB directory="${activemq.data}/kahadb">
> <filteredPersistenceAdapters>
> <!-- match internal queues -->
> <filteredKahaDB queue="ActiveMQ.>">
> <persistenceAdapter>
> <kahaDB journalDiskSyncStrategy="periodic"/>
> </persistenceAdapter>
> </filteredKahaDB>
> <filteredKahaDB topic="ActiveMQ.>">
> <persistenceAdapter>
> <kahaDB journalDiskSyncStrategy="periodic"/>
> </persistenceAdapter>
> </filteredKahaDB>
> <!-- kahaDB per destinations -->
> <filteredKahaDB perDestination="true">
> <persistenceAdapter>
> <kahaDB journalDiskSyncStrategy="periodic"
> concurrentStoreAndDispatchQueues="false"/>
> </persistenceAdapter>
> </filteredKahaDB>
> </filteredPersistenceAdapters>
> </mKahaDB>
> </persistenceAdapter>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage percentOfJvmHeap="70" />
> </memoryUsage>
> </systemUsage>
> </systemUsage>
> <transportConnectors>
> <transportConnector name="openwire"
> uri="tcp://127.0.0.1:61616?maximumConnections=1000&wireFormat.maxFrameSize=1048576"/>
> </transportConnectors>
> <shutdownHooks>
> <bean xmlns="http://www.springframework.org/schema/beans";
> class="org.apache.activemq.hooks.SpringContextHook" />
> </shutdownHooks>
> </broker>
> <import resource="jetty.xml"/>
> </beans>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
