[
https://issues.apache.org/jira/browse/AMQ-9504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17849076#comment-17849076
]
ritesh adval edited comment on AMQ-9504 at 5/23/24 6:42 PM:
------------------------------------------------------------
ok thx for the update. For us its a high priority issue, because the
functionality of multi kahadb is broken as soon as the broker is restarted
(with configuration we have in this issue).
Moreover it seems it would in fact causes DATA CORRUPTION because there are two
KahaDBPersistenceAdapter running on same directory when you turn off jmx. The
jmx=true exhibits the issue where we get an exception to register the same
mbean twice, but if we set jmx to false. then it silently would start two
KahaDBPersistenceAdapter and test will not fail.
if you set the jmx to false you will see below (this the log from test after we
restart the broker), note it goes into PageFile recovery after restart, there
will not be any exception thrown in the test but two KahaDBPersistenceAdapter
are allowed to run on same directory which is not good.
Also I think it would good to log a line in KahaDBPersistenceAdapter when it
starts on a directory... currently its very hard to know from the logs, how
many of these are configured and running on what directory.
2024-05-23 11:31:04,297 [main ] - INFO BrokerService
- Using Persistence Adapter:
MultiKahaDBPersistenceAdapter[/mnt/c/Users/ritesh.adval/work/external-repos/activemq/activemq-unit-tests/target/activemq-data/mKahaDB][KahaDBPersistenceAdapter[/mnt/c/Users/ritesh.adval/work/external-repos/activemq/activemq-unit-tests/target/activemq-data/mKahaDB/topic#3a#2f#2f#3e]|#3a#2f#2f#3e]]
2024-05-23 11:31:04,298 [main ] - INFO BrokerService
- Starting Persistence Adapter:
MultiKahaDBPersistenceAdapter[/mnt/c/Users/ritesh.adval/work/external-repos/activemq/activemq-unit-tests/target/activemq-data/mKahaDB][KahaDBPersistenceAdapter[/mnt/c/Users/ritesh.adval/work/external-repos/activemq/activemq-unit-tests/target/activemq-data/mKahaDB/topic#3a#2f#2f#3e]|#3a#2f#2f#3e]]
2024-05-23 11:31:04,331 [main ] - INFO KahaDBStore
- Starting KahaDBStore
2024-05-23 11:31:04,335 [main ] - INFO MessageDatabase
- Opening MessageDatabase
2024-05-23 11:31:04,413 [main ] - INFO MessageDatabase
- KahaDB is version 7
2024-05-23 11:31:04,551 [main ] - INFO KahaDBStore
- Starting KahaDBStore
2024-05-23 11:31:04,561 [main ] - INFO MessageDatabase
- Opening MessageDatabase
2024-05-23 11:31:04,619 [main ] - INFO MessageDatabase
- KahaDB is version 7
2024-05-23 11:31:04,631 [main ] - INFO KahaDBStore
- Starting KahaDBStore
2024-05-23 11:31:04,633 [main ] - INFO MessageDatabase
- Opening MessageDatabase
*2024-05-23 11:31:04,699 [e Page Recovery] - INFO PageFile
- Page File: target/activemq-data/mKahaDB/topic#3a#2f#2f#3e/db.data.
Recovering pageFile free list due to prior unclean shutdown..*
*2024-05-23 11:31:04,703 [main ] - INFO MessageDatabase
- KahaDB is version 7*
*2024-05-23 11:31:04,726 [e Page Recovery] - INFO PageFile
- Page File: target/activemq-data/mKahaDB/topic#3a#2f#2f#3e/db.data.
Recovered pageFile free list of size: 0*
*2024-05-23 11:31:04,730 [main ] - INFO MessageDatabase
- Recovering from the journal @1:622*
*2024-05-23 11:31:04,738 [main ] - INFO MessageDatabase
- Recovery replayed 1 operations from the journal in 0.018 seconds.*
2024-05-23 11:31:04,740 [main ] - INFO BrokerService
- Starting Temp Data Store
2024-05-23 11:31:04,742 [main ] - INFO PListStoreImpl
-
PListStore:[/mnt/c/Users/ritesh.adval/work/external-repos/activemq/activemq-unit-tests/target/activemq-data/localhost/tmp_storage]
started
2024-05-23 11:31:04,743 [main ] - INFO BrokerService
- Starting Job Scheduler Store
2024-05-23 11:31:04,744 [main ] - INFO BrokerService
- Persistence Adapter successfully started
was (Author: JIRAUSER305516):
ok thx for the update. For us its a high priority issue, because the
functionality of multi kahadb is broken as soon as the broker is restarted
(with configuration we have in this issue).
Moreover it seems it would in fact causes DATA CORRUPTION because there are two
KahaDBPersistenceAdapter running on same directory when you turn off jmx. The
jmx=true exhibits the issue where we get an exception to register the same
mbean twice, but if we set jmx to false. then it silently would start two
KahaDBPersistenceAdapter and test will not fail.
if you set the jmx to false you will see below (this the log from test after we
restart the broker), note it goes into PageFile recovery after restart, there
will not be any exception thrown in the test but two KahaDBPersistenceAdapter
are allowed to run on same directory which is not good.
Also I think it would good log a line in KahaDBPersistenceAdapter when it
starts on a directory... currently its very hard to know from logs how many of
these are configured and running.
2024-05-23 11:31:04,297 [main ] - INFO BrokerService
- Using Persistence Adapter:
MultiKahaDBPersistenceAdapter[/mnt/c/Users/ritesh.adval/work/external-repos/activemq/activemq-unit-tests/target/activemq-data/mKahaDB][KahaDBPersistenceAdapter[/mnt/c/Users/ritesh.adval/work/external-repos/activemq/activemq-unit-tests/target/activemq-data/mKahaDB/topic#3a#2f#2f#3e]]
2024-05-23 11:31:04,298 [main ] - INFO BrokerService
- Starting Persistence Adapter:
MultiKahaDBPersistenceAdapter[/mnt/c/Users/ritesh.adval/work/external-repos/activemq/activemq-unit-tests/target/activemq-data/mKahaDB][KahaDBPersistenceAdapter[/mnt/c/Users/ritesh.adval/work/external-repos/activemq/activemq-unit-tests/target/activemq-data/mKahaDB/topic#3a#2f#2f#3e]]
2024-05-23 11:31:04,331 [main ] - INFO KahaDBStore
- Starting KahaDBStore
2024-05-23 11:31:04,335 [main ] - INFO MessageDatabase
- Opening MessageDatabase
2024-05-23 11:31:04,413 [main ] - INFO MessageDatabase
- KahaDB is version 7
2024-05-23 11:31:04,551 [main ] - INFO KahaDBStore
- Starting KahaDBStore
2024-05-23 11:31:04,561 [main ] - INFO MessageDatabase
- Opening MessageDatabase
2024-05-23 11:31:04,619 [main ] - INFO MessageDatabase
- KahaDB is version 7
2024-05-23 11:31:04,631 [main ] - INFO KahaDBStore
- Starting KahaDBStore
2024-05-23 11:31:04,633 [main ] - INFO MessageDatabase
- Opening MessageDatabase
*2024-05-23 11:31:04,699 [e Page Recovery] - INFO PageFile
- Page File: target/activemq-data/mKahaDB/topic#3a#2f#2f#3e/db.data.
Recovering pageFile free list due to prior unclean shutdown..*
*2024-05-23 11:31:04,703 [main ] - INFO MessageDatabase
- KahaDB is version 7*
*2024-05-23 11:31:04,726 [e Page Recovery] - INFO PageFile
- Page File: target/activemq-data/mKahaDB/topic#3a#2f#2f#3e/db.data.
Recovered pageFile free list of size: 0*
*2024-05-23 11:31:04,730 [main ] - INFO MessageDatabase
- Recovering from the journal @1:622*
*2024-05-23 11:31:04,738 [main ] - INFO MessageDatabase
- Recovery replayed 1 operations from the journal in 0.018 seconds.*
2024-05-23 11:31:04,740 [main ] - INFO BrokerService
- Starting Temp Data Store
2024-05-23 11:31:04,742 [main ] - INFO PListStoreImpl
-
PListStore:[/mnt/c/Users/ritesh.adval/work/external-repos/activemq/activemq-unit-tests/target/activemq-data/localhost/tmp_storage]
started
2024-05-23 11:31:04,743 [main ] - INFO BrokerService
- Starting Job Scheduler Store
2024-05-23 11:31:04,744 [main ] - INFO BrokerService
- Persistence Adapter successfully started
> activemq multikahadb persistence adapter with topic wildcard filtered adapter
> and per destination filtered adapter causes broker failure on restart
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: AMQ-9504
> URL: https://issues.apache.org/jira/browse/AMQ-9504
> Project: ActiveMQ Classic
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.18.4, 6.1.2
> Reporter: ritesh adval
> Assignee: Christopher L. Shannon
> Priority: Major
> Fix For: 6.2.0, 5.18.5, 6.1.3
>
> Attachments: bugfix.patch, test.patch
>
>
> when using Multikahadb persistence adapter per documentation :
> [https://activemq.apache.org/components/classic/documentation/kahadb] it
> shows that you can use multiple filteredPersistenceAdapters but this does not
> work if you have two filtered adapter where one is using wildcard match for
> topics (or even a specific topic) and second filtered adapter using per
> destination filtered adapter.
> The idea being you want to use one kahadb instance for all the topics and per
> destination kahadb instance for all other destinations like queues. Something
> like this for illustration of the issue see test for more details. (note jmx
> needs to be enabled) :
> {code:java}
> <broker useJmx="true" brokerName="broker">
> <persistenceAdapter>
> <mKahaDB directory="${activemq.base}/data/kahadb">
> <filteredPersistenceAdapters>
> <!-- match all topics-->
> <filteredKahaDB topic=">">
> <usage>
> <storeUsage limit="1g" />
> </usage>
> <persistenceAdapter>
> <kahaDB journalMaxFileLength="32mb"/>
> </persistenceAdapter>
> </filteredKahaDB>
> <!-- match all destinations kahaDB per destinations -->
> <filteredKahaDB perDestination="true">
> <persistenceAdapter>
> <kahaDB journalMaxFileLength="32mb"/>
> </persistenceAdapter>
> </filteredKahaDB>
> </filteredPersistenceAdapters>
> </mKahaDB>
> </persistenceAdapter>
> </broker> {code}
> With this setting it works for the first time when broker is started. But as
> soon as you have atleast one topic created which uses wild card filtered
> adapter and you restart the broker, then what happens is there are two
> KahaDBPersistenceAdapter created one by the wildcard (">") topic filtered
> adapter and another one by the second per destination filtered adapter, and
> so second KahaDBPersistenceAdapter fails with below exception:
>
> [INFO] Running org.apache.activemq.bugs.MultiKahaDBMultipleFilteredAdapterTest
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 16.20
> s <<< FAILURE! – in
> org.apache.activemq.bugs.MultiKahaDBMultipleFilteredAdapterTest
> [ERROR]
> org.apache.activemq.bugs.MultiKahaDBMultipleFilteredAdapterTest.testTopicWildcardAndPerDestinationFilteredAdapter
> – Time elapsed: 11.08 s <<< ERROR!
> javax.management.InstanceAlreadyExistsException:
> org.apache.activemq:type=Broker,brokerName=localhost,service=PersistenceAdapter,instanceName=KahaDBPersistenceAdapter[/mnt/c/Users/ritesh.adval/work/external-repos/activemq/activemq-unit-tests/target/activemq-data/mKahaDB/topic#3a#2f#2f#3e_Index_/mnt/c/Users/ritesh.adval/work/external-repos/activemq/activemq-unit-tests/target/activemq-data/mKahaDB/topic#3a#2f#2f#3e|#3a#2f#2f#3e_Index_/mnt/c/Users/ritesh.adval/work/external-repos/activemq/activemq-unit-tests/target/activemq-data/mKahaDB/topic#3a#2f#2f#3e]
> at
> java.management/com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:436)
> at
> java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1855)
> at
> java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:955)
> at
> java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:890)
> at
> java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:320)
> at
> java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
> at
> org.apache.activemq.broker.jmx.ManagementContext.registerMBean(ManagementContext.java:415)
> at
> org.apache.activemq.broker.jmx.AnnotatedMBean.registerMBean(AnnotatedMBean.java:93)
> at
> org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.doStart(KahaDBPersistenceAdapter.java:251)
> at
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
> at
> org.apache.activemq.store.kahadb.MultiKahaDBPersistenceAdapter.doStart(MultiKahaDBPersistenceAdapter.java:381)
> at
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
> at
> org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:681)
> at
> org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:663)
> at
> org.apache.activemq.broker.BrokerService.start(BrokerService.java:627)
> at
> org.apache.activemq.bugs.MultiKahaDBMultipleFilteredAdapterTest.testTopicWildcardAndPerDestinationFilteredAdapter(MultiKahaDBMultipleFilteredAdapterTest.java:76)
>
> I have unit test for this and have a fix as well, attaching them with the
> issue (its for 5.18.4 release and I believe issue exists in latest 6.x
> release as well).
>
> the bug is that on restart/start of activemq, wildcard filtered adapter adds
> one KahaDBPersistenceAdapter when calling
> MultiKahaDBPersistenceAdapter.setFilteredPersistenceAdapters, and on the call
> to doStart of MultiKahaDBPersistenceAdapter, it uses per destination filtered
> adapter to add another KahaDBPersistenceAdapter for the same topic, and fails
> on jmx bean registration. The fix is to make sure to check if a
> KahaDBPersistenceAdapter already exists for given destination dir, when we do
> per destination filtered adapter processing and only add it if one does not
> exists already.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
