Thanks for the update ! Great you figured it out and always happy to help.
Regards JB On 05/11/2019 16:27, Jonathan Roby wrote: > Hi All > > > just an update: I figured out the issue. It was due to the software we use > (fedora-commons/tomcat) using its own set of configuration files and data > directories. So I was trying to debug the wrong set of config files and data > directories, thats why nothing seemed to work. And in the end, it was just > deleting the data directory. > > > Thanks Jean-Baptiste and Tim for your help > > Jon > > ________________________________ > From: Tim Bain <[email protected]> > Sent: Tuesday, November 5, 2019 7:34:10 AM > To: ActiveMQ Users > Subject: Re: org.apache.activemq.kaha.impl.index.? > > Is there any chance that a JAR from a newer version of ActiveMQ ended up on > the broker's classpath? If you delete the contents of the lib directory and > replace it with the contents from a freshly installed copy of 5.7.0, does > the behavior change? > > Also, can you share your activemq.xml with us? And does this behavior > change if you use the copy of activemq.xml from a fresh install? > > Tim > > On Sun, Nov 3, 2019, 2:35 PM Jonathan Roby <[email protected]> > wrote: > >> Hi >> >> >> I tried putting the kahadb directory on the local directory, it >> regenerated on startup but still gave me a BadMagicException >> >> ________________________________ >> From: Jean-Baptiste Onofré <[email protected]> >> Sent: Sunday, November 3, 2019 11:07:22 AM >> To: [email protected] >> Subject: Re: org.apache.activemq.kaha.impl.index.? >> >> Hi Jonathan, >> >> user mailing list is fine for this kind of question. >> >> IMHO, I'm pretty sure it's related to the filesystem. >> >> As Tim suggested, maybe you can try to put the kahadb directory on local >> filesystem instead of the SAN. It could be an interesting test. >> >> Regards >> JB >> >> On 03/11/2019 16:52, Jonathan Roby wrote: >>> Hi >>> >>> >>> Also, i'm sorry for the subject heading of my original email, I'm not >> sure what happened but its somewhat nonsensical. >>> >>> >>> Also, would anyone recommend I post this question to the developer >> mailing list, in case they might have other ideas? >>> >>> >>> I've asked our IT infrastructure team if they can check the VMs >> storage/machine layout for recent changes that could have caused this. >> hopefully t hey find something or I get a lucky break >>> >>> >>> Thank you everyone for your help so far >>> >>> Jon >>> >>> ________________________________ >>> From: Jonathan Roby >>> Sent: Sunday, November 3, 2019 9:02:56 AM >>> To: ActiveMQ Users >>> Subject: Re: org.apache.activemq.kaha.impl.index.? >>> >>> >>> Hi >>> >>> >>> I just checked and the directory is created exactly when the process >> tries to start. I've also checked and the ports used by the broker are not >> visible in the output of the netstat command. >>> >>> >>> I also tried moving the kahadb directory onto the local drive and still >> get the BadMagicException at the end of nested exception stacktrace >>> >>> >>> Thank you >>> >>> Jon >>> >>> ________________________________ >>> From: Tim Bain <[email protected]> >>> Sent: Sunday, November 3, 2019 8:06:29 AM >>> To: ActiveMQ Users >>> Subject: Re: org.apache.activemq.kaha.impl.index.? >>> >>> Following up on JB's line of questioning, is the directory only recreated >>> when (exactly when, to the second, with a matching timestamp) this broker >>> process is started? Might it be possible that a different broker process, >>> running elsewhere, might be creating the directory and its files? >>> >>> If you reconfigure this broker to store the KahaDB data file on local >> disk >>> instead of your SAN (just temporarily), does the process start >>> successfully? >>> >>> Tim >>> >>> On Sat, Nov 2, 2019, 1:56 PM Jonathan Roby <[email protected]> >>> wrote: >>> >>>> >>>> No, its not fixed. I wiped out the data folder and the persistant >> folder. >>>> both were regenerated but still is failing >>>> >>>> >>>> its weird as we have other instances set up exactly the same as this one >>>> that dont have this error. same versions, same everything >>>> >>>> >>>> so it sounds like you might be correct about it being the filesystem. i >>>> hope not as that can be tricky to debug >>>> >>>> >>>> Thank you for your help >>>> >>>> Jon >>>> >>>> ________________________________ >>>> From: Jean-Baptiste Onofré <[email protected]> >>>> Sent: Saturday, November 2, 2019 12:48:29 PM >>>> To: [email protected] >>>> Subject: Re: org.apache.activemq.kaha.impl.index.? >>>> >>>> I guess it's fixed with you complete remove the kahadb folder right ? >>>> >>>> I remember couple of users mentioned similar issue on virtual machine. >>>> >>>> Did you try ActiveMQ 5.15.10 ? We did a bunch of fixes and improvements >>>> on this version around kahadb. >>>> >>>> Regards >>>> JB >>>> >>>> On 02/11/2019 18:46, Jean-Baptiste Onofré wrote: >>>>> Hi, >>>>> >>>>> It's definitely something weird while reading file on the filesystem. >>>>> >>>>> Is it a local filesystem or shared one (NFS, CIFS, ...) ? >>>>> >>>>> Regards >>>>> JB >>>>> >>>>> On 02/11/2019 18:18, Jonathan Roby wrote: >>>>>> Hi >>>>>> >>>>>> >>>>>> as far as I know, the version of ActiveMQ hasn't been updated and i've >>>> now deleted the kahadb directory a few times (it regenerated) and even >>>> moved the broker path in case it was a drive error >>>>>> >>>>>> >>>>>> Jon >>>>>> >>>>>> ________________________________ >>>>>> From: Jean-Baptiste Onofré <[email protected]> >>>>>> Sent: Saturday, November 2, 2019 12:16:31 PM >>>>>> To: [email protected] >>>>>> Subject: Re: org.apache.activemq.kaha.impl.index.? >>>>>> >>>>>> Hi, >>>>>> >>>>>> Does the kahadb folder has been created with the same ActiveMQ version >>>>>> (and you updated ActiveMQ version later) ? >>>>>> >>>>>> It seems the kahadb index is not consistent. >>>>>> >>>>>> Regards >>>>>> JB >>>>>> >>>>>> On 02/11/2019 18:14, Jonathan Roby wrote: >>>>>>> Hi, i've just joined the list so apologies if i've not this >> correctly. >>>>>>> >>>>>>> >>>>>>> We run a number of islandora instances that use ActiveMQ 5.7.0 for >>>> message passing between components. Yesterday, one of our instances >> seemed >>>> to stop working. after several restarts of components and the system >>>> itself, I noticed that when the components restarted, it always >> produced a >>>> nested exception and the very last error was >>>> org.apache.activemq.kaha.impl.index.BadMagicException. >>>>>>> >>>>>>> >>>>>>> I've googled this and taken the advice given, which is usually to >>>> empty/delete the data directory or the kahadb directory and it still >> isnt >>>> working. All the other components seem to be working fine. >>>>>>> >>>>>>> >>>>>>> Thank you for any help anyone can give me. >>>>>>> >>>>>>> Jon >>>>>>> >>>>>>> >>>>>>> Here is the last part of the stack trace. >>>>>>> >>>>>>> >>>>>>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: >> null >>>>>>> at >>>> org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> >> org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> >> org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> >> org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> >> org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> >> org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> >> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> >> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> >> org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> >> org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> >> org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> >> org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> >> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> >> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> >> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> >> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> >> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> >> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> >> org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> >> org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> >> org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> >> org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> >> org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> >> org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> >> org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> >> org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> org.apache.activemq.broker.BrokerService.start(BrokerService.java:578) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at >>>> >> org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539) >>>> ~[activemq-core-5.7.0.jar:5.7.0] >>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> ~[na:1.8.0_162] >>>>>>> at >>>> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>> ~[na:1.8.0_162] >>>>>>> at >>>> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> ~[na:1.8.0_162] >>>>>>> at java.lang.reflect.Method.invoke(Method.java:498) >> ~[na:1.8.0_162] >>>>>>> at >>>> >> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340) >>>> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE] >>>>>>> at >>>> >> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293) >>>> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE] >>>>>>> at >>>> >> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130) >>>> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE] >>>>>>> >>>>>> >>>>>> -- >>>>>> Jean-Baptiste Onofré >>>>>> [email protected] >>>>>> http://blog.nanthrax.net >>>>>> Talend - http://www.talend.com >>>> Talend - A Cloud Data Integration Leader (modern ETL)< >>>> http://www.talend.com/> >>>> www.talend.com<http://www.talend.com> >>>> Talend Data Fabric offers a single suite of cloud apps for data >>>> integration and data integrity to help enterprises collect, govern, >>>> transform, and share data. >>>> >>>> >>>>>> >>>>> >>>> >>>> -- >>>> Jean-Baptiste Onofré >>>> [email protected] >>>> http://blog.nanthrax.net >>>> Talend - http://www.talend.com >>>> Talend - A Cloud Data Integration Leader (modern ETL)< >>>> http://www.talend.com/> >>>> www.talend.com<http://www.talend.com> >>>> Talend Data Fabric offers a single suite of cloud apps for data >>>> integration and data integrity to help enterprises collect, govern, >>>> transform, and share data. >>>> >>>> >>>> >>> >> >> -- >> Jean-Baptiste Onofré >> [email protected] >> http://blog.nanthrax.net >> Talend - http://www.talend.com >> > -- Jean-Baptiste Onofré [email protected] http://blog.nanthrax.net Talend - http://www.talend.com
