Hi Jonathan, user mailing list is fine for this kind of question.
IMHO, I'm pretty sure it's related to the filesystem. As Tim suggested, maybe you can try to put the kahadb directory on local filesystem instead of the SAN. It could be an interesting test. Regards JB On 03/11/2019 16:52, Jonathan Roby wrote: > Hi > > > Also, i'm sorry for the subject heading of my original email, I'm not sure > what happened but its somewhat nonsensical. > > > Also, would anyone recommend I post this question to the developer mailing > list, in case they might have other ideas? > > > I've asked our IT infrastructure team if they can check the VMs > storage/machine layout for recent changes that could have caused this. > hopefully t hey find something or I get a lucky break > > > Thank you everyone for your help so far > > Jon > > ________________________________ > From: Jonathan Roby > Sent: Sunday, November 3, 2019 9:02:56 AM > To: ActiveMQ Users > Subject: Re: org.apache.activemq.kaha.impl.index.? > > > Hi > > > I just checked and the directory is created exactly when the process tries to > start. I've also checked and the ports used by the broker are not visible in > the output of the netstat command. > > > I also tried moving the kahadb directory onto the local drive and still get > the BadMagicException at the end of nested exception stacktrace > > > Thank you > > Jon > > ________________________________ > From: Tim Bain <[email protected]> > Sent: Sunday, November 3, 2019 8:06:29 AM > To: ActiveMQ Users > Subject: Re: org.apache.activemq.kaha.impl.index.? > > Following up on JB's line of questioning, is the directory only recreated > when (exactly when, to the second, with a matching timestamp) this broker > process is started? Might it be possible that a different broker process, > running elsewhere, might be creating the directory and its files? > > If you reconfigure this broker to store the KahaDB data file on local disk > instead of your SAN (just temporarily), does the process start > successfully? > > Tim > > On Sat, Nov 2, 2019, 1:56 PM Jonathan Roby <[email protected]> > wrote: > >> >> No, its not fixed. I wiped out the data folder and the persistant folder. >> both were regenerated but still is failing >> >> >> its weird as we have other instances set up exactly the same as this one >> that dont have this error. same versions, same everything >> >> >> so it sounds like you might be correct about it being the filesystem. i >> hope not as that can be tricky to debug >> >> >> Thank you for your help >> >> Jon >> >> ________________________________ >> From: Jean-Baptiste Onofré <[email protected]> >> Sent: Saturday, November 2, 2019 12:48:29 PM >> To: [email protected] >> Subject: Re: org.apache.activemq.kaha.impl.index.? >> >> I guess it's fixed with you complete remove the kahadb folder right ? >> >> I remember couple of users mentioned similar issue on virtual machine. >> >> Did you try ActiveMQ 5.15.10 ? We did a bunch of fixes and improvements >> on this version around kahadb. >> >> Regards >> JB >> >> On 02/11/2019 18:46, Jean-Baptiste Onofré wrote: >>> Hi, >>> >>> It's definitely something weird while reading file on the filesystem. >>> >>> Is it a local filesystem or shared one (NFS, CIFS, ...) ? >>> >>> Regards >>> JB >>> >>> On 02/11/2019 18:18, Jonathan Roby wrote: >>>> Hi >>>> >>>> >>>> as far as I know, the version of ActiveMQ hasn't been updated and i've >> now deleted the kahadb directory a few times (it regenerated) and even >> moved the broker path in case it was a drive error >>>> >>>> >>>> Jon >>>> >>>> ________________________________ >>>> From: Jean-Baptiste Onofré <[email protected]> >>>> Sent: Saturday, November 2, 2019 12:16:31 PM >>>> To: [email protected] >>>> Subject: Re: org.apache.activemq.kaha.impl.index.? >>>> >>>> Hi, >>>> >>>> Does the kahadb folder has been created with the same ActiveMQ version >>>> (and you updated ActiveMQ version later) ? >>>> >>>> It seems the kahadb index is not consistent. >>>> >>>> Regards >>>> JB >>>> >>>> On 02/11/2019 18:14, Jonathan Roby wrote: >>>>> Hi, i've just joined the list so apologies if i've not this correctly. >>>>> >>>>> >>>>> We run a number of islandora instances that use ActiveMQ 5.7.0 for >> message passing between components. Yesterday, one of our instances seemed >> to stop working. after several restarts of components and the system >> itself, I noticed that when the components restarted, it always produced a >> nested exception and the very last error was >> org.apache.activemq.kaha.impl.index.BadMagicException. >>>>> >>>>> >>>>> I've googled this and taken the advice given, which is usually to >> empty/delete the data directory or the kahadb directory and it still isnt >> working. All the other components seem to be working fine. >>>>> >>>>> >>>>> Thank you for any help anyone can give me. >>>>> >>>>> Jon >>>>> >>>>> >>>>> Here is the last part of the stack trace. >>>>> >>>>> >>>>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: null >>>>> at >> org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.broker.BrokerService.start(BrokerService.java:578) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at >> org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539) >> ~[activemq-core-5.7.0.jar:5.7.0] >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> ~[na:1.8.0_162] >>>>> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> ~[na:1.8.0_162] >>>>> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> ~[na:1.8.0_162] >>>>> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162] >>>>> at >> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340) >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE] >>>>> at >> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293) >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE] >>>>> at >> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130) >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE] >>>>> >>>> >>>> -- >>>> Jean-Baptiste Onofré >>>> [email protected] >>>> http://blog.nanthrax.net >>>> Talend - http://www.talend.com >> Talend - A Cloud Data Integration Leader (modern ETL)< >> http://www.talend.com/> >> www.talend.com<http://www.talend.com> >> Talend Data Fabric offers a single suite of cloud apps for data >> integration and data integrity to help enterprises collect, govern, >> transform, and share data. >> >> >>>> >>> >> >> -- >> Jean-Baptiste Onofré >> [email protected] >> http://blog.nanthrax.net >> Talend - http://www.talend.com >> Talend - A Cloud Data Integration Leader (modern ETL)< >> http://www.talend.com/> >> www.talend.com<http://www.talend.com> >> Talend Data Fabric offers a single suite of cloud apps for data >> integration and data integrity to help enterprises collect, govern, >> transform, and share data. >> >> >> > -- Jean-Baptiste Onofré [email protected] http://blog.nanthrax.net Talend - http://www.talend.com
