Is there any chance that a JAR from a newer version of ActiveMQ ended up on the broker's classpath? If you delete the contents of the lib directory and replace it with the contents from a freshly installed copy of 5.7.0, does the behavior change?
Also, can you share your activemq.xml with us? And does this behavior change if you use the copy of activemq.xml from a fresh install? Tim On Sun, Nov 3, 2019, 2:35 PM Jonathan Roby <[email protected]> wrote: > Hi > > > I tried putting the kahadb directory on the local directory, it > regenerated on startup but still gave me a BadMagicException > > ________________________________ > From: Jean-Baptiste Onofré <[email protected]> > Sent: Sunday, November 3, 2019 11:07:22 AM > To: [email protected] > Subject: Re: org.apache.activemq.kaha.impl.index.? > > Hi Jonathan, > > user mailing list is fine for this kind of question. > > IMHO, I'm pretty sure it's related to the filesystem. > > As Tim suggested, maybe you can try to put the kahadb directory on local > filesystem instead of the SAN. It could be an interesting test. > > Regards > JB > > On 03/11/2019 16:52, Jonathan Roby wrote: > > Hi > > > > > > Also, i'm sorry for the subject heading of my original email, I'm not > sure what happened but its somewhat nonsensical. > > > > > > Also, would anyone recommend I post this question to the developer > mailing list, in case they might have other ideas? > > > > > > I've asked our IT infrastructure team if they can check the VMs > storage/machine layout for recent changes that could have caused this. > hopefully t hey find something or I get a lucky break > > > > > > Thank you everyone for your help so far > > > > Jon > > > > ________________________________ > > From: Jonathan Roby > > Sent: Sunday, November 3, 2019 9:02:56 AM > > To: ActiveMQ Users > > Subject: Re: org.apache.activemq.kaha.impl.index.? > > > > > > Hi > > > > > > I just checked and the directory is created exactly when the process > tries to start. I've also checked and the ports used by the broker are not > visible in the output of the netstat command. > > > > > > I also tried moving the kahadb directory onto the local drive and still > get the BadMagicException at the end of nested exception stacktrace > > > > > > Thank you > > > > Jon > > > > ________________________________ > > From: Tim Bain <[email protected]> > > Sent: Sunday, November 3, 2019 8:06:29 AM > > To: ActiveMQ Users > > Subject: Re: org.apache.activemq.kaha.impl.index.? > > > > Following up on JB's line of questioning, is the directory only recreated > > when (exactly when, to the second, with a matching timestamp) this broker > > process is started? Might it be possible that a different broker process, > > running elsewhere, might be creating the directory and its files? > > > > If you reconfigure this broker to store the KahaDB data file on local > disk > > instead of your SAN (just temporarily), does the process start > > successfully? > > > > Tim > > > > On Sat, Nov 2, 2019, 1:56 PM Jonathan Roby <[email protected]> > > wrote: > > > >> > >> No, its not fixed. I wiped out the data folder and the persistant > folder. > >> both were regenerated but still is failing > >> > >> > >> its weird as we have other instances set up exactly the same as this one > >> that dont have this error. same versions, same everything > >> > >> > >> so it sounds like you might be correct about it being the filesystem. i > >> hope not as that can be tricky to debug > >> > >> > >> Thank you for your help > >> > >> Jon > >> > >> ________________________________ > >> From: Jean-Baptiste Onofré <[email protected]> > >> Sent: Saturday, November 2, 2019 12:48:29 PM > >> To: [email protected] > >> Subject: Re: org.apache.activemq.kaha.impl.index.? > >> > >> I guess it's fixed with you complete remove the kahadb folder right ? > >> > >> I remember couple of users mentioned similar issue on virtual machine. > >> > >> Did you try ActiveMQ 5.15.10 ? We did a bunch of fixes and improvements > >> on this version around kahadb. > >> > >> Regards > >> JB > >> > >> On 02/11/2019 18:46, Jean-Baptiste Onofré wrote: > >>> Hi, > >>> > >>> It's definitely something weird while reading file on the filesystem. > >>> > >>> Is it a local filesystem or shared one (NFS, CIFS, ...) ? > >>> > >>> Regards > >>> JB > >>> > >>> On 02/11/2019 18:18, Jonathan Roby wrote: > >>>> Hi > >>>> > >>>> > >>>> as far as I know, the version of ActiveMQ hasn't been updated and i've > >> now deleted the kahadb directory a few times (it regenerated) and even > >> moved the broker path in case it was a drive error > >>>> > >>>> > >>>> Jon > >>>> > >>>> ________________________________ > >>>> From: Jean-Baptiste Onofré <[email protected]> > >>>> Sent: Saturday, November 2, 2019 12:16:31 PM > >>>> To: [email protected] > >>>> Subject: Re: org.apache.activemq.kaha.impl.index.? > >>>> > >>>> Hi, > >>>> > >>>> Does the kahadb folder has been created with the same ActiveMQ version > >>>> (and you updated ActiveMQ version later) ? > >>>> > >>>> It seems the kahadb index is not consistent. > >>>> > >>>> Regards > >>>> JB > >>>> > >>>> On 02/11/2019 18:14, Jonathan Roby wrote: > >>>>> Hi, i've just joined the list so apologies if i've not this > correctly. > >>>>> > >>>>> > >>>>> We run a number of islandora instances that use ActiveMQ 5.7.0 for > >> message passing between components. Yesterday, one of our instances > seemed > >> to stop working. after several restarts of components and the system > >> itself, I noticed that when the components restarted, it always > produced a > >> nested exception and the very last error was > >> org.apache.activemq.kaha.impl.index.BadMagicException. > >>>>> > >>>>> > >>>>> I've googled this and taken the advice given, which is usually to > >> empty/delete the data directory or the kahadb directory and it still > isnt > >> working. All the other components seem to be working fine. > >>>>> > >>>>> > >>>>> Thank you for any help anyone can give me. > >>>>> > >>>>> Jon > >>>>> > >>>>> > >>>>> Here is the last part of the stack trace. > >>>>> > >>>>> > >>>>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: > null > >>>>> at > >> org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> org.apache.activemq.broker.BrokerService.start(BrokerService.java:578) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> ~[na:1.8.0_162] > >>>>> at > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >> ~[na:1.8.0_162] > >>>>> at > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> ~[na:1.8.0_162] > >>>>> at java.lang.reflect.Method.invoke(Method.java:498) > ~[na:1.8.0_162] > >>>>> at > >> > org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340) > >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE] > >>>>> at > >> > org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293) > >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE] > >>>>> at > >> > org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130) > >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE] > >>>>> > >>>> > >>>> -- > >>>> Jean-Baptiste Onofré > >>>> [email protected] > >>>> http://blog.nanthrax.net > >>>> Talend - http://www.talend.com > >> Talend - A Cloud Data Integration Leader (modern ETL)< > >> http://www.talend.com/> > >> www.talend.com<http://www.talend.com> > >> Talend Data Fabric offers a single suite of cloud apps for data > >> integration and data integrity to help enterprises collect, govern, > >> transform, and share data. > >> > >> > >>>> > >>> > >> > >> -- > >> Jean-Baptiste Onofré > >> [email protected] > >> http://blog.nanthrax.net > >> Talend - http://www.talend.com > >> Talend - A Cloud Data Integration Leader (modern ETL)< > >> http://www.talend.com/> > >> www.talend.com<http://www.talend.com> > >> Talend Data Fabric offers a single suite of cloud apps for data > >> integration and data integrity to help enterprises collect, govern, > >> transform, and share data. > >> > >> > >> > > > > -- > Jean-Baptiste Onofré > [email protected] > http://blog.nanthrax.net > Talend - http://www.talend.com >
