Hi All
just an update: I figured out the issue. It was due to the software we use (fedora-commons/tomcat) using its own set of configuration files and data directories. So I was trying to debug the wrong set of config files and data directories, thats why nothing seemed to work. And in the end, it was just deleting the data directory. Thanks Jean-Baptiste and Tim for your help Jon ________________________________ From: Tim Bain <[email protected]> Sent: Tuesday, November 5, 2019 7:34:10 AM To: ActiveMQ Users Subject: Re: org.apache.activemq.kaha.impl.index.? Is there any chance that a JAR from a newer version of ActiveMQ ended up on the broker's classpath? If you delete the contents of the lib directory and replace it with the contents from a freshly installed copy of 5.7.0, does the behavior change? Also, can you share your activemq.xml with us? And does this behavior change if you use the copy of activemq.xml from a fresh install? Tim On Sun, Nov 3, 2019, 2:35 PM Jonathan Roby <[email protected]> wrote: > Hi > > > I tried putting the kahadb directory on the local directory, it > regenerated on startup but still gave me a BadMagicException > > ________________________________ > From: Jean-Baptiste Onofré <[email protected]> > Sent: Sunday, November 3, 2019 11:07:22 AM > To: [email protected] > Subject: Re: org.apache.activemq.kaha.impl.index.? > > Hi Jonathan, > > user mailing list is fine for this kind of question. > > IMHO, I'm pretty sure it's related to the filesystem. > > As Tim suggested, maybe you can try to put the kahadb directory on local > filesystem instead of the SAN. It could be an interesting test. > > Regards > JB > > On 03/11/2019 16:52, Jonathan Roby wrote: > > Hi > > > > > > Also, i'm sorry for the subject heading of my original email, I'm not > sure what happened but its somewhat nonsensical. > > > > > > Also, would anyone recommend I post this question to the developer > mailing list, in case they might have other ideas? > > > > > > I've asked our IT infrastructure team if they can check the VMs > storage/machine layout for recent changes that could have caused this. > hopefully t hey find something or I get a lucky break > > > > > > Thank you everyone for your help so far > > > > Jon > > > > ________________________________ > > From: Jonathan Roby > > Sent: Sunday, November 3, 2019 9:02:56 AM > > To: ActiveMQ Users > > Subject: Re: org.apache.activemq.kaha.impl.index.? > > > > > > Hi > > > > > > I just checked and the directory is created exactly when the process > tries to start. I've also checked and the ports used by the broker are not > visible in the output of the netstat command. > > > > > > I also tried moving the kahadb directory onto the local drive and still > get the BadMagicException at the end of nested exception stacktrace > > > > > > Thank you > > > > Jon > > > > ________________________________ > > From: Tim Bain <[email protected]> > > Sent: Sunday, November 3, 2019 8:06:29 AM > > To: ActiveMQ Users > > Subject: Re: org.apache.activemq.kaha.impl.index.? > > > > Following up on JB's line of questioning, is the directory only recreated > > when (exactly when, to the second, with a matching timestamp) this broker > > process is started? Might it be possible that a different broker process, > > running elsewhere, might be creating the directory and its files? > > > > If you reconfigure this broker to store the KahaDB data file on local > disk > > instead of your SAN (just temporarily), does the process start > > successfully? > > > > Tim > > > > On Sat, Nov 2, 2019, 1:56 PM Jonathan Roby <[email protected]> > > wrote: > > > >> > >> No, its not fixed. I wiped out the data folder and the persistant > folder. > >> both were regenerated but still is failing > >> > >> > >> its weird as we have other instances set up exactly the same as this one > >> that dont have this error. same versions, same everything > >> > >> > >> so it sounds like you might be correct about it being the filesystem. i > >> hope not as that can be tricky to debug > >> > >> > >> Thank you for your help > >> > >> Jon > >> > >> ________________________________ > >> From: Jean-Baptiste Onofré <[email protected]> > >> Sent: Saturday, November 2, 2019 12:48:29 PM > >> To: [email protected] > >> Subject: Re: org.apache.activemq.kaha.impl.index.? > >> > >> I guess it's fixed with you complete remove the kahadb folder right ? > >> > >> I remember couple of users mentioned similar issue on virtual machine. > >> > >> Did you try ActiveMQ 5.15.10 ? We did a bunch of fixes and improvements > >> on this version around kahadb. > >> > >> Regards > >> JB > >> > >> On 02/11/2019 18:46, Jean-Baptiste Onofré wrote: > >>> Hi, > >>> > >>> It's definitely something weird while reading file on the filesystem. > >>> > >>> Is it a local filesystem or shared one (NFS, CIFS, ...) ? > >>> > >>> Regards > >>> JB > >>> > >>> On 02/11/2019 18:18, Jonathan Roby wrote: > >>>> Hi > >>>> > >>>> > >>>> as far as I know, the version of ActiveMQ hasn't been updated and i've > >> now deleted the kahadb directory a few times (it regenerated) and even > >> moved the broker path in case it was a drive error > >>>> > >>>> > >>>> Jon > >>>> > >>>> ________________________________ > >>>> From: Jean-Baptiste Onofré <[email protected]> > >>>> Sent: Saturday, November 2, 2019 12:16:31 PM > >>>> To: [email protected] > >>>> Subject: Re: org.apache.activemq.kaha.impl.index.? > >>>> > >>>> Hi, > >>>> > >>>> Does the kahadb folder has been created with the same ActiveMQ version > >>>> (and you updated ActiveMQ version later) ? > >>>> > >>>> It seems the kahadb index is not consistent. > >>>> > >>>> Regards > >>>> JB > >>>> > >>>> On 02/11/2019 18:14, Jonathan Roby wrote: > >>>>> Hi, i've just joined the list so apologies if i've not this > correctly. > >>>>> > >>>>> > >>>>> We run a number of islandora instances that use ActiveMQ 5.7.0 for > >> message passing between components. Yesterday, one of our instances > seemed > >> to stop working. after several restarts of components and the system > >> itself, I noticed that when the components restarted, it always > produced a > >> nested exception and the very last error was > >> org.apache.activemq.kaha.impl.index.BadMagicException. > >>>>> > >>>>> > >>>>> I've googled this and taken the advice given, which is usually to > >> empty/delete the data directory or the kahadb directory and it still > isnt > >> working. All the other components seem to be working fine. > >>>>> > >>>>> > >>>>> Thank you for any help anyone can give me. > >>>>> > >>>>> Jon > >>>>> > >>>>> > >>>>> Here is the last part of the stack trace. > >>>>> > >>>>> > >>>>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: > null > >>>>> at > >> org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> org.apache.activemq.broker.BrokerService.start(BrokerService.java:578) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at > >> > org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539) > >> ~[activemq-core-5.7.0.jar:5.7.0] > >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> ~[na:1.8.0_162] > >>>>> at > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >> ~[na:1.8.0_162] > >>>>> at > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> ~[na:1.8.0_162] > >>>>> at java.lang.reflect.Method.invoke(Method.java:498) > ~[na:1.8.0_162] > >>>>> at > >> > org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340) > >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE] > >>>>> at > >> > org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293) > >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE] > >>>>> at > >> > org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130) > >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE] > >>>>> > >>>> > >>>> -- > >>>> Jean-Baptiste Onofré > >>>> [email protected] > >>>> http://blog.nanthrax.net > >>>> Talend - http://www.talend.com > >> Talend - A Cloud Data Integration Leader (modern ETL)< > >> http://www.talend.com/> > >> www.talend.com<http://www.talend.com> > >> Talend Data Fabric offers a single suite of cloud apps for data > >> integration and data integrity to help enterprises collect, govern, > >> transform, and share data. > >> > >> > >>>> > >>> > >> > >> -- > >> Jean-Baptiste Onofré > >> [email protected] > >> http://blog.nanthrax.net > >> Talend - http://www.talend.com > >> Talend - A Cloud Data Integration Leader (modern ETL)< > >> http://www.talend.com/> > >> www.talend.com<http://www.talend.com> > >> Talend Data Fabric offers a single suite of cloud apps for data > >> integration and data integrity to help enterprises collect, govern, > >> transform, and share data. > >> > >> > >> > > > > -- > Jean-Baptiste Onofré > [email protected] > http://blog.nanthrax.net > Talend - http://www.talend.com >
