Great, I'm glad you got it sorted out. Thanks for closing the loop and letting us know.
Tim On Tue, Nov 5, 2019, 8:27 AM Jonathan Roby <[email protected]> wrote: > Hi All > > > just an update: I figured out the issue. It was due to the software we > use (fedora-commons/tomcat) using its own set of configuration files and > data directories. So I was trying to debug the wrong set of config files > and data directories, thats why nothing seemed to work. And in the end, it > was just deleting the data directory. > > > Thanks Jean-Baptiste and Tim for your help > > Jon > > ________________________________ > From: Tim Bain <[email protected]> > Sent: Tuesday, November 5, 2019 7:34:10 AM > To: ActiveMQ Users > Subject: Re: org.apache.activemq.kaha.impl.index.? > > Is there any chance that a JAR from a newer version of ActiveMQ ended up on > the broker's classpath? If you delete the contents of the lib directory and > replace it with the contents from a freshly installed copy of 5.7.0, does > the behavior change? > > Also, can you share your activemq.xml with us? And does this behavior > change if you use the copy of activemq.xml from a fresh install? > > Tim > > On Sun, Nov 3, 2019, 2:35 PM Jonathan Roby <[email protected]> > wrote: > > > Hi > > > > > > I tried putting the kahadb directory on the local directory, it > > regenerated on startup but still gave me a BadMagicException > > > > ________________________________ > > From: Jean-Baptiste Onofré <[email protected]> > > Sent: Sunday, November 3, 2019 11:07:22 AM > > To: [email protected] > > Subject: Re: org.apache.activemq.kaha.impl.index.? > > > > Hi Jonathan, > > > > user mailing list is fine for this kind of question. > > > > IMHO, I'm pretty sure it's related to the filesystem. > > > > As Tim suggested, maybe you can try to put the kahadb directory on local > > filesystem instead of the SAN. It could be an interesting test. > > > > Regards > > JB > > > > On 03/11/2019 16:52, Jonathan Roby wrote: > > > Hi > > > > > > > > > Also, i'm sorry for the subject heading of my original email, I'm not > > sure what happened but its somewhat nonsensical. > > > > > > > > > Also, would anyone recommend I post this question to the developer > > mailing list, in case they might have other ideas? > > > > > > > > > I've asked our IT infrastructure team if they can check the VMs > > storage/machine layout for recent changes that could have caused this. > > hopefully t hey find something or I get a lucky break > > > > > > > > > Thank you everyone for your help so far > > > > > > Jon > > > > > > ________________________________ > > > From: Jonathan Roby > > > Sent: Sunday, November 3, 2019 9:02:56 AM > > > To: ActiveMQ Users > > > Subject: Re: org.apache.activemq.kaha.impl.index.? > > > > > > > > > Hi > > > > > > > > > I just checked and the directory is created exactly when the process > > tries to start. I've also checked and the ports used by the broker are > not > > visible in the output of the netstat command. > > > > > > > > > I also tried moving the kahadb directory onto the local drive and still > > get the BadMagicException at the end of nested exception stacktrace > > > > > > > > > Thank you > > > > > > Jon > > > > > > ________________________________ > > > From: Tim Bain <[email protected]> > > > Sent: Sunday, November 3, 2019 8:06:29 AM > > > To: ActiveMQ Users > > > Subject: Re: org.apache.activemq.kaha.impl.index.? > > > > > > Following up on JB's line of questioning, is the directory only > recreated > > > when (exactly when, to the second, with a matching timestamp) this > broker > > > process is started? Might it be possible that a different broker > process, > > > running elsewhere, might be creating the directory and its files? > > > > > > If you reconfigure this broker to store the KahaDB data file on local > > disk > > > instead of your SAN (just temporarily), does the process start > > > successfully? > > > > > > Tim > > > > > > On Sat, Nov 2, 2019, 1:56 PM Jonathan Roby <[email protected] > > > > > wrote: > > > > > >> > > >> No, its not fixed. I wiped out the data folder and the persistant > > folder. > > >> both were regenerated but still is failing > > >> > > >> > > >> its weird as we have other instances set up exactly the same as this > one > > >> that dont have this error. same versions, same everything > > >> > > >> > > >> so it sounds like you might be correct about it being the filesystem. > i > > >> hope not as that can be tricky to debug > > >> > > >> > > >> Thank you for your help > > >> > > >> Jon > > >> > > >> ________________________________ > > >> From: Jean-Baptiste Onofré <[email protected]> > > >> Sent: Saturday, November 2, 2019 12:48:29 PM > > >> To: [email protected] > > >> Subject: Re: org.apache.activemq.kaha.impl.index.? > > >> > > >> I guess it's fixed with you complete remove the kahadb folder right ? > > >> > > >> I remember couple of users mentioned similar issue on virtual machine. > > >> > > >> Did you try ActiveMQ 5.15.10 ? We did a bunch of fixes and > improvements > > >> on this version around kahadb. > > >> > > >> Regards > > >> JB > > >> > > >> On 02/11/2019 18:46, Jean-Baptiste Onofré wrote: > > >>> Hi, > > >>> > > >>> It's definitely something weird while reading file on the filesystem. > > >>> > > >>> Is it a local filesystem or shared one (NFS, CIFS, ...) ? > > >>> > > >>> Regards > > >>> JB > > >>> > > >>> On 02/11/2019 18:18, Jonathan Roby wrote: > > >>>> Hi > > >>>> > > >>>> > > >>>> as far as I know, the version of ActiveMQ hasn't been updated and > i've > > >> now deleted the kahadb directory a few times (it regenerated) and even > > >> moved the broker path in case it was a drive error > > >>>> > > >>>> > > >>>> Jon > > >>>> > > >>>> ________________________________ > > >>>> From: Jean-Baptiste Onofré <[email protected]> > > >>>> Sent: Saturday, November 2, 2019 12:16:31 PM > > >>>> To: [email protected] > > >>>> Subject: Re: org.apache.activemq.kaha.impl.index.? > > >>>> > > >>>> Hi, > > >>>> > > >>>> Does the kahadb folder has been created with the same ActiveMQ > version > > >>>> (and you updated ActiveMQ version later) ? > > >>>> > > >>>> It seems the kahadb index is not consistent. > > >>>> > > >>>> Regards > > >>>> JB > > >>>> > > >>>> On 02/11/2019 18:14, Jonathan Roby wrote: > > >>>>> Hi, i've just joined the list so apologies if i've not this > > correctly. > > >>>>> > > >>>>> > > >>>>> We run a number of islandora instances that use ActiveMQ 5.7.0 for > > >> message passing between components. Yesterday, one of our instances > > seemed > > >> to stop working. after several restarts of components and the system > > >> itself, I noticed that when the components restarted, it always > > produced a > > >> nested exception and the very last error was > > >> org.apache.activemq.kaha.impl.index.BadMagicException. > > >>>>> > > >>>>> > > >>>>> I've googled this and taken the advice given, which is usually to > > >> empty/delete the data directory or the kahadb directory and it still > > isnt > > >> working. All the other components seem to be working fine. > > >>>>> > > >>>>> > > >>>>> Thank you for any help anyone can give me. > > >>>>> > > >>>>> Jon > > >>>>> > > >>>>> > > >>>>> Here is the last part of the stack trace. > > >>>>> > > >>>>> > > >>>>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: > > null > > >>>>> at > > >> org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> > > > org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> > > > org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> > > > org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> > > > org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> > > > org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> > > > org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> > > > org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> > > > org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> > > > org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> > > > org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> > > > org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> > > > org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> > > > org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> > > > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> > > > org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> > > > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> > > > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> > > > org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> > > > org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> > > > org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> > > > org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> > > > org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> > > org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> > > > org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> > > > org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> org.apache.activemq.broker.BrokerService.start(BrokerService.java:578) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at > > >> > > > org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539) > > >> ~[activemq-core-5.7.0.jar:5.7.0] > > >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > >> ~[na:1.8.0_162] > > >>>>> at > > >> > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > >> ~[na:1.8.0_162] > > >>>>> at > > >> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > >> ~[na:1.8.0_162] > > >>>>> at java.lang.reflect.Method.invoke(Method.java:498) > > ~[na:1.8.0_162] > > >>>>> at > > >> > > > org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340) > > >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE] > > >>>>> at > > >> > > > org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293) > > >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE] > > >>>>> at > > >> > > > org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130) > > >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE] > > >>>>> > > >>>> > > >>>> -- > > >>>> Jean-Baptiste Onofré > > >>>> [email protected] > > >>>> http://blog.nanthrax.net > > >>>> Talend - http://www.talend.com > > >> Talend - A Cloud Data Integration Leader (modern ETL)< > > >> http://www.talend.com/> > > >> www.talend.com<http://www.talend.com> > > >> Talend Data Fabric offers a single suite of cloud apps for data > > >> integration and data integrity to help enterprises collect, govern, > > >> transform, and share data. > > >> > > >> > > >>>> > > >>> > > >> > > >> -- > > >> Jean-Baptiste Onofré > > >> [email protected] > > >> http://blog.nanthrax.net > > >> Talend - http://www.talend.com > > >> Talend - A Cloud Data Integration Leader (modern ETL)< > > >> http://www.talend.com/> > > >> www.talend.com<http://www.talend.com> > > >> Talend Data Fabric offers a single suite of cloud apps for data > > >> integration and data integrity to help enterprises collect, govern, > > >> transform, and share data. > > >> > > >> > > >> > > > > > > > -- > > Jean-Baptiste Onofré > > [email protected] > > http://blog.nanthrax.net > > Talend - http://www.talend.com > > >
