Hi All, Recently we have faced an issue where activeMQ process is shutting down and not able to start the process. We are using "master/slave" brokers and kahadb as a message storage.
We have found the "corrupt journal files" error message in activemq.log (Attached the screen shot). We have applied the below changes to ignore the corrupted journal files. <http://activemq.2283324.n4.nabble.com/file/n4698805/corrupt.png> activemq.xml <http://activemq.2283324.n4.nabble.com/file/n4698805/activemq.xml> <persistenceAdapter> <kahaDB directory="/data02/apphome/Broker" journalMaxFileLength="100mb" cleanupInterval="5000" ignoreMissingJournalfiles="true" checkForCorruptJournalFiles="true" checksumJournalFiles="true"/> </persistenceAdapter/> But after that, we are not able to restart the activeMQ process, we found the below exception in activemq.log file. 2015-05-18 11:26:47,900 | INFO | Connector clientconnection stopped | org.apache.activemq.broker.TransportConnector | ActiveMQ ShutdownHook 2015-05-18 11:26:47,937 | INFO | JobSchedulerStore:/data02/apphome/Broker/scheduler stopped | org.apache.activemq.store.kahadb.scheduler.JobSchedulerStoreImpl | ActiveMQ ShutdownHook 2015-05-18 11:26:47,951 | INFO | Broker Plugin org.apache.activemq.broker.util.RedeliveryPlugin stopped | org.apache.activemq.broker.BrokerPluginSupport | ActiveMQ ShutdownHook 2015-05-18 11:26:47,951 | INFO | PListStore:[/data02/apphome/Broker/tmp_storage] stopped | org.apache.activemq.store.kahadb.plist.PListStoreImpl | ActiveMQ ShutdownHook 2015-05-18 11:26:47,952 | INFO | Stopping async queue tasks | org.apache.activemq.store.kahadb.KahaDBStore | ActiveMQ ShutdownHook 2015-05-18 11:26:47,952 | INFO | Stopping async topic tasks | org.apache.activemq.store.kahadb.KahaDBStore | ActiveMQ ShutdownHook 2015-05-18 11:26:47,952 | INFO | Stopped KahaDB | org.apache.activemq.store.kahadb.KahaDBStore | ActiveMQ ShutdownHook 2015-05-18 11:26:48,139 | INFO | Apache ActiveMQ 5.9.0 (Broker, ID:deherwsvlws154.wsint.biz-41364-1431682528061-1:1) uptime 2 days 23 hours | org.apache.activemq.broker.BrokerService | ActiveMQ ShutdownHook 2015-05-18 11:26:48,139 | INFO | Apache ActiveMQ 5.9.0 (Broker, ID:deherwsvlws154.wsint.biz-41364-1431682528061-1:1) is shutdown | org.apache.activemq.broker.BrokerService | ActiveMQ ShutdownHook 2015-05-18 11:27:06,787 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@5999da01: startup date [Mon May 18 11:27:06 CEST 2015]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | WrapperSimpleAppMain 2015-05-18 11:27:08,659 | INFO | PListStore:[/data02/apphome/Broker/tmp_storage] started | org.apache.activemq.store.kahadb.plist.PListStoreImpl | WrapperSimpleAppMain 2015-05-18 11:27:09,364 | INFO | JobSchedulerStore:/data02/apphome/Broker/scheduler started | org.apache.activemq.store.kahadb.scheduler.JobSchedulerStoreImpl | WrapperSimpleAppMain 2015-05-18 11:27:09,365 | INFO | JobScheduler using directory: /data02/apphome/Broker/scheduler | org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain 2015-05-18 11:27:09,508 | INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[/data02/apphome/Broker] | org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain 2015-05-18 11:27:09,548 | INFO | JMX consoles can connect to service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi | org.apache.activemq.broker.jmx.ManagementContext | JMX connector 2015-05-18 11:27:47,926 | INFO | Corrupt journal records found in '/data02/apphome/Broker/db-21259.log' between offsets: 36319655..36319985 | org.apache.activemq.store.kahadb.disk.journal.Journal | WrapperSimpleAppMain 2015-05-18 11:27:49,958 | INFO | Corrupt journal records found in '/data02/apphome/Broker/db-21260.log' between offsets: 97434148..97606183 | org.apache.activemq.store.kahadb.disk.journal.Journal | WrapperSimpleAppMain 2015-05-18 11:27:50,477 | INFO | Corrupt journal records found in '/data02/apphome/Broker/db-21261.log' between offsets: 37403738..37533983 | org.apache.activemq.store.kahadb.disk.journal.Journal | WrapperSimpleAppMain 2015-05-18 11:28:06,946 | INFO | Corrupt journal records found in '/data02/apphome/Broker/db-21273.log' between offsets: 104495634..104647715 | org.apache.activemq.store.kahadb.disk.journal.Journal | WrapperSimpleAppMain 2015-05-18 11:28:13,519 | INFO | KahaDB is version 5 | org.apache.activemq.store.kahadb.MessageDatabase | WrapperSimpleAppMain 2015-05-18 11:28:13,793 | INFO | Recovering from the journal ... | org.apache.activemq.store.kahadb.MessageDatabase | WrapperSimpleAppMain 2015-05-18 11:28:13,795 | INFO | Recovery replayed 3 operations from the journal in 0.036 seconds. | org.apache.activemq.store.kahadb.MessageDatabase | WrapperSimpleAppMain 2015-05-18 11:28:13,814 | ERROR | Failed to start Apache ActiveMQ ([Broker, null], java.io.EOFException: Chunk stream does not exist, page: 19101 is marked free) | org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain 2015-05-18 11:28:13,818 | INFO | Apache ActiveMQ 5.9.0 (Broker, null) is shutting down | org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain 2015-05-18 11:28:13,818 | INFO | Connector clientconnection stopped | org.apache.activemq.broker.TransportConnector | WrapperSimpleAppMain 2015-05-18 11:28:13,834 | INFO | JobSchedulerStore:/data02/apphome/Broker/scheduler stopped | org.apache.activemq.store.kahadb.scheduler.JobSchedulerStoreImpl | WrapperSimpleAppMain 2015-05-18 11:28:13,834 | INFO | PListStore:[/data02/apphome/Broker/tmp_storage] stopped | org.apache.activemq.store.kahadb.plist.PListStoreImpl | WrapperSimpleAppMain 2015-05-18 11:28:13,835 | INFO | Stopping async queue tasks | org.apache.activemq.store.kahadb.KahaDBStore | WrapperSimpleAppMain 2015-05-18 11:28:13,835 | INFO | Stopping async topic tasks | org.apache.activemq.store.kahadb.KahaDBStore | WrapperSimpleAppMain 2015-05-18 11:28:13,835 | INFO | Stopped KahaDB | org.apache.activemq.store.kahadb.KahaDBStore | WrapperSimpleAppMain 2015-05-18 11:28:14,269 | INFO | Apache ActiveMQ 5.9.0 (Broker, null) uptime 1 minute | org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain 2015-05-18 11:28:14,269 | INFO | Apache ActiveMQ 5.9.0 (Broker, null) is shutdown | org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain 2015-05-18 11:28:14,274 | ERROR | Failed to load: class path resource [activemq.xml], reason: Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class path resource [activemq.xml]: Invocation of init method failed; nested exception is java.io.EOFException: Chunk stream does not exist, page: 19101 is marked free | org.apache.activemq.xbean.XBeanBrokerFactory | WrapperSimpleAppMain org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class path resource [activemq.xml]: Invocation of init method failed; nested exception is java.io.EOFException: Chunk stream does not exist, page: 19101 is marked free at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1482) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:295) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:292) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:628) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479) at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64) at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52) at org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:104) at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:104) at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:67) at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71) at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54) at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:87) at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57) at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:150) at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57) at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.activemq.console.Main.runTaskClass(Main.java:262) at org.apache.activemq.console.Main.main(Main.java:115) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240) at java.lang.Thread.run(Thread.java:744) Caused by: java.io.EOFException: Chunk stream does not exist, page: 19101 is marked free at org.apache.activemq.store.kahadb.disk.page.Transaction$2.readPage(Transaction.java:470) at org.apache.activemq.store.kahadb.disk.page.Transaction$2.<init>(Transaction.java:447) at org.apache.activemq.store.kahadb.disk.page.Transaction.openInputStream(Transaction.java:444) at org.apache.activemq.store.kahadb.disk.page.Transaction.load(Transaction.java:420) at org.apache.activemq.store.kahadb.disk.page.Transaction.load(Transaction.java:377) at org.apache.activemq.store.kahadb.disk.index.BTreeIndex.loadNode(BTreeIndex.java:262) at org.apache.activemq.store.kahadb.disk.index.BTreeNode.getChild(BTreeNode.java:225) at org.apache.activemq.store.kahadb.disk.index.BTreeNode.visit(BTreeNode.java:592) at org.apache.activemq.store.kahadb.disk.index.BTreeIndex.visit(BTreeIndex.java:236) at org.apache.activemq.store.kahadb.MessageDatabase.recoverIndex(MessageDatabase.java:744) at org.apache.activemq.store.kahadb.MessageDatabase$5.execute(MessageDatabase.java:602) at org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:779) at org.apache.activemq.store.kahadb.MessageDatabase.recover(MessageDatabase.java:599) at org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:389) at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:407) at org.apache.activemq.store.kahadb.MessageDatabase.doStart(MessageDatabase.java:260) at org.apache.activemq.store.kahadb.KahaDBStore.doStart(KahaDBStore.java:194) at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55) at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.doStart(KahaDBPersistenceAdapter.java:215) at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55) at org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:613) at org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:602) at org.apache.activemq.broker.BrokerService.start(BrokerService.java:567) at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:73) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1608) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1549) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1479) ... 33 more As this is very critical for us, we deleted the kahadb and restarted the activeMQ, it started working fine. We have gone through some of the cases in activemq issue tracker. https://issues.apache.org/jira/browse/AMQ-2935 This issue has been fixed in ActiveMQ 5.4.2. As per the solution in the ticket, it is not possible for me to make "SchedulerSupport" as false, which breaks redelievery policy. Initially we faced this issue in activeMQ 5.9 version. Recently we have upgraded activeMQ version 5.11.0 and still issue is exists in latest version also. I have attached activemq.xml file. -- View this message in context: http://activemq.2283324.n4.nabble.com/Unable-to-restart-the-activeMQ-with-the-error-Chunk-stream-doesn-t-exist-Page-is-marked-for-free-tp4698805.html Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
