[
https://issues.apache.org/jira/browse/AMQ-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Bish closed AMQ-6663.
-----------------------------
Resolution: Incomplete
No feedback on newer releases so unconfirmed if this is still an issue or not
and no reproducer.
> IO exceptions causing Broker Service shutdown; When broker restarts,
> active-mq ignores the configured location for kahadb
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: AMQ-6663
> URL: https://issues.apache.org/jira/browse/AMQ-6663
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker, KahaDB, Message Store
> Affects Versions: 5.11.0
> Reporter: Vidyashree
> Priority: Critical
>
> ActiveMQ version: 5.11.0.redhat-620143
> A-MQ configuration in activemq.xml
> <bean id="persistenceAdapter"
>
> class="org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter">
> <property name="directory"
> value="${activemq.home}/${id}/jms/kahaDb"/>
> <property name="enableJournalDiskSyncs" value="false" />
> <property name="journalMaxFileLength" value="33554432" />
> <property name="ignoreMissingJournalfiles" value="true"/>
> <property name="checkForCorruptJournalFiles" value="true"/>
> </bean>
> <bean id="brokerServices"
> class="org.apache.activemq.broker.BrokerService">
> <description>AMQ Broker</description>
> <property name="dataDirectory"
> value="${activemq.home}/${id}/jms"/>
> <property name="systemUsage" ref="systemUsage" />
> <property name="persistenceAdapter" ref="persistenceAdapter" />
> <property name="managementContext" ref="managementContext" />
> <property name="destinationPolicy" ref="destinationPolicy"/>
> <property name="useShutdownHook" value="false"/>
> </bean>
> Problem description:
> During durable subscription removal, we are observing IO exceptions as below,
> and it is resulting in shutdown of the broker. And we also observe change in
> path of the kahadb location to the active-mq default
> location({$home}/activemq-data/activemq-instance1/KahaDB). It should have
> taken the configured location from activemq.xml
> Stack trace from the EOFException :
> 2017-04-11 04:56:01,640 org.apache.activemq.store.kahadb.MessageDatabase
> ERROR (MessageDatabase.java:store:1010)
> [CamelJettyServer(0x57063e08)-3734787] [3734787] | KahaDB failed to store to
> Journal
> java.io.EOFException
> at java.io.DataInputStream.readFully(Unknown Source)
> at java.io.DataInputStream.readLong(Unknown Source)
> at
> org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller.readPayload(MessageDatabase.java:1988)
> at
> org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller.readPayload(MessageDatabase.java:1910)
> at
> org.apache.activemq.store.kahadb.disk.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:209)
> at
> org.apache.activemq.store.kahadb.disk.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:155)
> at
> org.apache.activemq.store.kahadb.disk.page.Transaction.load(Transaction.java:422)
> at
> org.apache.activemq.store.kahadb.disk.page.Transaction.load(Transaction.java:377)
> at
> org.apache.activemq.store.kahadb.disk.index.BTreeIndex.loadNode(BTreeIndex.java:266)
> at
> org.apache.activemq.store.kahadb.disk.index.BTreeIndex.getRoot(BTreeIndex.java:174)
> at
> org.apache.activemq.store.kahadb.disk.index.BTreeIndex.remove(BTreeIndex.java:194)
> at
> org.apache.activemq.store.kahadb.MessageDatabase.updateIndex(MessageDatabase.java:1469)
> at
> org.apache.activemq.store.kahadb.MessageDatabase.updateIndex(MessageDatabase.java:1500)
> at
> org.apache.activemq.store.kahadb.MessageDatabase$15.execute(MessageDatabase.java:1219)
> at
> org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:779)
> at
> org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:1216)
> at
> org.apache.activemq.store.kahadb.MessageDatabase$10.visit(MessageDatabase.java:1115)
> at
> org.apache.activemq.store.kahadb.data.KahaSubscriptionCommand.visit(KahaSubscriptionCommand.java:187)
> at
> org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:1082)
> at
> org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:989)
> at
> org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:969)
> at
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBTopicMessageStore.deleteSubscription(KahaDBStore.java:807)
> at
> org.apache.activemq.store.ProxyTopicMessageStore.deleteSubscription(ProxyTopicMessageStore.java:103)
> at
> org.apache.activemq.broker.region.Topic.deleteSubscription(Topic.java:201)
> at
> org.apache.activemq.broker.region.TopicRegion.removeSubscription(TopicRegion.java:220)
> at
> org.apache.activemq.broker.region.RegionBroker.removeSubscription(RegionBroker.java:448)
> at
> org.apache.activemq.broker.BrokerFilter.removeSubscription(BrokerFilter.java:117)
> at
> org.apache.activemq.advisory.AdvisoryBroker.removeSubscription(AdvisoryBroker.java:314)
> at
> org.apache.activemq.broker.BrokerFilter.removeSubscription(BrokerFilter.java:117)
> at
> org.apache.activemq.broker.BrokerFilter.removeSubscription(BrokerFilter.java:117)
> at
> org.apache.activemq.broker.MutableBrokerFilter.removeSubscription(MutableBrokerFilter.java:122)
> at
> org.apache.activemq.broker.jmx.BrokerView.destroyDurableSubscriber(BrokerView.java:449)
> at
> org.apache.camel.processor.DelegateSyncProcessor.process(DelegateSyncProcessor.java:63)
> at
> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)
> at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
> at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
> at
> org.apache.camel.component.jetty.CamelContinuationServlet.service(CamelContinuationServlet.java:150)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
> at
> org.eclipse.jetty.servlets.MultiPartFilter.doFilter(MultiPartFilter.java:131)
> at
> org.apache.camel.component.jetty.CamelFilterWrapper.doFilter(CamelFilterWrapper.java:43)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.eclipse.jetty.server.Server.handle(Server.java:370)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
> at
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Unknown Source)
> instance1 2017-04-11 04:56:01,646
> org.apache.activemq.util.DefaultIOExceptionHandler INFO
> (DefaultIOExceptionHandler.java:stopBroker:165)
> [CamelJettyServer(0x57063e08)-3734787] [3734787] | Stopping
> BrokerService[activemq-instance1] due to exception, java.io.EOFException
> java.io.EOFException
> at java.io.DataInputStream.readFully(Unknown Source)
> at java.io.DataInputStream.readLong(Unknown Source)
> at
> org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller.readPayload(MessageDatabase.java:1988)
>
> 1. Why did ActiveMQ consider the default location for KahaDB (Mentioned in
> IOHelper and MessageDatabase classes of activemq) instead of the configured
> path during broker restart?
> 2. When we restarted the system with the affected contents of kahaDB store,
> we found the below information from the active mq log
> org.apache.activemq.store.kahadb.MessageDatabase WARN
> (MessageDatabase.java:open:382) [main] [1] | Index corrupted. Recovering the
> index through journal replay. Cause:java.io.EOFException
> org.apache.activemq.store.kahadb.MessageDatabase DEBUG
> (MessageDatabase.java:open:384) [main] [1] | Index load failure
> - What could be the cause of index corruption?
> - What happens if there is index corruption in the KahaDB store and
> system does not restart?
> - We did not see any issue with disk such as disk is unavailable of no
> more space on the disk. So, in these conditions, how did the corruption
> happened in the system?
> Is it related to https://issues.apache.org/jira/browse/AMQ-6086?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
