[
https://issues.apache.org/jira/browse/AMQ-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15984947#comment-15984947
]
Vidyashree edited comment on AMQ-6663 at 4/27/17 5:54 AM:
----------------------------------------------------------
Hi Christopher,
Thanks for your comments.
We got to know about index corruption only after application restart. Before
that, during runtime, activemq changed the Kahadb location to the default
location instead of using the configured one after EOFException. Why did this
happen? There were no enough information for any possible errors in the logs
for this behavior of activemq. Also, what happens if there is index corruption
in the KahaDB store and system does not restart? We want to know the root cause
for index corruption and kahaDB directory path change.
was (Author: vshree):
Hi Christopher,
Thanks for your comments.
We got to know about index corruption only after application restart. Before
that, during runtime, activemq changed the Kahadb location to the default
location instead of using the configured one after EOFException. Why did this
happen? There were no enough information for any possible errors in the logs
for this behavior of activemq. Also, what happens if there is index corruption
in the KahaDB store and system does not restart?
> IO exceptions causing Broker Service shutdown; When broker restarts,
> active-mq ignores the configured location for kahadb
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: AMQ-6663
> URL: https://issues.apache.org/jira/browse/AMQ-6663
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker, KahaDB, Message Store
> Affects Versions: 5.11.0
> Reporter: Vidyashree
> Priority: Critical
>
> ActiveMQ version: 5.11.0.redhat-620143
> A-MQ configuration in activemq.xml
> <bean id="persistenceAdapter"
>
> class="org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter">
> <property name="directory"
> value="${activemq.home}/${id}/jms/kahaDb"/>
> <property name="enableJournalDiskSyncs" value="false" />
> <property name="journalMaxFileLength" value="33554432" />
> <property name="ignoreMissingJournalfiles" value="true"/>
> <property name="checkForCorruptJournalFiles" value="true"/>
> </bean>
> <bean id="brokerServices"
> class="org.apache.activemq.broker.BrokerService">
> <description>AMQ Broker</description>
> <property name="dataDirectory"
> value="${activemq.home}/${id}/jms"/>
> <property name="systemUsage" ref="systemUsage" />
> <property name="persistenceAdapter" ref="persistenceAdapter" />
> <property name="managementContext" ref="managementContext" />
> <property name="destinationPolicy" ref="destinationPolicy"/>
> <property name="useShutdownHook" value="false"/>
> </bean>
> Problem description:
> During durable subscription removal, we are observing IO exceptions as below,
> and it is resulting in shutdown of the broker. And we also observe change in
> path of the kahadb location to the active-mq default
> location({$home}/activemq-data/activemq-instance1/KahaDB). It should have
> taken the configured location from activemq.xml
> Stack trace from the EOFException :
> 2017-04-11 04:56:01,640 org.apache.activemq.store.kahadb.MessageDatabase
> ERROR (MessageDatabase.java:store:1010)
> [CamelJettyServer(0x57063e08)-3734787] [3734787] | KahaDB failed to store to
> Journal
> java.io.EOFException
> at java.io.DataInputStream.readFully(Unknown Source)
> at java.io.DataInputStream.readLong(Unknown Source)
> at
> org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller.readPayload(MessageDatabase.java:1988)
> at
> org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller.readPayload(MessageDatabase.java:1910)
> at
> org.apache.activemq.store.kahadb.disk.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:209)
> at
> org.apache.activemq.store.kahadb.disk.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:155)
> at
> org.apache.activemq.store.kahadb.disk.page.Transaction.load(Transaction.java:422)
> at
> org.apache.activemq.store.kahadb.disk.page.Transaction.load(Transaction.java:377)
> at
> org.apache.activemq.store.kahadb.disk.index.BTreeIndex.loadNode(BTreeIndex.java:266)
> at
> org.apache.activemq.store.kahadb.disk.index.BTreeIndex.getRoot(BTreeIndex.java:174)
> at
> org.apache.activemq.store.kahadb.disk.index.BTreeIndex.remove(BTreeIndex.java:194)
> at
> org.apache.activemq.store.kahadb.MessageDatabase.updateIndex(MessageDatabase.java:1469)
> at
> org.apache.activemq.store.kahadb.MessageDatabase.updateIndex(MessageDatabase.java:1500)
> at
> org.apache.activemq.store.kahadb.MessageDatabase$15.execute(MessageDatabase.java:1219)
> at
> org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:779)
> at
> org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:1216)
> at
> org.apache.activemq.store.kahadb.MessageDatabase$10.visit(MessageDatabase.java:1115)
> at
> org.apache.activemq.store.kahadb.data.KahaSubscriptionCommand.visit(KahaSubscriptionCommand.java:187)
> at
> org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:1082)
> at
> org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:989)
> at
> org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:969)
> at
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBTopicMessageStore.deleteSubscription(KahaDBStore.java:807)
> at
> org.apache.activemq.store.ProxyTopicMessageStore.deleteSubscription(ProxyTopicMessageStore.java:103)
> at
> org.apache.activemq.broker.region.Topic.deleteSubscription(Topic.java:201)
> at
> org.apache.activemq.broker.region.TopicRegion.removeSubscription(TopicRegion.java:220)
> at
> org.apache.activemq.broker.region.RegionBroker.removeSubscription(RegionBroker.java:448)
> at
> org.apache.activemq.broker.BrokerFilter.removeSubscription(BrokerFilter.java:117)
> at
> org.apache.activemq.advisory.AdvisoryBroker.removeSubscription(AdvisoryBroker.java:314)
> at
> org.apache.activemq.broker.BrokerFilter.removeSubscription(BrokerFilter.java:117)
> at
> org.apache.activemq.broker.BrokerFilter.removeSubscription(BrokerFilter.java:117)
> at
> org.apache.activemq.broker.MutableBrokerFilter.removeSubscription(MutableBrokerFilter.java:122)
> at
> org.apache.activemq.broker.jmx.BrokerView.destroyDurableSubscriber(BrokerView.java:449)
> at
> org.apache.camel.processor.DelegateSyncProcessor.process(DelegateSyncProcessor.java:63)
> at
> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)
> at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
> at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
> at
> org.apache.camel.component.jetty.CamelContinuationServlet.service(CamelContinuationServlet.java:150)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
> at
> org.eclipse.jetty.servlets.MultiPartFilter.doFilter(MultiPartFilter.java:131)
> at
> org.apache.camel.component.jetty.CamelFilterWrapper.doFilter(CamelFilterWrapper.java:43)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.eclipse.jetty.server.Server.handle(Server.java:370)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
> at
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Unknown Source)
> instance1 2017-04-11 04:56:01,646
> org.apache.activemq.util.DefaultIOExceptionHandler INFO
> (DefaultIOExceptionHandler.java:stopBroker:165)
> [CamelJettyServer(0x57063e08)-3734787] [3734787] | Stopping
> BrokerService[activemq-instance1] due to exception, java.io.EOFException
> java.io.EOFException
> at java.io.DataInputStream.readFully(Unknown Source)
> at java.io.DataInputStream.readLong(Unknown Source)
> at
> org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller.readPayload(MessageDatabase.java:1988)
>
> 1. Why did ActiveMQ consider the default location for KahaDB (Mentioned in
> IOHelper and MessageDatabase classes of activemq) instead of the configured
> path during broker restart?
> 2. When we restarted the system with the affected contents of kahaDB store,
> we found the below information from the active mq log
> org.apache.activemq.store.kahadb.MessageDatabase WARN
> (MessageDatabase.java:open:382) [main] [1] | Index corrupted. Recovering the
> index through journal replay. Cause:java.io.EOFException
> org.apache.activemq.store.kahadb.MessageDatabase DEBUG
> (MessageDatabase.java:open:384) [main] [1] | Index load failure
> - What could be the cause of index corruption?
> - What happens if there is index corruption in the KahaDB store and
> system does not restart?
> - We did not see any issue with disk such as disk is unavailable of no
> more space on the disk. So, in these conditions, how did the corruption
> happened in the system?
> Is it related to https://issues.apache.org/jira/browse/AMQ-6086?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
