[
https://issues.apache.org/jira/browse/AMQ-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15986547#comment-15986547
]
Christopher L. Shannon edited comment on AMQ-6663 at 4/27/17 12:51 PM:
-----------------------------------------------------------------------
I'm not sure why the default location would have changed as I don't think I've
seen that behavior reported before by other people. It's possible it's an
undiscovered bug caused by your situation but it would need to be investigated
more.
If the system does not restart because of index corruption you can always
delete the index files (starts with db.* such as db.data) and restart again.
However KahaDB is designed to detect a corruption and rebuild automatically so
that shouldn't be necessary. Index corruptions would generally be caused by
issues on the file system but as I said earlier it's impossible to tell you a
reason just by a stack trace as every system is different. But KahaDB is
designed so that it can recover from an index corruption in the rare event it
happens.
Is this a one time event or something that happens regularly? If this is
happening more than one time you should definitely upgrade to a new version as
there are probably close to 50 KahaDB fixes since version 5.11.0 and some of
those have addressed stability issues . I'm not guaranteeing that an upgrade
will solve the problem but it's always best to try the latest to see if the
issue still is around.
was (Author: christopher.l.shannon):
I'm not sure why the default location would have changed as I don't think I've
seen that behavior reported before by other people. It's possible it's an
undiscovered bug caused by your situation but it would need to be investigated
more.
If the system does not restart because of index corruption you can always
delete the index files (starts with db.* such as db.data) and restart again.
However KahaDB is designed to detect a corruption and rebuild automatically so
that shouldn't be necessary. Index corruptions would generally be caused by
issues on the file system but as I said earlier it's impossible to tell you a
reason just by a stack trace as every system is different. But KahaDB is
designed so that it can recover from an index corruption in the rare event it
happens.
Is this a one time event or something that happens regularly? If this is
happening more than one time you should definitely upgrade to a new version as
there are probably close to 50 KahaDB fixes since version 5.11.0 and some of
those have addressed issues stability. I'm not guaranteeing that an upgrade
will solve the problem but it's always best to try the latest to see if the
issue still is around.
> IO exceptions causing Broker Service shutdown; When broker restarts,
> active-mq ignores the configured location for kahadb
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: AMQ-6663
> URL: https://issues.apache.org/jira/browse/AMQ-6663
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker, KahaDB, Message Store
> Affects Versions: 5.11.0
> Reporter: Vidyashree
> Priority: Critical
>
> ActiveMQ version: 5.11.0.redhat-620143
> A-MQ configuration in activemq.xml
> <bean id="persistenceAdapter"
>
> class="org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter">
> <property name="directory"
> value="${activemq.home}/${id}/jms/kahaDb"/>
> <property name="enableJournalDiskSyncs" value="false" />
> <property name="journalMaxFileLength" value="33554432" />
> <property name="ignoreMissingJournalfiles" value="true"/>
> <property name="checkForCorruptJournalFiles" value="true"/>
> </bean>
> <bean id="brokerServices"
> class="org.apache.activemq.broker.BrokerService">
> <description>AMQ Broker</description>
> <property name="dataDirectory"
> value="${activemq.home}/${id}/jms"/>
> <property name="systemUsage" ref="systemUsage" />
> <property name="persistenceAdapter" ref="persistenceAdapter" />
> <property name="managementContext" ref="managementContext" />
> <property name="destinationPolicy" ref="destinationPolicy"/>
> <property name="useShutdownHook" value="false"/>
> </bean>
> Problem description:
> During durable subscription removal, we are observing IO exceptions as below,
> and it is resulting in shutdown of the broker. And we also observe change in
> path of the kahadb location to the active-mq default
> location({$home}/activemq-data/activemq-instance1/KahaDB). It should have
> taken the configured location from activemq.xml
> Stack trace from the EOFException :
> 2017-04-11 04:56:01,640 org.apache.activemq.store.kahadb.MessageDatabase
> ERROR (MessageDatabase.java:store:1010)
> [CamelJettyServer(0x57063e08)-3734787] [3734787] | KahaDB failed to store to
> Journal
> java.io.EOFException
> at java.io.DataInputStream.readFully(Unknown Source)
> at java.io.DataInputStream.readLong(Unknown Source)
> at
> org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller.readPayload(MessageDatabase.java:1988)
> at
> org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller.readPayload(MessageDatabase.java:1910)
> at
> org.apache.activemq.store.kahadb.disk.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:209)
> at
> org.apache.activemq.store.kahadb.disk.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:155)
> at
> org.apache.activemq.store.kahadb.disk.page.Transaction.load(Transaction.java:422)
> at
> org.apache.activemq.store.kahadb.disk.page.Transaction.load(Transaction.java:377)
> at
> org.apache.activemq.store.kahadb.disk.index.BTreeIndex.loadNode(BTreeIndex.java:266)
> at
> org.apache.activemq.store.kahadb.disk.index.BTreeIndex.getRoot(BTreeIndex.java:174)
> at
> org.apache.activemq.store.kahadb.disk.index.BTreeIndex.remove(BTreeIndex.java:194)
> at
> org.apache.activemq.store.kahadb.MessageDatabase.updateIndex(MessageDatabase.java:1469)
> at
> org.apache.activemq.store.kahadb.MessageDatabase.updateIndex(MessageDatabase.java:1500)
> at
> org.apache.activemq.store.kahadb.MessageDatabase$15.execute(MessageDatabase.java:1219)
> at
> org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:779)
> at
> org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:1216)
> at
> org.apache.activemq.store.kahadb.MessageDatabase$10.visit(MessageDatabase.java:1115)
> at
> org.apache.activemq.store.kahadb.data.KahaSubscriptionCommand.visit(KahaSubscriptionCommand.java:187)
> at
> org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:1082)
> at
> org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:989)
> at
> org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:969)
> at
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBTopicMessageStore.deleteSubscription(KahaDBStore.java:807)
> at
> org.apache.activemq.store.ProxyTopicMessageStore.deleteSubscription(ProxyTopicMessageStore.java:103)
> at
> org.apache.activemq.broker.region.Topic.deleteSubscription(Topic.java:201)
> at
> org.apache.activemq.broker.region.TopicRegion.removeSubscription(TopicRegion.java:220)
> at
> org.apache.activemq.broker.region.RegionBroker.removeSubscription(RegionBroker.java:448)
> at
> org.apache.activemq.broker.BrokerFilter.removeSubscription(BrokerFilter.java:117)
> at
> org.apache.activemq.advisory.AdvisoryBroker.removeSubscription(AdvisoryBroker.java:314)
> at
> org.apache.activemq.broker.BrokerFilter.removeSubscription(BrokerFilter.java:117)
> at
> org.apache.activemq.broker.BrokerFilter.removeSubscription(BrokerFilter.java:117)
> at
> org.apache.activemq.broker.MutableBrokerFilter.removeSubscription(MutableBrokerFilter.java:122)
> at
> org.apache.activemq.broker.jmx.BrokerView.destroyDurableSubscriber(BrokerView.java:449)
> at
> org.apache.camel.processor.DelegateSyncProcessor.process(DelegateSyncProcessor.java:63)
> at
> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)
> at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
> at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
> at
> org.apache.camel.component.jetty.CamelContinuationServlet.service(CamelContinuationServlet.java:150)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
> at
> org.eclipse.jetty.servlets.MultiPartFilter.doFilter(MultiPartFilter.java:131)
> at
> org.apache.camel.component.jetty.CamelFilterWrapper.doFilter(CamelFilterWrapper.java:43)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.eclipse.jetty.server.Server.handle(Server.java:370)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
> at
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Unknown Source)
> instance1 2017-04-11 04:56:01,646
> org.apache.activemq.util.DefaultIOExceptionHandler INFO
> (DefaultIOExceptionHandler.java:stopBroker:165)
> [CamelJettyServer(0x57063e08)-3734787] [3734787] | Stopping
> BrokerService[activemq-instance1] due to exception, java.io.EOFException
> java.io.EOFException
> at java.io.DataInputStream.readFully(Unknown Source)
> at java.io.DataInputStream.readLong(Unknown Source)
> at
> org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller.readPayload(MessageDatabase.java:1988)
>
> 1. Why did ActiveMQ consider the default location for KahaDB (Mentioned in
> IOHelper and MessageDatabase classes of activemq) instead of the configured
> path during broker restart?
> 2. When we restarted the system with the affected contents of kahaDB store,
> we found the below information from the active mq log
> org.apache.activemq.store.kahadb.MessageDatabase WARN
> (MessageDatabase.java:open:382) [main] [1] | Index corrupted. Recovering the
> index through journal replay. Cause:java.io.EOFException
> org.apache.activemq.store.kahadb.MessageDatabase DEBUG
> (MessageDatabase.java:open:384) [main] [1] | Index load failure
> - What could be the cause of index corruption?
> - What happens if there is index corruption in the KahaDB store and
> system does not restart?
> - We did not see any issue with disk such as disk is unavailable of no
> more space on the disk. So, in these conditions, how did the corruption
> happened in the system?
> Is it related to https://issues.apache.org/jira/browse/AMQ-6086?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
