KahaDB error: "Could not locate data file"
------------------------------------------
Key: AMQ-3120
URL: https://issues.apache.org/jira/browse/AMQ-3120
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.4.2
Environment: linux: CentOS 5.5
happens on both dual- and quad-cpu boxes, 4gb ram
java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)
Reporter: Dan Checkoway
I'm using an embedded broker (version 5.4.2) with persistence enabled. We're
pumping hundreds of millions of messages per day through this thing. Every
once in a while, all of a sudden the KahaDB directory starts growing
uncontrollably, and these errors start spewing out in the log over and over...
--------------------------
ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task;
AbstractStoreCursor - Failed to fill batch
java.lang.RuntimeException: java.io.IOException: Could not locate data file
/usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
at
org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
at
org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: Could not locate data file
/usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
at org.apache.kahadb.journal.Journal.read(Journal.java:592)
at
org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
at
org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
at
org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
at
org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
... 10 more
ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task; Queue -
Failed to page in more queue messages
java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException:
Could not locate data file
/usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:151)
at
org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
at
org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate
data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
... 9 more
Caused by: java.io.IOException: Could not locate data file
/usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
at org.apache.kahadb.journal.Journal.read(Journal.java:592)
at
org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
at
org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
at
org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
at
org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
... 10 more
--------------------------
It doesn't happen consistently, but it has happened numerous times. The only
way to "fix" it is to kill the app, blow away the KahaDB data directory, and
restart (ugh).
Unfortunately this issue is not reproduceable.
Here's the embedded broker config, fwiw:
--------------------------
<broker xmlns="http://activemq.apache.org/schema/core";
id="embeddedActivemqBroker"
useJmx="false"
persistent="true"
dataDirectory="/usr/local/embedded/activemq-data"
schedulerSupport="false"
advisorySupport="false"
enableStatistics="true">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue=">" producerFlowControl="false"
memoryLimit="20mb"/>
</policyEntries>
</policyMap>
</destinationPolicy>
</broker>
--------------------------
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
