In 5.4.2 its kahadb.. Bye, Norman
2011/1/6 dcheckoway <dchecko...@gmail.com>: > > Yeah, we're using whatever is the default persistence provider in 5.4.2. I > opened a fresh ticket for it: > > https://issues.apache.org/jira/browse/AMQ-3120 > > Thanks, > Dan > > On Thu, Jan 6, 2011 at 10:42 AM, Gary Tully [via ActiveMQ] < > ml-node+3177622-1458695149-140...@n4.nabble.com<ml-node%2b3177622-1458695149-140...@n4.nabble.com> >> wrote: > >> AMQ-3051 is using the older AMQPersistenceAdapter and NFS, so it seems >> to be different from your use case? >> >> On 6 January 2011 15:14, dcheckoway <[hidden >> email]<http://user/SendEmail.jtp?type=node&node=3177622&i=0>> >> wrote: >> >> > >> > Gary, >> > >> > Thanks for the reply and advice. I'll definitely try disabling >> > concurrentStoreAndDispatchQueues to see if that helps, and I'll set up an >> IO >> > exception handler. >> > >> > Unfortunately the issue can't reliably be reproduced. We have a cluster >> of >> > servers all running their own local embedded queue, and it happens on one >> >> > here, one there. >> > >> > That said, somebody else already opened a JIRA ticket for what looks like >> a >> > similar/nearly identical issue: >> > https://issues.apache.org/jira/browse/AMQ-3051 >> > >> > Dan >> > >> > >> > On Thu, Jan 6, 2011 at 9:29 AM, Gary Tully [via ActiveMQ] < >> > [hidden email] >> > <http://user/SendEmail.jtp?type=node&node=3177622&i=1><[hidden >> email] <http://user/SendEmail.jtp?type=node&node=3177622&i=2>> >> >> wrote: >> > >> >> One thought, there may be an issue with the default setting of true >> >> for concurrentStoreAndDispatchQueues that is causing the index to be >> >> out of sync with the store. >> >> >> >> It is worth trying to disable that, use: >> >> <persistenceAdapter> >> >> <kahaDB directory="/usr/local/embedded/activemq-data/kahadb" >> >> concurrentStoreAndDispatchQueues="false"/> >> >> </persistenceAdapter> >> >> >> >> How often is this occurring, to really get to the bottom of this a >> >> test case will be needed. >> >> Does that data file eventually get written, or are there any disk space >> >> issues? >> >> It may help to configure an IO exception handler just in case. >> >> http://activemq.apache.org/configurable-ioexception-handling.html >> >> >> >> Would be a good idea to create a jira issue to track this and attach >> >> what ever log and application information you have to it. >> >> >> >> On 6 January 2011 12:39, dcheckoway <[hidden email]< >> http://user/SendEmail.jtp?type=node&node=3177500&i=0>> >> >> wrote: >> >> >> >> > >> >> > I'm using an embedded broker (version 5.4.2) with persistence enabled. >> >> >> We're >> >> > pumping hundreds of millions of messages per day through this thing. >> >> Every >> >> > once in a while, all of a sudden the KahaDB directory starts growing >> >> > uncontrollably, and these errors start spewing out in the log over and >> >> >> > over... >> >> > >> >> > -------------------------- >> >> > ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task; >> >> > AbstractStoreCursor - Failed to fill batch >> >> > java.lang.RuntimeException: java.io.IOException: Could not locate data >> >> >> file >> >> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log >> >> > at >> >> > >> >> >> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265) >> >> >> >> >> > at >> >> > >> >> >> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148) >> >> >> >> >> > at >> >> > >> >> >> org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131) >> >> >> >> >> > at >> >> org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679) >> >> > at >> >> > >> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898) >> >> > at >> >> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425) >> >> > at >> >> > >> >> >> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122) >> >> >> >> >> > at >> >> > >> >> >> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43) >> >> > at >> >> > >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> >> >> >> >> > at >> >> > >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> >> >> >> >> > at java.lang.Thread.run(Thread.java:619) >> >> > Caused by: java.io.IOException: Could not locate data file >> >> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log >> >> > at >> org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345) >> >> >> >> > at org.apache.kahadb.journal.Journal.read(Journal.java:592) >> >> > at >> >> > >> >> >> org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786) >> >> >> >> >> > at >> >> > >> >> >> org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956) >> >> >> >> >> > at >> >> > >> >> >> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494) >> >> >> >> >> > at >> >> org.apache.kahadb.page.Transaction.execute(Transaction.java:728) >> >> > at >> >> > >> >> >> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485) >> >> >> >> >> > at >> >> > >> >> >> org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88) >> >> >> >> >> > at >> >> > >> >> >> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97) >> >> >> >> >> > at >> >> > >> >> >> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262) >> >> >> >> >> > ... 10 more >> >> > ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task; >> Queue >> >> - >> >> > Failed to page in more queue messages >> >> > java.lang.RuntimeException: java.lang.RuntimeException: >> >> java.io.IOException: >> >> > Could not locate data file >> >> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log >> >> > at >> >> > >> >> >> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:151) >> >> >> >> >> > at >> >> > >> >> >> org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131) >> >> >> >> >> > at >> >> org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679) >> >> > at >> >> > >> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898) >> >> > at >> >> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425) >> >> > at >> >> > >> >> >> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122) >> >> >> >> >> > at >> >> > >> >> >> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43) >> >> > at >> >> > >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> >> >> >> >> > at >> >> > >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> >> >> >> >> > at java.lang.Thread.run(Thread.java:619) >> >> > Caused by: java.lang.RuntimeException: java.io.IOException: Could not >> >> locate >> >> > data file >> /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log >> >> >> >> > at >> >> > >> >> >> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265) >> >> >> >> >> > at >> >> > >> >> >> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148) >> >> >> >> >> > ... 9 more >> >> > Caused by: java.io.IOException: Could not locate data file >> >> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log >> >> > at >> org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345) >> >> >> >> > at org.apache.kahadb.journal.Journal.read(Journal.java:592) >> >> > at >> >> > >> >> >> org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786) >> >> >> >> >> > at >> >> > >> >> >> org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956) >> >> >> >> >> > at >> >> > >> >> >> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494) >> >> >> >> >> > at >> >> org.apache.kahadb.page.Transaction.execute(Transaction.java:728) >> >> > at >> >> > >> >> >> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485) >> >> >> >> >> > at >> >> > >> >> >> org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88) >> >> >> >> >> > at >> >> > >> >> >> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97) >> >> >> >> >> > at >> >> > >> >> >> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262) >> >> >> >> >> > ... 10 more >> >> > -------------------------- >> >> > >> >> > It doesn't happen consistently, but it has happened numerous times. >> The >> >> > only way to "fix" it is to kill the app, blow away the KahaDB data >> >> > directory, and restart (ugh). >> >> > >> >> > Has anybody else seen this issue? Any idea why it happens? >> >> > >> >> > Here's the embedded broker config, fwiw: >> >> > -------------------------- >> >> > <broker xmlns="http://activemq.apache.org/schema/core"; >> >> > id="embeddedActivemqBroker" >> >> > useJmx="false" >> >> > persistent="true" >> >> > dataDirectory="/usr/local/embedded/activemq-data" >> >> > schedulerSupport="false" >> >> > advisorySupport="false" >> >> > enableStatistics="true"> >> >> > <destinationPolicy> >> >> > <policyMap> >> >> > <policyEntries> >> >> > <policyEntry queue=">" producerFlowControl="false" >> >> > memoryLimit="20mb"/> >> >> > </policyEntries> >> >> > </policyMap> >> >> > </destinationPolicy> >> >> > </broker> >> >> > -------------------------- >> >> > >> >> > Help?! >> >> > >> >> > ----- >> >> > Dan Checkoway >> >> > dcheckoway gmail com >> >> > -- >> >> > View this message in context: >> >> >> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html?by-user=t> >> < >> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html?by-user=t<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html?by-user=t&by-user=t>> >> >> >> > Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> >> > >> >> >> >> >> >> >> >> -- >> >> http://blog.garytully.com >> >> http://fusesource.com >> >> >> >> >> >> ------------------------------ >> >> View message @ >> >> >> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177500.html<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177500.html?by-user=t> >> >> >> >> >> >> >> > >> > >> > ----- >> > Dan Checkoway >> > dcheckoway gmail com >> > -- >> > View this message in context: >> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177577.html<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177577.html?by-user=t> >> >> > Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> > >> >> >> >> -- >> http://blog.garytully.com >> http://fusesource.com >> >> >> ------------------------------ >> View message @ >> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177622.html >> >> To unsubscribe from KahaDB error: "Could not locate data file", click >> here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=3177338&code=ZGNoZWNrb3dheUBnbWFpbC5jb218MzE3NzMzOHw3MDc4NzEwMTU=>. >> >> > > > ----- > Dan Checkoway > dcheckoway gmail com > -- > View this message in context: > http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177745.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. >
