[
https://issues.apache.org/jira/browse/AMQ-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294961#comment-13294961
]
Gary Tully commented on AMQ-3866:
---------------------------------
There is a periodic checkpoint of the producer audit to the journal, this could
be the reason for the increase in size but it would not prevent an unreferenced
journal data file form getting removed.
First thing to do is recreate the index by replaying the journal from the
start. Do this by stopping the broker, deleting the db.data and db.redo file in
the kahadb persistence adapter directory and restarting.
This will ensure that the index is correct after the upgrade to 5.6
The key to understanding why a data file remains in your point 13) - but the
information w.r.t to garbage collection is lost in those entries about the
sequenceid index. I think with a rebuild of the index, those should disappear.
> kahadb log files not getting cleaned up
> ---------------------------------------
>
> Key: AMQ-3866
> URL: https://issues.apache.org/jira/browse/AMQ-3866
> Project: ActiveMQ
> Issue Type: Bug
> Components: Message Store
> Affects Versions: 5.5.1, 5.6.0
> Reporter: Alexander Vallens
> Priority: Critical
>
> A few weeks ago we experienced an event where several hundred thousand
> messages were posted to a queue and slowly consumed over several days until
> accidentally purged. Ever since then our kahadb has been growing on the order
> of 12-20 bytes/10 seconds, corresponding to additional entries being added to
> the kahadb log files that are not being cleaned up (this is even the case in
> an isolated test environment with no subscribers/consumers/activity). Having
> searched and found AMQ-3131, AMQ-3736, AMQ-3424, we attempted an upgrade on a
> clone of the affected system from 5.5.1 to 5.6.0, with no luck. Along the way
> here are the steps we took:
> 1) I cloned our ActiveMQ virtual machine “hot” (i.e. without shutdown) and
> was able to get an isolated (different hostname, IP, VLAN), but functioning,
> “copy” of the production ActiveMQ instance
> 2) startup is slow, given the sheer size of the kahadb, as shown in the
> startup logging:
> 2012-05-29 19:59:11,780 | INFO | Recovering from the journal ... |
> org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 20:05:28,352 | WARN | Duplicate message add attempt rejected.
> Destination: Consumer.SchedulerEvents.VirtualTopic.scheduler/events, Message
> id: ID:<FQDN>-55381-1337386136713-0:18:1:1:1 | org.apache.a
> ctivemq.store.kahadb.MessageDatabase | main
> 2012-05-29 20:05:28,494 | WARN | Duplicate message add attempt rejected.
> Destination: Consumer.SchedulerEvents.VirtualTopic.scheduler/events, Message
> id: ID:<FQDN>-55381-1337386136713-0:25:1:1:1 | org.apache.a
> ctivemq.store.kahadb.MessageDatabase | main
> 2012-05-29 20:05:28,528 | WARN | Duplicate message add attempt rejected.
> Destination: Consumer.SchedulerEvents.VirtualTopic.scheduler/events, Message
> id: ID:<FQDN>-55381-1337386136713-0:32:1:1:1 | org.apache.a
> ctivemq.store.kahadb.MessageDatabase | main
> 2012-05-29 20:05:34,513 | INFO | Recovery replayed 2325747 operations from
> the journal in 382.795 seconds. |
> org.apache.activemq.store.kahadb.MessageDatabase | main
> 3) I was able to eliminate the 3 “duplicate message” warnings by purging
> the “Consumer.SchedulerEvents.VirtualTopic.scheduler/events” queue (I suspect
> there were unconsumed messages, perhaps partially uploaded, in that queue at
> time of clone)
> 4) after the above steps, I was able to observe that kahadb continued to
> grow at the same rate as the actual production ActiveMQ instancw, ~12-20
> bytes every 10s
> 5) during the first normal shutdown of activemq, I observed the following
> errors:
> java.lang.IllegalStateException: PageFile is not loaded
> at org.apache.kahadb.page.PageFile.assertLoaded(PageFile.java:715)
> at org.apache.kahadb.page.PageFile.tx(PageFile.java:239)
> at
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.resetBatching(KahaDBStore.java:510)
> at
> org.apache.activemq.store.ProxyMessageStore.resetBatching(ProxyMessageStore.java:93)
> at
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.resetBatch(QueueStorePrefetch.java:85)
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.stop(AbstractStoreCursor.java:67)
> at
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.stop(StoreQueueCursor.java:84)
> at org.apache.activemq.broker.region.Queue.stop(Queue.java:872)
> at
> org.apache.activemq.broker.region.AbstractRegion.stop(AbstractRegion.java:110)
> at
> org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:41)
> at
> org.apache.activemq.broker.region.RegionBroker.doStop(RegionBroker.java:729)
> at
> org.apache.activemq.broker.jmx.ManagedRegionBroker.doStop(ManagedRegionBroker.java:113)
> at
> org.apache.activemq.broker.region.RegionBroker.stop(RegionBroker.java:213)
> at org.apache.activemq.broker.BrokerFilter.stop(BrokerFilter.java:161)
> at
> org.apache.activemq.broker.scheduler.SchedulerBroker.stop(SchedulerBroker.java:104)
> at org.apache.activemq.broker.BrokerFilter.stop(BrokerFilter.java:161)
> at org.apache.activemq.broker.BrokerFilter.stop(BrokerFilter.java:161)
> at
> org.apache.activemq.broker.TransactionBroker.stop(TransactionBroker.java:114)
> at
> org.apache.activemq.broker.BrokerService$3.stop(BrokerService.java:1795)
> at
> org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:41)
> at
> org.apache.activemq.broker.BrokerService.stop(BrokerService.java:587)
> at
> org.apache.activemq.console.command.StartCommand$1.run(StartCommand.java:135)
> 2012-05-29 20:22:09,112 | ERROR | Failed to reset batching |
> org.apache.activemq.store.kahadb.KahaDBStore | Thread-45
> 6) subsequent shutdowns, however, operated normally without error
> 7) even after various attempts at shutdown and start back up, the
> aforementioned journal recovery was time consuming:
> 2012-05-29 20:25:06,780 | INFO | Recovering from the journal ... |
> org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 20:31:39,821 | INFO | Recovery replayed 2325954 operations from
> the journal in 393.074 seconds. |
> org.apache.activemq.store.kahadb.MessageDatabase | main
> 8) and the kahadb growth issue persisted
> 9) I went ahead and successfully upgraded to ActiveMQ v5.6.0 (verified
> queues & messages in scheduled queue survived upgrade)
> 10) first attempt to start activemq after 5.6.0 upgrade included following
> errors:
> 2012-05-29 20:55:36,825 | ERROR | Failed to start ActiveMQ JMS Message Broker
> (localhost, null). Reason: java.lang.NegativeArraySizeException |
> org.apache.activemq.broker.BrokerService | main
> java.lang.NegativeArraySizeException
> at
> org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:188)
> at
> org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:147)
> at org.apache.kahadb.page.Transaction.load(Transaction.java:412)
> at org.apache.kahadb.page.Transaction.load(Transaction.java:367)
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:262)
> at org.apache.kahadb.index.BTreeIndex.getRoot(BTreeIndex.java:174)
> at org.apache.kahadb.index.BTreeIndex.iterator(BTreeIndex.java:228)
> at
> org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller$1.execute(MessageDatabase.java:1654)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:769)
> at
> org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller.readPayload(MessageDatabase.java:1642)
> at
> org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller.readPayload(MessageDatabase.java:1627)
> at
> org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:201)
> at
> org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:147)
> at org.apache.kahadb.page.Transaction.load(Transaction.java:412)
> at org.apache.kahadb.page.Transaction.load(Transaction.java:367)
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:262)
> at org.apache.kahadb.index.BTreeIndex.getRoot(BTreeIndex.java:174)
> at org.apache.kahadb.index.BTreeIndex.iterator(BTreeIndex.java:228)
> at
> org.apache.activemq.store.kahadb.KahaDBStore$6.execute(KahaDBStore.java:943)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:769)
> at
> org.apache.activemq.store.kahadb.KahaDBStore.getDestinations(KahaDBStore.java:941)
> at
> org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.getDestinations(KahaDBPersistenceAdapter.java:122)
>
> at
> org.apache.activemq.broker.region.DestinationFactoryImpl.getDestinations(DestinationFactoryImpl.java:68)
> at
> org.apache.activemq.broker.BrokerService.startVirtualConsumerDestinations(BrokerService.java:2369)
> at
> org.apache.activemq.broker.BrokerService.startDestinations(BrokerService.java:2198)
> at
> org.apache.activemq.broker.BrokerService.start(BrokerService.java:527)
> at
> org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:60)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1544)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1485)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1417)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
> at
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
> at
> org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585)
> at
> org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
> at
> org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
> at
> org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)
> at
> org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)
>
> at
> org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:108)
> at
> org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:108)
> at
> org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:72)
> at
> org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)
> at
> org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)
> at
> org.apache.activemq.console.command.StartCommand.startBroker(StartCommand.java:115)
> at
> org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:74)
> at
> org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
> at
> org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:148)
> at
> org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
> at
> org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:90)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.activemq.console.Main.runTaskClass(Main.java:257)
> at org.apache.activemq.console.Main.main(Main.java:111)
> 2012-05-29 20:55:36,873 | INFO | ActiveMQ Message Broker (localhost, null)
> is shutting down | org.apache.activemq.broker.BrokerService | main
> 2012-05-29 20:55:36,873 | INFO | Connector openwire Stopped |
> org.apache.activemq.broker.TransportConnector | main
> 2012-05-29 20:55:36,873 | INFO | Connector stomp Stopped |
> org.apache.activemq.broker.TransportConnector | main
> 2012-05-29 20:55:36,897 | INFO |
> PListStore:[/opt/activemq/data/localhost/tmp_storage ] stopped |
> org.apache.activemq.store.kahadb.plist.PListStore | main
> 2012-05-29 20:55:36,897 | INFO | Stopping async queue tasks |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2012-05-29 20:55:36,897 | INFO | Stopping async topic tasks |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2012-05-29 20:55:36,897 | INFO | Stopped KahaDB |
> org.apache.activemq.store.kahadb.KahaDBStore | main
> 2012-05-29 20:55:37,297 | INFO | ActiveMQ JMS Message Broker (localhost,
> null) stopped | org.apache.activemq.broker.BrokerService | main
> 2012-05-29 20:55:37,300 | ERROR | Failed to load: class path resource
> [activemq.xml], reason: Error creating bean with name
> 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class path
> resource [activemq.xml]: Invoc
> ation of init method failed; nested exception is
> java.lang.NegativeArraySizeException |
> org.apache.activemq.xbean.XBeanBrokerFactory | main
> org.springframework.beans.factory.BeanCreationException: Error creating bean
> with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class
> path resource [activemq.xml]: Invocation of init method failed; nested e
> xception is java.lang.NegativeArraySizeException
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1420)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
> at
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
> at
> org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585)
> at
> org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
> at
> org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
> at
> org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)
> at
> org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)
> at
> org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:108)
> at
> org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:108)
> at
> org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:72)
> at
> org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)
> at
> org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)
> at
> org.apache.activemq.console.command.StartCommand.startBroker(StartCommand.java:115)
> at
> org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:74)
> at
> org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
> at
> org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:148)
> at
> org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
> at
> org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:90)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.activemq.console.Main.runTaskClass(Main.java:257)
> at org.apache.activemq.console.Main.main(Main.java:111)
> Caused by: java.lang.NegativeArraySizeException
> at
> org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:188)
> at
> org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:147)
> at org.apache.kahadb.page.Transaction.load(Transaction.java:412)
> at org.apache.kahadb.page.Transaction.load(Transaction.java:367)
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:262)
> at org.apache.kahadb.index.BTreeIndex.getRoot(BTreeIndex.java:174)
> at org.apache.kahadb.index.BTreeIndex.iterator(BTreeIndex.java:228)
> at
> org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller$1.execute(MessageDatabase.java:1654)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:769)
> at
> org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller.readPayload(MessageDatabase.java:1642)
>
> at
> org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller.readPayload(MessageDatabase.java:1627)
> at
> org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:201)
> at
> org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:147)
> at org.apache.kahadb.page.Transaction.load(Transaction.java:412)
> at org.apache.kahadb.page.Transaction.load(Transaction.java:367)
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:262)
> at org.apache.kahadb.index.BTreeIndex.getRoot(BTreeIndex.java:174)
> at org.apache.kahadb.index.BTreeIndex.iterator(BTreeIndex.java:228)
> at
> org.apache.activemq.store.kahadb.KahaDBStore$6.execute(KahaDBStore.java:943)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:769)
> at
> org.apache.activemq.store.kahadb.KahaDBStore.getDestinations(KahaDBStore.java:941)
> at
> org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.getDestinations(KahaDBPersistenceAdapter.java:122)
> at
> org.apache.activemq.broker.region.DestinationFactoryImpl.getDestinations(DestinationFactoryImpl.java:68)
> at
> org.apache.activemq.broker.BrokerService.startVirtualConsumerDestinations(BrokerService.java:2369)
> at
> org.apache.activemq.broker.BrokerService.startDestinations(BrokerService.java:2198)
> at
> org.apache.activemq.broker.BrokerService.start(BrokerService.java:527)
> at
> org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:60)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1544)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1485)
>
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1417)
> ... 28 more
> 11) after the above errors, ActiveMQ was dead; another attempt to start
> worked, yielding the following (also logged in previous attempt):
> 2012-05-29 21:58:01,317 | INFO | KahaDB is version 4 |
> org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 21:58:01,396 | INFO | Recovering from the journal ... |
> org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 21:58:27,601 | INFO | @2207:10939359, 100000 entries recovered ..
> | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 21:58:42,788 | INFO | @2222:10328575, 200000 entries recovered ..
> | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 21:58:59,234 | INFO | @2242:14360209, 300000 entries recovered ..
> | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 21:59:13,628 | INFO | @2254:1365404, 400000 entries recovered ..
> | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 21:59:27,466 | INFO | @2264:30387363, 500000 entries recovered ..
> | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 21:59:43,739 | INFO | @2281:28756842, 600000 entries recovered ..
> | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:00:13,389 | INFO | @2320:26490711, 700000 entries recovered ..
> | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:00:47,350 | INFO | @2371:21837918, 800000 entries recovered ..
> | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:01:17,752 | INFO | @2412:10702462, 900000 entries recovered ..
> | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:01:51,288 | INFO | @2465:32070937, 1000000 entries recovered
> .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:02:25,203 | INFO | @2519:21797589, 1100000 entries recovered
> .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:02:55,188 | INFO | @2564:13289299, 1200000 entries recovered
> .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:03:05,786 | INFO | @2572:23712519, 1300000 entries recovered
> .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:03:15,375 | INFO | @2578:22618226, 1400000 entries recovered
> .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:03:27,246 | INFO | @2585:31399538, 1500000 entries recovered
> .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:03:38,213 | INFO | @2593:7102179, 1600000 entries recovered ..
> | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:03:48,285 | INFO | @2600:11040885, 1700000 entries recovered
> .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:03:55,053 | INFO | @2602:28573821, 1800000 entries recovered
> .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:04:01,331 | INFO | @2603:32850602, 1900000 entries recovered
> .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:04:06,689 | INFO | @2605:3582539, 2000000 entries recovered ..
> | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:04:12,435 | INFO | @2606:7920235, 2100000 entries recovered ..
> | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:04:17,500 | INFO | @2607:12248010, 2200000 entries recovered
> .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:04:25,521 | INFO | @2617:28414362, 2300000 entries recovered
> .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:04:28,459 | INFO | Recovery replayed 2326036 operations from
> the journal in 387.094 seconds. |
> org.apache.activemq.store.kahadb.MessageDatabase | main
> 12) unfortunately, I confirmed that even after upgrade, kahadb continues to
> grow at about the same rate as before
> 13) I then applied the logging parameters as described in
> http://activemq.apache.org/why-do-kahadb-log-files-remain-after-cleanup.html.
> . With debugging spun up we see A LOT (I can’t get a count because log files
> rotate too fast, but in the 6 logs that remain I get 31,978 entries) of the
> following:
> 2012-05-29 23:18:41,925 | DEBUG | message not found in sequence id index:
> ID:<FQDN>-59596-1337475432262-0:35:1:1:1202 |
> org.apache.activemq.store.kahadb.MessageDatabase | main
> Google and a search on issues.apache.org reveal nothing on this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
