[
https://issues.apache.org/activemq/browse/AMQ-1795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=43520#action_43520
]
Gary Tully commented on AMQ-1795:
---------------------------------
Still having problems with this:
I find that the following change helps but is not bulit proof. I think it makes
sense to use the mark as the boundary point for deletion but on a slower
machine the MissingDataFileTest still shows the error stack trace. A multi core
machine does not show the problem.
Also, the test case only complets when the memory usage limit is omitted or
increased to 1024*1024 which may indicate a limitation of the test case.
Decreasing the memory usage caused the problem to show up earlier, which does
not now seem to be the case. So there is some progress.
The short of it is that collection of inUse file Ids is not sufficiently stable
for the cleanup method. I still do not know where the problem lies though. I
need another set of eyes.
Index: src/main/java/org/apache/activemq/store/amq/AMQPersistenceAdapter.java
===================================================================
--- src/main/java/org/apache/activemq/store/amq/AMQPersistenceAdapter.java
(revision 668601)
+++ src/main/java/org/apache/activemq/store/amq/AMQPersistenceAdapter.java
(working copy)
@@ -425,6 +425,7 @@
}
Integer lastDataFile = asyncDataManager.getCurrentDataFileId();
inProgress.add(lastDataFile);
+ lastDataFile = asyncDataManager.getMark().getDataFileId();
inProgress.addAll(referenceStoreAdapter.getReferenceFileIdsInUse());
Location lastActiveTx = transactionStore.checkpoint();
if (lastActiveTx != null) {
> in use data files removed from data store under load
> ----------------------------------------------------
>
> Key: AMQ-1795
> URL: https://issues.apache.org/activemq/browse/AMQ-1795
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: all
> Reporter: Gary Tully
> Assignee: Rob Davies
> Fix For: 5.2.0
>
> Attachments: AMQ-1795.test_and_partial_fix
>
>
> Under load, I see the error below.
> the problem is that
> org.apache.activemq.store.amq.AMQPersistenceAdapter.cleanup() does not
> capture all of the in use files.
> I have made some changes that improves the situation but there is still a
> window, that seems to be related to spooling messages when a memory limit is
> reached.
> 2008-06-12 14:55:16,689 [main ] INFO BrokerService
> - Using Persistence Adapter: AMQPersistenceAdapter(activemq-data\localhost)
> 2008-06-12 14:55:16,704 [main ] INFO AMQPersistenceAdapter
> - AMQStore starting using directory: activemq-data\localhost
> 2008-06-12 14:55:17,361 [main ] INFO AMQPersistenceAdapter
> - Journal deleted:
> 2008-06-12 14:55:17,361 [main ] INFO KahaStore
> - Kaha Store successfully deleted data directory
> activemq-data\localhost\kr-store\data
> 2008-06-12 14:55:17,392 [main ] INFO KahaStore
> - Kaha Store successfully deleted data directory
> activemq-data\localhost\kr-store\state
> 2008-06-12 14:55:17,392 [main ] INFO KahaStore
> - Kaha Store using data directory activemq-data\localhost\kr-store\state
> 2008-06-12 14:55:17,470 [main ] INFO AMQPersistenceAdapter
> - Active data files: []
> 2008-06-12 14:55:17,705 [main ] INFO BrokerService
> - ActiveMQ null JMS Message Broker (localhost) is starting
> 2008-06-12 14:55:17,705 [main ] INFO BrokerService
> - For help or more information please see: http://activemq.apache.org/
> 2008-06-12 14:55:17,814 [JMX connector ] INFO ManagementContext
> - JMX consoles can connect to
> service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
> 2008-06-12 14:55:17,924 [main ] INFO KahaStore
> - Kaha Store using data directory activemq-data\localhost\kr-store\data
> 2008-06-12 14:55:17,939 [main ] INFO TransportServerThreadSupport
> - Listening for connections at: tcp://salthill:61616
> 2008-06-12 14:55:17,939 [main ] INFO TransportConnector
> - Connector Default Started
> 2008-06-12 14:55:17,939 [main ] INFO BrokerService
> - ActiveMQ JMS Message Broker (localhost,
> ID:salthill-2790-1213278917752-0:0) started
> 2008-06-12 14:55:17,939 [main ] INFO MissingDataFileTest
> - Starting broker..
> 2008-06-12 14:55:58,516 [age Thread Pool] ERROR AsyncDataManager
> - Looking for key 551 but not found in fileMap: {1014=data-1014 number =
> 1014 , length = 1960 refCount = 6, 607=data-607 number = 607 , length = 1266
> refCount = 1, 928=data-928 number = 928 , .. .. 622 , length = 1352 refCount
> = 2, 328=data-328 number = 328 , length = 1266 refCount = 1, 256=data-256
> number = 256 , length = 1264 refCount = 1, 92=data-92 number = 92 , length =
> 1264 refCount = 1, 404=data-404 number = 404 , length = 1266 refCount = 1}
> 2008-06-12 14:55:58,563 [age Thread Pool] ERROR AbstractStoreCursor
> - Failed to fill batch
> java.io.IOException: Failed to read to journal for: offset = 0, file = 551,
> size = -1, type = 0. Reason: java.io.IOException: Could not locate data file
> data--551
> at
> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:33)
> at
> org.apache.activemq.store.amq.AMQPersistenceAdapter.createReadException(AMQPersistenceAdapter.java:635)
> at
> org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:521)
> at
> org.apache.activemq.store.amq.AMQMessageStore.getMessage(AMQMessageStore.java:432)
> at
> org.apache.activemq.store.amq.RecoveryListenerAdapter.recoverMessageReference(RecoveryListenerAdapter.java:54)
> at
> org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverReference(KahaReferenceStore.java:82)
> at
> org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverNextMessages(KahaReferenceStore.java:120)
> at
> org.apache.activemq.store.amq.AMQMessageStore.recoverNextMessages(AMQMessageStore.java:530)
> at
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:75)
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:188)
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.onUsageChanged(AbstractStoreCursor.java:157)
> at org.apache.activemq.usage.Usage$1.run(Usage.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
> at java.lang.Thread.run(Thread.java:595)
> Caused by: java.io.IOException: Could not locate data file data--551
> at
> org.apache.activemq.kaha.impl.async.AsyncDataManager.getDataFile(AsyncDataManager.java:303)
> at
> org.apache.activemq.kaha.impl.async.AsyncDataManager.read(AsyncDataManager.java:613)
> at
> org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:518)
> ... 12 more
> 2008-06-12 14:55:58,563 [age Thread Pool] ERROR AbstractStoreCursor
> - Failed to fill batch
> java.lang.RuntimeException: java.io.IOException: Failed to read to journal
> for: offset = 0, file = 551, size = -1, type = 0. Reason:
> java.io.IOException: Could not locate data file data--551
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:191)
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.onUsageChanged(AbstractStoreCursor.java:157)
> at org.apache.activemq.usage.Usage$1.run(Usage.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
> at java.lang.Thread.run(Thread.java:595)
> Caused by: java.io.IOException: Failed to read to journal for: offset = 0,
> file = 551, size = -1, type = 0. Reason: java.io.IOException: Could not
> locate data file data--551
> at
> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:33)
> at
> org.apache.activemq.store.amq.AMQPersistenceAdapter.createReadException(AMQPersistenceAdapter.java:635)
> at
> org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:521)
> at
> org.apache.activemq.store.amq.AMQMessageStore.getMessage(AMQMessageStore.java:432)
> at
> org.apache.activemq.store.amq.RecoveryListenerAdapter.recoverMessageReference(RecoveryListenerAdapter.java:54)
> at
> org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverReference(KahaReferenceStore.java:82)
> at
> org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverNextMessages(KahaReferenceStore.java:120)
> at
> org.apache.activemq.store.amq.AMQMessageStore.recoverNextMessages(AMQMessageStore.java:530)
> at
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:75)
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:188)
> ... 5 more
> Caused by: java.io.IOException: Could not locate data file data--551
> at
> org.apache.activemq.kaha.impl.async.AsyncDataManager.getDataFile(AsyncDataManager.java:303)
> at
> org.apache.activemq.kaha.impl.async.AsyncDataManager.read(AsyncDataManager.java:613)
> at
> org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:518)
> ... 12 more
> 2008-06-12 15:00:18,531 [main ] INFO BrokerService
> - ActiveMQ Message Broker (localhost, ID:salthill-2790-1213278917752-0:0) is
> shutting down
> 2008-06-12 15:00:20,438 [main ] INFO TransportConnector
> - Connector Default Stopped
> 2008-06-12 15:00:20,641 [main ] INFO BrokerService
> - ActiveMQ JMS Message Broker (localhost,
> ID:salthill-2790-1213278917752-0:0) stopped
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
