[
https://issues.apache.org/activemq/browse/AMQ-3051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Keith Brown updated AMQ-3051:
-----------------------------
Description:
We are consistently seeing the following exception in our Production
environment. At the current time we are seeing over 200K messages on the queue
and no consumers can consume any messages. Due to issues with the kahadb not
deleting files, we are using the AMQStore for persistence. We are using a NFS
drive for the output of our persistent store.
The process cannot then be restarted as it will fail trying to load back in the
file store. The messages stored in the queue are critical, a way to recover
these messages is important also.
2010-11-24 08:20:44,015 | ERROR | Failed to page in more queue messages |
org.apache.activemq.broker.region.Queue |
Queue:Consumer.CaseManagement.VirtualTopic.intralinks.eventService
java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException:
Failed to read to journal for: offset = 168158, file = 17376, size = -1, type =
0. Reason: java.io.IOException: Could not locate data file data-17376
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
at
org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1681)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1845)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1432)
at
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
at
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
Caused by: java.lang.RuntimeException: java.io.IOException: Failed to read to
journal for: offset = 168158, file = 17376, size = -1, type = 0. Reason:
java.io.IOException: Could not locate data file data-17376
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:261)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:110)
... 6 more
Caused by: java.io.IOException: Failed to read to journal for: offset = 168158,
file = 17376, size = -1, type = 0. Reason: java.io.IOException: Could not
locate data file data-17376
at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:33)
at
org.apache.activemq.store.amq.AMQPersistenceAdapter.createReadException(AMQPersistenceAdapter.java:680)
at
org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:564)
at
org.apache.activemq.store.amq.AMQMessageStore.getMessage(AMQMessageStore.java:444)
at
org.apache.activemq.store.amq.RecoveryListenerAdapter.recoverMessageReference(RecoveryListenerAdapter.java:59)
at
org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverReference(KahaReferenceStore.java:89)
at
org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverNextMessages(KahaReferenceStore.java:127)
at
org.apache.activemq.store.amq.AMQMessageStore.recoverNextMessages(AMQMessageStore.java:537)
at
org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:258)
... 7 more
Caused by: java.io.IOException: Could not locate data file data-17376
at
org.apache.activemq.kaha.impl.async.AsyncDataManager.getDataFile(AsyncDataManager.java:315)
at
org.apache.activemq.kaha.impl.async.AsyncDataManager.read(AsyncDataManager.java:637)
at
org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:561)
... 14 more
was:
We are consistently seeing the following exception in our Production
environment. At the current time we are seeing over 200K messages on the queue
and no consumers can consume any messages. Due to issues with the kahadb not
deleting files, we are using the AMQStore for persistence. We are using a NFS
drive for the output of our persistent store.
2010-11-24 08:20:44,015 | ERROR | Failed to page in more queue messages |
org.apache.activemq.broker.region.Queue |
Queue:Consumer.CaseManagement.VirtualTopic.intralinks.eventService
java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException:
Failed to read to journal for: offset = 168158, file = 17376, size = -1, type =
0. Reason: java.io.IOException: Could not locate data file data-17376
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
at
org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1681)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1845)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1432)
at
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
at
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
Caused by: java.lang.RuntimeException: java.io.IOException: Failed to read to
journal for: offset = 168158, file = 17376, size = -1, type = 0. Reason:
java.io.IOException: Could not locate data file data-17376
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:261)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:110)
... 6 more
Caused by: java.io.IOException: Failed to read to journal for: offset = 168158,
file = 17376, size = -1, type = 0. Reason: java.io.IOException: Could not
locate data file data-17376
at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:33)
at
org.apache.activemq.store.amq.AMQPersistenceAdapter.createReadException(AMQPersistenceAdapter.java:680)
at
org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:564)
at
org.apache.activemq.store.amq.AMQMessageStore.getMessage(AMQMessageStore.java:444)
at
org.apache.activemq.store.amq.RecoveryListenerAdapter.recoverMessageReference(RecoveryListenerAdapter.java:59)
at
org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverReference(KahaReferenceStore.java:89)
at
org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverNextMessages(KahaReferenceStore.java:127)
at
org.apache.activemq.store.amq.AMQMessageStore.recoverNextMessages(AMQMessageStore.java:537)
at
org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:258)
... 7 more
Caused by: java.io.IOException: Could not locate data file data-17376
at
org.apache.activemq.kaha.impl.async.AsyncDataManager.getDataFile(AsyncDataManager.java:315)
at
org.apache.activemq.kaha.impl.async.AsyncDataManager.read(AsyncDataManager.java:637)
at
org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:561)
... 14 more
> Failed to read journal - Could not locate data file
> ---------------------------------------------------
>
> Key: AMQ-3051
> URL: https://issues.apache.org/activemq/browse/AMQ-3051
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.1
> Environment: Solaris 10 running on Sparc
> Reporter: Keith Brown
> Priority: Blocker
> Attachments: activemq-master-a.xml
>
>
> We are consistently seeing the following exception in our Production
> environment. At the current time we are seeing over 200K messages on the
> queue and no consumers can consume any messages. Due to issues with the
> kahadb not deleting files, we are using the AMQStore for persistence. We are
> using a NFS drive for the output of our persistent store.
> The process cannot then be restarted as it will fail trying to load back in
> the file store. The messages stored in the queue are critical, a way to
> recover these messages is important also.
> 2010-11-24 08:20:44,015 | ERROR | Failed to page in more queue messages |
> org.apache.activemq.broker.region.Queue |
> Queue:Consumer.CaseManagement.VirtualTopic.intralinks.eventService
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException:
> Failed to read to journal for: offset = 168158, file = 17376, size = -1, type
> = 0. Reason: java.io.IOException: Could not locate data file data-17376
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
> at
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1681)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1845)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1432)
> at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
> at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> Caused by: java.lang.RuntimeException: java.io.IOException: Failed to read to
> journal for: offset = 168158, file = 17376, size = -1, type = 0. Reason:
> java.io.IOException: Could not locate data file data-17376
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:261)
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:110)
> ... 6 more
> Caused by: java.io.IOException: Failed to read to journal for: offset =
> 168158, file = 17376, size = -1, type = 0. Reason: java.io.IOException: Could
> not locate data file data-17376
> at
> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:33)
> at
> org.apache.activemq.store.amq.AMQPersistenceAdapter.createReadException(AMQPersistenceAdapter.java:680)
> at
> org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:564)
> at
> org.apache.activemq.store.amq.AMQMessageStore.getMessage(AMQMessageStore.java:444)
> at
> org.apache.activemq.store.amq.RecoveryListenerAdapter.recoverMessageReference(RecoveryListenerAdapter.java:59)
> at
> org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverReference(KahaReferenceStore.java:89)
> at
> org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverNextMessages(KahaReferenceStore.java:127)
> at
> org.apache.activemq.store.amq.AMQMessageStore.recoverNextMessages(AMQMessageStore.java:537)
> at
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:258)
> ... 7 more
> Caused by: java.io.IOException: Could not locate data file data-17376
> at
> org.apache.activemq.kaha.impl.async.AsyncDataManager.getDataFile(AsyncDataManager.java:315)
> at
> org.apache.activemq.kaha.impl.async.AsyncDataManager.read(AsyncDataManager.java:637)
> at
> org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:561)
> ... 14 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
