Hey all,
We're using ActiveMQ 4.0.2 on a Solaris 10 (sparc) machine to act as a
re-broadcaster
of messages from our production system (WebSphere MQ) for our developers.
Every once in a while (like, twice a day), I'm seeing errors like the
one below.
Under some circumstances, this appears to cause ActiveMQ to stop
recieving messages from
the producers which are mirroring messages from WebSphereMQ. At this
point, we get backlogs
of messages and nasty out-of-memory problems further upstream.
The entire ActiveMQ process is running on an NFSv3 volume, so I'm not
sure how derby likes
playing with that. We're process around 500k messages a day, between
1kb and 20mb.
Anyone seen this before? I haven't yet tried turning of peristance,
but I strongly suspect that it'll
make this go away (at the expense of losing messages, which is
something we don't want to do).
daniel
2006-11-21 11:38:04,298 [eckpoint Worker] ERROR
JournalPersistenceAdapter - Failed to checkpoint a message store:
edu.emory.mathcs.backport.java.util.concurrent.ExecutionException:
java.io.IOException: Not started.
edu.emory.mathcs.backport.java.util.concurrent.ExecutionException:
java.io.IOException: Not started.
at
edu.emory.mathcs.backport.java.util.concurrent.FutureTask.getResult(FutureTask.java:299)
at
edu.emory.mathcs.backport.java.util.concurrent.FutureTask.get(FutureTask.java:118)
at
org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:380)
at
org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:123)
at
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:88)
at
org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25)
at
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39)
Caused by: java.io.IOException: Not started.
at
org.apache.activemq.store.jdbc.TransactionContext.rollback(TransactionContext.java:165)
at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.rollbackTransaction(JDBCPersistenceAdapter.java:309)
at
org.apache.activemq.store.journal.JournalPersistenceAdapter.rollbackTransaction(JournalPersistenceAdapter.java:191)
at
org.apache.activemq.util.TransactionTemplate.run(TransactionTemplate.java:62)
at
org.apache.activemq.store.journal.JournalMessageStore.checkpoint(JournalMessageStore.java:247)
at
org.apache.activemq.store.journal.JournalTopicMessageStore.checkpoint(JournalTopicMessageStore.java:156)
at
org.apache.activemq.store.journal.JournalPersistenceAdapter$5.call(JournalPersistenceAdapter.java:367)
at
edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:176)
at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
2006-11-21 11:38:04,301 [eckpoint Worker] ERROR
JournalPersistenceAdapter - Failed to mark the Journal:
org.apache.activeio.journal.InvalidRecordLocationException: The location
is less than the last mark.
org.apache.activeio.journal.InvalidRecordLocationException: The location
is less than the last mark.
at
org.apache.activeio.journal.active.JournalImpl.setMark(JournalImpl.java:340)
at
org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:397)
at
org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:123)
at
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:88)
at
org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25)
at
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39)
