This morning I got a lot of these exceptions in our server log. What
could be the cause ? Any thoughts ?
It appears that once these start happening it cannot consume any more
messages off the queue even after a restart. I had to delete
activemq-data directory and restart, but this means we lost messages.
012-09-25 05:25:48,446 ERROR [BrokerService[localhost] Task-2008]
org.apache.activemq.broker.region.cursors.AbstractSto
reCursor.fillBatch(AbstractStoreCursor.java:262) -
QueueStorePrefetch1291856639 - Failed to fill batch
java.lang.NullPointerException
at
org.apache.kahadb.index.BTreeNode$BTreeIterator.findNextPage(BTreeNode.java:110)
at
org.apache.kahadb.index.BTreeNode$BTreeIterator.hasNext(BTreeNode.java:121)
at
org.apache.activemq.store.kahadb.MessageDatabase$MessageOrderIndex$MessageOrderIterator.hasNext(MessageDatabase.java:2344)
at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:491)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
at
org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
at
org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
at
org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
at
org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
at
org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:127)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Inspecting the code I see this:
while( current!=null ) {
if( nextIndex >= current.keys.length ) {
// we need to roll to the next leaf..
if( current.next >= 0 ) {
current = index.loadNode(tx, current.next, null);
assert !current.isBranch() : "Should have
linked to the next leaf node.";
nextIndex=0;
} else {
break;
}
} else {
nextEntry = new
KeyValueEntry(current.keys[nextIndex], current.values[nextIndex]);
nextIndex++;
break;
}
}
But looking at this, I don't see how can there be an NPE on the line
that says this:
nextEntry = new
KeyValueEntry(current.keys[nextIndex], current.values[nextIndex]);
(that happens to be line 110)…
Any help is greatly appreciated.
--
Regards,
Oleg Dulin
NYC Java Big Data Engineer
http://www.olegdulin.com/
