[jira] [Updated] (AMQ-7009) ActiveMQ stop delivering messages
[ https://issues.apache.org/jira/browse/AMQ-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nezih BEN FREDJ updated AMQ-7009: - Attachment: MemoryMessageStoreQueueCursorTest.java > ActiveMQ stop delivering messages > - > > Key: AMQ-7009 > URL: https://issues.apache.org/jira/browse/AMQ-7009 > Project: ActiveMQ > Issue Type: Bug >Affects Versions: 5.15.4 >Reporter: Nezih BEN FREDJ >Priority: Critical > Attachments: MemoryMessageStore.java, > MemoryMessageStoreQueueCursorTest.java, activemq.log > > > We have a problem with ActiveMQ 5.15.4, it stop deliver messages to clients. > This happens completely randomly and we are not able to give a test case that > reproduce the problem > Activemq is configured to use the « memoryPersistenceAdapter » > > When examining memory dumps, we identified incoherent situation in the class > MemoryMessageStore that can lead to stop delivering messages. When : > - MemoryMessageStore.recoverNextMessages(...) is called > - « lastBatchId » is not null > - « messageTable » does not contains any entry with « lastBatchId » as key > no messages are recovered > > We added logs to try to understand how it is possible to have a non null « > lastBatchId » and « messageTable » with no entry having « lastBatchId » as > key. > > We noticed that the method setBatch is called with a non null « messageId » > parameter, but no message with this id is inserted in « messageTable ». After > that, when the method MemoryMessageStore.recoverNextMessages(...) is called, > it does nothing and no messages are recoverd. And the system go in a what > looks like an endless loop. > > We are testing a temporary solution by resetting « lastBatchId » to null when > the incoherent situation is detected. > > Can you help us resolving the problem at the source and not just get around. > I joined modified source code of the class MemoryMessageStore and an extract > of the log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMQ-7009) ActiveMQ stop delivering messages
[ https://issues.apache.org/jira/browse/AMQ-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nezih BEN FREDJ updated AMQ-7009: - Attachment: (was: MemoryMessageStoreQueueCursor.java) > ActiveMQ stop delivering messages > - > > Key: AMQ-7009 > URL: https://issues.apache.org/jira/browse/AMQ-7009 > Project: ActiveMQ > Issue Type: Bug >Affects Versions: 5.15.4 >Reporter: Nezih BEN FREDJ >Priority: Critical > Attachments: MemoryMessageStore.java, > MemoryMessageStoreQueueCursorTest.java, activemq.log > > > We have a problem with ActiveMQ 5.15.4, it stop deliver messages to clients. > This happens completely randomly and we are not able to give a test case that > reproduce the problem > Activemq is configured to use the « memoryPersistenceAdapter » > > When examining memory dumps, we identified incoherent situation in the class > MemoryMessageStore that can lead to stop delivering messages. When : > - MemoryMessageStore.recoverNextMessages(...) is called > - « lastBatchId » is not null > - « messageTable » does not contains any entry with « lastBatchId » as key > no messages are recovered > > We added logs to try to understand how it is possible to have a non null « > lastBatchId » and « messageTable » with no entry having « lastBatchId » as > key. > > We noticed that the method setBatch is called with a non null « messageId » > parameter, but no message with this id is inserted in « messageTable ». After > that, when the method MemoryMessageStore.recoverNextMessages(...) is called, > it does nothing and no messages are recoverd. And the system go in a what > looks like an endless loop. > > We are testing a temporary solution by resetting « lastBatchId » to null when > the incoherent situation is detected. > > Can you help us resolving the problem at the source and not just get around. > I joined modified source code of the class MemoryMessageStore and an extract > of the log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMQ-7009) ActiveMQ stop delivering messages
[ https://issues.apache.org/jira/browse/AMQ-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558093#comment-16558093 ] Nezih BEN FREDJ commented on AMQ-7009: -- Explanation : when activating TRACE level logs for the class "AbstractStoreCursor", I noticed the next two lines in the log file : 2018-07-18 12:02:06,136 | TRACE | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@3a5f8f9a:retour.event.box4,batchResetNeeded=false,size=0,cacheEnabled=true,maxBatchSize:1,hasSpace:false,pendingCachedIds.size:0,lastSyncCachedId:ID:vhqve41.pprod.helios.cp-54155-1531902423423-1:10:2:1:282,lastSyncCachedId-seq:75,lastAsyncCachedId:null,lastAsyncCachedId-seq:null,store=org.apache.activemq.store.memory.MemoryMessageStore@349876a1 - disabling cache on add ID:vhqve41.pprod.helios.cp-54155-1531902423423-1:3:1:1:370 76 | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | ActiveMQ Transport: tcp:///174.4.111.217:35610@ 2018-07-18 12:02:06,136 | WARN | @@@NBF@@@ setBatch called with param messageId : (ID:vhqve41.pprod.helios.cp-54155-1531902423423-1:10:2:1:282). lastBatchId : (null) the new lastBatchId : (ID:vhqve41.pprod.helios.cp-54155-1531902423423-1:10:2:1:282) is NOT in messageTable | org.apache.activemq.store.memory.MemoryMessageStore | ActiveMQ Transport: tcp:///174.4.111.217:35610@ The second line shows that setBatch is called with a parameter "messageId" corresponding to a probably old message that is already removed from store. The first line shows that just befor calling setBatch, we had (hasSpace:false while size=0) which mean : there is no messages in the queue but there is no free space too !!!. It seems like messages are consumed but their space is not rendered or not yet rendered at that moment. > ActiveMQ stop delivering messages > - > > Key: AMQ-7009 > URL: https://issues.apache.org/jira/browse/AMQ-7009 > Project: ActiveMQ > Issue Type: Bug >Affects Versions: 5.15.4 >Reporter: Nezih BEN FREDJ >Priority: Critical > Attachments: MemoryMessageStore.java, > MemoryMessageStoreQueueCursor.java, activemq.log > > > We have a problem with ActiveMQ 5.15.4, it stop deliver messages to clients. > This happens completely randomly and we are not able to give a test case that > reproduce the problem > Activemq is configured to use the « memoryPersistenceAdapter » > > When examining memory dumps, we identified incoherent situation in the class > MemoryMessageStore that can lead to stop delivering messages. When : > - MemoryMessageStore.recoverNextMessages(...) is called > - « lastBatchId » is not null > - « messageTable » does not contains any entry with « lastBatchId » as key > no messages are recovered > > We added logs to try to understand how it is possible to have a non null « > lastBatchId » and « messageTable » with no entry having « lastBatchId » as > key. > > We noticed that the method setBatch is called with a non null « messageId » > parameter, but no message with this id is inserted in « messageTable ». After > that, when the method MemoryMessageStore.recoverNextMessages(...) is called, > it does nothing and no messages are recoverd. And the system go in a what > looks like an endless loop. > > We are testing a temporary solution by resetting « lastBatchId » to null when > the incoherent situation is detected. > > Can you help us resolving the problem at the source and not just get around. > I joined modified source code of the class MemoryMessageStore and an extract > of the log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMQ-7009) ActiveMQ stop delivering messages
[
https://issues.apache.org/jira/browse/AMQ-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555745#comment-16555745
]
Nezih BEN FREDJ commented on AMQ-7009:
--
In attachment, a unit test reproducing the
issue.[^MemoryMessageStoreQueueCursor.java].
To make the memoryMessageStore more robust we can modify methods setBatch and
recoverNextMessages as follow :
@Override
public void setBatch(MessageId messageId) {
synchronized (messageTable) {
lastBatchId = messageId;
if(!messageTable.containsKey(lastBatchId)) {
lastBatchId = null;
}
}
}
@Override
public void recoverNextMessages(int maxReturned, MessageRecoveryListener
listener) throws Exception {
synchronized (messageTable) {
boolean containsLastBatchId = messageTable.containsKey(lastBatchId);
if(!containsLastBatchId) {
lastBatchId = null;
}
boolean pastLackBatch = lastBatchId == null;
for (Map.Entry entry : messageTable.entrySet())
{
if (pastLackBatch) {
Object msg = entry.getValue();
lastBatchId = entry.getKey();
if (msg.getClass() == MessageId.class) {
listener.recoverMessageReference((MessageId) msg);
} else {
listener.recoverMessage((Message) msg);
}
} else {
pastLackBatch = entry.getKey().equals(lastBatchId);
}
}
}
}
> ActiveMQ stop delivering messages
> -
>
> Key: AMQ-7009
> URL: https://issues.apache.org/jira/browse/AMQ-7009
> Project: ActiveMQ
> Issue Type: Bug
>Affects Versions: 5.15.4
>Reporter: Nezih BEN FREDJ
>Priority: Critical
> Attachments: MemoryMessageStore.java,
> MemoryMessageStoreQueueCursor.java, activemq.log
>
>
> We have a problem with ActiveMQ 5.15.4, it stop deliver messages to clients.
> This happens completely randomly and we are not able to give a test case that
> reproduce the problem
> Activemq is configured to use the « memoryPersistenceAdapter »
>
> When examining memory dumps, we identified incoherent situation in the class
> MemoryMessageStore that can lead to stop delivering messages. When :
> - MemoryMessageStore.recoverNextMessages(...) is called
> - « lastBatchId » is not null
> - « messageTable » does not contains any entry with « lastBatchId » as key
> no messages are recovered
>
> We added logs to try to understand how it is possible to have a non null «
> lastBatchId » and « messageTable » with no entry having « lastBatchId » as
> key.
>
> We noticed that the method setBatch is called with a non null « messageId »
> parameter, but no message with this id is inserted in « messageTable ». After
> that, when the method MemoryMessageStore.recoverNextMessages(...) is called,
> it does nothing and no messages are recoverd. And the system go in a what
> looks like an endless loop.
>
> We are testing a temporary solution by resetting « lastBatchId » to null when
> the incoherent situation is detected.
>
> Can you help us resolving the problem at the source and not just get around.
> I joined modified source code of the class MemoryMessageStore and an extract
> of the log file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AMQ-7009) ActiveMQ stop delivering messages
[ https://issues.apache.org/jira/browse/AMQ-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nezih BEN FREDJ updated AMQ-7009: - Priority: Critical (was: Major) > ActiveMQ stop delivering messages > - > > Key: AMQ-7009 > URL: https://issues.apache.org/jira/browse/AMQ-7009 > Project: ActiveMQ > Issue Type: Bug >Affects Versions: 5.15.4 >Reporter: Nezih BEN FREDJ >Priority: Critical > Attachments: MemoryMessageStore.java, > MemoryMessageStoreQueueCursor.java, activemq.log > > > We have a problem with ActiveMQ 5.15.4, it stop deliver messages to clients. > This happens completely randomly and we are not able to give a test case that > reproduce the problem > Activemq is configured to use the « memoryPersistenceAdapter » > > When examining memory dumps, we identified incoherent situation in the class > MemoryMessageStore that can lead to stop delivering messages. When : > - MemoryMessageStore.recoverNextMessages(...) is called > - « lastBatchId » is not null > - « messageTable » does not contains any entry with « lastBatchId » as key > no messages are recovered > > We added logs to try to understand how it is possible to have a non null « > lastBatchId » and « messageTable » with no entry having « lastBatchId » as > key. > > We noticed that the method setBatch is called with a non null « messageId » > parameter, but no message with this id is inserted in « messageTable ». After > that, when the method MemoryMessageStore.recoverNextMessages(...) is called, > it does nothing and no messages are recoverd. And the system go in a what > looks like an endless loop. > > We are testing a temporary solution by resetting « lastBatchId » to null when > the incoherent situation is detected. > > Can you help us resolving the problem at the source and not just get around. > I joined modified source code of the class MemoryMessageStore and an extract > of the log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMQ-7009) ActiveMQ stop delivering messages
[ https://issues.apache.org/jira/browse/AMQ-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nezih BEN FREDJ updated AMQ-7009: - Attachment: MemoryMessageStoreQueueCursor.java > ActiveMQ stop delivering messages > - > > Key: AMQ-7009 > URL: https://issues.apache.org/jira/browse/AMQ-7009 > Project: ActiveMQ > Issue Type: Bug >Affects Versions: 5.15.4 >Reporter: Nezih BEN FREDJ >Priority: Major > Attachments: MemoryMessageStore.java, > MemoryMessageStoreQueueCursor.java, activemq.log > > > We have a problem with ActiveMQ 5.15.4, it stop deliver messages to clients. > This happens completely randomly and we are not able to give a test case that > reproduce the problem > Activemq is configured to use the « memoryPersistenceAdapter » > > When examining memory dumps, we identified incoherent situation in the class > MemoryMessageStore that can lead to stop delivering messages. When : > - MemoryMessageStore.recoverNextMessages(...) is called > - « lastBatchId » is not null > - « messageTable » does not contains any entry with « lastBatchId » as key > no messages are recovered > > We added logs to try to understand how it is possible to have a non null « > lastBatchId » and « messageTable » with no entry having « lastBatchId » as > key. > > We noticed that the method setBatch is called with a non null « messageId » > parameter, but no message with this id is inserted in « messageTable ». After > that, when the method MemoryMessageStore.recoverNextMessages(...) is called, > it does nothing and no messages are recoverd. And the system go in a what > looks like an endless loop. > > We are testing a temporary solution by resetting « lastBatchId » to null when > the incoherent situation is detected. > > Can you help us resolving the problem at the source and not just get around. > I joined modified source code of the class MemoryMessageStore and an extract > of the log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMQ-7009) ActiveMQ stop delivering messages
[ https://issues.apache.org/jira/browse/AMQ-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16546177#comment-16546177 ] Nezih BEN FREDJ commented on AMQ-7009: -- Hello [~tabish121], Unfortunately, we already spent a lot of time trying to understand why AMQ stop deliver messages, and we don't have resources and time to do more analysis (The problam has began 3 years ago). We should get by, by uncommenting the line 135 of the method "recoverNextMessages(...)". Thanks, Nezih > ActiveMQ stop delivering messages > - > > Key: AMQ-7009 > URL: https://issues.apache.org/jira/browse/AMQ-7009 > Project: ActiveMQ > Issue Type: Bug >Affects Versions: 5.15.4 >Reporter: Nezih BEN FREDJ >Priority: Major > Attachments: MemoryMessageStore.java, activemq.log > > > We have a problem with ActiveMQ 5.15.4, it stop deliver messages to clients. > This happens completely randomly and we are not able to give a test case that > reproduce the problem > Activemq is configured to use the « memoryPersistenceAdapter » > > When examining memory dumps, we identified incoherent situation in the class > MemoryMessageStore that can lead to stop delivering messages. When : > - MemoryMessageStore.recoverNextMessages(...) is called > - « lastBatchId » is not null > - « messageTable » does not contains any entry with « lastBatchId » as key > no messages are recovered > > We added logs to try to understand how it is possible to have a non null « > lastBatchId » and « messageTable » with no entry having « lastBatchId » as > key. > > We noticed that the method setBatch is called with a non null « messageId » > parameter, but no message with this id is inserted in « messageTable ». After > that, when the method MemoryMessageStore.recoverNextMessages(...) is called, > it does nothing and no messages are recoverd. And the system go in a what > looks like an endless loop. > > We are testing a temporary solution by resetting « lastBatchId » to null when > the incoherent situation is detected. > > Can you help us resolving the problem at the source and not just get around. > I joined modified source code of the class MemoryMessageStore and an extract > of the log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMQ-7009) ActiveMQ stop delivering messages
[ https://issues.apache.org/jira/browse/AMQ-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nezih BEN FREDJ updated AMQ-7009: - Attachment: (was: MemoryMessageStore.java) > ActiveMQ stop delivering messages > - > > Key: AMQ-7009 > URL: https://issues.apache.org/jira/browse/AMQ-7009 > Project: ActiveMQ > Issue Type: Bug >Affects Versions: 5.15.4 >Reporter: Nezih BEN FREDJ >Priority: Critical > Attachments: MemoryMessageStore.java, activemq.log > > > We have a problem with ActiveMQ 5.15.4, it stop deliver messages to clients. > This happens completely randomly and we are not able to give a test case that > reproduce the problem > Activemq is configured to use the « memoryPersistenceAdapter » > > When examining memory dumps, we identified incoherent situation in the class > MemoryMessageStore that can lead to stop delivering messages. When : > - MemoryMessageStore.recoverNextMessages(...) is called > - « lastBatchId » is not null > - « messageTable » does not contains any entry with « lastBatchId » as key > no messages are recovered > > We added logs to try to understand how it is possible to have a non null « > lastBatchId » and « messageTable » with no entry having « lastBatchId » as > key. > > We noticed that the method setBatch is called with a non null « messageId » > parameter, but no message with this id is inserted in « messageTable ». After > that, when the method MemoryMessageStore.recoverNextMessages(...) is called, > it does nothing and no messages are recoverd. And the system go in a what > looks like an endless loop. > > We are testing a temporary solution by resetting « lastBatchId » to null when > the incoherent situation is detected. > > Can you help us resolving the problem at the source and not just get around. > I joined modified source code of the class MemoryMessageStore and an extract > of the log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMQ-7009) ActiveMQ stop delivering messages
[ https://issues.apache.org/jira/browse/AMQ-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nezih BEN FREDJ updated AMQ-7009: - Attachment: activemq.log > ActiveMQ stop delivering messages > - > > Key: AMQ-7009 > URL: https://issues.apache.org/jira/browse/AMQ-7009 > Project: ActiveMQ > Issue Type: Bug >Affects Versions: 5.15.4 >Reporter: Nezih BEN FREDJ >Priority: Critical > Attachments: MemoryMessageStore.java, activemq.log > > > We have a problem with ActiveMQ 5.15.4, it stop deliver messages to clients. > This happens completely randomly and we are not able to give a test case that > reproduce the problem > Activemq is configured to use the « memoryPersistenceAdapter » > > When examining memory dumps, we identified incoherent situation in the class > MemoryMessageStore that can lead to stop delivering messages. When : > - MemoryMessageStore.recoverNextMessages(...) is called > - « lastBatchId » is not null > - « messageTable » does not contains any entry with « lastBatchId » as key > no messages are recovered > > We added logs to try to understand how it is possible to have a non null « > lastBatchId » and « messageTable » with no entry having « lastBatchId » as > key. > > We noticed that the method setBatch is called with a non null « messageId » > parameter, but no message with this id is inserted in « messageTable ». After > that, when the method MemoryMessageStore.recoverNextMessages(...) is called, > it does nothing and no messages are recoverd. And the system go in a what > looks like an endless loop. > > We are testing a temporary solution by resetting « lastBatchId » to null when > the incoherent situation is detected. > > Can you help us resolving the problem at the source and not just get around. > I joined modified source code of the class MemoryMessageStore and an extract > of the log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMQ-7009) ActiveMQ stop delivering messages
[ https://issues.apache.org/jira/browse/AMQ-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nezih BEN FREDJ updated AMQ-7009: - Attachment: (was: activemq.log) > ActiveMQ stop delivering messages > - > > Key: AMQ-7009 > URL: https://issues.apache.org/jira/browse/AMQ-7009 > Project: ActiveMQ > Issue Type: Bug >Affects Versions: 5.15.4 >Reporter: Nezih BEN FREDJ >Priority: Critical > Attachments: MemoryMessageStore.java, activemq.log > > > We have a problem with ActiveMQ 5.15.4, it stop deliver messages to clients. > This happens completely randomly and we are not able to give a test case that > reproduce the problem > Activemq is configured to use the « memoryPersistenceAdapter » > > When examining memory dumps, we identified incoherent situation in the class > MemoryMessageStore that can lead to stop delivering messages. When : > - MemoryMessageStore.recoverNextMessages(...) is called > - « lastBatchId » is not null > - « messageTable » does not contains any entry with « lastBatchId » as key > no messages are recovered > > We added logs to try to understand how it is possible to have a non null « > lastBatchId » and « messageTable » with no entry having « lastBatchId » as > key. > > We noticed that the method setBatch is called with a non null « messageId » > parameter, but no message with this id is inserted in « messageTable ». After > that, when the method MemoryMessageStore.recoverNextMessages(...) is called, > it does nothing and no messages are recoverd. And the system go in a what > looks like an endless loop. > > We are testing a temporary solution by resetting « lastBatchId » to null when > the incoherent situation is detected. > > Can you help us resolving the problem at the source and not just get around. > I joined modified source code of the class MemoryMessageStore and an extract > of the log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AMQ-7009) ActiveMQ stop delivering messages
[ https://issues.apache.org/jira/browse/AMQ-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545272#comment-16545272 ] Nezih BEN FREDJ edited comment on AMQ-7009 at 7/16/18 3:02 PM: --- We have done the upgrade to the latest version 5.15.4 and the problem does persist. So I updated the description of the problem above. In attachement, the new version of the class modified to generate some helpful logs. [^MemoryMessageStore.java] was (Author: nezih): We have done the upgrade to the latest version 5.15.4 and the problem does persist. In attachement, the new version of the class modified to generate some helpful logs. [^MemoryMessageStore.java] > ActiveMQ stop delivering messages > - > > Key: AMQ-7009 > URL: https://issues.apache.org/jira/browse/AMQ-7009 > Project: ActiveMQ > Issue Type: Bug >Affects Versions: 5.15.4 >Reporter: Nezih BEN FREDJ >Priority: Critical > Attachments: MemoryMessageStore.java, MemoryMessageStore.java, > activemq.log > > > We have a problem with ActiveMQ 5.15.4, it stop deliver messages to clients. > This happens completely randomly and we are not able to give a test case that > reproduce the problem > Activemq is configured to use the « memoryPersistenceAdapter » > > When examining memory dumps, we identified incoherent situation in the class > MemoryMessageStore that can lead to stop delivering messages. When : > - MemoryMessageStore.recoverNextMessages(...) is called > - « lastBatchId » is not null > - « messageTable » does not contains any entry with « lastBatchId » as key > no messages are recovered > > We added logs to try to understand how it is possible to have a non null « > lastBatchId » and « messageTable » with no entry having « lastBatchId » as > key. > > We noticed that the method setBatch is called with a non null « messageId » > parameter, but no message with this id is inserted in « messageTable ». After > that, when the method MemoryMessageStore.recoverNextMessages(...) is called, > it does nothing and no messages are recoverd. And the system go in a what > looks like an endless loop. > > We are testing a temporary solution by resetting « lastBatchId » to null when > the incoherent situation is detected. > > Can you help us resolving the problem at the source and not just get around. > I joined modified source code of the class MemoryMessageStore and an extract > of the log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMQ-7009) ActiveMQ stop delivering messages
[ https://issues.apache.org/jira/browse/AMQ-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nezih BEN FREDJ updated AMQ-7009: - Description: We have a problem with ActiveMQ 5.15.4, it stop deliver messages to clients. This happens completely randomly and we are not able to give a test case that reproduce the problem Activemq is configured to use the « memoryPersistenceAdapter » When examining memory dumps, we identified incoherent situation in the class MemoryMessageStore that can lead to stop delivering messages. When : - MemoryMessageStore.recoverNextMessages(...) is called - « lastBatchId » is not null - « messageTable » does not contains any entry with « lastBatchId » as key no messages are recovered We added logs to try to understand how it is possible to have a non null « lastBatchId » and « messageTable » with no entry having « lastBatchId » as key. We noticed that the method setBatch is called with a non null « messageId » parameter, but no message with this id is inserted in « messageTable ». After that, when the method MemoryMessageStore.recoverNextMessages(...) is called, it does nothing and no messages are recoverd. And the system go in a what looks like an endless loop. We are testing a temporary solution by resetting « lastBatchId » to null when the incoherent situation is detected. Can you help us resolving the problem at the source and not just get around. I joined modified source code of the class MemoryMessageStore and an extract of the log file. was: We have a problem with ActiveMQ 5.13.4, it stop deliver messages to clients. This happens completely randomly and we are not able to give a test case that reproduce the problem Activemq is configured to use the « memoryPersistenceAdapter » When examining memory dumps, we identified incoherent situation in the class MemoryMessageStore that can lead to stop delivering messages. When : - MemoryMessageStore.recoverNextMessages(...) is called - « lastBatchId » is not null - « messageTable » does not contains any entry with « lastBatchId » as key no messages are recovered We added logs to try to understand how it is possible to have a non null « lastBatchId » and « messageTable » with no entry having « lastBatchId » as key. We noticed that the method setBatch is called with a non null « messageId » parameter, but no message with this id is inserted in « messageTable ». After that, when the method MemoryMessageStore.recoverNextMessages(...) is called, it does nothing and no messages are recoverd. And the system go in a what looks like an endless loop. We are testing a temporary solution by resetting « lastBatchId » to null when the incoherent situation is detected. Can you help us resolving the problem at the source and not just get around. I joined modified source code of the class MemoryMessageStore and an extract of the log file. > ActiveMQ stop delivering messages > - > > Key: AMQ-7009 > URL: https://issues.apache.org/jira/browse/AMQ-7009 > Project: ActiveMQ > Issue Type: Bug >Affects Versions: 5.15.4 >Reporter: Nezih BEN FREDJ >Priority: Critical > Attachments: MemoryMessageStore.java, MemoryMessageStore.java, > activemq.log > > > We have a problem with ActiveMQ 5.15.4, it stop deliver messages to clients. > This happens completely randomly and we are not able to give a test case that > reproduce the problem > Activemq is configured to use the « memoryPersistenceAdapter » > > When examining memory dumps, we identified incoherent situation in the class > MemoryMessageStore that can lead to stop delivering messages. When : > - MemoryMessageStore.recoverNextMessages(...) is called > - « lastBatchId » is not null > - « messageTable » does not contains any entry with « lastBatchId » as key > no messages are recovered > > We added logs to try to understand how it is possible to have a non null « > lastBatchId » and « messageTable » with no entry having « lastBatchId » as > key. > > We noticed that the method setBatch is called with a non null « messageId » > parameter, but no message with this id is inserted in « messageTable ». After > that, when the method MemoryMessageStore.recoverNextMessages(...) is called, > it does nothing and no messages are recoverd. And the system go in a what > looks like an endless loop. > > We are testing a temporary solution by resetting « lastBatchId » to null when > the incoherent situation is detected. > > Can you help us resolving the problem at the source and not just get around. > I joined modified source code of the class MemoryMessageStore and an extract > of the log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMQ-7009) ActiveMQ stop delivering messages
[ https://issues.apache.org/jira/browse/AMQ-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nezih BEN FREDJ updated AMQ-7009: - Affects Version/s: (was: 5.13.4) 5.15.4 > ActiveMQ stop delivering messages > - > > Key: AMQ-7009 > URL: https://issues.apache.org/jira/browse/AMQ-7009 > Project: ActiveMQ > Issue Type: Bug >Affects Versions: 5.15.4 >Reporter: Nezih BEN FREDJ >Priority: Critical > Attachments: MemoryMessageStore.java, MemoryMessageStore.java, > activemq.log > > > We have a problem with ActiveMQ 5.13.4, it stop deliver messages to clients. > This happens completely randomly and we are not able to give a test case that > reproduce the problem > Activemq is configured to use the « memoryPersistenceAdapter » > > When examining memory dumps, we identified incoherent situation in the class > MemoryMessageStore that can lead to stop delivering messages. When : > - MemoryMessageStore.recoverNextMessages(...) is called > - « lastBatchId » is not null > - « messageTable » does not contains any entry with « lastBatchId » as key > no messages are recovered > > We added logs to try to understand how it is possible to have a non null « > lastBatchId » and « messageTable » with no entry having « lastBatchId » as > key. > > We noticed that the method setBatch is called with a non null « messageId » > parameter, but no message with this id is inserted in « messageTable ». After > that, when the method MemoryMessageStore.recoverNextMessages(...) is called, > it does nothing and no messages are recoverd. And the system go in a what > looks like an endless loop. > > We are testing a temporary solution by resetting « lastBatchId » to null when > the incoherent situation is detected. > > Can you help us resolving the problem at the source and not just get around. > I joined modified source code of the class MemoryMessageStore and an extract > of the log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMQ-7009) ActiveMQ stop delivering messages
[ https://issues.apache.org/jira/browse/AMQ-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nezih BEN FREDJ updated AMQ-7009: - Priority: Critical (was: Major) > ActiveMQ stop delivering messages > - > > Key: AMQ-7009 > URL: https://issues.apache.org/jira/browse/AMQ-7009 > Project: ActiveMQ > Issue Type: Bug >Affects Versions: 5.13.4 >Reporter: Nezih BEN FREDJ >Priority: Critical > Attachments: MemoryMessageStore.java, MemoryMessageStore.java, > activemq.log > > > We have a problem with ActiveMQ 5.13.4, it stop deliver messages to clients. > This happens completely randomly and we are not able to give a test case that > reproduce the problem > Activemq is configured to use the « memoryPersistenceAdapter » > > When examining memory dumps, we identified incoherent situation in the class > MemoryMessageStore that can lead to stop delivering messages. When : > - MemoryMessageStore.recoverNextMessages(...) is called > - « lastBatchId » is not null > - « messageTable » does not contains any entry with « lastBatchId » as key > no messages are recovered > > We added logs to try to understand how it is possible to have a non null « > lastBatchId » and « messageTable » with no entry having « lastBatchId » as > key. > > We noticed that the method setBatch is called with a non null « messageId » > parameter, but no message with this id is inserted in « messageTable ». After > that, when the method MemoryMessageStore.recoverNextMessages(...) is called, > it does nothing and no messages are recoverd. And the system go in a what > looks like an endless loop. > > We are testing a temporary solution by resetting « lastBatchId » to null when > the incoherent situation is detected. > > Can you help us resolving the problem at the source and not just get around. > I joined modified source code of the class MemoryMessageStore and an extract > of the log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AMQ-7009) ActiveMQ stop delivering messages
[ https://issues.apache.org/jira/browse/AMQ-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545272#comment-16545272 ] Nezih BEN FREDJ edited comment on AMQ-7009 at 7/16/18 2:45 PM: --- We have done the upgrade to the latest version 5.15.4 and the problem does persist. In attachement, the new version of the class modified to generate some helpful logs. [^MemoryMessageStore.java] was (Author: nezih): We have done the upgrade to the latest version 5.15.4 and the problem do persist. In attachement, the new version of the class modified to generate some helpful logs. [^MemoryMessageStore.java] > ActiveMQ stop delivering messages > - > > Key: AMQ-7009 > URL: https://issues.apache.org/jira/browse/AMQ-7009 > Project: ActiveMQ > Issue Type: Bug >Affects Versions: 5.13.4 >Reporter: Nezih BEN FREDJ >Priority: Major > Attachments: MemoryMessageStore.java, MemoryMessageStore.java, > activemq.log > > > We have a problem with ActiveMQ 5.13.4, it stop deliver messages to clients. > This happens completely randomly and we are not able to give a test case that > reproduce the problem > Activemq is configured to use the « memoryPersistenceAdapter » > > When examining memory dumps, we identified incoherent situation in the class > MemoryMessageStore that can lead to stop delivering messages. When : > - MemoryMessageStore.recoverNextMessages(...) is called > - « lastBatchId » is not null > - « messageTable » does not contains any entry with « lastBatchId » as key > no messages are recovered > > We added logs to try to understand how it is possible to have a non null « > lastBatchId » and « messageTable » with no entry having « lastBatchId » as > key. > > We noticed that the method setBatch is called with a non null « messageId » > parameter, but no message with this id is inserted in « messageTable ». After > that, when the method MemoryMessageStore.recoverNextMessages(...) is called, > it does nothing and no messages are recoverd. And the system go in a what > looks like an endless loop. > > We are testing a temporary solution by resetting « lastBatchId » to null when > the incoherent situation is detected. > > Can you help us resolving the problem at the source and not just get around. > I joined modified source code of the class MemoryMessageStore and an extract > of the log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMQ-7009) ActiveMQ stop delivering messages
[ https://issues.apache.org/jira/browse/AMQ-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545272#comment-16545272 ] Nezih BEN FREDJ commented on AMQ-7009: -- We have done the upgrade to the latest version 5.15.4 and the problem do persist. In attachement, the new version of the class modified to generate some helpful logs. [^MemoryMessageStore.java] > ActiveMQ stop delivering messages > - > > Key: AMQ-7009 > URL: https://issues.apache.org/jira/browse/AMQ-7009 > Project: ActiveMQ > Issue Type: Bug >Affects Versions: 5.13.4 >Reporter: Nezih BEN FREDJ >Priority: Major > Attachments: MemoryMessageStore.java, MemoryMessageStore.java, > activemq.log > > > We have a problem with ActiveMQ 5.13.4, it stop deliver messages to clients. > This happens completely randomly and we are not able to give a test case that > reproduce the problem > Activemq is configured to use the « memoryPersistenceAdapter » > > When examining memory dumps, we identified incoherent situation in the class > MemoryMessageStore that can lead to stop delivering messages. When : > - MemoryMessageStore.recoverNextMessages(...) is called > - « lastBatchId » is not null > - « messageTable » does not contains any entry with « lastBatchId » as key > no messages are recovered > > We added logs to try to understand how it is possible to have a non null « > lastBatchId » and « messageTable » with no entry having « lastBatchId » as > key. > > We noticed that the method setBatch is called with a non null « messageId » > parameter, but no message with this id is inserted in « messageTable ». After > that, when the method MemoryMessageStore.recoverNextMessages(...) is called, > it does nothing and no messages are recoverd. And the system go in a what > looks like an endless loop. > > We are testing a temporary solution by resetting « lastBatchId » to null when > the incoherent situation is detected. > > Can you help us resolving the problem at the source and not just get around. > I joined modified source code of the class MemoryMessageStore and an extract > of the log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMQ-7009) ActiveMQ stop delivering messages
[ https://issues.apache.org/jira/browse/AMQ-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nezih BEN FREDJ updated AMQ-7009: - Attachment: MemoryMessageStore.java > ActiveMQ stop delivering messages > - > > Key: AMQ-7009 > URL: https://issues.apache.org/jira/browse/AMQ-7009 > Project: ActiveMQ > Issue Type: Bug >Affects Versions: 5.13.4 >Reporter: Nezih BEN FREDJ >Priority: Major > Attachments: MemoryMessageStore.java, MemoryMessageStore.java, > activemq.log > > > We have a problem with ActiveMQ 5.13.4, it stop deliver messages to clients. > This happens completely randomly and we are not able to give a test case that > reproduce the problem > Activemq is configured to use the « memoryPersistenceAdapter » > > When examining memory dumps, we identified incoherent situation in the class > MemoryMessageStore that can lead to stop delivering messages. When : > - MemoryMessageStore.recoverNextMessages(...) is called > - « lastBatchId » is not null > - « messageTable » does not contains any entry with « lastBatchId » as key > no messages are recovered > > We added logs to try to understand how it is possible to have a non null « > lastBatchId » and « messageTable » with no entry having « lastBatchId » as > key. > > We noticed that the method setBatch is called with a non null « messageId » > parameter, but no message with this id is inserted in « messageTable ». After > that, when the method MemoryMessageStore.recoverNextMessages(...) is called, > it does nothing and no messages are recoverd. And the system go in a what > looks like an endless loop. > > We are testing a temporary solution by resetting « lastBatchId » to null when > the incoherent situation is detected. > > Can you help us resolving the problem at the source and not just get around. > I joined modified source code of the class MemoryMessageStore and an extract > of the log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMQ-7009) ActiveMQ stop delivering messages
[ https://issues.apache.org/jira/browse/AMQ-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nezih BEN FREDJ updated AMQ-7009: - Priority: Critical (was: Major) > ActiveMQ stop delivering messages > - > > Key: AMQ-7009 > URL: https://issues.apache.org/jira/browse/AMQ-7009 > Project: ActiveMQ > Issue Type: Bug >Affects Versions: 5.13.4 >Reporter: Nezih BEN FREDJ >Priority: Critical > Attachments: MemoryMessageStore.java, activemq.log > > > We have a problem with ActiveMQ 5.13.4, it stop deliver messages to clients. > This happens completely randomly and we are not able to give a test case that > reproduce the problem > Activemq is configured to use the « memoryPersistenceAdapter » > > When examining memory dumps, we identified incoherent situation in the class > MemoryMessageStore that can lead to stop delivering messages. When : > - MemoryMessageStore.recoverNextMessages(...) is called > - « lastBatchId » is not null > - « messageTable » does not contains any entry with « lastBatchId » as key > no messages are recovered > > We added logs to try to understand how it is possible to have a non null « > lastBatchId » and « messageTable » with no entry having « lastBatchId » as > key. > > We noticed that the method setBatch is called with a non null « messageId » > parameter, but no message with this id is inserted in « messageTable ». After > that, when the method MemoryMessageStore.recoverNextMessages(...) is called, > it does nothing and no messages are recoverd. And the system go in a what > looks like an endless loop. > > We are testing a temporary solution by resetting « lastBatchId » to null when > the incoherent situation is detected. > > Can you help us resolving the problem at the source and not just get around. > I joined modified source code of the class MemoryMessageStore and an extract > of the log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AMQ-7009) ActiveMQ stop delivering messages
Nezih BEN FREDJ created AMQ-7009: Summary: ActiveMQ stop delivering messages Key: AMQ-7009 URL: https://issues.apache.org/jira/browse/AMQ-7009 Project: ActiveMQ Issue Type: Bug Affects Versions: 5.13.4 Reporter: Nezih BEN FREDJ Attachments: MemoryMessageStore.java, activemq.log We have a problem with ActiveMQ 5.13.4, it stop deliver messages to clients. This happens completely randomly and we are not able to give a test case that reproduce the problem Activemq is configured to use the « memoryPersistenceAdapter » When examining memory dumps, we identified incoherent situation in the class MemoryMessageStore that can lead to stop delivering messages. When : - MemoryMessageStore.recoverNextMessages(...) is called - « lastBatchId » is not null - « messageTable » does not contains any entry with « lastBatchId » as key no messages are recovered We added logs to try to understand how it is possible to have a non null « lastBatchId » and « messageTable » with no entry having « lastBatchId » as key. We noticed that the method setBatch is called with a non null « messageId » parameter, but no message with this id is inserted in « messageTable ». After that, when the method MemoryMessageStore.recoverNextMessages(...) is called, it does nothing and no messages are recoverd. And the system go in a what looks like an endless loop. We are testing a temporary solution by resetting « lastBatchId » to null when the incoherent situation is detected. Can you help us resolving the problem at the source and not just get around. I joined modified source code of the class MemoryMessageStore and an extract of the log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
