[
https://issues.apache.org/jira/browse/AMQ-9654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17957559#comment-17957559
]
Matt Pavlovich edited comment on AMQ-9654 at 11/20/25 6:34 PM:
---------------------------------------------------------------
I'm narrowing down a candidate to this code block:
Observations
1. A log or counter in an else {} could assist in providing verification that
this is the area with the issue.
2. Could a consistency error with dropIfLive also result in the duplicate from
store? If the pagedInMessages reference is not getting removed on dropMessages,
that would seem to make sense.
{noformat}
private void dropMessage(ConnectionContext context, QueueMessageReference
reference) {
//use dropIfLive so we only process the statistics at most one time
if (reference.dropIfLive()) {
destinationStatistics.getDequeues().increment();
destinationStatistics.getMessages().decrement();
final var tmpMessageFlowStats =
destinationStatistics.getMessageFlowStats();
if(tmpMessageFlowStats != null) {
Message tmpMessage = reference.getMessage();
tmpMessageFlowStats.dequeueStats(context.getClientId(),
tmpMessage.getMessageId().toString(), tmpMessage.getTimestamp(),
tmpMessage.getBrokerInTime(), tmpMessage.getBrokerOutTime());
}
if(isAdvancedNetworkStatisticsEnabled() && context.getConnection()
!= null && context.getConnection().isNetworkConnection()) {
destinationStatistics.getNetworkDequeues().increment();
}
pagedInMessagesLock.writeLock().lock();
try {
pagedInMessages.remove(reference);
} finally {
pagedInMessagesLock.writeLock().unlock();
}
}
}
{noformat}
Consider improved locking semantics?
ref:
https://github.com/apache/activemq/blob/main/activemq-broker/src/main/java/org/apache/activemq/broker/region/IndirectMessageReference.java
Update:
Message referenceCount increment/decrement is managed with a synchronized(this)
block.
was (Author: mattrpav):
I'm narrowing down a candidate to this code block:
Observations
1. A log or counter in an else {} could assist in providing verification that
this is the area with the issue.
2. Could a consistency error with dropIfLive also result in the duplicate from
store? If the pagedInMessages reference is not getting removed on dropMessages,
that would seem to make sense.
{noformat}
private void dropMessage(ConnectionContext context, QueueMessageReference
reference) {
//use dropIfLive so we only process the statistics at most one time
if (reference.dropIfLive()) {
destinationStatistics.getDequeues().increment();
destinationStatistics.getMessages().decrement();
final var tmpMessageFlowStats =
destinationStatistics.getMessageFlowStats();
if(tmpMessageFlowStats != null) {
Message tmpMessage = reference.getMessage();
tmpMessageFlowStats.dequeueStats(context.getClientId(),
tmpMessage.getMessageId().toString(), tmpMessage.getTimestamp(),
tmpMessage.getBrokerInTime(), tmpMessage.getBrokerOutTime());
}
if(isAdvancedNetworkStatisticsEnabled() && context.getConnection()
!= null && context.getConnection().isNetworkConnection()) {
destinationStatistics.getNetworkDequeues().increment();
}
pagedInMessagesLock.writeLock().lock();
try {
pagedInMessages.remove(reference);
} finally {
pagedInMessagesLock.writeLock().unlock();
}
}
}
{noformat}
Consider improved locking semantics?
ref:
https://github.com/apache/activemq/blob/main/activemq-broker/src/main/java/org/apache/activemq/broker/region/IndirectMessageReference.java
> Queue size incorrect even when queue is empty
> ---------------------------------------------
>
> Key: AMQ-9654
> URL: https://issues.apache.org/jira/browse/AMQ-9654
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.17.6
> Reporter: Matt Pavlovich
> Assignee: Matt Pavlovich
> Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Environment:
> Linux OS
> ActiveMQ 5.17.6 using network of brokers (2-3 brokers per cluster)
> Cloud deployment with high frequency of network inconsistency
> 1. Queue size may reflect a small number of messages are still pending on the
> queue (1 or 2), but there are no messages present.
> 2. Broker restart corrects the messages
> 3. No clear indications of exception or error in broker or client-side logs
> 4. Application consuming messages using multiple consumers and
> single-message-per-transaction using local transaction ack mode. (Camel JMS,
> Spring, etc)
> 5. Across dozens of brokers processing 100's of millions of messages per
> year, this will occur a handful of times in all.
> Suspect queue size counter consistency issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
For further information, visit: https://activemq.apache.org/contact
