Radek Kraus created AMQ-9813:
--------------------------------
Summary: Incorrect QueueSize if Non-Persistent messages with TTL
is used
Key: AMQ-9813
URL: https://issues.apache.org/jira/browse/AMQ-9813
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.19.1, 6.2.0
Reporter: Radek Kraus
Sometimes, I detect a problem with incorrect QueueSize, when non-persitent
messages with TTL is used.
*UseCase/Symptoms*
* produce/consume non-persistent messages with defined TTL (expiration)
* producerFlowControl=false
* consumers "are slow", it means that some messages are expired (broker side
or client side)
* QueueSize sometimes reports non-zero value even when all messages are
consumed/expired
* Broker does not "push" next messages to consumers
** JMX method {{browseAsTable()}} does not provide messages
** JMX method {{purge()}} does not remove messages (QueueSize is still
non-zero, nothing is deleted)
* It seems like messages are expired, but QueueSize still reports non-zero
values
* after Broker restart QueueSize is reset to zero (expected for non-persitent
messages)
*Analyse*
* I guess, that problem was introduced by AMQ-5785 (by
[8b23e07|https://github.com/apache/activemq/commit/8b23e07], where deadlock was
solved)
* there is a valid AMQ-5785 comment without next reaction
* what happens in [8b23e07|https://github.com/apache/activemq/commit/8b23e07]
commit (nearly the same what is written in AMQ-5785 comment):
** contract of {{expireOldMessages()}} method was changed to don't invoke
{{discardExpiredMessage(MessageReference)}} (which is finally responsible to
"process expired message")
** the invocation of {{discardExpiredMessage(MessageReference)}} was moved out
of synchronized section (deadlock prevention), see {{onUsageChanged(Usage, int,
int)}} method
** but the *problem is that {{expireOldMessages()}} method is invoked from
next methods* ({{{}tryAddMessageLast(MessageReference, long){}}},
{{{}addMessageFirst(MessageReference){}}}), where
*{{discardExpiredMessage(MessageReference)}} method is no longer invoked*
(after change), *what it is problem from my point of view*
* IMHO "message expiration process" can be invoked:
** from {{onUsageChanged(Usage, int, int)}} method ({{{}UsageListener{}}}),
asynchronously triggered by Usage.setPercentUsage(int) (thread pool executor)
** or "directly" from {{tryAddMessageLast(MessageReference, long)}} (in case
when {{OrderedPendingList}} is used)
** it depends on situations (timings), which method is invoked "at first"
** when {{tryAddMessageLast(MessageReference, long)}} method is invoked, then
expired messages are only removed from {{{}memoryList{}}}, but
{{discardExpiredMessage(MessageReference)}} is never invoked (what it is the
problem)
*PR* (it will be added after I will get AMQ issue number)
* I added "missing" invocation of {{discardExpiredMessage(MessageReference)}}
method into {{{}tryAddMessageLast(MessageReference, long){}}},
{{addMessageFirst(MessageReference)}} methods
* I tried to use same "postponed" strategy (outside of synchronization) like
was already done in original commit (see {{{}onUsageChanged(Usage, int,
int){}}}) method
* I tried to write JUnit test, but it is complicated, because it depends on
timings (from my expiriences TTL must be very small to simulate the problem,
see {{{}FullDestinationMemoryMessageExpirationTest{}}})
Could you please check/analyze my findings?
In each case (IMHO) [8b23e07|https://github.com/apache/activemq/commit/8b23e07]
breaks previous contract of {{expireOldMessages()}} methods, which can caused
that expired messages are not "processed", when expiration is triggered by
{{{}tryAddMessageLast(MessageReference, long){}}},
{{addMessageFirst(MessageReference)}} methods.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
For further information, visit: https://activemq.apache.org/contact
