[
https://issues.apache.org/jira/browse/ARTEMIS-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17494212#comment-17494212
]
ASF subversion and git services commented on ARTEMIS-3647:
----------------------------------------------------------
Commit 8a9f326b25e5feaedcb1393f8d2a07a9d9432c48 in activemq-artemis's branch
refs/heads/main from AntonRoskvist
[ https://gitbox.apache.org/repos/asf?p=activemq-artemis.git;h=8a9f326 ]
ARTEMIS-3647 - OpenWire, remove rolledbackMessageRef on Ack
> rolledbackMessageRefs can grow until OOM with OpenWire clients
> --------------------------------------------------------------
>
> Key: ARTEMIS-3647
> URL: https://issues.apache.org/jira/browse/ARTEMIS-3647
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Reporter: Anton Roskvist
> Priority: Major
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> {color:#1d1c1d}In my use case I have quite a few long lived OpenWire
> consumers. I noticed that over time the heap usage increases. Looking through
> a heap dump, I found that memory is held in "rolledbackMessageRefs". In one
> case holding as much as 1.6GB of data with 0 messages on queue.
> Disconnecting the consumer and then reconnecting released the memory.
> Clients are running Spring with transactions. The clients affected by this
> have some small issue receiving messages such that some of them are retried a
> couple of times before getting processed properly.
> I suspect that "rolledbackMessageRefs"{color} are not getting cleared with
> the message ref once it's finally processed for some reason.
> {-}{color:#1d1c1d}I have not found a way to reproduce this yet and it happens
> over several days.
> {color}{-}{color:#1d1c1d}UPDATE: I can easily reproduce this by setting up a
> standalone Artemis broker with "out-of-the-box"-configuration and using these
> tools:{color} -- [https://github.com/erik-wramner/JmsTools] (AmqJmsConsumer
> and optionally AmqJmsProducer)
> 1. Start the broker
> 2. Send 100k messages to "queue://TEST"
> {code:java}
> # java -jar JmsTools/shaded-jars/AmqJmsProducer.jar -url
> "tcp://localhost:61616" -user USER -pw PASSWORD -queue TEST -count
> 100000{code}
> 3. Receive one more message than produced and do a rollback on 30% of them
> (unrealistic, but means this can be done in minutes instead of days. Receive
> one more to ensure consumer stays live)
> {code:java}
> # java -jar JmsTools/shaded-jars/AmqJmsConsumer.jar -url
> "tcp://localhost:61616?jms.prefetchPolicy.all=100&jms.nonBlockingRedelivery=true"
> -user USER -pw PASSWORD -queue TEST -count 100001 -rollback 30{code}
> 4. Wait until no more messages are left on "queue://TEST" (a few might be on
> DLQ but that's okay)
> 5. Get a heap dump with the consumer still connected
> {code:java}
> # jmap -dump:format=b,file=dump.hprof Artemis_PID{code}
> 6. Running "Leak suspects" with MAT will show a (relatively) large amount of
> memory held by {color:#1d1c1d}"rolledbackMessageRefs"{color} for the consumer
> connected to queue://TEST
> The consumer is run with "jms.nonBlockingRedelivery=true" to speed things up,
> though it should not be strictly needed.
> As an added bonus this also shows that the prefetch limit
> "jms.prefetchPolicy.all=100" is not respected while messages are in the
> redelivery process which can easily be seen in the consoles
> "Attributes"-section for the queue. This is also true for the default
> prefetch value of 1000.
> Br,
> Anton
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
