Hi list,
We have recently upgraded the ActiveMQ broker on one of our test servers
from version 5.9 to 5.12.1. After the upgrade, we get lots of
OutOfMemoryErrors from the broker and the broker becomes unusable.
We increased the JVM memory from 128 MB (which is sufficient on the 5.9
production server) to 256 MB, but that did not help. We then changed the
pending queue policy to "fileQueueCursor" and the pending (durable)
subscriber policy to "fileCursor" and "fileDurableSubscriberCursor",
respectively, to let the messages be written to disk instead of kept in
memory. This seemed to improve stability, but after a while the OOM errors
returned.
Since we saw lots of messages in the DLQ (which were never retrieved from
there), we disabled the DLQ by these configuration lines:
<deadLetterStrategy>
<discarding/>
</deadLetterStrategy>
We also tried to reduce the memory usage using the following lines:
<memoryUsage>
<memoryUsage percentOfJvmHeap="..."/>
</memoryUsage>
We tried different settings for the "percentOfJvmHeap", but did not see any
effect.
In our setup, we have a pretty fast producer and a slow consumer, so it is
quite clear that messages pile up in the broker. However, the messages are
small, and when I checked the queues and topics using a JMX connection, the
total memory consumption is far below the assigned JVM memory.
Maybe I totally misunderstood how the broker works: I always thought that
the broker receives messages from the producers and stores them (either in
memory or on disk). The broker should decouple the consumers from the
producers and deliver the messages to the consumers at whatever speed they
can handle them. This would explain why the broker needs some storage space,
but this should be on disk, so why does it need all the memory?
Another observation that might be related: The producer seems to be slowed
down by the broker, so throttling seems to occur. This is contradictory to
my assumption how the broker works, but on the other hand emphasizes that
messages pile up in the broker. In the producer, we use
"setAlwaysSyncSend(true)" because we need to avoid that messages are lost
(see
http://tmielke.blogspot.de/2011/11/loosing-messages-despite-of-sending.html).
This may be a cause for the problem, but I do not see an alternative
solution.
Can you give me some hints how we can change our configuration (or
architecture?) to avoid the OOM errors? Thanks a lot in advance.
Regards,
Christian
--
View this message in context:
http://activemq.2283324.n4.nabble.com/Broker-OOM-tp4703695.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
