This is a case of "working as intended", if the software that uses durable subscribers will not reconnect messages will be accumulating in internal queue (this object is not visible as queue, but in webconsole in "subscribers" tab). And this messages plus journal files that cannot be dropped because they are still in use is what causing the storage to get full.
Remeber that size of stored messages does not equal size of storage on the disk, if there is a message assosciated with journal file, whole file will remain on disk for a while, until journal reorganization kicks in, but this itself is imperfect. You can set timetolive for disconnected durable subscribers in activemq.xml Check https://activemq.apache.org/manage-durable-subscribers It can also help to change size of journal files to smaller (default is 32Mb). And for sanity make sure that messages delivered to broker have expiry set and if not enforce it (there is a plugin for this ) pt., 10 mar 2023, 23:08 użytkownik George Sexton <geor...@mhsoftware.com.invalid> napisał: > I've made some headway on this, but could still use some help. > > I went through and deleted all of the Offline Durable Topic Subscribers. > That freed up a little space (1%) in the store. > > I notice there's a parameter /JournalFilesForFullRecovery that shows//a > value of 6500+. When I deleted the Offline Subscriber, the number > reduced about 75. After I deleted the offline subscribers, I restarted > the Broker. I saw that where previously, the number of journal entries > processed during startup/recover was over 300,000, during this restart > cycle, there were none, and the broker came online very quickly./ > // > /I've been monitoring it for a while. DLQs show 0, and it's done maybe > 10 or so posts. So, things are "working", but the store percent used is > staying at 99%, and the number of JournalFilesForFullRecovery is still > staying very high at 6440./ > > /Does anyone have any ideas on how to pursue this? At this point, I'm > thinking I should bring up a new MQ Broker and change my app to point to > it./ > > /George///// > > // > > On 3/9/2023 4:18 PM, George Sexton wrote: > > I'm having problems now with an ActiveMQ instance running on AWS. I'm > > running version 5.15.15 on an mq.m5.large instance, which is 4 cores > > and 16GB of memory. > > > > There's a couple of DLQs that are empty, and one topic that sees maybe > > 50 messages a day. The AMQP connector shows 20 connections, Active > > Durable Subscribers is 12, and Offline Durable Topic Subscribers is > > something on the order of 400 or so. > > > > The store has been steadily increasing, around 9% per day and now it's > > at 100%. The one topic has had maybe 700 messages. > > > > The configuration is very simple, it's: > > > > <?xml version="1.0" encoding="UTF-8" standalone="yes"?> > > <broker advisorySupport="false" > > xmlns=http://activemq.apache.org/schema/core> > > <plugins> > > <forcePersistencyModeBrokerPlugin persistenceFlag="true"/> > > <statisticsBrokerPlugin/> > > <timeStampingBrokerPlugin ttlCeiling="86400000" > > zeroExpirationOverride="86400000"/> > > </plugins> > > </broker> > > > > Logging shows: > > > > 2023-03-09 23:12:14,774 | INFO | > > Usage(default:store:queue://ActiveMQ.DLQ:store) percentUsage=0%, > > usage=214796216383, limit=214748364800, > > percentUsageMinDelta=1%;Parent:Usage(default:store) percentUsage=100%, > > usage=214796216383, limit=214748364800, percentUsageMinDelta=1%: > > Persistent store is Full, 100% of 214748364800. Stopping producer > > (ID:.mq.us-west-2.amazonaws.com-X:44:0:0) > > to prevent flooding queue://ActiveMQ.DLQ. > > Seehttp://activemq.apache.org/producer-flow-control.html for more > > info (blocking for: 1472s) | > > org.apache.activemq.broker.region.Queue | > > ActiveMQ Broker[MyApp] Scheduler > > > > Doing a little math shows it's writing something like 226KB/Second. > > > > I've noticed Network utilization seems very high. > > > > Can anyone give me some ideas? Should I be looking at clients? Is this > > some sort of internal file corruption? Should I just kill this and > > deploy a totally new ActiveMQ instance? > > > > Thanks for any help. > > > > > -- > George Sexton > (303) 438 9585 x102 > MH Software, Inc.
