Hi, after some experiments, it seems that maybe one part of the problem is that temp-queue messages are not spooled to disk. Is there a possibility to tell ActiveMQ 5.5 to spool them to the temp store as well?
Another issue I noticed, after purging a queue with non-persistent messages, the space in the temp-store seems to not be freed. Best regards, Martin On Mon, Aug 27, 2012 at 1:05 PM, Martin C. <mart...@gmx.at> wrote: > Hi, > > in one of the instances I didn't find anything in the broker log, just in > the client log, in the second log I found the same log entry in both client > and server: > > javax.jms.ResourceAllocationException: Usage Manager Memory Limit reached. > Stopping producer (ID:hostname:26:33:1) to prevent flooding > temp-queue://e1413fd6-2e46-76d5-dc62-df3260914cac:0 > > What would you suggest to use as a persistent <pendingQueuePolicy/> in my > case, if I'd just want messages to go to disk if the memory-usage is > approached? > > Thanks in advance! > > Best regards, > Martin > > On Mon, Aug 27, 2012 at 12:25 PM, Gary Tully <gary.tu...@gmail.com> wrote: > >> what is out put to the broker log when it stops processing? It should >> report the limit that is reached. >> >> The pendingQueueCursor will stop caching messages when the systemusage >> limit reaches 70%, but it won't spool to disk. The persistent messages >> will just remain in the store till they are needed. >> >> On 27 August 2012 10:59, Martin C. <mart...@gmx.at> wrote: >> > Hi Gary, >> > >> > sorry for getting back to this rather old issue, but I just got an >> issue in >> > the field (twice) with the configuration below. The broker stopped >> > accepting messages after it reached the 64MB memory limit, instead of >> > spooling them out to disk. It basically is the previously discussed >> > configuration with the <pendingQueuePolicy /> section removed. >> > >> > After increasing the MemoryLimit of the broker via JMX, the messages got >> > processed again. The temp-store usage was at 0, the persistent store >> usage >> > at 11%. So, from my understanding, the broker should not have stopped >> > accepting messages. Any hints? Or should I fall back to using >> > <fileQueueCursor /> again? >> > >> > I am still on 5.5.0 and cannot upgrade that easily. >> > >> > Thanks in advance! >> > >> > Best regards, >> > Martin >> > >> > The relevant parts of the configuration again. >> > >> > <broker xmlns="http://activemq.apache.org/schema/core"; >> > brokerName="localhost" dataDirectory="${activemq.base}/data" >> > populateJMSXUserID="true" useJmx="true"> >> > >> > <destinationPolicy> >> > <policyMap> >> > <policyEntries> >> > <policyEntry topic=">" producerFlowControl="false" >> > gcInactiveDestinations="false" inactiveTimoutBeforeGC="60000"> >> > <vmCursor /> >> > </policyEntry> >> > <policyEntry queue=">" producerFlowControl="false" >> > gcInactiveDestinations="false" inactiveTimoutBeforeGC="86400000"> >> > <deadLetterStrategy> >> > <individualDeadLetterStrategy queuePrefix="DLQ." >> > useQueueForQueueMessages="true" >> > processExpired="true" processNonPersistent="false" /> >> > </deadLetterStrategy> >> > </policyEntry> >> > </policyEntries> >> > </policyMap> >> > </destinationPolicy> >> > >> > <persistenceAdapter> >> > <kahaDB directory="${activemq.base}/data/kahadb" >> > checkForCorruptJournalFiles="true" checksumJournalFiles="true" >> > concurrentStoreAndDispatchTopics="false" >> > concurrentStoreAndDispatchQueues="false" >> > /> >> > </persistenceAdapter> >> > >> > <systemUsage> >> > <systemUsage sendFailIfNoSpace="true"> >> > <memoryUsage> >> > <memoryUsage limit="64 mb" /> >> > </memoryUsage> >> > <storeUsage> >> > <storeUsage limit="10 gb" /> >> > </storeUsage> >> > <tempUsage> >> > <tempUsage limit="1 gb" /> >> > </tempUsage> >> > </systemUsage> >> > </systemUsage> >> > </broker> >> > >> > On Wed, Nov 16, 2011 at 1:50 PM, Gary Tully <gary.tu...@gmail.com> >> wrote: >> > >> >> yes. >> >> >> >> On 16 November 2011 12:01, Martin C. <mart...@gmx.at> wrote: >> >> > Hi Gary, >> >> > >> >> > thanks for the very fast response. I will change the configuration >> >> > accordingly. So basically you say I should remove the >> >> > <pendingQueuePolicy> as a whole? >> >> > >> >> > Best regards, >> >> > Martin >> >> > >> >> > On Wed, Nov 16, 2011 at 11:34 AM, Gary Tully <gary.tu...@gmail.com> >> >> wrote: >> >> >> You should not be using the file pending message cursor in this >> case. >> >> >> The default store cursor is best, it will use a file cursor for non >> >> >> persistent messages and leave excess (exceed the cache) persistent >> >> >> messages in the store. >> >> >> When it is specified as the default cursor in configuration, it is >> >> >> used for both persistent and non persistent messages. >> >> >> >> >> >> The file pending message cursor replays all messages on startup such >> >> >> that it can do an append to the cursor on the next add but it is >> only >> >> >> intended for really slow stores, where reading from the local file >> >> >> system is faster than reading from the store. This is not the case >> >> >> with kahaDB. >> >> >> >> >> >> On 16 November 2011 10:26, Martin C. <mart...@gmx.at> wrote: >> >> >>> Hi, >> >> >>> >> >> >>> I a have a question regarding fileQueueCursor and systemUsage on >> >> >>> ActiveMQ 5.5.0: Below you see my configuration, which limits the >> >> >>> memory to 64MB, store usage to 10GB and tempUsage to 1GB. >> >> >>> >> >> >>> I have a KahaDB with about 300000 messages, using 330MB in the >> kahadb >> >> >>> directory, all messages are persistent. If I start ActiveMQ, I see >> the >> >> >>> tmp_storage directory grow to 1GB, where it finally blocks and >> >> >>> recovery of the messages starts hanging: >> >> >>> >> >> >>> [...] >> >> >>> DEBUG | default:memory: usage change from: 2% of available memory, >> to: >> >> >>> 1% of available memory >> >> >>> DEBUG | default:memory:queue://local.queue.myqueue:memory: usage >> >> >>> change from: 1% of available memory, to: 0% of available memory >> >> >>> DEBUG | default:memory: usage change from: 1% of available memory, >> to: >> >> >>> 0% of available memory >> >> >>> INFO | cursor for queue://local.queue.myqueue has recovered 50000 >> >> >>> messages. 15% complete >> >> >>> INFO | cursor for queue://local.queue.myqueue has recovered 100000 >> >> >>> messages. 31% complete >> >> >>> INFO | cursor for queue://local.queue.myqueue has recovered 150000 >> >> >>> messages. 47% complete >> >> >>> INFO | cursor for queue://local.queue.myqueue has recovered 200000 >> >> >>> messages. 62% complete >> >> >>> >> >> >>> The memory usage goes up to around 75%, where it starts dropping >> down >> >> >>> to the 0% from the logfile above. This is where it keeps hanging, >> with >> >> >>> the tmp_storage directory at 1GB diskspace. >> >> >>> >> >> >>> If I increase the tempUsage to 10GB, the startup works fine >> >> >>> (tmp_storage becomes about 1.4GB in this case). Can you explain to >> me, >> >> >>> why I need to increase tempUsage in order to restart my broker >> with a >> >> >>> lot of persistent messages left? As far as I understood the >> tempUsage >> >> >>> is for storing non-persistent messages, persistent messages should >> be >> >> >>> affected by storeUsage? >> >> >>> >> >> >>> Thanks in advance! >> >> >>> >> >> >>> Best regards, >> >> >>> Martin >> >> >>> >> >> >>> PS: the relevant parts from my configuration >> >> >>> >> >> >>> <broker xmlns="http://activemq.apache.org/schema/core"; >> >> >>> brokerName="localhost" dataDirectory="${activemq.base}/data" >> >> >>> populateJMSXUserID="true" useJmx="true"> >> >> >>> >> >> >>> <destinationPolicy> >> >> >>> <policyMap> >> >> >>> <policyEntries> >> >> >>> <policyEntry topic=">" producerFlowControl="false" >> >> >>> gcInactiveDestinations="false" inactiveTimoutBeforeGC="60000"> >> >> >>> <vmCursor /> >> >> >>> </policyEntry> >> >> >>> <policyEntry queue=">" producerFlowControl="false" >> >> >>> gcInactiveDestinations="false" inactiveTimoutBeforeGC="86400000"> >> >> >>> <deadLetterStrategy> >> >> >>> <individualDeadLetterStrategy queuePrefix="DLQ." >> >> >>> useQueueForQueueMessages="true" >> >> >>> processExpired="true" processNonPersistent="false" >> /> >> >> >>> </deadLetterStrategy> >> >> >>> <pendingQueuePolicy> >> >> >>> <fileQueueCursor /> >> >> >>> </pendingQueuePolicy> >> >> >>> </policyEntry> >> >> >>> </policyEntries> >> >> >>> </policyMap> >> >> >>> </destinationPolicy> >> >> >>> >> >> >>> <persistenceAdapter> >> >> >>> <kahaDB directory="${activemq.base}/data/kahadb" >> >> >>> checkForCorruptJournalFiles="true" >> checksumJournalFiles="true" >> >> >>> concurrentStoreAndDispatchTopics="false" >> >> >>> concurrentStoreAndDispatchQueues="false" >> >> >>> /> >> >> >>> </persistenceAdapter> >> >> >>> >> >> >>> <systemUsage> >> >> >>> <systemUsage sendFailIfNoSpace="true"> >> >> >>> <memoryUsage> >> >> >>> <memoryUsage limit="64 mb" /> >> >> >>> </memoryUsage> >> >> >>> <storeUsage> >> >> >>> <storeUsage limit="10 gb" /> >> >> >>> </storeUsage> >> >> >>> <tempUsage> >> >> >>> <tempUsage limit="1 gb" /> >> >> >>> </tempUsage> >> >> >>> </systemUsage> >> >> >>> </systemUsage> >> >> >>> </broker> >> >> >>> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> http://fusesource.com >> >> >> http://blog.garytully.com >> >> >> >> >> > >> >> >> >> >> >> >> >> -- >> >> http://fusesource.com >> >> http://blog.garytully.com >> >> >> >> >> >> -- >> http://fusesource.com >> http://blog.garytully.com >> > >
