Sorry Joe, you want to reply to users@activemq.apache.org, looks like
my usage of reply all confused things.
Looks like you want to increase the temp store system limit, not sure
off the top of my head what the default is, but it's probably similar
to the commented out setting in the default activemq.xml. With these
settings I can put 1M messages onto a queue with no consumer using the
default 512MB heap limit:
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="true"
memoryLimit="1mb"/>
<policyEntry queue=">" producerFlowControl="true"
memoryLimit="1mb"/>
</policyEntries>
</policyMap>
</destinationPolicy>
<!-- snip -->
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="20 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="10 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="10 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
When using the default store based cursor (and KahaDB for your
persistence adapter) the tempUsage limit tracks the size of
data/localhost/tmp_storage, though not sure if it just looks at the
one datafile or includes the whole directory, using the example
producer to push 1M messages gives me a temp store size of 5.5G.
In general flow control prevents some limit being reached, be it the
memory limit on a destination, the overall system memory limit or the
store/temp filesize limits, in a way that you wouldn't perhaps need to
address in your producer, i.e. if the limit is reached, your producer
blocks or slows down a bit, perhaps giving a consumer time to drain
the destination that the producer is sending messages to. Message
cursors are another way to avoid high memory usage in the broker, by
spooling messages someplace other than memory (unless of course you're
using the vmcursor) as they're received. In your case I suspect the
cursor is doing it's job but you have to increase the temp size limit,
as that's what kicks in when you're pushing non-persistent messages.
If you're pushing persistent messages then it'd be the store size
limit.
Hope that helps!
On Tue, Oct 5, 2010 at 2:11 PM, Joe Niski <joe.ni...@nwea.org> wrote:
> Thanks, Stan, that was a good suggestion.
>
> We set it to 1MB. Additional background - for testing purposes, we've set
> the JVM heap to 256MB (it's 1.5GB in our UAT and Production environments,
> 32-bit Sun jvm). We're using Kahadb for our message store.
>
> Our test messages are roughly 1kB in size. We ran tests with no consumer on
> the queue, in order to force the messageCursor into working. We monitored
> ActiveMQ with the web console and jconsole, in addition to tailing the log
> files for the producer and ActiveMQ.
>
> With producerFlowControl OFF and fileQueueCursor enabled, we could send a
> bit over 24,000 messages to the queue, at which point the jvm heap was
> exhausted and the producer could no longer connect.
>
> With producerFlowControl ON and fileQueueCursor enabled, we could send 105
> messages, at which point producerFlowControl blocked the producer.
> Activating a consumer on the queue got things flowing, as expected.
>
> But this question remains:
> 1) Is it possible to use messageCursors (either store-based or file-based)
> in concert with producerFlowControl, so producers can continue even when the
> queue's memoryLimit is reached?
>
> And i have these new questions:
> 2) Is the memoryLimit for queues and topics applied only when
> producerFlowControl is enabled?
> 3) If the answer to (2) is "no", then why do i see ActiveMQ using up all
> available heap when i configure the fileQueueCursor?
> 4) Are messageCursors intended to work when producerFlowControl is enabled
> and the queue memoryLimit is reached?
> 5) If the answer to (4) is "yes", then why do i see producers blocked when
> the memoryLimit is reached?
>
> The online documentation is ambiguous about how messageCursors and
> producerFlowControl are supposed to work together. In the latest release of
> "ActiveMQ in Action", messageCursors aren't even mentioned.
>
> Thanks in advance,
> Joe
>
> Joe Niski
> IS Development | NWEA
>
> PHONE 503.212.3382 | FAX 503.639.7873
>
> NWEA.ORG | Partnering to Help All Kids Learn
>
> On 10/01/2010 03:18 PM, Stan Lewis wrote:
>
> It could be that you're hitting the system memory usage limit. Try
> setting a lower memory limit on your queue like 1mb with the
> pendingQueuePolicy.
>
> On Fri, Oct 1, 2010 at 6:13 PM, Joe Niski <joe.ni...@nwea.org> wrote:
>> Well, i may have spoken too soon. Tests with a queue's destination policy
>> configured like this:
>>
>> <policyEntry queue=">" producerFlowControl="true"
>> memoryLimit="64mb">
>> <pendingQueuePolicy>
>> <fileQueueCursor/>
>> </pendingQueuePolicy>
>> </policyEntry>
>>
>> indicate that the pendingQueuePolicy is ignored if producerFlowControl is
>> enabled. The same test (sending 10,000 messages to a queue that has no
>> consumers) locks up at the same message count, regardless of whether there's
>> a pendingQueuePolicy or not, with this error in the log:
>> 2010-10-01 09:39:51,081 | INFO | Kaha Store using data directory
>> /opt/activemq/data/localhost/tmp_storage |
>> org.apache.activemq.kaha.impl.KahaStore |
>> default:memory:queue://inbound.tt2etl.q:memory Usage Thread Pool
>> 2010-10-01 09:41:04,855 | INFO | Usage Manager memory limit reached on
>> queue://inbound.tt2etl.q. Producers will be throttled to the rate at which
>> messages are removed from this destination to prevent flooding it. See
>> http://activemq.apache.org/producer-flow-control.html for more info |
>> org.apache.activemq.broker.region.Queue | ActiveMQ Transport:
>> tcp:///192.168.22.78:52741
>>
>> Setting producerFlowControl="false" changes the behavior - messages
>> continue to be accepted by the queue, until a consumer starts or AMQ
>> actually runs out of heap and the producer's connection is broken.
