[ https://issues.apache.org/activemq/browse/AMQ-688?page=all ]
Christopher A. Larrieu reassigned AMQ-688:
------------------------------------------
Assign To: Christopher A. Larrieu
> Avoid blocking producers
> ------------------------
>
> Key: AMQ-688
> URL: https://issues.apache.org/activemq/browse/AMQ-688
> Project: ActiveMQ
> Type: New Feature
> Components: Broker
> Versions: 4.0 RC 2
> Reporter: Christopher A. Larrieu
> Assignee: Christopher A. Larrieu
> Fix For: incubation
>
> Original Estimate: 8 weeks
> Remaining: 8 weeks
>
> Our main goal
> is to avoid stalled producers by addressing the main culprit: too many
> undispatched messages
> in the broker's memory. Our motivation is to handle significant --though
> temporary-- imbalances
> between production and consumption rates.
> Reaching this goal entails specific broker modifications:
> 1. When memory gets tight, start dropping undispatched non-persistent
> messages. This is the
> first-cut attempt to maintain throughput of persistent messages.
> Unlike the approach documented at
> http://docs.codehaus.org/display/ACTIVEMQ/Slow+Consumer+Handling,
> the message dropping will only occur after the UsageManager reaches capacity.
> Non-persistent
> messages in dispatch lists will be dropped according to per-destination
> policy. Subscriptions
> can purge their own messages triggered via callback from the UsageManager.
> 2. Evict messages if memory remains tight, to be fetched from backing store
> prior to dispatch.
> ActiveMQ already supports this for persistent messages on Topics with durable
> subscriptions.
> If a consumer's prefetch buffer is full, the splash-over messages remain as
> IndirectMessageReference
> objects in the dispatch list, with the actual message body loaded from store
> on demand. I
> believe we can extend this approach for Queues as well.
> 3. Improve the efficiency with which evicted messages are loaded back into
> memory.
> Currently, they are loaded one at a time as needed. It would make sense to
> batch-load message
> sets periodically. This will require a significant shift in responsibilities
> between objects,
> since an IndirectMessageReference doesn't know about other instances that can
> be loaded in
> mass.
>
> The goal will be to keep each subscription dispatch list stocked with a
> decent number of messages
> in-memory to reasonably trade-off between it's consumer's performance and
> resource usage in
> the broker. As with everything else, we can implement this as a strategy
> class with the first
> cut implementing a simple resource allocation strategy: divvy up available
> memory amongst
> all subscriptions and keep that memory filled with messages for dispatch. I
> envision a worker
> task assuming responsibility for keeping these lists filled.
> 4. Even with the above modifications, we still can't entirely avoid blocked
> producers, so
> we'd like to add client-configurable time-outs to provide a bound for the
> time a producer
> can remain stalled.
> Maybe this should be a new attribute of ActiveMQConnection:
> maxProducerFlowControlWait. Calls
> to UsageManager.waitForSpace can take this quantity as an argument. Failure
> to reach sufficient
> space for the new message will throw an exception back up the stack and
> across the wire, letting
> the producer know that the message was not delivered.
> 5. Finally, we need to extend disk support for Topics that have only
> non-durable subscribers,
> otherwise their dispatch lists can fill up with persistent messages. In
> order to maintain
> compliance with JMS, it would be nice to provide some alternative to dropping
> persistent messages.
> One possible first cut is to layer this on top of a DurableTopicSubscription
> by creating an
> anonymous subscriber for every Topic that has only non-durable subscriptions.
> When all such
> subscriptions terminate, the broker can remove the anonymous subscriber.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
