[ https://issues.apache.org/activemq/browse/AMQ-688?page=comments#action_36039 ]
Christopher A. Larrieu commented on AMQ-688: -------------------------------------------- While experimenting with some code to force Queues to evict stored messages from memory, I discovered that a bug in the Journal was causing messages that should have been evicted not to be. Somehow the list of listeners on the UsageManager was getting trampled, so that the journal was never getting notification that memory utilization was getting high. I updated to the most recent revision of the source (r392487) and now persistent messages do get evicted from Queues as expected, and producers don't get stalled or throttled by consumers. > Avoid blocking producers > ------------------------ > > Key: AMQ-688 > URL: https://issues.apache.org/activemq/browse/AMQ-688 > Project: ActiveMQ > Type: New Feature > Components: Broker > Versions: 4.0 RC 2 > Reporter: Christopher A. Larrieu > Fix For: incubation > > Original Estimate: 8 weeks > Remaining: 8 weeks > > Our main goal > is to avoid stalled producers by addressing the main culprit: too many > undispatched messages > in the broker's memory. Our motivation is to handle significant --though > temporary-- imbalances > between production and consumption rates. > Reaching this goal entails specific broker modifications: > 1. When memory gets tight, start dropping undispatched non-persistent > messages. This is the > first-cut attempt to maintain throughput of persistent messages. > Unlike the approach documented at > http://docs.codehaus.org/display/ACTIVEMQ/Slow+Consumer+Handling, > the message dropping will only occur after the UsageManager reaches capacity. > Non-persistent > messages in dispatch lists will be dropped according to per-destination > policy. Subscriptions > can purge their own messages triggered via callback from the UsageManager. > 2. Evict messages if memory remains tight, to be fetched from backing store > prior to dispatch. > ActiveMQ already supports this for persistent messages on Topics with durable > subscriptions. > If a consumer's prefetch buffer is full, the splash-over messages remain as > IndirectMessageReference > objects in the dispatch list, with the actual message body loaded from store > on demand. I > believe we can extend this approach for Queues as well. > 3. Improve the efficiency with which evicted messages are loaded back into > memory. > Currently, they are loaded one at a time as needed. It would make sense to > batch-load message > sets periodically. This will require a significant shift in responsibilities > between objects, > since an IndirectMessageReference doesn't know about other instances that can > be loaded in > mass. > > The goal will be to keep each subscription dispatch list stocked with a > decent number of messages > in-memory to reasonably trade-off between it's consumer's performance and > resource usage in > the broker. As with everything else, we can implement this as a strategy > class with the first > cut implementing a simple resource allocation strategy: divvy up available > memory amongst > all subscriptions and keep that memory filled with messages for dispatch. I > envision a worker > task assuming responsibility for keeping these lists filled. > 4. Even with the above modifications, we still can't entirely avoid blocked > producers, so > we'd like to add client-configurable time-outs to provide a bound for the > time a producer > can remain stalled. > Maybe this should be a new attribute of ActiveMQConnection: > maxProducerFlowControlWait. Calls > to UsageManager.waitForSpace can take this quantity as an argument. Failure > to reach sufficient > space for the new message will throw an exception back up the stack and > across the wire, letting > the producer know that the message was not delivered. > 5. Finally, we need to extend disk support for Topics that have only > non-durable subscribers, > otherwise their dispatch lists can fill up with persistent messages. In > order to maintain > compliance with JMS, it would be nice to provide some alternative to dropping > persistent messages. > One possible first cut is to layer this on top of a DurableTopicSubscription > by creating an > anonymous subscriber for every Topic that has only non-durable subscriptions. > When all such > subscriptions terminate, the broker can remove the anonymous subscriber. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
