Re: handling slow consumers on non-persistent topics and AMQ-688
On 7/4/06, Hiram Chirino [EMAIL PROTECTED] wrote: The problem is there is a high likelyhood that we will need to also extend the MessageStore interface to provide some cursoring ablilities. So the change is not going to just be sefcontained in the broker region package. Ah I see - in that case I completely agree :). How about we only make a branch when we decide we have to do some major surgery, like changing the MessageStore? Maybe we can do a basic spool-to-disk implementation which is not optimal but works without major surgery? Or do you think its better to solve this via changing the MesaageStore contract, to deal with spooling of non-persistent messages anyway? -- James --- http://radio.weblogs.com/0112098/
Re: handling slow consumers on non-persistent topics and AMQ-688
These are all good questions.. And that's why I kinda was think a branch would be good since it would alow folks to go crazy figuring out the best solution and not have to work about breaking things while the changes are in progress. On 7/4/06, James Strachan [EMAIL PROTECTED] wrote: On 7/4/06, Hiram Chirino [EMAIL PROTECTED] wrote: The problem is there is a high likelyhood that we will need to also extend the MessageStore interface to provide some cursoring ablilities. So the change is not going to just be sefcontained in the broker region package. Ah I see - in that case I completely agree :). How about we only make a branch when we decide we have to do some major surgery, like changing the MessageStore? Maybe we can do a basic spool-to-disk implementation which is not optimal but works without major surgery? Or do you think its better to solve this via changing the MesaageStore contract, to deal with spooling of non-persistent messages anyway? -- James --- http://radio.weblogs.com/0112098/ -- Regards, Hiram Blog: http://hiramchirino.com
Re: handling slow consumers on non-persistent topics and AMQ-688
BTW AMQ-792 is now fixed - details here... http://incubator.apache.org/activemq/consumer-dispatch-async.html so the only real pending issue so far that I'm aware of for dealing with slow consumers is AMQ-791 (spooling to disk)... http://issues.apache.org/activemq/browse/AMQ-791 Unless folks have any other use cases or requirements? On 7/3/06, James Strachan [EMAIL PROTECTED] wrote: While perusing JIRA I spotted this issue again... http://issues.apache.org/activemq/browse/AMQ-688 I know its an issue close to folks at Amazon's hearts. Dealing with slow consumers is a fascinating problem for a messaging system; its quite a tricky problem :). Here's some background on the issue... http://incubator.apache.org/activemq/slow-consumers.html together with the currently supported features - to allow messages to be discarded on slow consumers using a pluggable algorithm... http://incubator.apache.org/activemq/slow-consumer-handling.html Now for all consumers we fill up prefetch buffers as quickly as possible... http://incubator.apache.org/activemq/what-is-the-prefetch-limit-for.html so there's always a buffer of messages per consumer. For non-durable topics once these messages are written to a socket they are evicted from RAM; so we already have some support for slow consumers. I wanted to start a discussion on both AMQ-688 and to see if folks had other requirements for handling slow consumers to try decide what features stragegies we should add next in this area. One of the first requirements folks ask for is that rather than blocking permanently the non-persistent topic engine until RAM is cleared, that at a certain threshhold we start spooling to disk. I've raised a separate JIRA issue for this specific feature request... http://issues.apache.org/activemq/browse/AMQ-791 Another issue some folks have hit in the past is that for high performance and to minimise context switches in the broker, we tend to use the current thread in the broker to dispatch to all the non-durable topic consumers so a slow/blocked consumer can appear to 'block' the producer. I've raised this issue to track that feature http://issues.apache.org/activemq/browse/AMQ-792 I just wondered if folks had any other issues or requirements with the whole slow consumers and non-durable topics they'd like to discuss? Is there any requirements we won't have covered if the above two JIRAs are fixed -- James --- http://radio.weblogs.com/0112098/ -- James --- http://radio.weblogs.com/0112098/
Re: handling slow consumers on non-persistent topics and AMQ-688
I think that fixing these issues are going to require substancial work on internals of the broker. I want to try to take a stab fixing this stuff but since it's going to take a few iterations to get right, how about we branch trunk and work there to avoid breaking everbody else? Any sugestions for the branch name? Regards, Hiram On 7/3/06, James Strachan [EMAIL PROTECTED] wrote: While perusing JIRA I spotted this issue again... http://issues.apache.org/activemq/browse/AMQ-688 I know its an issue close to folks at Amazon's hearts. Dealing with slow consumers is a fascinating problem for a messaging system; its quite a tricky problem :). Here's some background on the issue... http://incubator.apache.org/activemq/slow-consumers.html together with the currently supported features - to allow messages to be discarded on slow consumers using a pluggable algorithm... http://incubator.apache.org/activemq/slow-consumer-handling.html Now for all consumers we fill up prefetch buffers as quickly as possible... http://incubator.apache.org/activemq/what-is-the-prefetch-limit-for.html so there's always a buffer of messages per consumer. For non-durable topics once these messages are written to a socket they are evicted from RAM; so we already have some support for slow consumers. I wanted to start a discussion on both AMQ-688 and to see if folks had other requirements for handling slow consumers to try decide what features stragegies we should add next in this area. One of the first requirements folks ask for is that rather than blocking permanently the non-persistent topic engine until RAM is cleared, that at a certain threshhold we start spooling to disk. I've raised a separate JIRA issue for this specific feature request... http://issues.apache.org/activemq/browse/AMQ-791 Another issue some folks have hit in the past is that for high performance and to minimise context switches in the broker, we tend to use the current thread in the broker to dispatch to all the non-durable topic consumers so a slow/blocked consumer can appear to 'block' the producer. I've raised this issue to track that feature http://issues.apache.org/activemq/browse/AMQ-792 I just wondered if folks had any other issues or requirements with the whole slow consumers and non-durable topics they'd like to discuss? Is there any requirements we won't have covered if the above two JIRAs are fixed -- James --- http://radio.weblogs.com/0112098/ -- Regards, Hiram Blog: http://hiramchirino.com
handling slow consumers on non-persistent topics and AMQ-688
While perusing JIRA I spotted this issue again... http://issues.apache.org/activemq/browse/AMQ-688 I know its an issue close to folks at Amazon's hearts. Dealing with slow consumers is a fascinating problem for a messaging system; its quite a tricky problem :). Here's some background on the issue... http://incubator.apache.org/activemq/slow-consumers.html together with the currently supported features - to allow messages to be discarded on slow consumers using a pluggable algorithm... http://incubator.apache.org/activemq/slow-consumer-handling.html Now for all consumers we fill up prefetch buffers as quickly as possible... http://incubator.apache.org/activemq/what-is-the-prefetch-limit-for.html so there's always a buffer of messages per consumer. For non-durable topics once these messages are written to a socket they are evicted from RAM; so we already have some support for slow consumers. I wanted to start a discussion on both AMQ-688 and to see if folks had other requirements for handling slow consumers to try decide what features stragegies we should add next in this area. One of the first requirements folks ask for is that rather than blocking permanently the non-persistent topic engine until RAM is cleared, that at a certain threshhold we start spooling to disk. I've raised a separate JIRA issue for this specific feature request... http://issues.apache.org/activemq/browse/AMQ-791 Another issue some folks have hit in the past is that for high performance and to minimise context switches in the broker, we tend to use the current thread in the broker to dispatch to all the non-durable topic consumers so a slow/blocked consumer can appear to 'block' the producer. I've raised this issue to track that feature http://issues.apache.org/activemq/browse/AMQ-792 I just wondered if folks had any other issues or requirements with the whole slow consumers and non-durable topics they'd like to discuss? Is there any requirements we won't have covered if the above two JIRAs are fixed -- James --- http://radio.weblogs.com/0112098/