Re: handling slow consumers on non-persistent topics and AMQ-688

2006-07-04 Thread James Strachan 

On 7/4/06, Hiram Chirino [EMAIL PROTECTED] wrote:

The problem is there is a high likelyhood that we will need to also extend
the MessageStore interface to provide some cursoring ablilities.  So the
change is not going to just be sefcontained in the broker region package.


Ah I see - in that case I completely agree :).

How about we only make a branch when we decide we have to do some
major surgery, like changing the MessageStore? Maybe we can do a basic
spool-to-disk implementation which is not optimal but works without
major surgery? Or do you think its better to solve this via changing
the MesaageStore contract, to deal with spooling of non-persistent
messages anyway?

--

James
---
http://radio.weblogs.com/0112098/

Re: handling slow consumers on non-persistent topics and AMQ-688

2006-07-04 Thread Hiram Chirino 

These are all good questions.. And that's why I kinda was think a branch
would be good since it would alow folks to go crazy figuring out the best
solution and not have to work about breaking things while the changes are in
progress.

On 7/4/06, James Strachan [EMAIL PROTECTED] wrote:


On 7/4/06, Hiram Chirino [EMAIL PROTECTED] wrote:
 The problem is there is a high likelyhood that we will need to also
extend
 the MessageStore interface to provide some cursoring ablilities.  So the
 change is not going to just be sefcontained in the broker region
package.

Ah I see - in that case I completely agree :).

How about we only make a branch when we decide we have to do some
major surgery, like changing the MessageStore? Maybe we can do a basic
spool-to-disk implementation which is not optimal but works without
major surgery? Or do you think its better to solve this via changing
the MesaageStore contract, to deal with spooling of non-persistent
messages anyway?

--

James
---
http://radio.weblogs.com/0112098/





--
Regards,
Hiram

Blog: http://hiramchirino.com

Re: handling slow consumers on non-persistent topics and AMQ-688

2006-07-03 Thread Hiram Chirino 

I think that fixing these issues are going to require substancial work on
internals of the broker.  I want to try to take a stab fixing this stuff but
since it's going to take a few iterations to get right, how about we branch
trunk and work there to avoid breaking everbody else?  Any sugestions for
the branch name?

Regards,
Hiram

On 7/3/06, James Strachan [EMAIL PROTECTED] wrote:


While perusing JIRA I spotted this issue again...
http://issues.apache.org/activemq/browse/AMQ-688
I know its an issue close to folks at Amazon's hearts.

Dealing with slow consumers is a fascinating problem for a messaging
system; its quite a tricky problem :). Here's some background on the
issue...
http://incubator.apache.org/activemq/slow-consumers.html

together with the currently supported features - to allow messages to
be discarded on slow consumers using a pluggable algorithm...
http://incubator.apache.org/activemq/slow-consumer-handling.html

Now for all consumers we fill up prefetch buffers as quickly as
possible...
http://incubator.apache.org/activemq/what-is-the-prefetch-limit-for.html

so there's always a buffer of messages per consumer. For non-durable
topics once these messages are written to a socket they are evicted
from RAM; so we already have some support for slow consumers.

I wanted to start a discussion on both AMQ-688 and to see if folks had
other requirements for handling slow consumers  to try decide what
features  stragegies we should add next in this area.

One of the first requirements folks ask for is that rather than
blocking permanently the non-persistent topic engine until RAM is
cleared, that at a certain threshhold we start spooling to disk. I've
raised a separate JIRA issue for this specific feature request...

http://issues.apache.org/activemq/browse/AMQ-791

Another issue some folks have hit in the past is that for high
performance and to minimise context switches in the broker, we tend to
use the current thread in the broker to dispatch to all the
non-durable topic consumers so a slow/blocked consumer can appear to
'block' the producer.

I've raised this issue to track that feature
http://issues.apache.org/activemq/browse/AMQ-792

I just wondered if folks had any other issues or requirements with the
whole slow consumers and non-durable topics they'd like to discuss? Is
there any requirements we won't have covered if the above two JIRAs
are fixed
--

James
---
http://radio.weblogs.com/0112098/





--
Regards,
Hiram

Blog: http://hiramchirino.com