On Apr 7, 8:22 pm, Ray Krueger <[email protected]> wrote:
> > I guess the biggest difference I see people talking about when talking
> > messaging and job queues is that with messaging they expect all
> > workers to receive all messages. With the 'normal' beanstalkd usage as
> > a job queue, we expect each worker to see a single 'next job' message
> > while avoiding all the duplicates. The job queue doesn't work as well
> > for one to many subscribers, which it seems many people use
> > queue/messaging systems for.
>
> I don't think that's true. In the messaging world there are queues for
> point-to-point and there are topics (using JMS terms sorry) for
> publish-subscribe. When people use a messaging implementation such as
> amqp or jms for work queueing you do it with a Queue that can have
> many senders and many listeners. The queue guarantees that only one
> listener gets a message when it gets put on the queue. As far as I
> now, this is exactly how beanstalk works.
>
> The only thing beanstalkd is missing here is the persistence feature
> available in the messaging providers already on the market. Any time
> I've tried to talk about beanstalkd with people they never seem to be
> able to get past the lack of persistence. As soon as they say "if
> beanstalk goes down all my messages are lost?", the conversation is
> over.
>
> > The fact that beanstalkd is used for queuing FIFO made sense to me,
> > but with many other projects focusing on the one to many queues (which
> > really seems like setting up multiple queues, one for every
> > subscriber), it might be good to try to clarify the wording a bit on
> > the site.
>
> One of the systems I worked on relied on both messaging paradigms.
> System-A would send a message out on a topic for anyone who cared to
> hear about it. System-B would receive a copy of that message and put
> it into a work queue where one of the many instances of System-C would
> pick it up and work on it.
There are a few different groupings of communication style, which have
thier own semantics and uses. (note these may or may not be terms made
up by me, but the ideas are solid, i just don't want to hunt down
references)
1. one-to-one: this is the boring case, in which a sends a message to
b.
2. one-to-many: i push out a message, and anyone listening gets it.
(broadcast/multicast are minor variations on the theme)
3. many-to-one: A gets messages from B,C,D etc as they arrive in the
queue, the key is that the reciever gets them all in the same place.
4. many-to-many: A message from A,B,C, or D goes to all the rest.
Like in a walkie talkie system :)
5. one-to-ANY: a message from A gets to B xor C xor D, and it doesnt
matter which.
For all of these it is trivial to transform one to the other. (for
example everything above has been implemented in tcp, which is
inherently one-to-one), however that doesn't make it good to do so.
The important thing tho is that 1-4 above are all degenerate cases of
4 (conversely built up from 1), so they can be mixed and matched with
just a few changes/abstractions. To use 5 to get the others, you have
to set up a one-to-one system, and then build on top of that,
requiring 2 transforms. This can start to get expensive.
They all also have different use cases. I personally am a big fan of
lots of separate tools for different functions (this is why i hate
memcache based queues). The point I'm trying to make is, beanstalk
does some things very well. It could do the others also, but why not
use a hammer for the nails, and screwdriver for the screws? A lot of
messaging systems are very much set up for the 1-4 cases. They do a
great job. Beanstalk is really set up for the 5 case. This isn't bad,
nor does it mean that they can't be used in place of each other.
Another thing to address here is persistance. It is nice, but computer
people (actually anyone with the engineering mindset...) get hung up
on the concept of guarantee. There are many cases where persistence at
the message server is the right answer, but there are many cases where
it is the lazy (sort of wrong) and wrong answer. The best example is
the 0mq guys. Their message system has no store and forward concept.
heck, if you are not connected to the right channel when a message is
sent, you will never get it. In their world (finance) this is
desirable, since the stock price 100 ms ago can lose you a lot of
money now. I've frequently had disccusions like this before:
me "use brand X non-persistant queue"
them "omg! no persistence, it's evil"
me "does your problem need persistence for real?"
them "oh... uh.. actually no"
Basically persistence is frequently an assumed need that goes
unexamined in many cases. (of course if persistence is actually
needed, the above doesn't apply... I'm not knocking the concept).
In short, all of this is just a long-winded expansion on my earlier
reply to Keith (about my co-worker and I deciding they should have
included bonjur/zeroconf and D-BUS to the list).
Regards,
Erich
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"beanstalk-talk" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/beanstalk-talk?hl=en
-~----------~----~----~----~------~----~------~--~---
