Re: [Openstack] Burrow (queue service)

[Sandy Walsh]

 Let me know if there are any questions I didn't answer thoroughly enough.

Looks like a fun project Eric. I only got caught up on the ML this weekend and 
I'm behind again already. 

Some questions that I jotted down from the previous discussions were:

1. Will broadcast queues be supported? 

2. What are the guarantees of idempotence in the event of retries/errors/etc? 
(not applicable to broadcast obviously).

3. Could there be support for WAN clusters? i.e. only forward certain queues to 
remote locations to keep chatter down?

4. Are all operations assumed to be one-way, or is it assumed that return 
values are part of each call. i.e. do I have to set up other queues for return 
values as callbacks or is that part of the framework? Is it optional?

Thanks
-S


Confidentiality Notice: This e-mail message (including any attached or
embedded documents) is intended for the exclusive and confidential use of the
individual or entity to which this message is addressed, and unless otherwise
expressly indicated, is confidential and privileged information of Rackspace.
Any dissemination, distribution or copying of the enclosed material is 
prohibited.
If you receive this transmission in error, please notify us immediately by 
e-mail
at ab...@rackspace.com, and delete the original message.
Your cooperation is appreciated.


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Burrow (queue service)

[Eric Day]

Hi Sandy,

On Thu, Feb 24, 2011 at 07:42:34PM +, Sandy Walsh wrote:
 Looks like a fun project Eric. I only got caught up on the ML this weekend 
 and I'm behind again already. 

It's a never-ending battle. I find routing all messages from jaypipes
into /dev/null helps. :)

 1. Will broadcast queues be supported? 

As we discussed on IRC, this can be one of two this. Broadcast
(unreliable) or replication (reliable). The former is part of the
service. The latter is possible using workers from the start, but
in the future we'll probably add a special router API for these
specialized workers. This will look kind of like exchanges in AMQP. For
more info on both, see:

http://wiki.openstack.org/QueueService#Multi-cast_Event_Notifications
http://wiki.openstack.org/QueueService#Other_Features

 2. What are the guarantees of idempotence in the event of retries/errors/etc? 
 (not applicable to broadcast obviously).

Basically, for messages that were added with persistence, we error on
the side of duplicating the message instead of loosing the message. The
client can be fully aware of this. If a delete fails (server holding
the message is down), it can ignore (and duplicates are fine), revert
the work, or mark that a duplicate may be coming (mark the message
ID somewhere to check later when the duplicate may show up).

 3. Could there be support for WAN clusters? i.e. only forward certain queues 
 to remote locations to keep chatter down?

Sure, we can construct any topology we want. For a public queue
service, I'm thinking having a separate cluster per zone (whatever
that means for deployment), and then clients can use the local zone
or remote zone, probably preferring the local for speed.

 4. Are all operations assumed to be one-way, or is it assumed that return 
 values are part of each call. i.e. do I have to set up other queues for 
 return values as callbacks or is that part of the framework? Is it optional?

Right now it's all async. A sync operation with a 60 second timeout
could be done with:

PUT /account/queue/message_id
GET /account/queue/message_id_result?wait=60

And you would write your worker to PUT the result in a message named
message_id_result.

-Eric

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Burrow (queue service)

[Jay Pipes]

On Thu, Feb 24, 2011 at 3:56 PM, Eric Day e...@oddments.org wrote:
 Hi Sandy,

 On Thu, Feb 24, 2011 at 07:42:34PM +, Sandy Walsh wrote:
 Looks like a fun project Eric. I only got caught up on the ML this weekend 
 and I'm behind again already.

 It's a never-ending battle. I find routing all messages from jaypipes
 into /dev/null helps. :)

I heard that.

-jay

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp