Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-24 Thread Flavio Percoco
On 09/23/2014 11:59 PM, Joe Gordon wrote: On Tue, Sep 23, 2014 at 9:13 AM, Zane Bitter zbit...@redhat.com mailto:zbit...@redhat.com wrote: On 22/09/14 22:04, Joe Gordon wrote: To me this is less about valid or invalid choices. The Zaqar team is comparing Zaqar to

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-24 Thread Flavio Percoco
On 09/24/2014 03:48 AM, Devananda van der Veen wrote: I've taken a bit of time out of this thread, and I'd like to jump back in now and attempt to summarize what I've learned and hopefully frame it in such a way that it helps us to answer the question Thierry asked: I *loved* it! Thanks a

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-24 Thread Flavio Percoco
On 09/24/2014 12:06 AM, Joe Gordon wrote: On Tue, Sep 23, 2014 at 2:40 AM, Flavio Percoco fla...@redhat.com mailto:fla...@redhat.com wrote: On 09/23/2014 05:13 AM, Clint Byrum wrote: Excerpts from Joe Gordon's message of 2014-09-22 19:04:03 -0700: [snip] To

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-24 Thread Zane Bitter
On 23/09/14 17:59, Joe Gordon wrote: Zaqar is aiming for low latency per message, SQS doesn't appear to be. I've seen no evidence that Zaqar is actually aiming for that. There are waaay lower-latency ways to implement messaging if you don't care about durability (you wouldn't do

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-24 Thread Zane Bitter
On 23/09/14 19:29, Devananda van der Veen wrote: On Mon, Sep 22, 2014 at 5:47 PM, Zane Bitter zbit...@redhat.com wrote: On 22/09/14 17:06, Joe Gordon wrote: If 50,000 messages per second doesn't count as small-to-moderate then Zaqar does not fulfill a major SQS use case. It's not a drop-in

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-24 Thread Gordon Sim
Apologies in advance for possible repetition and pedantry... On 09/24/2014 02:48 AM, Devananda van der Veen wrote: 2. Single Delivery - each message must be processed *exactly* once Example: Using a queue to process votes. Every vote must be counted only once. It is also important to

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-23 Thread Gordon Sim
On 09/22/2014 05:58 PM, Zane Bitter wrote: On 22/09/14 10:11, Gordon Sim wrote: As I understand it, pools don't help scaling a given queue since all the messages for that queue must be in the same pool. At present traffic through different Zaqar queues are essentially entirely orthogonal

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-23 Thread Flavio Percoco
On 09/23/2014 05:13 AM, Clint Byrum wrote: Excerpts from Joe Gordon's message of 2014-09-22 19:04:03 -0700: [snip] To me this is less about valid or invalid choices. The Zaqar team is comparing Zaqar to SQS, but after digging into the two of them, zaqar barely looks like SQS. Zaqar doesn't

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-23 Thread Flavio Percoco
On 09/23/2014 10:58 AM, Gordon Sim wrote: On 09/22/2014 05:58 PM, Zane Bitter wrote: On 22/09/14 10:11, Gordon Sim wrote: As I understand it, pools don't help scaling a given queue since all the messages for that queue must be in the same pool. At present traffic through different Zaqar

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-23 Thread Zane Bitter
On 23/09/14 08:58, Flavio Percoco wrote: I believe the guarantee is still useful and it currently does not represent an issue for the service nor the user. 2 things could happen to FIFO in the future: 1. It's made optional and we allow users to opt-in in a per flavor basis. (I personally don't

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-23 Thread Zane Bitter
On 22/09/14 22:04, Joe Gordon wrote: To me this is less about valid or invalid choices. The Zaqar team is comparing Zaqar to SQS, but after digging into the two of them, zaqar barely looks like SQS. Zaqar doesn't guarantee what IMHO is the most important parts of SQS: the message will be

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-23 Thread Fox, Kevin M
Subject: Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues On 09/23/2014 10:58 AM, Gordon Sim wrote: On 09/22/2014 05:58 PM, Zane Bitter wrote: On 22/09/14 10:11, Gordon Sim wrote: As I understand it, pools don't help scaling a given queue since all the messages

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-23 Thread Flavio Percoco
@lists.openstack.org Subject: Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues On 09/23/2014 10:58 AM, Gordon Sim wrote: On 09/22/2014 05:58 PM, Zane Bitter wrote: On 22/09/14 10:11, Gordon Sim wrote: As I understand it, pools don't help scaling a given queue since all

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-23 Thread Joe Gordon
On Tue, Sep 23, 2014 at 9:13 AM, Zane Bitter zbit...@redhat.com wrote: On 22/09/14 22:04, Joe Gordon wrote: To me this is less about valid or invalid choices. The Zaqar team is comparing Zaqar to SQS, but after digging into the two of them, zaqar barely looks like SQS. Zaqar doesn't

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-23 Thread Joe Gordon
On Tue, Sep 23, 2014 at 2:40 AM, Flavio Percoco fla...@redhat.com wrote: On 09/23/2014 05:13 AM, Clint Byrum wrote: Excerpts from Joe Gordon's message of 2014-09-22 19:04:03 -0700: [snip] To me this is less about valid or invalid choices. The Zaqar team is comparing Zaqar to SQS, but

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-23 Thread Clint Byrum
Excerpts from Joe Gordon's message of 2014-09-23 14:59:33 -0700: On Tue, Sep 23, 2014 at 9:13 AM, Zane Bitter zbit...@redhat.com wrote: On 22/09/14 22:04, Joe Gordon wrote: To me this is less about valid or invalid choices. The Zaqar team is comparing Zaqar to SQS, but after digging

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-23 Thread Devananda van der Veen
On Mon, Sep 22, 2014 at 5:47 PM, Zane Bitter zbit...@redhat.com wrote: On 22/09/14 17:06, Joe Gordon wrote: If 50,000 messages per second doesn't count as small-to-moderate then Zaqar does not fulfill a major SQS use case. It's not a drop-in replacement, but as I mentioned you can recreate

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-23 Thread Devananda van der Veen
I've taken a bit of time out of this thread, and I'd like to jump back in now and attempt to summarize what I've learned and hopefully frame it in such a way that it helps us to answer the question Thierry asked: On Fri, Sep 19, 2014 at 2:00 AM, Thierry Carrez thie...@openstack.org wrote: The

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-22 Thread Gordon Sim
On 09/19/2014 09:13 PM, Zane Bitter wrote: SQS offers very, very limited guarantees, and it's clear that the reason for that is to make it massively, massively scalable in the way that e.g. S3 is scalable while also remaining comparably durable (S3 is supposedly designed for 11 nines, BTW).

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-22 Thread Zane Bitter
On 22/09/14 10:11, Gordon Sim wrote: On 09/19/2014 09:13 PM, Zane Bitter wrote: SQS offers very, very limited guarantees, and it's clear that the reason for that is to make it massively, massively scalable in the way that e.g. S3 is scalable while also remaining comparably durable (S3 is

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-22 Thread Joe Gordon
On Mon, Sep 22, 2014 at 9:58 AM, Zane Bitter zbit...@redhat.com wrote: On 22/09/14 10:11, Gordon Sim wrote: On 09/19/2014 09:13 PM, Zane Bitter wrote: SQS offers very, very limited guarantees, and it's clear that the reason for that is to make it massively, massively scalable in the way

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-22 Thread Zane Bitter
On 22/09/14 17:06, Joe Gordon wrote: On Mon, Sep 22, 2014 at 9:58 AM, Zane Bitter zbit...@redhat.com wrote: On 22/09/14 10:11, Gordon Sim wrote: On 09/19/2014 09:13 PM, Zane Bitter wrote: SQS offers very, very limited guarantees, and it's clear that the reason for that is to make it

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-22 Thread Joe Gordon
On Mon, Sep 22, 2014 at 5:47 PM, Zane Bitter zbit...@redhat.com wrote: On 22/09/14 17:06, Joe Gordon wrote: On Mon, Sep 22, 2014 at 9:58 AM, Zane Bitter zbit...@redhat.com wrote: On 22/09/14 10:11, Gordon Sim wrote: On 09/19/2014 09:13 PM, Zane Bitter wrote: SQS offers very, very

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-22 Thread Joe Gordon
On Mon, Sep 22, 2014 at 7:04 PM, Joe Gordon joe.gord...@gmail.com wrote: On Mon, Sep 22, 2014 at 5:47 PM, Zane Bitter zbit...@redhat.com wrote: On 22/09/14 17:06, Joe Gordon wrote: On Mon, Sep 22, 2014 at 9:58 AM, Zane Bitter zbit...@redhat.com wrote: On 22/09/14 10:11, Gordon Sim

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-22 Thread Clint Byrum
Excerpts from Joe Gordon's message of 2014-09-22 19:04:03 -0700: On Mon, Sep 22, 2014 at 5:47 PM, Zane Bitter zbit...@redhat.com wrote: On 22/09/14 17:06, Joe Gordon wrote: On Mon, Sep 22, 2014 at 9:58 AM, Zane Bitter zbit...@redhat.com wrote: On 22/09/14 10:11, Gordon Sim wrote:

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-22 Thread Joe Gordon
On Mon, Sep 22, 2014 at 8:13 PM, Clint Byrum cl...@fewbar.com wrote: Excerpts from Joe Gordon's message of 2014-09-22 19:04:03 -0700: On Mon, Sep 22, 2014 at 5:47 PM, Zane Bitter zbit...@redhat.com wrote: On 22/09/14 17:06, Joe Gordon wrote: On Mon, Sep 22, 2014 at 9:58 AM, Zane

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-19 Thread Flavio Percoco
On 09/18/2014 09:25 PM, Gordon Sim wrote: On 09/18/2014 03:45 PM, Flavio Percoco wrote: On 09/18/2014 04:09 PM, Gordon Sim wrote: Is the replication synchronous or asynchronous with respect to client calls? E.g. will the response to a post of messages be returned only once the replication of

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-19 Thread Flavio Percoco
On 09/18/2014 07:19 PM, Joe Gordon wrote: On Thu, Sep 18, 2014 at 7:45 AM, Flavio Percoco fla...@redhat.com mailto:fla...@redhat.com wrote: On 09/18/2014 04:09 PM, Gordon Sim wrote: On 09/18/2014 12:31 PM, Flavio Percoco wrote: On 09/17/2014 10:36 PM, Joe Gordon wrote:

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-19 Thread Flavio Percoco
On 09/18/2014 07:16 PM, Devananda van der Veen wrote: On Thu, Sep 18, 2014 at 8:54 AM, Devananda van der Veen devananda@gmail.com wrote: On Thu, Sep 18, 2014 at 7:45 AM, Flavio Percoco fla...@redhat.com wrote: On 09/18/2014 04:09 PM, Gordon Sim wrote: On 09/18/2014 12:31 PM, Flavio

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-19 Thread Thierry Carrez
Joe Gordon wrote: On Thu, Sep 18, 2014 at 9:02 AM, Devananda van der Veen devananda@gmail.com mailto:devananda@gmail.com wrote: - guaranteed message order - not distributing work across a configurable number of back ends These are scale-limiting design choices which are

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-19 Thread Flavio Percoco
On 09/19/2014 11:00 AM, Thierry Carrez wrote: Joe Gordon wrote: On Thu, Sep 18, 2014 at 9:02 AM, Devananda van der Veen devananda@gmail.com mailto:devananda@gmail.com wrote: - guaranteed message order - not distributing work across a configurable number of back ends

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-19 Thread Eoghan Glynn
Hi All, My understanding of Zaqar is that it's like SQS. SQS uses distributed queues, which have a few unusual properties [0]: Message Order Amazon SQS makes a best effort to preserve order in messages, but due to the distributed nature of the queue, we cannot guarantee you will

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-19 Thread Clint Byrum
Excerpts from Eoghan Glynn's message of 2014-09-19 04:23:55 -0700: Hi All, My understanding of Zaqar is that it's like SQS. SQS uses distributed queues, which have a few unusual properties [0]: Message Order Amazon SQS makes a best effort to preserve order in messages, but

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-19 Thread Gordon Sim
On 09/19/2014 08:53 AM, Flavio Percoco wrote: On 09/18/2014 09:25 PM, Gordon Sim wrote: I don't see that the claim mechanism brings any stronger guarantee, it just offers a competing consumer behaviour where browsing is non-competing (non-destructive). In both cases you require the client to be

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-19 Thread Vipul Sabhaya
On Fri, Sep 19, 2014 at 4:23 AM, Eoghan Glynn egl...@redhat.com wrote: Hi All, My understanding of Zaqar is that it's like SQS. SQS uses distributed queues, which have a few unusual properties [0]: Message Order Amazon SQS makes a best effort to preserve order in messages, but

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-19 Thread Zane Bitter
On 18/09/14 10:55, Flavio Percoco wrote: On 09/18/2014 04:24 PM, Clint Byrum wrote: Great job highlighting what our friends over at Amazon are doing. It's clear from these snippets, and a few other pieces of documentation for SQS I've read, that the Amazon team approached SQS from a _massive_

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-18 Thread Flavio Percoco
On 09/17/2014 10:36 PM, Joe Gordon wrote: Hi All, My understanding of Zaqar is that it's like SQS. SQS uses distributed queues, which have a few unusual properties [0]: Message Order Amazon SQS makes a best effort to preserve order in messages, but due to the distributed nature

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-18 Thread Gordon Sim
On 09/18/2014 12:31 PM, Flavio Percoco wrote: On 09/17/2014 10:36 PM, Joe Gordon wrote: My understanding of Zaqar is that it's like SQS. SQS uses distributed queues, which have a few unusual properties [0]: Message Order Amazon SQS makes a best effort to preserve order in messages, but

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-18 Thread Clint Byrum
Great job highlighting what our friends over at Amazon are doing. It's clear from these snippets, and a few other pieces of documentation for SQS I've read, that the Amazon team approached SQS from a _massive_ scaling perspective. I think what may be forcing a lot of this frustration with Zaqar

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-18 Thread Flavio Percoco
On 09/18/2014 04:09 PM, Gordon Sim wrote: On 09/18/2014 12:31 PM, Flavio Percoco wrote: On 09/17/2014 10:36 PM, Joe Gordon wrote: My understanding of Zaqar is that it's like SQS. SQS uses distributed queues, which have a few unusual properties [0]: Message Order Amazon SQS makes a

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-18 Thread Flavio Percoco
On 09/18/2014 04:24 PM, Clint Byrum wrote: Great job highlighting what our friends over at Amazon are doing. It's clear from these snippets, and a few other pieces of documentation for SQS I've read, that the Amazon team approached SQS from a _massive_ scaling perspective. I think what may

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-18 Thread Steve Lewis
On September 18, 2014 7:45 AM, Flavio Percoco wrote: On 09/18/2014 04:09 PM, Gordon Sim wrote: However, as far as consuming messages is concerned, it can guarantee once-and-only-once and/or at-least-once delivery depending on the message pattern used to consume messages. Using pop or claims

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-18 Thread Flavio Percoco
On 09/18/2014 05:42 PM, Steve Lewis wrote: On September 18, 2014 7:45 AM, Flavio Percoco wrote: On 09/18/2014 04:09 PM, Gordon Sim wrote: However, as far as consuming messages is concerned, it can guarantee once-and-only-once and/or at-least-once delivery depending on the message pattern

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-18 Thread Devananda van der Veen
On Thu, Sep 18, 2014 at 7:45 AM, Flavio Percoco fla...@redhat.com wrote: On 09/18/2014 04:09 PM, Gordon Sim wrote: On 09/18/2014 12:31 PM, Flavio Percoco wrote: Zaqar guarantees FIFO. To be more precise, it does that relying on the storage backend ability to do so as well. Depending on the

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-18 Thread Devananda van der Veen
On Thu, Sep 18, 2014 at 7:55 AM, Flavio Percoco fla...@redhat.com wrote: On 09/18/2014 04:24 PM, Clint Byrum wrote: Great job highlighting what our friends over at Amazon are doing. It's clear from these snippets, and a few other pieces of documentation for SQS I've read, that the Amazon team

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-18 Thread Joe Gordon
On Thu, Sep 18, 2014 at 9:02 AM, Devananda van der Veen devananda@gmail.com wrote: On Thu, Sep 18, 2014 at 7:55 AM, Flavio Percoco fla...@redhat.com wrote: On 09/18/2014 04:24 PM, Clint Byrum wrote: Great job highlighting what our friends over at Amazon are doing. It's clear from

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-18 Thread Devananda van der Veen
On Thu, Sep 18, 2014 at 8:54 AM, Devananda van der Veen devananda@gmail.com wrote: On Thu, Sep 18, 2014 at 7:45 AM, Flavio Percoco fla...@redhat.com wrote: On 09/18/2014 04:09 PM, Gordon Sim wrote: On 09/18/2014 12:31 PM, Flavio Percoco wrote: Zaqar guarantees FIFO. To be more precise, it

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-18 Thread Joe Gordon
On Thu, Sep 18, 2014 at 7:45 AM, Flavio Percoco fla...@redhat.com wrote: On 09/18/2014 04:09 PM, Gordon Sim wrote: On 09/18/2014 12:31 PM, Flavio Percoco wrote: On 09/17/2014 10:36 PM, Joe Gordon wrote: My understanding of Zaqar is that it's like SQS. SQS uses distributed queues, which

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

2014-09-18 Thread Gordon Sim
On 09/18/2014 03:45 PM, Flavio Percoco wrote: On 09/18/2014 04:09 PM, Gordon Sim wrote: Is the replication synchronous or asynchronous with respect to client calls? E.g. will the response to a post of messages be returned only once the replication of those messages is confirmed? Likewise when