Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]
On 09/16/2014 08:55 AM, Flavio Percoco wrote: pub/sub doesn't necessarily guarantees messages delivery, it really depends on the implementation. As I understand it, the model for pub-sub in Zaqar is to have multiple subscribers polling the queue with gets, and have the messages removed from the queue only when they expire. Is that right? If the ttl of the messages is long enough, a subscriber can start getting the queue from where they left off (if they have or can recover their last used marker) or from the head of the queue. So although not acknowledged, subscribers can retry on failover providing they do so before the message expires. That said, there are ways to guarantee that depending on the method used. For example, if the subscriber is a webhook, we can use the response status code to ack the message. if it has a persistent connection like websocket or even (long|short)-poll an ack may be needed. In the pub-sub case, to remove a message based on acks you need to wait until all known subscribers have acked it. With the current model there is no explicit concept of subscriber (nor of ack in the non-competing consumer case). Without changing that I don't think you can use the response of a webhook anyway (unless of course there are not get style subscribers on the queue). ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]
On 09/15/2014 09:33 PM, Clint Byrum wrote: Excerpts from Zane Bitter's message of 2014-09-15 12:05:09 -0700: On 15/09/14 13:28, Clint Byrum wrote: Excerpts from Flavio Percoco's message of 2014-09-15 00:57:05 -0700: On 09/12/2014 07:13 PM, Clint Byrum wrote: Excerpts from Thierry Carrez's message of 2014-09-12 02:16:42 -0700: Clint Byrum wrote: Excerpts from Flavio Percoco's message of 2014-09-11 04:14:30 -0700: Is Zaqar being optimized as a *queuing* service? I'd say no. Our goal is to optimize Zaqar for delivering messages and supporting different messaging patterns. Awesome! Just please don't expect people to get excited about it for the lighter weight queueing workloads that you've claimed as use cases. I totally see Horizon using it to keep events for users. I see Heat using it for stack events as well. I would bet that Trove would benefit from being able to communicate messages to users. But I think in between Zaqar and the backends will likely be a lighter weight queue-only service that the users can just subscribe to when they don't want an inbox. And I think that lighter weight queue service is far more important for OpenStack than the full blown random access inbox. I think the reason such a thing has not appeared is because we were all sort of running into but Zaqar is already incubated. Now that we've fleshed out the difference, I think those of us that need a lightweight multi-tenant queue service should add it to OpenStack. Separately. I hope that doesn't offend you and the rest of the excellent Zaqar developers. It is just a different thing. Should we remove all the semantics that allow people to use Zaqar as a queue service? I don't think so either. Again, the semantics are there because Zaqar is using them to do its job. Whether other folks may/may not use Zaqar as a queue service is out of our control. This doesn't mean the project is broken. No, definitely not broken. It just isn't actually necessary for many of the stated use cases. Clint, If I read you correctly, you're basically saying the Zaqar is overkill for a lot of people who only want a multi-tenant queue service. It's doing A+B. Why does that prevent people who only need A from using it ? Is it that it's actually not doing A well, from a user perspective ? Like the performance sucks, or it's missing a key primitive ? Is it that it's unnecessarily complex to deploy, from a deployer perspective, and that something only doing A would be simpler, while covering most of the use cases? Is it something else ? I want to make sure I understand your objection. In the user perspective it might make sense to pursue both options as separate projects. In the deployer perspective case, having a project doing A+B and a project doing A doesn't solve anything. So this affects the decision we have to take next Tuesday... I believe that Zaqar does two things, inbox semantics, and queue semantics. I believe the queueing is a side-effect of needing some kind of queue to enable users to store and subscribe to messages in the inbox. What I'd rather see is an API for queueing, and an API for inboxes which integrates well with the queueing API. For instance, if a user says give me an inbox I think Zaqar should return a queue handle for sending into the inbox the same way Nova gives you a Neutron port if you don't give it one. You might also ask for a queue to receive push messages from the inbox. Point being, the queues are not the inbox, and the inbox is not the queues. However, if I just want a queue, just give me a queue. Don't store my messages in a randomly addressable space, and don't saddle the deployer with the burden of such storage. Put the queue API in front of a scalable message queue and give me a nice simple HTTP API. Users would likely be thrilled. Heat, Nova, Ceilometer, probably Trove and Sahara, could all make use of just this. Only Horizon seems to need a place to keep the messages around while users inspect them. Whether that is two projects, or one, separation between the two API's, and thus two very different types of backends, is something I think will lead to more deployers wanting to deploy both, so that they can bill usage appropriately and so that their users can choose wisely. This is one of the use-cases we designed flavors for. One of the mail ideas behind flavors is giving the user the choice of where they want their messages to be stored. This certainly requires the deployer to have installed stores that are good for each job. For example, based on the current existing drivers, a deployer could have configured a high-throughput flavor on top of a redis node that has been configured to perform for this job. Alongside to this flavor, the deployer could've configured a flavor that features durability on top of mongodb or redis. When the user creates the queue/bucket/inbox/whatever they want to put their messages into,
Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]
On 09/12/2014 07:13 PM, Clint Byrum wrote: Excerpts from Thierry Carrez's message of 2014-09-12 02:16:42 -0700: Clint Byrum wrote: Excerpts from Flavio Percoco's message of 2014-09-11 04:14:30 -0700: Is Zaqar being optimized as a *queuing* service? I'd say no. Our goal is to optimize Zaqar for delivering messages and supporting different messaging patterns. Awesome! Just please don't expect people to get excited about it for the lighter weight queueing workloads that you've claimed as use cases. I totally see Horizon using it to keep events for users. I see Heat using it for stack events as well. I would bet that Trove would benefit from being able to communicate messages to users. But I think in between Zaqar and the backends will likely be a lighter weight queue-only service that the users can just subscribe to when they don't want an inbox. And I think that lighter weight queue service is far more important for OpenStack than the full blown random access inbox. I think the reason such a thing has not appeared is because we were all sort of running into but Zaqar is already incubated. Now that we've fleshed out the difference, I think those of us that need a lightweight multi-tenant queue service should add it to OpenStack. Separately. I hope that doesn't offend you and the rest of the excellent Zaqar developers. It is just a different thing. Should we remove all the semantics that allow people to use Zaqar as a queue service? I don't think so either. Again, the semantics are there because Zaqar is using them to do its job. Whether other folks may/may not use Zaqar as a queue service is out of our control. This doesn't mean the project is broken. No, definitely not broken. It just isn't actually necessary for many of the stated use cases. Clint, If I read you correctly, you're basically saying the Zaqar is overkill for a lot of people who only want a multi-tenant queue service. It's doing A+B. Why does that prevent people who only need A from using it ? Is it that it's actually not doing A well, from a user perspective ? Like the performance sucks, or it's missing a key primitive ? Is it that it's unnecessarily complex to deploy, from a deployer perspective, and that something only doing A would be simpler, while covering most of the use cases? Is it something else ? I want to make sure I understand your objection. In the user perspective it might make sense to pursue both options as separate projects. In the deployer perspective case, having a project doing A+B and a project doing A doesn't solve anything. So this affects the decision we have to take next Tuesday... I believe that Zaqar does two things, inbox semantics, and queue semantics. I believe the queueing is a side-effect of needing some kind of queue to enable users to store and subscribe to messages in the inbox. What I'd rather see is an API for queueing, and an API for inboxes which integrates well with the queueing API. For instance, if a user says give me an inbox I think Zaqar should return a queue handle for sending into the inbox the same way Nova gives you a Neutron port if you don't give it one. You might also ask for a queue to receive push messages from the inbox. Point being, the queues are not the inbox, and the inbox is not the queues. However, if I just want a queue, just give me a queue. Don't store my messages in a randomly addressable space, and don't saddle the deployer with the burden of such storage. Put the queue API in front of a scalable message queue and give me a nice simple HTTP API. Users would likely be thrilled. Heat, Nova, Ceilometer, probably Trove and Sahara, could all make use of just this. Only Horizon seems to need a place to keep the messages around while users inspect them. Whether that is two projects, or one, separation between the two API's, and thus two very different types of backends, is something I think will lead to more deployers wanting to deploy both, so that they can bill usage appropriately and so that their users can choose wisely. This is one of the use-cases we designed flavors for. One of the mail ideas behind flavors is giving the user the choice of where they want their messages to be stored. This certainly requires the deployer to have installed stores that are good for each job. For example, based on the current existing drivers, a deployer could have configured a high-throughput flavor on top of a redis node that has been configured to perform for this job. Alongside to this flavor, the deployer could've configured a flavor that features durability on top of mongodb or redis. When the user creates the queue/bucket/inbox/whatever they want to put their messages into, they'll be able to choose where those messages should be stored into based on their needs. I do understand your objection is not against Zaqar being able to do this now or not but whether an integrate API for both kind of semantics
Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]
Excerpts from Flavio Percoco's message of 2014-09-15 00:57:05 -0700: On 09/12/2014 07:13 PM, Clint Byrum wrote: Excerpts from Thierry Carrez's message of 2014-09-12 02:16:42 -0700: Clint Byrum wrote: Excerpts from Flavio Percoco's message of 2014-09-11 04:14:30 -0700: Is Zaqar being optimized as a *queuing* service? I'd say no. Our goal is to optimize Zaqar for delivering messages and supporting different messaging patterns. Awesome! Just please don't expect people to get excited about it for the lighter weight queueing workloads that you've claimed as use cases. I totally see Horizon using it to keep events for users. I see Heat using it for stack events as well. I would bet that Trove would benefit from being able to communicate messages to users. But I think in between Zaqar and the backends will likely be a lighter weight queue-only service that the users can just subscribe to when they don't want an inbox. And I think that lighter weight queue service is far more important for OpenStack than the full blown random access inbox. I think the reason such a thing has not appeared is because we were all sort of running into but Zaqar is already incubated. Now that we've fleshed out the difference, I think those of us that need a lightweight multi-tenant queue service should add it to OpenStack. Separately. I hope that doesn't offend you and the rest of the excellent Zaqar developers. It is just a different thing. Should we remove all the semantics that allow people to use Zaqar as a queue service? I don't think so either. Again, the semantics are there because Zaqar is using them to do its job. Whether other folks may/may not use Zaqar as a queue service is out of our control. This doesn't mean the project is broken. No, definitely not broken. It just isn't actually necessary for many of the stated use cases. Clint, If I read you correctly, you're basically saying the Zaqar is overkill for a lot of people who only want a multi-tenant queue service. It's doing A+B. Why does that prevent people who only need A from using it ? Is it that it's actually not doing A well, from a user perspective ? Like the performance sucks, or it's missing a key primitive ? Is it that it's unnecessarily complex to deploy, from a deployer perspective, and that something only doing A would be simpler, while covering most of the use cases? Is it something else ? I want to make sure I understand your objection. In the user perspective it might make sense to pursue both options as separate projects. In the deployer perspective case, having a project doing A+B and a project doing A doesn't solve anything. So this affects the decision we have to take next Tuesday... I believe that Zaqar does two things, inbox semantics, and queue semantics. I believe the queueing is a side-effect of needing some kind of queue to enable users to store and subscribe to messages in the inbox. What I'd rather see is an API for queueing, and an API for inboxes which integrates well with the queueing API. For instance, if a user says give me an inbox I think Zaqar should return a queue handle for sending into the inbox the same way Nova gives you a Neutron port if you don't give it one. You might also ask for a queue to receive push messages from the inbox. Point being, the queues are not the inbox, and the inbox is not the queues. However, if I just want a queue, just give me a queue. Don't store my messages in a randomly addressable space, and don't saddle the deployer with the burden of such storage. Put the queue API in front of a scalable message queue and give me a nice simple HTTP API. Users would likely be thrilled. Heat, Nova, Ceilometer, probably Trove and Sahara, could all make use of just this. Only Horizon seems to need a place to keep the messages around while users inspect them. Whether that is two projects, or one, separation between the two API's, and thus two very different types of backends, is something I think will lead to more deployers wanting to deploy both, so that they can bill usage appropriately and so that their users can choose wisely. This is one of the use-cases we designed flavors for. One of the mail ideas behind flavors is giving the user the choice of where they want their messages to be stored. This certainly requires the deployer to have installed stores that are good for each job. For example, based on the current existing drivers, a deployer could have configured a high-throughput flavor on top of a redis node that has been configured to perform for this job. Alongside to this flavor, the deployer could've configured a flavor that features durability on top of mongodb or redis. When the user creates the queue/bucket/inbox/whatever they want to put their messages into, they'll be able to choose where those messages should be stored
Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]
On 15/09/14 13:28, Clint Byrum wrote: Excerpts from Flavio Percoco's message of 2014-09-15 00:57:05 -0700: On 09/12/2014 07:13 PM, Clint Byrum wrote: Excerpts from Thierry Carrez's message of 2014-09-12 02:16:42 -0700: Clint Byrum wrote: Excerpts from Flavio Percoco's message of 2014-09-11 04:14:30 -0700: Is Zaqar being optimized as a *queuing* service? I'd say no. Our goal is to optimize Zaqar for delivering messages and supporting different messaging patterns. Awesome! Just please don't expect people to get excited about it for the lighter weight queueing workloads that you've claimed as use cases. I totally see Horizon using it to keep events for users. I see Heat using it for stack events as well. I would bet that Trove would benefit from being able to communicate messages to users. But I think in between Zaqar and the backends will likely be a lighter weight queue-only service that the users can just subscribe to when they don't want an inbox. And I think that lighter weight queue service is far more important for OpenStack than the full blown random access inbox. I think the reason such a thing has not appeared is because we were all sort of running into but Zaqar is already incubated. Now that we've fleshed out the difference, I think those of us that need a lightweight multi-tenant queue service should add it to OpenStack. Separately. I hope that doesn't offend you and the rest of the excellent Zaqar developers. It is just a different thing. Should we remove all the semantics that allow people to use Zaqar as a queue service? I don't think so either. Again, the semantics are there because Zaqar is using them to do its job. Whether other folks may/may not use Zaqar as a queue service is out of our control. This doesn't mean the project is broken. No, definitely not broken. It just isn't actually necessary for many of the stated use cases. Clint, If I read you correctly, you're basically saying the Zaqar is overkill for a lot of people who only want a multi-tenant queue service. It's doing A+B. Why does that prevent people who only need A from using it ? Is it that it's actually not doing A well, from a user perspective ? Like the performance sucks, or it's missing a key primitive ? Is it that it's unnecessarily complex to deploy, from a deployer perspective, and that something only doing A would be simpler, while covering most of the use cases? Is it something else ? I want to make sure I understand your objection. In the user perspective it might make sense to pursue both options as separate projects. In the deployer perspective case, having a project doing A+B and a project doing A doesn't solve anything. So this affects the decision we have to take next Tuesday... I believe that Zaqar does two things, inbox semantics, and queue semantics. I believe the queueing is a side-effect of needing some kind of queue to enable users to store and subscribe to messages in the inbox. What I'd rather see is an API for queueing, and an API for inboxes which integrates well with the queueing API. For instance, if a user says give me an inbox I think Zaqar should return a queue handle for sending into the inbox the same way Nova gives you a Neutron port if you don't give it one. You might also ask for a queue to receive push messages from the inbox. Point being, the queues are not the inbox, and the inbox is not the queues. However, if I just want a queue, just give me a queue. Don't store my messages in a randomly addressable space, and don't saddle the deployer with the burden of such storage. Put the queue API in front of a scalable message queue and give me a nice simple HTTP API. Users would likely be thrilled. Heat, Nova, Ceilometer, probably Trove and Sahara, could all make use of just this. Only Horizon seems to need a place to keep the messages around while users inspect them. Whether that is two projects, or one, separation between the two API's, and thus two very different types of backends, is something I think will lead to more deployers wanting to deploy both, so that they can bill usage appropriately and so that their users can choose wisely. This is one of the use-cases we designed flavors for. One of the mail ideas behind flavors is giving the user the choice of where they want their messages to be stored. This certainly requires the deployer to have installed stores that are good for each job. For example, based on the current existing drivers, a deployer could have configured a high-throughput flavor on top of a redis node that has been configured to perform for this job. Alongside to this flavor, the deployer could've configured a flavor that features durability on top of mongodb or redis. When the user creates the queue/bucket/inbox/whatever they want to put their messages into, they'll be able to choose where those messages should be stored into based on their needs. I do understand your objection is not against Zaqar being able to do this now or not but
Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]
Excerpts from Zane Bitter's message of 2014-09-15 12:05:09 -0700: On 15/09/14 13:28, Clint Byrum wrote: Excerpts from Flavio Percoco's message of 2014-09-15 00:57:05 -0700: On 09/12/2014 07:13 PM, Clint Byrum wrote: Excerpts from Thierry Carrez's message of 2014-09-12 02:16:42 -0700: Clint Byrum wrote: Excerpts from Flavio Percoco's message of 2014-09-11 04:14:30 -0700: Is Zaqar being optimized as a *queuing* service? I'd say no. Our goal is to optimize Zaqar for delivering messages and supporting different messaging patterns. Awesome! Just please don't expect people to get excited about it for the lighter weight queueing workloads that you've claimed as use cases. I totally see Horizon using it to keep events for users. I see Heat using it for stack events as well. I would bet that Trove would benefit from being able to communicate messages to users. But I think in between Zaqar and the backends will likely be a lighter weight queue-only service that the users can just subscribe to when they don't want an inbox. And I think that lighter weight queue service is far more important for OpenStack than the full blown random access inbox. I think the reason such a thing has not appeared is because we were all sort of running into but Zaqar is already incubated. Now that we've fleshed out the difference, I think those of us that need a lightweight multi-tenant queue service should add it to OpenStack. Separately. I hope that doesn't offend you and the rest of the excellent Zaqar developers. It is just a different thing. Should we remove all the semantics that allow people to use Zaqar as a queue service? I don't think so either. Again, the semantics are there because Zaqar is using them to do its job. Whether other folks may/may not use Zaqar as a queue service is out of our control. This doesn't mean the project is broken. No, definitely not broken. It just isn't actually necessary for many of the stated use cases. Clint, If I read you correctly, you're basically saying the Zaqar is overkill for a lot of people who only want a multi-tenant queue service. It's doing A+B. Why does that prevent people who only need A from using it ? Is it that it's actually not doing A well, from a user perspective ? Like the performance sucks, or it's missing a key primitive ? Is it that it's unnecessarily complex to deploy, from a deployer perspective, and that something only doing A would be simpler, while covering most of the use cases? Is it something else ? I want to make sure I understand your objection. In the user perspective it might make sense to pursue both options as separate projects. In the deployer perspective case, having a project doing A+B and a project doing A doesn't solve anything. So this affects the decision we have to take next Tuesday... I believe that Zaqar does two things, inbox semantics, and queue semantics. I believe the queueing is a side-effect of needing some kind of queue to enable users to store and subscribe to messages in the inbox. What I'd rather see is an API for queueing, and an API for inboxes which integrates well with the queueing API. For instance, if a user says give me an inbox I think Zaqar should return a queue handle for sending into the inbox the same way Nova gives you a Neutron port if you don't give it one. You might also ask for a queue to receive push messages from the inbox. Point being, the queues are not the inbox, and the inbox is not the queues. However, if I just want a queue, just give me a queue. Don't store my messages in a randomly addressable space, and don't saddle the deployer with the burden of such storage. Put the queue API in front of a scalable message queue and give me a nice simple HTTP API. Users would likely be thrilled. Heat, Nova, Ceilometer, probably Trove and Sahara, could all make use of just this. Only Horizon seems to need a place to keep the messages around while users inspect them. Whether that is two projects, or one, separation between the two API's, and thus two very different types of backends, is something I think will lead to more deployers wanting to deploy both, so that they can bill usage appropriately and so that their users can choose wisely. This is one of the use-cases we designed flavors for. One of the mail ideas behind flavors is giving the user the choice of where they want their messages to be stored. This certainly requires the deployer to have installed stores that are good for each job. For example, based on the current existing drivers, a deployer could have configured a high-throughput flavor on top of a redis node that has been configured to perform for this job. Alongside to this flavor, the deployer could've configured a flavor that features durability on top of mongodb or redis. When the user creates the
Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]
Clint Byrum wrote: Excerpts from Flavio Percoco's message of 2014-09-11 04:14:30 -0700: Is Zaqar being optimized as a *queuing* service? I'd say no. Our goal is to optimize Zaqar for delivering messages and supporting different messaging patterns. Awesome! Just please don't expect people to get excited about it for the lighter weight queueing workloads that you've claimed as use cases. I totally see Horizon using it to keep events for users. I see Heat using it for stack events as well. I would bet that Trove would benefit from being able to communicate messages to users. But I think in between Zaqar and the backends will likely be a lighter weight queue-only service that the users can just subscribe to when they don't want an inbox. And I think that lighter weight queue service is far more important for OpenStack than the full blown random access inbox. I think the reason such a thing has not appeared is because we were all sort of running into but Zaqar is already incubated. Now that we've fleshed out the difference, I think those of us that need a lightweight multi-tenant queue service should add it to OpenStack. Separately. I hope that doesn't offend you and the rest of the excellent Zaqar developers. It is just a different thing. Should we remove all the semantics that allow people to use Zaqar as a queue service? I don't think so either. Again, the semantics are there because Zaqar is using them to do its job. Whether other folks may/may not use Zaqar as a queue service is out of our control. This doesn't mean the project is broken. No, definitely not broken. It just isn't actually necessary for many of the stated use cases. Clint, If I read you correctly, you're basically saying the Zaqar is overkill for a lot of people who only want a multi-tenant queue service. It's doing A+B. Why does that prevent people who only need A from using it ? Is it that it's actually not doing A well, from a user perspective ? Like the performance sucks, or it's missing a key primitive ? Is it that it's unnecessarily complex to deploy, from a deployer perspective, and that something only doing A would be simpler, while covering most of the use cases? Is it something else ? I want to make sure I understand your objection. In the user perspective it might make sense to pursue both options as separate projects. In the deployer perspective case, having a project doing A+B and a project doing A doesn't solve anything. So this affects the decision we have to take next Tuesday... Cheers, -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]
On Wed, 2014-09-10 at 14:51 +0200, Thierry Carrez wrote: Flavio Percoco wrote: [...] Based on the feedback from the meeting[3], the current main concern is: - Do we need a messaging service with a feature-set akin to SQS+SNS? [...] I think we do need, as Samuel puts it, some sort of durable message-broker/queue-server thing. It's a basic application building block. Some claim it's THE basic application building block, more useful than database provisioning. It's definitely a layer above pure IaaS, so if we end up splitting OpenStack into layers this clearly won't be in the inner one. But I think IaaS+ basic application building blocks belong in OpenStack one way or another. That's the reason I supported Designate (everyone needs DNS) and Trove (everyone needs DBs). With that said, I think yesterday there was a concern that Zaqar might not fill the some sort of durable message-broker/queue-server thing role well. The argument goes something like: if it was a queue-server then it should actually be built on top of Rabbit; if it was a message-broker it should be built on top of postfix/dovecot; the current architecture is only justified because it's something in between, so it's broken. I guess I don't mind that much zaqar being something in between: unless I misunderstood, exposing extra primitives doesn't prevent the queue-server use case from being filled. Even considering the message-broker case, I'm also not convinced building it on top of postfix/dovecot would be a net win compared to building it on top of Redis, to be honest. AFAICT, this part of the debate boils down to the following argument: If Zaqar implemented messaging-as-a-service with only queuing semantics (and no random access semantics), it's design would naturally be dramatically different and simply implement a multi-tenant REST API in front of AMQP queues like this: https://www.dropbox.com/s/yonloa9ytlf8fdh/ZaqarQueueOnly.png?dl=0 and that this architecture would allow for dramatically improved throughput for end-users while not making the cost of providing the service prohibitive to operators. You can't dismiss that argument out-of-hand, but I wonder (a) whether the claimed performance improvement is going to make a dramatic difference to the SQS-like use case and (b) whether backing this thing with an RDBMS and multiple highly available, durable AMQP broker clusters is going to be too much of a burden on operators for whatever performance improvements it does gain. But the troubling part of this debate is where we repeatedly batter the Zaqar team with hypotheses like these and appear to only barely entertain their carefully considered justification for their design decisions like: https://wiki.openstack.org/wiki/Frequently_asked_questions_%28Zaqar%29#Is_Zaqar_a_provisioning_service_or_a_data_API.3F https://wiki.openstack.org/wiki/Frequently_asked_questions_%28Zaqar%29#What_messaging_patterns_does_Zaqar_support.3F I would like to see an SQS-like API provided by OpenStack, I accept the reasons for Zaqar's design decisions to date, I respect that those decisions were made carefully by highly competent members of our community and I expect Zaqar to evolve (like all projects) in the years ahead based on more real-world feedback, new hypotheses or ideas, and lessons learned from trying things out. Mark. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]
On 09/11/2014 07:46 AM, Flavio Percoco wrote: On 09/10/2014 03:18 PM, Gordon Sim wrote: On 09/10/2014 09:58 AM, Flavio Percoco wrote: Other OpenStack components can integrate with Zaqar to surface events to end users and to communicate with guest agents that run in the over-cloud layer. I may be misunderstanding the last sentence, but I think *direct* integration of other OpenStack services with Zaqar would be a bad idea. Wouldn't this be better done through olso.messaging's notifications in some way? and/or through some standard protocol (and there's more than one to choose from)? Communicating through a specific, fixed messaging system, with its own unique protocol is actually a step backwards in my opinion, especially for things that you want to keep as loosely coupled as possible. This is exactly why various standard protocols emerged. Yes and no. The answer is yes most of the time but there are use cases, like the ones mentioned here[0], that make zaqar a good tool for the job. I certainly wasn't saying that Zaqar is not a good tool. I was merely stating that - in my opinion - wiring it in as the only tool would be a mistake. [0] https://etherpad.openstack.org/p/zaqar-integrated-projects-use-cases Again, Zaqar might be great for those cases, but none of them describe features that are unique to Zaqar, so other solutions could also fit. All I'm saying is that if the channel between openstack services and users is configurable, that will give users more choice (as well as operators) and that - in my opinion - would be a good thing. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]
2014-09-12 10:44 GMT-03:00 Gordon Sim g...@redhat.com: On 09/11/2014 07:46 AM, Flavio Percoco wrote: On 09/10/2014 03:18 PM, Gordon Sim wrote: On 09/10/2014 09:58 AM, Flavio Percoco wrote: Other OpenStack components can integrate with Zaqar to surface events to end users and to communicate with guest agents that run in the over-cloud layer. I may be misunderstanding the last sentence, but I think *direct* integration of other OpenStack services with Zaqar would be a bad idea. Wouldn't this be better done through olso.messaging's notifications in some way? and/or through some standard protocol (and there's more than one to choose from)? Communicating through a specific, fixed messaging system, with its own unique protocol is actually a step backwards in my opinion, especially for things that you want to keep as loosely coupled as possible. This is exactly why various standard protocols emerged. Yes and no. The answer is yes most of the time but there are use cases, like the ones mentioned here[0], that make zaqar a good tool for the job. I certainly wasn't saying that Zaqar is not a good tool. I was merely stating that - in my opinion - wiring it in as the only tool would be a mistake. Fair enough. Zaqar is just one of the possibilities and it's crafted to work with OpenStack. If users prefer to use a different tool, it's totally fine. I guess that operators will choose what it best fit their needs. [0] https://etherpad.openstack.org/p/zaqar-integrated-projects-use-cases Again, Zaqar might be great for those cases, but none of them describe features that are unique to Zaqar, so other solutions could also fit. All I'm saying is that if the channel between openstack services and users is configurable, that will give users more choice (as well as operators) and that - in my opinion - would be a good thing. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]
Excerpts from Mark McLoughlin's message of 2014-09-12 03:27:42 -0700: On Wed, 2014-09-10 at 14:51 +0200, Thierry Carrez wrote: Flavio Percoco wrote: [...] Based on the feedback from the meeting[3], the current main concern is: - Do we need a messaging service with a feature-set akin to SQS+SNS? [...] I think we do need, as Samuel puts it, some sort of durable message-broker/queue-server thing. It's a basic application building block. Some claim it's THE basic application building block, more useful than database provisioning. It's definitely a layer above pure IaaS, so if we end up splitting OpenStack into layers this clearly won't be in the inner one. But I think IaaS+ basic application building blocks belong in OpenStack one way or another. That's the reason I supported Designate (everyone needs DNS) and Trove (everyone needs DBs). With that said, I think yesterday there was a concern that Zaqar might not fill the some sort of durable message-broker/queue-server thing role well. The argument goes something like: if it was a queue-server then it should actually be built on top of Rabbit; if it was a message-broker it should be built on top of postfix/dovecot; the current architecture is only justified because it's something in between, so it's broken. I guess I don't mind that much zaqar being something in between: unless I misunderstood, exposing extra primitives doesn't prevent the queue-server use case from being filled. Even considering the message-broker case, I'm also not convinced building it on top of postfix/dovecot would be a net win compared to building it on top of Redis, to be honest. AFAICT, this part of the debate boils down to the following argument: If Zaqar implemented messaging-as-a-service with only queuing semantics (and no random access semantics), it's design would naturally be dramatically different and simply implement a multi-tenant REST API in front of AMQP queues like this: https://www.dropbox.com/s/yonloa9ytlf8fdh/ZaqarQueueOnly.png?dl=0 and that this architecture would allow for dramatically improved throughput for end-users while not making the cost of providing the service prohibitive to operators. You can't dismiss that argument out-of-hand, but I wonder (a) whether the claimed performance improvement is going to make a dramatic difference to the SQS-like use case and (b) whether backing this thing with an RDBMS and multiple highly available, durable AMQP broker clusters is going to be too much of a burden on operators for whatever performance improvements it does gain. Having had experience taking queue-only data out of RDBMS's and even SMTP solutions, and putting them into queues, I can say that it was generally quite a bit more reliable and cheaper to maintain. However, as I've been thinking about this more, I am concerned about the complexity of trying to use a stateless protocol like HTTP for reliable delivery, given that these queues all use a session model that relies on connection persistence. That may very well invalidate my hypothesis. But the troubling part of this debate is where we repeatedly batter the Zaqar team with hypotheses like these and appear to only barely entertain their carefully considered justification for their design decisions like: https://wiki.openstack.org/wiki/Frequently_asked_questions_%28Zaqar%29#Is_Zaqar_a_provisioning_service_or_a_data_API.3F https://wiki.openstack.org/wiki/Frequently_asked_questions_%28Zaqar%29#What_messaging_patterns_does_Zaqar_support.3F I would like to see an SQS-like API provided by OpenStack, I accept the reasons for Zaqar's design decisions to date, I respect that those decisions were made carefully by highly competent members of our community and I expect Zaqar to evolve (like all projects) in the years ahead based on more real-world feedback, new hypotheses or ideas, and lessons learned from trying things out. I have read those and I truly believe that the Zaqar team, who I believe are already a valuable part of the OpenStack family, are doing good work. Seriously, I believe it is valuable as is and I trust them to do what they have stated they will do. Let me explain my position again. Heat is in dire need of a way to communicate with instances that is efficient. It has no need for a full messaging stack.. just a way for users to have things pushed from Heat to their instances efficiently. So, to reiterate why I keep going on about this: If a messaging service is to become an integrated part of OpenStack's release, we should think carefully about the ramifications for operators _and_ users of not having a light weight queue-only option, when that seems to fit _most_ of the use cases. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]
On 09/10/2014 03:18 PM, Gordon Sim wrote: On 09/10/2014 09:58 AM, Flavio Percoco wrote: To clarify the doubts of what Zaqar is or it's not, let me quote what's written in the project's overview section[0]: Zaqar is a multi-tenant cloud messaging service for web developers. How are different tenants isolated from each other? Can different tenants access the same queue? If so, what does Zaqar do to prevent one tenant from negatively affecting the other? If not, how is communication with other tenants achieved. Most messaging systems allow authorisation to be used to restrict what a particular user can access and quotas to restrict their resource consumption. What does Zaqar do differently? Zaqar keeps queues/groups isolated in a per-tenant basis. As of now, there's still no way to make 2 tenants access the same group of messages. However, we've already discussed - we'll likely work on that during Kilo - a way to provide a more fine-grained access control on messages. The service features a fully RESTful API, which developers can use to send messages between various components of their SaaS and mobile applications, by using a variety of communication patterns. Underlying this API is an efficient messaging engine designed with scalability and security in mind. Other OpenStack components can integrate with Zaqar to surface events to end users and to communicate with guest agents that run in the over-cloud layer. I may be misunderstanding the last sentence, but I think *direct* integration of other OpenStack services with Zaqar would be a bad idea. Wouldn't this be better done through olso.messaging's notifications in some way? and/or through some standard protocol (and there's more than one to choose from)? Communicating through a specific, fixed messaging system, with its own unique protocol is actually a step backwards in my opinion, especially for things that you want to keep as loosely coupled as possible. This is exactly why various standard protocols emerged. Yes and no. The answer is yes most of the time but there are use cases, like the ones mentioned here[0], that make zaqar a good tool for the job. [0] https://etherpad.openstack.org/p/zaqar-integrated-projects-use-cases -- @flaper87 Flavio Percoco ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]
On 09/10/2014 03:45 PM, Gordon Sim wrote: On 09/10/2014 01:51 PM, Thierry Carrez wrote: I think we do need, as Samuel puts it, some sort of durable message-broker/queue-server thing. It's a basic application building block. Some claim it's THE basic application building block, more useful than database provisioning. It's definitely a layer above pure IaaS, so if we end up splitting OpenStack into layers this clearly won't be in the inner one. But I think IaaS+ basic application building blocks belong in OpenStack one way or another. That's the reason I supported Designate (everyone needs DNS) and Trove (everyone needs DBs). With that said, I think yesterday there was a concern that Zaqar might not fill the some sort of durable message-broker/queue-server thing role well. The argument goes something like: if it was a queue-server then it should actually be built on top of Rabbit; if it was a message-broker it should be built on top of postfix/dovecot; the current architecture is only justified because it's something in between, so it's broken. What is the distinction between a message broker and a queue server? To me those terms both imply something broadly similar (message broker perhaps being a little bit more generic). I could see Zaqar perhaps as somewhere between messaging and data-storage. I agree with Gordon here. I really don't know how to say this without creating more confusion. Zaqar is a messaging service. Messages are the most important entity in Zaqar. This, however, does not forbid anyone to use Zaqar as a queue. It has the required semantics, it guarantees FIFO and other queuing specific patterns. This doesn't mean Zaqar is trying to do something outside its scope, it comes for free. Is Zaqar being optimized as a *queuing* service? I'd say no. Our goal is to optimize Zaqar for delivering messages and supporting different messaging patterns. Should we remove all the semantics that allow people to use Zaqar as a queue service? I don't think so either. Again, the semantics are there because Zaqar is using them to do its job. Whether other folks may/may not use Zaqar as a queue service is out of our control. This doesn't mean the project is broken. There are of course quite a lot of durable message-broker/queue-server things around already. I understood Zaqar to have been created to address perceived limitations in existing solutions (e.g. requiring less 'babysitting', being 'designed for the cloud' etc). All solutions certainly have their limitations. Zaqar has limitations with respect to existing solutions also. Agreed, again. Zaqar has a long way ahead but a clear goal. New features will be proposed by the community once, hopefully, it's adopted by more and more users. The project will obviously evaluated them all and make sure that whatever gets implemented fits in the project goals. So while I agree that there is great value in a basic building block for 'messaging as a service' I think the ideal solution would allow different variations, tailored to different patterns of use with a common API for provisioning, managing and monitoring coupled with support for standard protocols. I agree there's lots of space for other messaging related services. One of the ideas that has come up quite a few times is having a service that provisions message brokers (or other messaging technologies). However, I believe that's a complete different service and it's out of the scope of what Zaqar wants to do. That said, I'm really looking forward to see a project like that emerge in the future. Thanks a lot for your feedback, Flavio -- @flaper87 Flavio Percoco ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]
Excerpts from Flavio Percoco's message of 2014-09-11 04:14:30 -0700: On 09/10/2014 03:45 PM, Gordon Sim wrote: On 09/10/2014 01:51 PM, Thierry Carrez wrote: I think we do need, as Samuel puts it, some sort of durable message-broker/queue-server thing. It's a basic application building block. Some claim it's THE basic application building block, more useful than database provisioning. It's definitely a layer above pure IaaS, so if we end up splitting OpenStack into layers this clearly won't be in the inner one. But I think IaaS+ basic application building blocks belong in OpenStack one way or another. That's the reason I supported Designate (everyone needs DNS) and Trove (everyone needs DBs). With that said, I think yesterday there was a concern that Zaqar might not fill the some sort of durable message-broker/queue-server thing role well. The argument goes something like: if it was a queue-server then it should actually be built on top of Rabbit; if it was a message-broker it should be built on top of postfix/dovecot; the current architecture is only justified because it's something in between, so it's broken. What is the distinction between a message broker and a queue server? To me those terms both imply something broadly similar (message broker perhaps being a little bit more generic). I could see Zaqar perhaps as somewhere between messaging and data-storage. I agree with Gordon here. I really don't know how to say this without creating more confusion. Zaqar is a messaging service. Messages are the most important entity in Zaqar. This, however, does not forbid anyone to use Zaqar as a queue. It has the required semantics, it guarantees FIFO and other queuing specific patterns. This doesn't mean Zaqar is trying to do something outside its scope, it comes for free. It comes with a huge cost actually, so saying it comes for free is a misrepresentation. It is a side effect of developing a superset of queueing. But that superset is only useful to a small number of your stated use cases. Many of your use cases (including the one I've been involved with, Heat pushing metadata to servers) are entirely served by the much simpler, much lighter weight, pure queueing service. Is Zaqar being optimized as a *queuing* service? I'd say no. Our goal is to optimize Zaqar for delivering messages and supporting different messaging patterns. Awesome! Just please don't expect people to get excited about it for the lighter weight queueing workloads that you've claimed as use cases. I totally see Horizon using it to keep events for users. I see Heat using it for stack events as well. I would bet that Trove would benefit from being able to communicate messages to users. But I think in between Zaqar and the backends will likely be a lighter weight queue-only service that the users can just subscribe to when they don't want an inbox. And I think that lighter weight queue service is far more important for OpenStack than the full blown random access inbox. I think the reason such a thing has not appeared is because we were all sort of running into but Zaqar is already incubated. Now that we've fleshed out the difference, I think those of us that need a lightweight multi-tenant queue service should add it to OpenStack. Separately. I hope that doesn't offend you and the rest of the excellent Zaqar developers. It is just a different thing. Should we remove all the semantics that allow people to use Zaqar as a queue service? I don't think so either. Again, the semantics are there because Zaqar is using them to do its job. Whether other folks may/may not use Zaqar as a queue service is out of our control. This doesn't mean the project is broken. No, definitely not broken. It just isn't actually necessary for many of the stated use cases. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]
On 09/09/2014 09:03 PM, Monty Taylor wrote: On 09/04/2014 01:30 AM, Clint Byrum wrote: Excerpts from Flavio Percoco's message of 2014-09-04 00:08:47 -0700: Greetings, Last Tuesday the TC held the first graduation review for Zaqar. During the meeting some concerns arose. I've listed those concerns below with some comments hoping that it will help starting a discussion before the next meeting. In addition, I've added some comments about the project stability at the bottom and an etherpad link pointing to a list of use cases for Zaqar. Hi Flavio. This was an interesting read. As somebody whose attention has recently been drawn to Zaqar, I am quite interested in seeing it graduate. # Concerns - Concern on operational burden of requiring NoSQL deploy expertise to the mix of openstack operational skills For those of you not familiar with Zaqar, it currently supports 2 nosql drivers - MongoDB and Redis - and those are the only 2 drivers it supports for now. This will require operators willing to use Zaqar to maintain a new (?) NoSQL technology in their system. Before expressing our thoughts on this matter, let me say that: 1. By removing the SQLAlchemy driver, we basically removed the chance for operators to use an already deployed OpenStack-technology 2. Zaqar won't be backed by any AMQP based messaging technology for now. Here's[0] a summary of the research the team (mostly done by Victoria) did during Juno 3. We (OpenStack) used to require Redis for the zmq matchmaker 4. We (OpenStack) also use memcached for caching and as the oslo caching lib becomes available - or a wrapper on top of dogpile.cache - Redis may be used in place of memcached in more and more deployments. 5. Ceilometer's recommended storage driver is still MongoDB, although Ceilometer has now support for sqlalchemy. (Please correct me if I'm wrong). That being said, it's obvious we already, to some extent, promote some NoSQL technologies. However, for the sake of the discussion, lets assume we don't. I truly believe, with my OpenStack (not Zaqar's) hat on, that we can't keep avoiding these technologies. NoSQL technologies have been around for years and we should be prepared - including OpenStack operators - to support these technologies. Not every tool is good for all tasks - one of the reasons we removed the sqlalchemy driver in the first place - therefore it's impossible to keep an homogeneous environment for all services. I whole heartedly agree that non traditional storage technologies that are becoming mainstream are good candidates for use cases where SQL based storage gets in the way. I wish there wasn't so much FUD (warranted or not) about MongoDB, but that is the reality we live in. With this, I'm not suggesting to ignore the risks and the extra burden this adds but, instead of attempting to avoid it completely by not evolving the stack of services we provide, we should probably work on defining a reasonable subset of NoSQL services we are OK with supporting. This will help making the burden smaller and it'll give operators the option to choose. [0] http://blog.flaper87.com/post/marconi-amqp-see-you-later/ - Concern on should we really reinvent a queue system rather than piggyback on one As mentioned in the meeting on Tuesday, Zaqar is not reinventing message brokers. Zaqar provides a service akin to SQS from AWS with an OpenStack flavor on top. [0] I think Zaqar is more like SMTP and IMAP than AMQP. You're not really trying to connect two processes in real time. You're trying to do fully asynchronous messaging with fully randomized access to any message. Perhaps somebody should explore whether the approaches taken by large scale IMAP providers could be applied to Zaqar. Anyway, I can't imagine writing a system to intentionally use the semantics of IMAP and SMTP. I'd be very interested in seeing actual use cases for it, apologies if those have been posted before. It seems like you're EITHER describing something called XMPP that has at least one open source scalable backend called ejabberd. OR, you've actually hit the nail on the head with bringing up SMTP and IMAP but for some reason that feels strange. SMTP and IMAP already implement every feature you've described, as well as retries/failover/HA and a fully end to end secure transport (if installed properly) If you don't actually set them up to run as a public messaging interface but just as a cloud-local exchange, then you could get by with very low overhead for a massive throughput - it can very easily be run on a single machine for Sean's simplicity, and could just as easily be scaled out using well known techniques for public cloud sized deployments? So why not use existing daemons that do this? You could still use the REST API you've got, but instead of writing it to a mongo backend and trying to implement all of the things that already exist in
Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]
Flavio Percoco wrote: [...] Based on the feedback from the meeting[3], the current main concern is: - Do we need a messaging service with a feature-set akin to SQS+SNS? [...] I think we do need, as Samuel puts it, some sort of durable message-broker/queue-server thing. It's a basic application building block. Some claim it's THE basic application building block, more useful than database provisioning. It's definitely a layer above pure IaaS, so if we end up splitting OpenStack into layers this clearly won't be in the inner one. But I think IaaS+ basic application building blocks belong in OpenStack one way or another. That's the reason I supported Designate (everyone needs DNS) and Trove (everyone needs DBs). With that said, I think yesterday there was a concern that Zaqar might not fill the some sort of durable message-broker/queue-server thing role well. The argument goes something like: if it was a queue-server then it should actually be built on top of Rabbit; if it was a message-broker it should be built on top of postfix/dovecot; the current architecture is only justified because it's something in between, so it's broken. I guess I don't mind that much zaqar being something in between: unless I misunderstood, exposing extra primitives doesn't prevent the queue-server use case from being filled. Even considering the message-broker case, I'm also not convinced building it on top of postfix/dovecot would be a net win compared to building it on top of Redis, to be honest. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]
On 09/10/2014 09:58 AM, Flavio Percoco wrote: To clarify the doubts of what Zaqar is or it's not, let me quote what's written in the project's overview section[0]: Zaqar is a multi-tenant cloud messaging service for web developers. How are different tenants isolated from each other? Can different tenants access the same queue? If so, what does Zaqar do to prevent one tenant from negatively affecting the other? If not, how is communication with other tenants achieved. Most messaging systems allow authorisation to be used to restrict what a particular user can access and quotas to restrict their resource consumption. What does Zaqar do differently? It combines the ideas pioneered by Amazon's SQS product with additional semantics to support event broadcasting. The service features a fully RESTful API, which developers can use to send messages between various components of their SaaS and mobile applications, by using a variety of communication patterns. Underlying this API is an efficient messaging engine designed with scalability and security in mind. Other OpenStack components can integrate with Zaqar to surface events to end users and to communicate with guest agents that run in the over-cloud layer. I may be misunderstanding the last sentence, but I think *direct* integration of other OpenStack services with Zaqar would be a bad idea. Wouldn't this be better done through olso.messaging's notifications in some way? and/or through some standard protocol (and there's more than one to choose from)? Communicating through a specific, fixed messaging system, with its own unique protocol is actually a step backwards in my opinion, especially for things that you want to keep as loosely coupled as possible. This is exactly why various standard protocols emerged. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]
On 09/10/2014 01:51 PM, Thierry Carrez wrote: I think we do need, as Samuel puts it, some sort of durable message-broker/queue-server thing. It's a basic application building block. Some claim it's THE basic application building block, more useful than database provisioning. It's definitely a layer above pure IaaS, so if we end up splitting OpenStack into layers this clearly won't be in the inner one. But I think IaaS+ basic application building blocks belong in OpenStack one way or another. That's the reason I supported Designate (everyone needs DNS) and Trove (everyone needs DBs). With that said, I think yesterday there was a concern that Zaqar might not fill the some sort of durable message-broker/queue-server thing role well. The argument goes something like: if it was a queue-server then it should actually be built on top of Rabbit; if it was a message-broker it should be built on top of postfix/dovecot; the current architecture is only justified because it's something in between, so it's broken. What is the distinction between a message broker and a queue server? To me those terms both imply something broadly similar (message broker perhaps being a little bit more generic). I could see Zaqar perhaps as somewhere between messaging and data-storage. There are of course quite a lot of durable message-broker/queue-server things around already. I understood Zaqar to have been created to address perceived limitations in existing solutions (e.g. requiring less 'babysitting', being 'designed for the cloud' etc). All solutions certainly have their limitations. Zaqar has limitations with respect to existing solutions also. So while I agree that there is great value in a basic building block for 'messaging as a service' I think the ideal solution would allow different variations, tailored to different patterns of use with a common API for provisioning, managing and monitoring coupled with support for standard protocols. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]
Excerpts from Gordon Sim's message of 2014-09-10 06:18:52 -0700: On 09/10/2014 09:58 AM, Flavio Percoco wrote: To clarify the doubts of what Zaqar is or it's not, let me quote what's written in the project's overview section[0]: Zaqar is a multi-tenant cloud messaging service for web developers. How are different tenants isolated from each other? Can different tenants access the same queue? If so, what does Zaqar do to prevent one tenant from negatively affecting the other? If not, how is communication with other tenants achieved. Most messaging systems allow authorisation to be used to restrict what a particular user can access and quotas to restrict their resource consumption. What does Zaqar do differently? It combines the ideas pioneered by Amazon's SQS product with additional semantics to support event broadcasting. The service features a fully RESTful API, which developers can use to send messages between various components of their SaaS and mobile applications, by using a variety of communication patterns. Underlying this API is an efficient messaging engine designed with scalability and security in mind. Other OpenStack components can integrate with Zaqar to surface events to end users and to communicate with guest agents that run in the over-cloud layer. I may be misunderstanding the last sentence, but I think *direct* integration of other OpenStack services with Zaqar would be a bad idea. Wouldn't this be better done through olso.messaging's notifications in some way? and/or through some standard protocol (and there's more than one to choose from)? It's not direct, nobody is suggesting that. What people are suggesting is that a user would be able to tell Nova to put any messages that would want to deliver in a _user_ focused queue/inbox. This has nothing to do with oslo.messaging. Users don't want many options for backends. They want a simple message passing interface so they don't have to babysit one and choose one. Certainly the undercloud Zaqar API could be based on the existing oslo.messaging notifications. A simple daemon that sits between the oslo notifications firehose and Zaqar's user queues would be quite efficient. However, putting the whole burden of talking directly to a notification bus on the users is unnecessarily complex... especially if they use Java and have no idea what oslo is. Communicating through a specific, fixed messaging system, with its own unique protocol is actually a step backwards in my opinion, especially for things that you want to keep as loosely coupled as possible. This is exactly why various standard protocols emerged. You're thinking like an operator. Think like an application developer. They're asking you how do I subscribe to notifications about _just my instances_ from Nova?, not how do I pump 40,000 messages per second through a message bus that I fully control? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]
On 09/10/2014 07:29 PM, Clint Byrum wrote: Excerpts from Gordon Sim's message of 2014-09-10 06:18:52 -0700: On 09/10/2014 09:58 AM, Flavio Percoco wrote: Other OpenStack components can integrate with Zaqar to surface events to end users and to communicate with guest agents that run in the over-cloud layer. I may be misunderstanding the last sentence, but I think *direct* integration of other OpenStack services with Zaqar would be a bad idea. Wouldn't this be better done through olso.messaging's notifications in some way? and/or through some standard protocol (and there's more than one to choose from)? It's not direct, nobody is suggesting that. What people are suggesting is that a user would be able to tell Nova to put any messages that would want to deliver in a _user_ focused queue/inbox. This has nothing to do with oslo.messaging. Users don't want many options for backends. They want a simple message passing interface so they don't have to babysit one and choose one. Certainly the undercloud Zaqar API could be based on the existing oslo.messaging notifications. A simple daemon that sits between the oslo notifications firehose and Zaqar's user queues would be quite efficient. Right, that's what I meant. The interface applications have to it would of course not be oslo.messaging notifications. However, putting the whole burden of talking directly to a notification bus on the users is unnecessarily complex... especially if they use Java and have no idea what oslo is. Agreed. I actually think for applications, making them accessible through standard protocols would be an advantage. E.g. for java as you mention, there are JMS clients for AMQP or STOMP. Thats not to say that Zaqar's protocol as built on top of HTTP is not also an option, just that - in my opinion - it shouldn't be the only possible one. (I.e. Nova shouldn't be directly tied to Zaqar as the only way for users to get relevant message from it). Communicating through a specific, fixed messaging system, with its own unique protocol is actually a step backwards in my opinion, especially for things that you want to keep as loosely coupled as possible. This is exactly why various standard protocols emerged. You're thinking like an operator. Think like an application developer. They're asking you how do I subscribe to notifications about _just my instances_ from Nova?, not how do I pump 40,000 messages per second through a message bus that I fully control? I'm not sure why you think that from my statement. I wasn't talking about performance. I was talking about choice both for deployment *and* application developers. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
