Re: [openstack-dev] [Zaqar][all] Zaqar will stay... Lots of work ahead
On 31/05/15 18:12 +, Ildikó Váncsa wrote: Hi All, I would like to ask about the user-facing notifications part of the list. Do you have a roadmap for this? Is this driven by the Zaqar team? What are the next steps? Hey, This will require cross-project efforts and we (the Zaqar team) would love to get some help here. If anyone is willing to drive the spec and sync, the Zaqar team would be happy to help with other tasks. I think it'd be really beneficial to start doing it in one of the services (Heat?) as a PoC, while we discuss the cross-project spec and feed it with the things we'll learn from the PoC. Would you like to help with this? Cheers, Flavio Thanks and Best Regards, Ildikó -Original Message- From: Clint Byrum [mailto:cl...@fewbar.com] Sent: May 27, 2015 18:42 To: openstack-dev Subject: Re: [openstack-dev] [Zaqar][all] Zaqar will stay... Lots of work ahead Excerpts from Flavio Percoco's message of 2015-05-26 01:28:06 -0700: Greetings, TL;DR: Thanks everyone for your feedback. Based on the discussed plans at the summit - I'll be writing more about these later - Zaqar will stick around and play its role in the community. Summit Summary == I'm happy to say that several use cases were discussed at the summit and the difference from previous summits is that we left the room with some action items to make them happen. Cross-project user-facing notifications === https://etherpad.openstack.org/p/liberty-cross-project-user-notificati ons Besides brainstorming a bit on what things should/should not be notified and what format should be used, we also talked a bit about the available technologies that could be used for this tasks. Zaqar was among those and, AFAICT, at the end of the session we agreed on giving this a try. It'll likely not happen as fast as we want but the action item out of this session was to write a cross-project spec describing the things discussed and the technology that will be adopted. My takeaway from that session was that there's need for something like yagi to filter the backend notifications into user-consumable tenant-scoped messages, and that Zaqar would be an interesting target for those messages along with Atom feeds or perhaps other pub/sub oriented things that deployers would be comfortable hosting for their users. Heat + Zaqar The 2 main areas where Zaqar will be used in Heat are Software Config and Hooks. The minimum requirements (server side) for this are in place already. There's some work to do on the client side that the team will get to asap. The bigger one to me is just user-notificiation which I think is covered in the cross project session, but it's worth noting that Heat is one of the projects that already does some user notification and suffers problems because of it (the events API is what I mean here). Next Steps == In light of the above, and as mentioned in the TL;DR, Zaqar will stick around and the team, as promised, will focus on making those integrations happen. The team is small, which means we'll carefully pick the tasks we'll be spending time on. As a first step, we should restore our meetings and get to work right away. To favor our contributors in NZ, next week's meeting will be at 21:00 UTC and we'll keep it at that time for 2 weeks. For the Zaqar team (and folks interested), I'll be sending out further emails to sync on the work to do. Special thanks for all the folks that showed interest, participated in sessions and that are committed on making this happen. Thanks for setting things up for success before the summit. I think we all went into the discussions with an open mind because we knew where the team stood. == Crazy idea section == One thing I never had a chance to discuss with any of the Zaqar devs that I would find interesting is an email-only backend for Zaqar. Basically make Zaqar an HTTP-to-email gateway. There are quite a few hyper-scale options for SMTP and IMAP, and they're inherently multi-tenant, so I'd find it interesting to see if the full Zaqar API could be mapped onto that. This would probably be more comfortable to scale for some deployers than Redis or MongoDB, and might have the nice side-effect that a deployer could expose IMAP IDLE for efficient end-user subscription, and it could also allow Zaqar to serve as email-as-a-service for senders too (to prevent getting all your vms' IPs added to spam lists overnight. ;) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org
Re: [openstack-dev] [Zaqar][all] Zaqar will stay... Lots of work ahead
On 28/05/15 16:48 +0100, Gordon Sim wrote: On 05/27/2015 05:08 PM, Hayes, Graham wrote: It was agreed that Zaqar would look at a oslo_messaging driver for the services that did not agree with the 3 options presented. Another option there would be to add amqp 1.0 support to zaqar, and then you could use the amqp 1.0 driver with zaqar as the back-end. (Support for an existing protocol would be beneficial for the general messaging as a service use case also.) FWIW, this is still in the long-term roadmap for Zaqar. This is something we looked into a couple of cycles ago that we'd love to get done. I do think an AMQP *1.0* would help with some use cases. Cheers, Flavio -- @flaper87 Flavio Percoco pgp0DDoBJumoi.pgp Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar][all] Zaqar will stay... Lots of work ahead
On 27/05/15 11:19 -0700, Clint Byrum wrote: Excerpts from Zane Bitter's message of 2015-05-27 10:40:48 -0700: On 27/05/15 12:42, Clint Byrum wrote: == Crazy idea section == One thing I never had a chance to discuss with any of the Zaqar devs that I would find interesting is an email-only backend for Zaqar. Basically make Zaqar an HTTP-to-email gateway. There are quite a few hyper-scale options for SMTP and IMAP, and they're inherently multi-tenant, so I'd find it interesting to see if the full Zaqar API could be mapped onto that. This would probably be more comfortable to scale for some deployers than Redis or MongoDB, and might have the nice side-effect that a deployer could expose IMAP IDLE for efficient end-user subscription, Can you guarantee delivery end-to-end (and still get the scaling benefits)? Because AIUI SMTP is only best effort, and that makes this idea a non-starter IMHO. So yes and no. If you are going to set your Zaqar backend up to forward messages across the internet, then no, of course you cannot guarantee delivery. That is not what I am suggesting as the default configuration. However, in a closed system like a Zaqar backend SMTP would be, you have control over things like retries and bandwidth, so you can at least guarantee that if the intended destination is available, the message will make it there. If you have 100 SMTP servers enqueing for 1000 destination queue servers, all with 1Gbit network between them, you'd simply set your retry interval much lower than for the internet, and allow infinite or very very high retries before giving up on messages. The scaling comes from the asynchronous queueing. One can have guaranteed delivery with asynchronous queueing. Not such a crazy idea, TBH. This has come up in the past and as Monty said the other day, he and I should sit down somewhere in EU and get a PoC done. That said, I think this would work better as a notifications publisher - the SNS side of Zaqar - than a specific Zaqar storage backend. Right now, we just have webhook publishers that push messages to registered URLs. Instead of an URL, I'd expect there to be an smtp URL that Zaqar would send messages to. I guess that's what you were suggesting, right? Just want to make sure we're talking about the same thing. Flavio __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- @flaper87 Flavio Percoco pgpnjbLjhU_l_.pgp Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar][all] Zaqar will stay... Lots of work ahead
On 06/01/2015 04:34 AM, Flavio Percoco wrote: On 31/05/15 18:12 +, Ildikó Váncsa wrote: Hi All, I would like to ask about the user-facing notifications part of the list. Do you have a roadmap for this? Is this driven by the Zaqar team? What are the next steps? Hey, This will require cross-project efforts and we (the Zaqar team) would love to get some help here. If anyone is willing to drive the spec and sync, the Zaqar team would be happy to help with other tasks. I think it'd be really beneficial to start doing it in one of the services (Heat?) as a PoC, while we discuss the cross-project spec and feed it with the things we'll learn from the PoC. Would you like to help with this? Indeed it will require tons of cross-project work. Heat is going to try to get Zaqar notifications to our users in Liberty, and I've posted specs for the message format side of the problem[1][2]. There is also a cross-project spec for user notifications[3]. Heat plans on adopting this format after sufficient review, and we welcome other projects to look over the format and provide feedback to help it work for their use case as well. [1]: https://review.openstack.org/#/c/186436/ [2]: https://review.openstack.org/#/c/186555/ [3]: https://review.openstack.org/#/c/185822/ ...[snip]... -- Ryan Brown / Software Engineer, Openstack / Red Hat, Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar][all] Zaqar will stay... Lots of work ahead
Hi Ryan and Flavio, Thanks for the quick response and the pointers, I will check out the reviews to catch up. Best Regards, Ildikó -Original Message- From: Ryan Brown [mailto:rybr...@redhat.com] Sent: June 01, 2015 16:34 To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Zaqar][all] Zaqar will stay... Lots of work ahead On 06/01/2015 04:34 AM, Flavio Percoco wrote: On 31/05/15 18:12 +, Ildikó Váncsa wrote: Hi All, I would like to ask about the user-facing notifications part of the list. Do you have a roadmap for this? Is this driven by the Zaqar team? What are the next steps? Hey, This will require cross-project efforts and we (the Zaqar team) would love to get some help here. If anyone is willing to drive the spec and sync, the Zaqar team would be happy to help with other tasks. I think it'd be really beneficial to start doing it in one of the services (Heat?) as a PoC, while we discuss the cross-project spec and feed it with the things we'll learn from the PoC. Would you like to help with this? Indeed it will require tons of cross-project work. Heat is going to try to get Zaqar notifications to our users in Liberty, and I've posted specs for the message format side of the problem[1][2]. There is also a cross-project spec for user notifications[3]. Heat plans on adopting this format after sufficient review, and we welcome other projects to look over the format and provide feedback to help it work for their use case as well. [1]: https://review.openstack.org/#/c/186436/ [2]: https://review.openstack.org/#/c/186555/ [3]: https://review.openstack.org/#/c/185822/ ...[snip]... -- Ryan Brown / Software Engineer, Openstack / Red Hat, Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: OpenStack-dev- requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar][all] Zaqar will stay... Lots of work ahead
There is a spec to get this work started. It's currently a Heat spec: https://review.openstack.org/#/c/186436/ That spec is for defining a basic schema to start with. -Alex On Mon, Jun 1, 2015 at 4:48 AM, Flavio Percoco fla...@redhat.com wrote: On 28/05/15 16:48 +0100, Gordon Sim wrote: On 05/27/2015 05:08 PM, Hayes, Graham wrote: It was agreed that Zaqar would look at a oslo_messaging driver for the services that did not agree with the 3 options presented. Another option there would be to add amqp 1.0 support to zaqar, and then you could use the amqp 1.0 driver with zaqar as the back-end. (Support for an existing protocol would be beneficial for the general messaging as a service use case also.) FWIW, this is still in the long-term roadmap for Zaqar. This is something we looked into a couple of cycles ago that we'd love to get done. I do think an AMQP *1.0* would help with some use cases. Cheers, Flavio -- @flaper87 Flavio Percoco __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar][all] Zaqar will stay... Lots of work ahead
Hi All, I would like to ask about the user-facing notifications part of the list. Do you have a roadmap for this? Is this driven by the Zaqar team? What are the next steps? Thanks and Best Regards, Ildikó -Original Message- From: Clint Byrum [mailto:cl...@fewbar.com] Sent: May 27, 2015 18:42 To: openstack-dev Subject: Re: [openstack-dev] [Zaqar][all] Zaqar will stay... Lots of work ahead Excerpts from Flavio Percoco's message of 2015-05-26 01:28:06 -0700: Greetings, TL;DR: Thanks everyone for your feedback. Based on the discussed plans at the summit - I'll be writing more about these later - Zaqar will stick around and play its role in the community. Summit Summary == I'm happy to say that several use cases were discussed at the summit and the difference from previous summits is that we left the room with some action items to make them happen. Cross-project user-facing notifications === https://etherpad.openstack.org/p/liberty-cross-project-user-notificati ons Besides brainstorming a bit on what things should/should not be notified and what format should be used, we also talked a bit about the available technologies that could be used for this tasks. Zaqar was among those and, AFAICT, at the end of the session we agreed on giving this a try. It'll likely not happen as fast as we want but the action item out of this session was to write a cross-project spec describing the things discussed and the technology that will be adopted. My takeaway from that session was that there's need for something like yagi to filter the backend notifications into user-consumable tenant-scoped messages, and that Zaqar would be an interesting target for those messages along with Atom feeds or perhaps other pub/sub oriented things that deployers would be comfortable hosting for their users. Heat + Zaqar The 2 main areas where Zaqar will be used in Heat are Software Config and Hooks. The minimum requirements (server side) for this are in place already. There's some work to do on the client side that the team will get to asap. The bigger one to me is just user-notificiation which I think is covered in the cross project session, but it's worth noting that Heat is one of the projects that already does some user notification and suffers problems because of it (the events API is what I mean here). Next Steps == In light of the above, and as mentioned in the TL;DR, Zaqar will stick around and the team, as promised, will focus on making those integrations happen. The team is small, which means we'll carefully pick the tasks we'll be spending time on. As a first step, we should restore our meetings and get to work right away. To favor our contributors in NZ, next week's meeting will be at 21:00 UTC and we'll keep it at that time for 2 weeks. For the Zaqar team (and folks interested), I'll be sending out further emails to sync on the work to do. Special thanks for all the folks that showed interest, participated in sessions and that are committed on making this happen. Thanks for setting things up for success before the summit. I think we all went into the discussions with an open mind because we knew where the team stood. == Crazy idea section == One thing I never had a chance to discuss with any of the Zaqar devs that I would find interesting is an email-only backend for Zaqar. Basically make Zaqar an HTTP-to-email gateway. There are quite a few hyper-scale options for SMTP and IMAP, and they're inherently multi-tenant, so I'd find it interesting to see if the full Zaqar API could be mapped onto that. This would probably be more comfortable to scale for some deployers than Redis or MongoDB, and might have the nice side-effect that a deployer could expose IMAP IDLE for efficient end-user subscription, and it could also allow Zaqar to serve as email-as-a-service for senders too (to prevent getting all your vms' IPs added to spam lists overnight. ;) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar][all] Zaqar will stay... Lots of work ahead
I don't know AMQP very well. At the protocol level, are Exchanges, Queues, etc things? Are fanout, durability, and other things required or optional to be implemented? Thanks, Kevin From: Gordon Sim [g...@redhat.com] Sent: Thursday, May 28, 2015 10:05 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Zaqar][all] Zaqar will stay... Lots of work ahead On 05/28/2015 05:14 PM, Fox, Kevin M wrote: Thats like saying you should implement a sql engine inside of Berkeley DB since so many folks like sql... Not a good fit. If you want AMQP, get an AMQP server. Zaqar's intentionally lighter weight then that. Hardly any OpenStack services use but a fraction of AMQP though, so getting oslo.messaging to support that simple subset of AMQP on top of Zaqar is much more reasonable then getting Zaqar to support all of AMQP. AMQP 1.0 is just a protocol for asynchronously transferring messages between processes. Supporting it doesn't require that zaqar provide any functionality that it doesn't want to. It is simply another protocol for accessing exactly the same service. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar][all] Zaqar will stay... Lots of work ahead
On 05/28/2015 05:14 PM, Fox, Kevin M wrote: Thats like saying you should implement a sql engine inside of Berkeley DB since so many folks like sql... Not a good fit. If you want AMQP, get an AMQP server. Zaqar's intentionally lighter weight then that. Hardly any OpenStack services use but a fraction of AMQP though, so getting oslo.messaging to support that simple subset of AMQP on top of Zaqar is much more reasonable then getting Zaqar to support all of AMQP. AMQP 1.0 is just a protocol for asynchronously transferring messages between processes. Supporting it doesn't require that zaqar provide any functionality that it doesn't want to. It is simply another protocol for accessing exactly the same service. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar][all] Zaqar will stay... Lots of work ahead
On 05/28/2015 07:20 PM, Fox, Kevin M wrote: I don't know AMQP very well. At the protocol level, are Exchanges, Queues, etc things? Are fanout, durability, and other things required or optional to be implemented? AMQP 1.0 is very different from the pre 1.0 versions. Where 0.8/0.9 focus on defining the internals of a particular broker implementation (queues, exchanges, bindings etc) and 1.0 does not dictate any of that and focuses instead on the protocol for transferring messages between processes. So supporting AMQP 1.0 wouldn't require zaqar to adopt any new model. It's just an alternative protocol/encoding on the wire for getting messages into (and out of) the zaqar service. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar][all] Zaqar will stay... Lots of work ahead
Thats like saying you should implement a sql engine inside of Berkeley DB since so many folks like sql... Not a good fit. If you want AMQP, get an AMQP server. Zaqar's intentionally lighter weight then that. Hardly any OpenStack services use but a fraction of AMQP though, so getting oslo.messaging to support that simple subset of AMQP on top of Zaqar is much more reasonable then getting Zaqar to support all of AMQP. Thanks, Kevin From: Gordon Sim [g...@redhat.com] Sent: Thursday, May 28, 2015 8:48 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Zaqar][all] Zaqar will stay... Lots of work ahead On 05/27/2015 05:08 PM, Hayes, Graham wrote: It was agreed that Zaqar would look at a oslo_messaging driver for the services that did not agree with the 3 options presented. Another option there would be to add amqp 1.0 support to zaqar, and then you could use the amqp 1.0 driver with zaqar as the back-end. (Support for an existing protocol would be beneficial for the general messaging as a service use case also.) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar][all] Zaqar will stay... Lots of work ahead
On 05/27/2015 05:08 PM, Hayes, Graham wrote: It was agreed that Zaqar would look at a oslo_messaging driver for the services that did not agree with the 3 options presented. Another option there would be to add amqp 1.0 support to zaqar, and then you could use the amqp 1.0 driver with zaqar as the back-end. (Support for an existing protocol would be beneficial for the general messaging as a service use case also.) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar][all] Zaqar will stay... Lots of work ahead
Hi, Thanks for writing down the summit notes Flavio. I'm really glad that there are so many great features to work on in this cycle. I'll be there to help as much as possible with every one of them. I would also like to add that we will be doing a lot of work in the client side, updating it to cover latest features and working on improving the CLI with the new Outreachy intern, Doraly. Looking forward for the next meeting, Victoria El mar., 26 may. 2015 a las 9:40, Flavio Percoco (fla...@redhat.com) escribió: On 26/05/15 08:23 -0400, Ryan Brown wrote: On 05/26/2015 04:28 AM, Flavio Percoco wrote: [snip] As a first step, we should restore our meetings and get to work right away. To favor our contributors in NZ, next week's meeting will be at 21:00 UTC and we'll keep it at that time for 2 weeks. For those who didn't know what day the Zaqar meetings are normally, they are on Mondays according to the OpenStack calendar (next meeting on June 1). Damn, I always forget something. Yes, meetings are on Mondays with alternate times (15:00 and 21:00 UTC). We'll do 21:00 UTC for 2 weeks to have enough time to sync with folks from NZ. Cheers, Flavio -- @flaper87 Flavio Percoco __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar][all] Zaqar will stay... Lots of work ahead
Excerpts from Zane Bitter's message of 2015-05-27 10:40:48 -0700: On 27/05/15 12:42, Clint Byrum wrote: == Crazy idea section == One thing I never had a chance to discuss with any of the Zaqar devs that I would find interesting is an email-only backend for Zaqar. Basically make Zaqar an HTTP-to-email gateway. There are quite a few hyper-scale options for SMTP and IMAP, and they're inherently multi-tenant, so I'd find it interesting to see if the full Zaqar API could be mapped onto that. This would probably be more comfortable to scale for some deployers than Redis or MongoDB, and might have the nice side-effect that a deployer could expose IMAP IDLE for efficient end-user subscription, Can you guarantee delivery end-to-end (and still get the scaling benefits)? Because AIUI SMTP is only best effort, and that makes this idea a non-starter IMHO. So yes and no. If you are going to set your Zaqar backend up to forward messages across the internet, then no, of course you cannot guarantee delivery. That is not what I am suggesting as the default configuration. However, in a closed system like a Zaqar backend SMTP would be, you have control over things like retries and bandwidth, so you can at least guarantee that if the intended destination is available, the message will make it there. If you have 100 SMTP servers enqueing for 1000 destination queue servers, all with 1Gbit network between them, you'd simply set your retry interval much lower than for the internet, and allow infinite or very very high retries before giving up on messages. The scaling comes from the asynchronous queueing. One can have guaranteed delivery with asynchronous queueing. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar][all] Zaqar will stay... Lots of work ahead
On 05/26/2015 09:29 AM, Flavio Percoco wrote: Greetings, TL;DR: Thanks everyone for your feedback. Based on the discussed plans at the summit - I'll be writing more about these later - Zaqar will stick around and play its role in the community. Summit Summary == I'm happy to say that several use cases were discussed at the summit and the difference from previous summits is that we left the room with some action items to make them happen. Cross-project user-facing notifications === https://etherpad.openstack.org/p/liberty-cross-project-user-notifications Besides brainstorming a bit on what things should/should not be notified and what format should be used, we also talked a bit about the available technologies that could be used for this tasks. Zaqar was among those and, AFAICT, at the end of the session we agreed on giving this a try. It'll likely not happen as fast as we want but the action item out of this session was to write a cross-project spec describing the things discussed and the technology that will be adopted. Heat + Zaqar The 2 main areas where Zaqar will be used in Heat are Software Config and Hooks. The minimum requirements (server side) for this are in place already. There's some work to do on the client side that the team will get to asap. Sahara (or other guest agent based services) + Zaqar We discussed 3 different ways to enable services to communicate with their guest agents using Zaqar: 1) Using notification hooks: Assuming the guest agents doesn't need to communicate with the controller, the controller can register a notification hook that will push messages to the guest agent. 2) Inject keystone credentials: The controller would inject keystone credentials into the VM to allow the guest agent to send/receive messages using Zaqar. 3) PreSigned URLs: The controller injects a PreSigned URL in the controller that will grant the guest agent access to a specific tenant/queue with either read or readwrite access. I think it is important to note that these 3 options are for a sub set of guest agent based services. It was agreed that Zaqar would look at a oslo_messaging driver for the services that did not agree with the 3 options presented. Hallway Discussions === We had a chance to talk to some other folks from teams like Horizon that were also interested in doing some actual integration work with Zaqar as well. Not to mention that some other folks from the puppet team showed interest in helping out with the creation of these manifests. Next Steps == In light of the above, and as mentioned in the TL;DR, Zaqar will stick around and the team, as promised, will focus on making those integrations happen. The team is small, which means we'll carefully pick the tasks we'll be spending time on. As a first step, we should restore our meetings and get to work right away. To favor our contributors in NZ, next week's meeting will be at 21:00 UTC and we'll keep it at that time for 2 weeks. For the Zaqar team (and folks interested), I'll be sending out further emails to sync on the work to do. Special thanks for all the folks that showed interest, participated in sessions and that are committed on making this happen. Lets now make it happen, Flavio -- *Graham Hayes *Software Engineer DNS as a Service HP Public Cloud - Platform Services GPG Key: 7D28E972 graham.ha...@hp.com mailto:graham.ha...@hp.com M +353 87 377 8315 P +353 1 524 2175 Dublin, Ireland HP http://www.hp.com/ __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar][all] Zaqar will stay... Lots of work ahead
On 27/05/15 12:42, Clint Byrum wrote: == Crazy idea section == One thing I never had a chance to discuss with any of the Zaqar devs that I would find interesting is an email-only backend for Zaqar. Basically make Zaqar an HTTP-to-email gateway. There are quite a few hyper-scale options for SMTP and IMAP, and they're inherently multi-tenant, so I'd find it interesting to see if the full Zaqar API could be mapped onto that. This would probably be more comfortable to scale for some deployers than Redis or MongoDB, and might have the nice side-effect that a deployer could expose IMAP IDLE for efficient end-user subscription, Can you guarantee delivery end-to-end (and still get the scaling benefits)? Because AIUI SMTP is only best effort, and that makes this idea a non-starter IMHO. cheers, Zane. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar][all] Zaqar will stay... Lots of work ahead
Excerpts from Flavio Percoco's message of 2015-05-26 01:28:06 -0700: Greetings, TL;DR: Thanks everyone for your feedback. Based on the discussed plans at the summit - I'll be writing more about these later - Zaqar will stick around and play its role in the community. Summit Summary == I'm happy to say that several use cases were discussed at the summit and the difference from previous summits is that we left the room with some action items to make them happen. Cross-project user-facing notifications === https://etherpad.openstack.org/p/liberty-cross-project-user-notifications Besides brainstorming a bit on what things should/should not be notified and what format should be used, we also talked a bit about the available technologies that could be used for this tasks. Zaqar was among those and, AFAICT, at the end of the session we agreed on giving this a try. It'll likely not happen as fast as we want but the action item out of this session was to write a cross-project spec describing the things discussed and the technology that will be adopted. My takeaway from that session was that there's need for something like yagi to filter the backend notifications into user-consumable tenant-scoped messages, and that Zaqar would be an interesting target for those messages along with Atom feeds or perhaps other pub/sub oriented things that deployers would be comfortable hosting for their users. Heat + Zaqar The 2 main areas where Zaqar will be used in Heat are Software Config and Hooks. The minimum requirements (server side) for this are in place already. There's some work to do on the client side that the team will get to asap. The bigger one to me is just user-notificiation which I think is covered in the cross project session, but it's worth noting that Heat is one of the projects that already does some user notification and suffers problems because of it (the events API is what I mean here). Next Steps == In light of the above, and as mentioned in the TL;DR, Zaqar will stick around and the team, as promised, will focus on making those integrations happen. The team is small, which means we'll carefully pick the tasks we'll be spending time on. As a first step, we should restore our meetings and get to work right away. To favor our contributors in NZ, next week's meeting will be at 21:00 UTC and we'll keep it at that time for 2 weeks. For the Zaqar team (and folks interested), I'll be sending out further emails to sync on the work to do. Special thanks for all the folks that showed interest, participated in sessions and that are committed on making this happen. Thanks for setting things up for success before the summit. I think we all went into the discussions with an open mind because we knew where the team stood. == Crazy idea section == One thing I never had a chance to discuss with any of the Zaqar devs that I would find interesting is an email-only backend for Zaqar. Basically make Zaqar an HTTP-to-email gateway. There are quite a few hyper-scale options for SMTP and IMAP, and they're inherently multi-tenant, so I'd find it interesting to see if the full Zaqar API could be mapped onto that. This would probably be more comfortable to scale for some deployers than Redis or MongoDB, and might have the nice side-effect that a deployer could expose IMAP IDLE for efficient end-user subscription, and it could also allow Zaqar to serve as email-as-a-service for senders too (to prevent getting all your vms' IPs added to spam lists overnight. ;) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Zaqar][all] Zaqar will stay... Lots of work ahead
Greetings, TL;DR: Thanks everyone for your feedback. Based on the discussed plans at the summit - I'll be writing more about these later - Zaqar will stick around and play its role in the community. Summit Summary == I'm happy to say that several use cases were discussed at the summit and the difference from previous summits is that we left the room with some action items to make them happen. Cross-project user-facing notifications === https://etherpad.openstack.org/p/liberty-cross-project-user-notifications Besides brainstorming a bit on what things should/should not be notified and what format should be used, we also talked a bit about the available technologies that could be used for this tasks. Zaqar was among those and, AFAICT, at the end of the session we agreed on giving this a try. It'll likely not happen as fast as we want but the action item out of this session was to write a cross-project spec describing the things discussed and the technology that will be adopted. Heat + Zaqar The 2 main areas where Zaqar will be used in Heat are Software Config and Hooks. The minimum requirements (server side) for this are in place already. There's some work to do on the client side that the team will get to asap. Sahara (or other guest agent based services) + Zaqar We discussed 3 different ways to enable services to communicate with their guest agents using Zaqar: 1) Using notification hooks: Assuming the guest agents doesn't need to communicate with the controller, the controller can register a notification hook that will push messages to the guest agent. 2) Inject keystone credentials: The controller would inject keystone credentials into the VM to allow the guest agent to send/receive messages using Zaqar. 3) PreSigned URLs: The controller injects a PreSigned URL in the controller that will grant the guest agent access to a specific tenant/queue with either read or readwrite access. Hallway Discussions === We had a chance to talk to some other folks from teams like Horizon that were also interested in doing some actual integration work with Zaqar as well. Not to mention that some other folks from the puppet team showed interest in helping out with the creation of these manifests. Next Steps == In light of the above, and as mentioned in the TL;DR, Zaqar will stick around and the team, as promised, will focus on making those integrations happen. The team is small, which means we'll carefully pick the tasks we'll be spending time on. As a first step, we should restore our meetings and get to work right away. To favor our contributors in NZ, next week's meeting will be at 21:00 UTC and we'll keep it at that time for 2 weeks. For the Zaqar team (and folks interested), I'll be sending out further emails to sync on the work to do. Special thanks for all the folks that showed interest, participated in sessions and that are committed on making this happen. Lets now make it happen, Flavio -- @flaper87 Flavio Percoco pgp21BCBY9xBK.pgp Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar][all] Zaqar will stay... Lots of work ahead
On 05/26/2015 04:28 AM, Flavio Percoco wrote: Greetings, TL;DR: Thanks everyone for your feedback. Based on the discussed plans at the summit - I'll be writing more about these later - Zaqar will stick around and play its role in the community. \o/ [snip] == Next Steps == In light of the above, and as mentioned in the TL;DR, Zaqar will stick around and the team, as promised, will focus on making those integrations happen. The team is small, which means we'll carefully pick the tasks we'll be spending time on. As a first step, we should restore our meetings and get to work right away. To favor our contributors in NZ, next week's meeting will be at 21:00 UTC and we'll keep it at that time for 2 weeks. For those who didn't know what day the Zaqar meetings are normally, they are on Mondays according to the OpenStack calendar (next meeting on June 1). For the Zaqar team (and folks interested), I'll be sending out further emails to sync on the work to do. Special thanks for all the folks that showed interest, participated in sessions and that are committed on making this happen. Lets now make it happen, Flavio Thanks for the mail Flavio, Ryan -- Ryan Brown / Software Engineer, Openstack / Red Hat, Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Zaqar][all] Zaqar will stay... Lots of work ahead
On 26/05/15 08:23 -0400, Ryan Brown wrote: On 05/26/2015 04:28 AM, Flavio Percoco wrote: [snip] As a first step, we should restore our meetings and get to work right away. To favor our contributors in NZ, next week's meeting will be at 21:00 UTC and we'll keep it at that time for 2 weeks. For those who didn't know what day the Zaqar meetings are normally, they are on Mondays according to the OpenStack calendar (next meeting on June 1). Damn, I always forget something. Yes, meetings are on Mondays with alternate times (15:00 and 21:00 UTC). We'll do 21:00 UTC for 2 weeks to have enough time to sync with folks from NZ. Cheers, Flavio -- @flaper87 Flavio Percoco pgpWFWObxyEAm.pgp Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
