Re: [openstack-dev] Oslo messaging vs zaqar
On 23/09/2014 12:58 PM, "Clint Byrum" wrote: > > Excerpts from Geoff O'Callaghan's message of 2014-09-22 17:30:47 -0700: > > On 23/09/2014 1:59 AM, "Clint Byrum" wrote: > > > > > > Geoff, do you expect all of our users to write all of their messaging > > > code in Python? > > > > > > oslo.messaging is a _python_ library. > > > > > > Zaqar is a service with a REST API -- accessible to any application. > > > > No I do not. I am suggesting thaf a well designed, scalable and robust > > messaging layer can meet the requirements of both as well as a number of > > other openstack servuces. How the messaging layer is consumed isn't the > > issue. > > > > Below is what I originally posted. > > > > > > > > > > It seems to my casual view that we could have one and scale that and > > use it > > > > for SQS style messages, internal messaging (which could include logging) > > > > all managed by message schemas and QoS. This would give a very robust > > and > > > > flexible system for endpoints to consume. > > > > > > > > Is there a plan to consolidate? > > > > Sorry for the snark George. I was very confused by the text above, and > I still am. I am confused because consolidation requires commonalities, > of which to my mind, there are almost none other than the relationship > to the very abstract term "messaging". Hahaha.. now you're calling me george ;) Don't worry dude. I'm only joking and I even liked sharknado. Any way I was hoping zaqar had a greater scope than it appears to have. I'll watch the progress. > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Oslo messaging vs zaqar
On 09/22/2014 09:25 PM, Doug Hellmann wrote: > > On Sep 22, 2014, at 12:19 PM, Gordon Sim wrote: > >> On 09/22/2014 03:40 PM, Geoff O'Callaghan wrote: >>> That being said I'm not sure why a well constructed zaqar with an rpc >>> interface couldn't meet the requirements of oslo.messsaging and much more. >> >> What Zaqar is today and what it might become may of course be different >> things but as it stands today, Zaqar relies on polling which in my opinion >> is not a natural fit for RPC[1]. Though using an intermediary for >> routing/addressing can be of benefit, store and forward is not necessary and >> in my opinion even gets in the way[2]. >> >> Notifications on the other hand can benefit from store and forward and may >> be less latency sensitive, alleviating the polling concerns. >> >> One of the use cases I've heard cited for Zaqar is as an inbox for recording >> certain sets of relevant events sent out by other open stack services. In my >> opinion using oslo.messaging's notification API on the openstack service >> side of this would seem - to me at least - quite sensible, even if the >> events are then stored in (or forwarded to) Zaqar and accessed by users >> through Zaqar's own protocol. > > I agree that the notification features of oslo.messaging are more likely to > be useful through Zaqar than the RPC API. Our internal notifications may > include information we wouldn’t want to leak outside of a cloud, but a > notification driver for oslo.messaging that talked to Zaqar and took into > account tenant-based addressing in some way might make a lot of sense. > I won't get into much detail on this right now but I agree and it was discussed already - at the Juno summit, I believe. Thanks, Flavio > Doug > >> >> >> >> [1] The latency of an RPC call as perceived by the client is going to depend >> heavily on the polling frequency; to get lower latency, you'll need to pool >> more frequently both on the server and on the client. However polling more >> frequently results in increased load even when no requests are being made. >> >> [2] I am of the view that reliable RPC is best handled by replaying the >> request from the client when needed, rather than trying to make the request >> and reply messages durably recorded, replicated and reliably delivered. >> Doing so is more scalable and simpler. An end-to-end acknowledgement for the >> request (rather than a broker taking responsibility and acknowledging the >> request independent of delivery status) makes it easier to detect failures >> and trigger a resend. >> >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- @flaper87 Flavio Percoco ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Oslo messaging vs zaqar
Excerpts from Geoff O'Callaghan's message of 2014-09-22 17:30:47 -0700: > On 23/09/2014 1:59 AM, "Clint Byrum" wrote: > > > > Geoff, do you expect all of our users to write all of their messaging > > code in Python? > > > > oslo.messaging is a _python_ library. > > > > Zaqar is a service with a REST API -- accessible to any application. > > No I do not. I am suggesting thaf a well designed, scalable and robust > messaging layer can meet the requirements of both as well as a number of > other openstack servuces. How the messaging layer is consumed isn't the > issue. > > Below is what I originally posted. > > > > > > > It seems to my casual view that we could have one and scale that and > use it > > > for SQS style messages, internal messaging (which could include logging) > > > all managed by message schemas and QoS. This would give a very robust > and > > > flexible system for endpoints to consume. > > > > > > Is there a plan to consolidate? > Sorry for the snark George. I was very confused by the text above, and I still am. I am confused because consolidation requires commonalities, of which to my mind, there are almost none other than the relationship to the very abstract term "messaging". ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Oslo messaging vs zaqar
On 23/09/2014 1:59 AM, "Clint Byrum" wrote: > > Geoff, do you expect all of our users to write all of their messaging > code in Python? > > oslo.messaging is a _python_ library. > > Zaqar is a service with a REST API -- accessible to any application. No I do not. I am suggesting thaf a well designed, scalable and robust messaging layer can meet the requirements of both as well as a number of other openstack servuces. How the messaging layer is consumed isn't the issue. Below is what I originally posted. > > > > It seems to my casual view that we could have one and scale that and use it > > for SQS style messages, internal messaging (which could include logging) > > all managed by message schemas and QoS. This would give a very robust and > > flexible system for endpoints to consume. > > > > Is there a plan to consolidate? Tks Geoff ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Oslo messaging vs zaqar
On Sep 22, 2014, at 12:19 PM, Gordon Sim wrote: > On 09/22/2014 03:40 PM, Geoff O'Callaghan wrote: >> That being said I'm not sure why a well constructed zaqar with an rpc >> interface couldn't meet the requirements of oslo.messsaging and much more. > > What Zaqar is today and what it might become may of course be different > things but as it stands today, Zaqar relies on polling which in my opinion is > not a natural fit for RPC[1]. Though using an intermediary for > routing/addressing can be of benefit, store and forward is not necessary and > in my opinion even gets in the way[2]. > > Notifications on the other hand can benefit from store and forward and may be > less latency sensitive, alleviating the polling concerns. > > One of the use cases I've heard cited for Zaqar is as an inbox for recording > certain sets of relevant events sent out by other open stack services. In my > opinion using oslo.messaging's notification API on the openstack service side > of this would seem - to me at least - quite sensible, even if the events are > then stored in (or forwarded to) Zaqar and accessed by users through Zaqar's > own protocol. I agree that the notification features of oslo.messaging are more likely to be useful through Zaqar than the RPC API. Our internal notifications may include information we wouldn’t want to leak outside of a cloud, but a notification driver for oslo.messaging that talked to Zaqar and took into account tenant-based addressing in some way might make a lot of sense. Doug > > > > [1] The latency of an RPC call as perceived by the client is going to depend > heavily on the polling frequency; to get lower latency, you'll need to pool > more frequently both on the server and on the client. However polling more > frequently results in increased load even when no requests are being made. > > [2] I am of the view that reliable RPC is best handled by replaying the > request from the client when needed, rather than trying to make the request > and reply messages durably recorded, replicated and reliably delivered. Doing > so is more scalable and simpler. An end-to-end acknowledgement for the > request (rather than a broker taking responsibility and acknowledging the > request independent of delivery status) makes it easier to detect failures > and trigger a resend. > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Oslo messaging vs zaqar
Geoff, do you expect all of our users to write all of their messaging code in Python? oslo.messaging is a _python_ library. Zaqar is a service with a REST API -- accessible to any application. As Zane's sarcastic reply implied, these are as related as sharks are to tornados. Could they be combined? Yes [1]. But the only result would be dead people and sharks strewn about the landscape. [1] http://www.imdb.com/title/tt2724064/ Excerpts from Geoff O'Callaghan's message of 2014-09-20 01:17:45 -0700: > Hi all, > I'm just trying to understand the messaging strategy in openstack.It > seems we have at least 2 messaging layers. > > Oslo.messaging and zaqar, Can someone explain to me why there are two? > To quote from the zaqar faq : > - > How does Zaqar compare to oslo.messaging? > > oslo.messsaging is an RPC library used throughout OpenStack to manage > distributed commands by sending messages through different messaging > layers. Oslo Messaging was originally developed as an abstraction over > AMQP, but has since added support for ZeroMQ. > > As opposed to Oslo Messaging, Zaqar is a messaging service for the over and > under cloud. As a service, it is meant to be consumed by using libraries > for different languages. Zaqar currently supports 1 protocol (HTTP) and > sits on top of other existing technologies (MongoDB as of version 1.0). > > It seems to my casual view that we could have one and scale that and use it > for SQS style messages, internal messaging (which could include logging) > all managed by message schemas and QoS. This would give a very robust and > flexible system for endpoints to consume. > > Is there a plan to consolidate? > > Rgds > Geoff ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Oslo messaging vs zaqar
On 09/22/2014 03:40 PM, Geoff O'Callaghan wrote: That being said I'm not sure why a well constructed zaqar with an rpc interface couldn't meet the requirements of oslo.messsaging and much more. What Zaqar is today and what it might become may of course be different things but as it stands today, Zaqar relies on polling which in my opinion is not a natural fit for RPC[1]. Though using an intermediary for routing/addressing can be of benefit, store and forward is not necessary and in my opinion even gets in the way[2]. Notifications on the other hand can benefit from store and forward and may be less latency sensitive, alleviating the polling concerns. One of the use cases I've heard cited for Zaqar is as an inbox for recording certain sets of relevant events sent out by other open stack services. In my opinion using oslo.messaging's notification API on the openstack service side of this would seem - to me at least - quite sensible, even if the events are then stored in (or forwarded to) Zaqar and accessed by users through Zaqar's own protocol. [1] The latency of an RPC call as perceived by the client is going to depend heavily on the polling frequency; to get lower latency, you'll need to pool more frequently both on the server and on the client. However polling more frequently results in increased load even when no requests are being made. [2] I am of the view that reliable RPC is best handled by replaying the request from the client when needed, rather than trying to make the request and reply messages durably recorded, replicated and reliably delivered. Doing so is more scalable and simpler. An end-to-end acknowledgement for the request (rather than a broker taking responsibility and acknowledging the request independent of delivery status) makes it easier to detect failures and trigger a resend. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Oslo messaging vs zaqar
On 22/09/14 10:40, Geoff O'Callaghan wrote: So On 22/09/2014 10:01 PM, "Zane Bitter" wrote: On 20/09/14 04:17, Geoff O'Callaghan wrote: Hi all, I'm just trying to understand the messaging strategy in openstack.It seems we have at least 2 messaging layers. Oslo.messaging and zaqar, Can someone explain to me why there are two? Is there a plan to consolidate? I'm trying to understand the database strategy in OpenStack. It seems we have at least 2 database layers - sqlalchemy and Trove. Can anyone explain to me why there are two? Is there a plan to consolidate? So the answer is because we can ;) Not a great answer, but an answer nonetheless. :) No, the answer is that they're completely different things :) That being said I'm not sure why a well constructed zaqar with an rpc interface couldn't meet the requirements of oslo.messsaging and much more.It seems I need to dig some more. Usually when people talk about "consolidation" they mean "why isn't Zaqar just a front-end to oslo.messaging?". If you mean that there should be a Zaqar back-end to oslo.messaging (alongside the existing AMQP and ZeroMQ back-ends) then that is a stronger possibility. (In my increasingly tortured analogy I guess this would be equivalent to using Trove in the undercloud to provision the RDBMS for other OpenStack services, which is a perfectly respectable idea). That said, I'm not sure if the semantics fit. Most uses of oslo.messaging AFAIK are RPC-style calls (I'm not sure what the percentage breakdown of call vs. cast is, but I believe it's heavily weighted in favour of the former). So basically it's mostly used for synchronous stuff. To me, the big selling point of Zaqar is (or at least IMHO should be - see discussion in other thread) that it is end-to-end reliable even for completely asynchronous delivery. If that's not a requirement for OpenStack services then the stuff Zaqar does to achieve it (writing each message to multiple disks in a cluster before delivering it) is probably wasted overhead for this particular application. tl;dr it's possible but probably inefficient due to differing requirements. cheers, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Oslo messaging vs zaqar
So On 22/09/2014 10:01 PM, "Zane Bitter" wrote: > > On 20/09/14 04:17, Geoff O'Callaghan wrote: >> >> Hi all, >> I'm just trying to understand the messaging strategy in openstack.It >> seems we have at least 2 messaging layers. >> >> Oslo.messaging and zaqar, Can someone explain to me why there are two? >> >> Is there a plan to consolidate? > > > I'm trying to understand the database strategy in OpenStack. It seems we have at least 2 database layers - sqlalchemy and Trove. > > Can anyone explain to me why there are two? > > > Is there a plan to consolidate? > > So the answer is because we can ;) Not a great answer, but an answer nonetheless. :) That being said I'm not sure why a well constructed zaqar with an rpc interface couldn't meet the requirements of oslo.messsaging and much more.It seems I need to dig some more. Thanks all for taking the time to reply. Geoff ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Oslo messaging vs zaqar
On 20/09/14 04:17, Geoff O'Callaghan wrote: Hi all, I'm just trying to understand the messaging strategy in openstack.It seems we have at least 2 messaging layers. Oslo.messaging and zaqar, Can someone explain to me why there are two? Is there a plan to consolidate? I'm trying to understand the database strategy in OpenStack. It seems we have at least 2 database layers - sqlalchemy and Trove. Can anyone explain to me why there are two? Is there a plan to consolidate? cheers, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Oslo messaging vs zaqar
On 09/22/2014 02:35 PM, Gordon Sim wrote: > On 09/22/2014 10:56 AM, Flavio Percoco wrote: >> What I meant is that oslo.messaging is an rpc library and it depends on >> few very specific message delivery patterns that are somehow tight/based >> on AMQP semantics. > > RPC at it's core is the request-response pattern which is directly > supported by many messaging technologies and implementable over almost > any form of communication (it's mostly just convention for addressing). > > In addition the oslo.messaging model offers the ability to invoke on one > of a group of servers, which is the task-queue pattern and again is very > general. > > One-way messages are also supported either unicast (to a specific > server) or broadcast (to all servers in a group). > > I don't think any of these patterns are in any way tightly bound or > specific to AMQP. > > In fact Zaqar offers (mostly) the same patterns (there is no way to > indicate the 'address' to reply to defined by the spec itself, but that > could be easily added). > >> Implementing Zaqar's API in oslo.messaging would be >> like trying to add an AMQP driver to Zaqar. > > I don't think it would be. AMQP and Zaqar are wire protocols offering > very similar levels of abstraction (sending, consuming, browsing, > acknowledging messages). By contrast oslo.messaging is a language level > API, generally at a slightly higher level of abstraction, mapping method > invocation to particular messaging patterns between processes. > > Implementing Zaqar's model over oslo.messaging would be like > implementing AMQP's model over oslo.messaging, i.e. reimplementing a > general purpose message-oriented abstraction on top of an API intended > to hide such a model behind message invocation. Though technically > possible it seems a little pointless (and I don't think anyone is > suggesting otherwise). I was referring to the messaging/storage technologies both projects target, which IMHO, are different. > > Zaqar drivers are really providing different implementations of > (distributed) message storage. AMQP (and I'm talking primarily about > version 1.0) is not intended for that purpose. It's intended to control > the transfer of messages between processes. Exposing AMQP as an > alternative interface for publishing/receiving/consuming messages > through Zaqar on the other hand would be simple. Somehow, I keep failing at explaining things here. The point is that IMHO, it doesn't make sense to merge both projects because they both have different goals and purposes. Cheers, Flavio -- @flaper87 Flavio Percoco ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Oslo messaging vs zaqar
On 09/22/2014 10:56 AM, Flavio Percoco wrote: What I meant is that oslo.messaging is an rpc library and it depends on few very specific message delivery patterns that are somehow tight/based on AMQP semantics. RPC at it's core is the request-response pattern which is directly supported by many messaging technologies and implementable over almost any form of communication (it's mostly just convention for addressing). In addition the oslo.messaging model offers the ability to invoke on one of a group of servers, which is the task-queue pattern and again is very general. One-way messages are also supported either unicast (to a specific server) or broadcast (to all servers in a group). I don't think any of these patterns are in any way tightly bound or specific to AMQP. In fact Zaqar offers (mostly) the same patterns (there is no way to indicate the 'address' to reply to defined by the spec itself, but that could be easily added). Implementing Zaqar's API in oslo.messaging would be like trying to add an AMQP driver to Zaqar. I don't think it would be. AMQP and Zaqar are wire protocols offering very similar levels of abstraction (sending, consuming, browsing, acknowledging messages). By contrast oslo.messaging is a language level API, generally at a slightly higher level of abstraction, mapping method invocation to particular messaging patterns between processes. Implementing Zaqar's model over oslo.messaging would be like implementing AMQP's model over oslo.messaging, i.e. reimplementing a general purpose message-oriented abstraction on top of an API intended to hide such a model behind message invocation. Though technically possible it seems a little pointless (and I don't think anyone is suggesting otherwise). Zaqar drivers are really providing different implementations of (distributed) message storage. AMQP (and I'm talking primarily about version 1.0) is not intended for that purpose. It's intended to control the transfer of messages between processes. Exposing AMQP as an alternative interface for publishing/receiving/consuming messages through Zaqar on the other hand would be simple. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Oslo messaging vs zaqar
On 09/22/2014 11:04 AM, Gordon Sim wrote: > On 09/22/2014 08:00 AM, Flavio Percoco wrote: >> oslo messaging is highly tight to AMQP semantics whereas Zaqar is not. > > The API for oslo.messaging is actually quite high level and could easily > be mapped to different messaging technologies. > > There is some terminology that comes from older versions of AMQP, e.g. > the use of 'exchanges' as essentially a namespace, but I don't myself > believe these tie the API in anyway to the original implementation from > which the naming arose. > > In what way do you see olso.messaging as being tied to AMQP semantics? I'm pretty sure it can be mapped to different messaging technologies (there's a zmq driver). I could see myself experimenting on a Zaqar driver for oslo.messaging in the future. What I meant is that oslo.messaging is an rpc library and it depends on few very specific message delivery patterns that are somehow tight/based on AMQP semantics. Implementing Zaqar's API in oslo.messaging would be like trying to add an AMQP driver to Zaqar. Probably "highly tight to AMQP" wasn't the best way to express this, sorry about that. Cheers, Flavio -- @flaper87 Flavio Percoco ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Oslo messaging vs zaqar
On 09/22/2014 08:00 AM, Flavio Percoco wrote: oslo messaging is highly tight to AMQP semantics whereas Zaqar is not. The API for oslo.messaging is actually quite high level and could easily be mapped to different messaging technologies. There is some terminology that comes from older versions of AMQP, e.g. the use of 'exchanges' as essentially a namespace, but I don't myself believe these tie the API in anyway to the original implementation from which the naming arose. In what way do you see olso.messaging as being tied to AMQP semantics? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Oslo messaging vs zaqar
On 09/20/2014 10:17 AM, Geoff O'Callaghan wrote: > Hi all, > I'm just trying to understand the messaging strategy in openstack.It > seems we have at least 2 messaging layers. > > Oslo.messaging and zaqar, Can someone explain to me why there are > two?To quote from the zaqar faq : > - > How does Zaqar compare to oslo.messaging? > > oslo.messsaging is an RPC library used throughout OpenStack to manage > distributed commands by sending messages through different messaging > layers. Oslo Messaging was originally developed as an abstraction over > AMQP, but has since added support for ZeroMQ. > > As opposed to Oslo Messaging, Zaqar is a messaging service for the over > and under cloud. As a service, it is meant to be consumed by using > libraries for different languages. Zaqar currently supports 1 protocol > (HTTP) and sits on top of other existing technologies (MongoDB as of > version 1.0). > > It seems to my casual view that we could have one and scale that and use > it for SQS style messages, internal messaging (which could include > logging) all managed by message schemas and QoS. This would give a very > robust and flexible system for endpoints to consume. > > Is there a plan to consolidate? Hi Geoff, No, there's no plan to consolidate. As mentioned in the FAQ, oslo.messaging is a messaging *library* whereas Zaqar is a messaging *service*. Moreover, oslo messaging is highly tight to AMQP semantics whereas Zaqar is not. Note that I'm not saying this isn't technically possible, I'm saying these 2 projects have different goals, visions and scope, hence they weren't merged nor they will. Cheers, Flavio -- @flaper87 Flavio Percoco ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] Oslo messaging vs zaqar
Hi all, I'm just trying to understand the messaging strategy in openstack.It seems we have at least 2 messaging layers. Oslo.messaging and zaqar, Can someone explain to me why there are two? To quote from the zaqar faq : - How does Zaqar compare to oslo.messaging? oslo.messsaging is an RPC library used throughout OpenStack to manage distributed commands by sending messages through different messaging layers. Oslo Messaging was originally developed as an abstraction over AMQP, but has since added support for ZeroMQ. As opposed to Oslo Messaging, Zaqar is a messaging service for the over and under cloud. As a service, it is meant to be consumed by using libraries for different languages. Zaqar currently supports 1 protocol (HTTP) and sits on top of other existing technologies (MongoDB as of version 1.0). It seems to my casual view that we could have one and scale that and use it for SQS style messages, internal messaging (which could include logging) all managed by message schemas and QoS. This would give a very robust and flexible system for endpoints to consume. Is there a plan to consolidate? Rgds Geoff ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
