Re: [openstack-dev] live-snapshot/cloning of virtual machines
On Aug 27, 2013, at 18:52 , Russell Bryant rbry...@redhat.commailto:rbry...@redhat.com wrote: On 08/27/2013 10:53 AM, Alessandro Pilotti wrote: That's IMO a different story: backporting a driver is usually quite trivial as it affects only one service (nova-compute) and one interaction point with Nova (the driver's interface). Between Havana and Grizzly for example, the entire Hyper-V driver can be backported without substantial issues. On the deployment side, we have to care only about updating the code which runs con the compute nodes, using vanilla OpenStack components on the controller and remaining nodes. Backporting the public APIs is a whole different story, it affects way more components that need to be deployed (nova-api as a minimum of course), with way more interaction points that might incur into patching hell. Do you really know that? This is pretty hand wavy. I think you're making this backport out to be _way_ more complicated than it is. I don't see why it's any more complicated than a virt driver feature backport. What about publishing the API as blacklisted by default? This way it would be available only to users that enable it explicitly, while still supporting the scenario described above. It still makes no sense to me to merge an API for a feature that can't be used. I just committed a fully working implementation of the live-snapshot blueprint in the Hyper-V driver. The tests haven't been published yet (I need to clean them up a bit before). Here's the patch: https://review.openstack.org/#/c/44595/ My plan was to write it and publish it at the beginning of the next cycle, but I was wondering if with this we could save the live-snapshot APIs in Havana, so I hurried up a bit. :-) I know that it's awfully late to bring it up for Havana, if it's not possible to accept it, no problem at all of course, I'm going to bring it back at the beginning of the Icehouse cycle. The results in terms of boot times are quite impressive. This feature, especially combined with the Hyper-V RemoteFX feature in Havana (host GPU access in the instances) can bring VDI snenarios to another level, just to name one of the use cases. We are already at work on the cloudbase-init side to support the initialization on the client side. The compute node takes of course care of changing Mac addresses and attaching a new config drive (when needed). Alessandro -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] live-snapshot/cloning of virtual machines
On 08/26/2013 08:15 PM, Tim Smith wrote: Hi all, On Mon, Aug 19, 2013 at 11:49 PM, Bob Ball bob.b...@citrix.com mailto:bob.b...@citrix.com wrote: I agree with the below from a XenServer perspective. As with vmware, XenServer supports live snapshotting and creating multiple clones from that live snapshot. I understand that there is a XenAPI equivalent in the works and therefore would argue the API changes need to be accepted as a minimum. Can nova technical leadership provide clarification on the current standing of this blueprint? Two hypervisor vendors have expressed plans for supporting this feature, and one has specifically requested that the API changes be merged, but it appears that both the API changeset [1] and novaclient support [2] have both been rejected pending libvirt support (which has assumedly been ruled out for the Havana release). [1] https://review.openstack.org/#/c/34036/ [2] https://review.openstack.org/#/c/43777/ In order to minimize the feature divergence between hypervisors, I'd also argue that we should accept the libvirt implementation even if it uses unsupported APIs - perhaps disabled by default with a suitable warning that it isn't considered safe by libvirt/QEmu. It's understandable that changes to the libvirt driver would be held back until libvirt/qemu-upstream support for live snapshotting is established (if ever), but given that other vendors whose release cadences don't necessarily align with the nova release schedule have expressed plans to support the interface it's unclear why lack of libvirt driver support would block the entire blueprint. Two other driver maintainers have expressed interest in it, but AFAIK, there are not implementations of this feature ready for review and merging for these drivers. Given that's the case, it doesn't make any sense to me to merge the API with no ability to use it. I'm only saying it should wait until it can be merged with something that makes it usable. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] live-snapshot/cloning of virtual machines
On 08/27/2013 10:06 AM, Alessandro Pilotti wrote: We are also planning to implement the live snapshot feature in the Hyper-V driver during the next release cycle. I'm personally in favour of publishing the APIs in Havana, as this would provide a stable baseline at the beginning of the release cycle and also The API is published already. What matters even more than the API for you as a driver maintainer is the driver interface, which is actually already merged. It went in before it became clear the libvirt patch wouldn't go in, but I don't think there's any reason to remove it now. give the ability to users and third parties to backport the driver's feature to Havana (outside of the official repo of course). If you're backporting stuff anyway, you can backport the API patch, as well. I see no sense in delivering an API to *everyone* that can't be used. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] live-snapshot/cloning of virtual machines
On Tue, Aug 27, 2013 at 10:34 AM, Russell Bryant rbry...@redhat.com wrote: On 08/27/2013 10:06 AM, Alessandro Pilotti wrote: We are also planning to implement the live snapshot feature in the Hyper-V driver during the next release cycle. I'm personally in favour of publishing the APIs in Havana, as this would provide a stable baseline at the beginning of the release cycle and also The API is published already. What matters even more than the API for you as a driver maintainer is the driver interface, which is actually already merged. It went in before it became clear the libvirt patch wouldn't go in, but I don't think there's any reason to remove it now. Since the API is published already, where is the harm in offering a backing implementation of it? This completes the picture and leaves only the virt driver maintainers to finish up the work and they can do that in their own time based on their own priorities and release schedule. Ultimately the API implementation and the virt driver work is being done by two distinct groups. I don't think its beneficial to block one group's efforts because another group has different priorities. Especially since both groups have expressed a desire to see the work in. give the ability to users and third parties to backport the driver's feature to Havana (outside of the official repo of course). If you're backporting stuff anyway, you can backport the API patch, as well. I see no sense in delivering an API to *everyone* that can't be used. Why require the additional hassle of backporting the API patch (which affects a different sets of nodes / services than backporting pure driver support)? Especially since the API patch simply fills in the implementation of the published API. I understand that in the current outlook, Icehouse will be the release where this feature really shines because it'll be supported by most of the virt drivers. However in the 6 months of the Havana release there is a rough patch of the functionality available in Vish's libvirt patch. Yes, it is not the long term solution, unsupported by libvirt maintainers and comes with a bunch of caveats around its use. But early adopters can certainly use this patch to experiment with this API and see what interesting workflows come out of it. That way they will be ready for when Icehouse lands with full support and it is ready for primetime. Thanks, David Scannell ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] live-snapshot/cloning of virtual machines
On 08/27/2013 10:53 AM, Alessandro Pilotti wrote: That's IMO a different story: backporting a driver is usually quite trivial as it affects only one service (nova-compute) and one interaction point with Nova (the driver's interface). Between Havana and Grizzly for example, the entire Hyper-V driver can be backported without substantial issues. On the deployment side, we have to care only about updating the code which runs con the compute nodes, using vanilla OpenStack components on the controller and remaining nodes. Backporting the public APIs is a whole different story, it affects way more components that need to be deployed (nova-api as a minimum of course), with way more interaction points that might incur into patching hell. Do you really know that? This is pretty hand wavy. I think you're making this backport out to be _way_ more complicated than it is. I don't see why it's any more complicated than a virt driver feature backport. What about publishing the API as blacklisted by default? This way it would be available only to users that enable it explicitly, while still supporting the scenario described above. It still makes no sense to me to merge an API for a feature that can't be used. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] live-snapshot/cloning of virtual machines
On 08/27/2013 12:04 PM, Alessandro Pilotti wrote: On Aug 27, 2013, at 18:52 , Russell Bryant rbry...@redhat.com mailto:rbry...@redhat.com wrote: On 08/27/2013 10:53 AM, Alessandro Pilotti wrote: That's IMO a different story: backporting a driver is usually quite trivial as it affects only one service (nova-compute) and one interaction point with Nova (the driver's interface). Between Havana and Grizzly for example, the entire Hyper-V driver can be backported without substantial issues. On the deployment side, we have to care only about updating the code which runs con the compute nodes, using vanilla OpenStack components on the controller and remaining nodes. Backporting the public APIs is a whole different story, it affects way more components that need to be deployed (nova-api as a minimum of course), with way more interaction points that might incur into patching hell. Do you really know that? This is pretty hand wavy. I think you're making this backport out to be _way_ more complicated than it is. I don't see why it's any more complicated than a virt driver feature backport. No, that's why I used might instead of will :-) More important than the coding issue, there's the deployment and support issue for additional components that need to be mantained outside of the main code repo. What about publishing the API as blacklisted by default? This way it would be available only to users that enable it explicitly, while still supporting the scenario described above. It still makes no sense to me to merge an API for a feature that can't be used. This depends on the definition of can't be used: It's a feature that can't be used in the Havana code base, but I can assure you that we would do good use of it by backporting the I code. Used by code *in the tree*. If you're backporting anything, you're on your own. I find it completely unreasonable to ask the upstream project to worry about supporting that kind of thing. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] live-snapshot/cloning of virtual machines
On 08/27/2013 12:04 PM, Tim Smith wrote: On Tue, Aug 27, 2013 at 8:52 AM, Russell Bryant rbry...@redhat.com mailto:rbry...@redhat.com wrote: What about publishing the API as blacklisted by default? This way it would be available only to users that enable it explicitly, while still supporting the scenario described above. It still makes no sense to me to merge an API for a feature that can't be used. While it's true that there won't be an in-tree driver that supports the API for this release cycle, we have a commercial driver that supports it (https://github.com/gridcentric/cobalt). Having the API standardized in Havana would ensure that client support is immediately available for our users as well as for the other hypervisor vendors should they release a supporting driver in the next 9 months. I believe there is precedent for publishing a nova API for those purposes. IMO, to be the healthiest project we can be, we must focus on what code is actually a part of Nova. If you'd like to submit your changes for inclusion into Nova, then we can talk. What you are seeing here is a part of the pain of maintaining a fork. I am not OK with shifting part of that burden on to the upstream project when it doesn't help the upstream project *at all*. When we have supporting code to make the feature usable, then the API can go in. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] live-snapshot/cloning of virtual machines
On Tue, Aug 27, 2013 at 12:13:49PM -0400, Russell Bryant wrote: On 08/27/2013 12:04 PM, Tim Smith wrote: On Tue, Aug 27, 2013 at 8:52 AM, Russell Bryant rbry...@redhat.com mailto:rbry...@redhat.com wrote: What about publishing the API as blacklisted by default? This way it would be available only to users that enable it explicitly, while still supporting the scenario described above. It still makes no sense to me to merge an API for a feature that can't be used. While it's true that there won't be an in-tree driver that supports the API for this release cycle, we have a commercial driver that supports it (https://github.com/gridcentric/cobalt). Having the API standardized in Havana would ensure that client support is immediately available for our users as well as for the other hypervisor vendors should they release a supporting driver in the next 9 months. I believe there is precedent for publishing a nova API for those purposes. IMO, to be the healthiest project we can be, we must focus on what code is actually a part of Nova. If you'd like to submit your changes for inclusion into Nova, then we can talk. What you are seeing here is a part of the pain of maintaining a fork. I am not OK with shifting part of that burden on to the upstream project when it doesn't help the upstream project *at all*. When we have supporting code to make the feature usable, then the API can go in. Totally agreed with this. Supporting APIs in Nova with no in-tree users, to satisfy the requirements of out of tree drivers should be an explicit non-goal of the community IMHO. If a 3rd party does not wish to contribute their code to Nova codebase, then it is expected that they take on all the burden of doing the extra integration work their fork/branch implies. For a code review POV, I would also not be satisfied doing review of APIs without illustration of an in-tree driver wired up to the wire. Doing API design is hard work, and I've been burnt too many times on other projects adding APIs without in tree users, which then had to be thrown away or replaced when the in-tree user finally arrived. So I would be very much against adding any APIs without in-tree users, even ignoring the fact that I think the live VM cloning concept as a whole is flawed. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] live-snapshot/cloning of virtual machines
While it's true that there won't be an in-tree driver that supports the API for this release cycle, we have a commercial driver that supports it ( https://github.com/gridcentric/cobalt). IMHO, out of tree virt drivers are completely out of scope here. We change the virt driver API at will, adding, removing, and changing things without any concern for what it may do to anything outside of our tree. Until we commit to a stable virt driver API, the above argument holds no weight for me. Having the API standardized in Havana would ensure that client support is immediately available for our users as well as for the other hypervisor vendors should they release a supporting driver in the next 9 months. I believe there is precedent for publishing a nova API for those purposes. Having an API codified (and committed-to) before we have a working implementation that uses it in Nova is asking for trouble, IMHO. Personally, I think the hardest bit is already in the tree, which is the piece that allows the API to communicate with a fictional live-snapshot-supporting virt driver. Backporting the API and writing a virt driver to the interface that is there is cake compared to what it would take to bolt on the plumbing part. IIRC, the plumbing was speculatively merged ahead of the libvirt and API pieces, but perhaps the safest (and least confusing) thing for us to do would be to yank it out prior to Havana? --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] live-snapshot/cloning of virtual machines
On Tue, Aug 27, 2013 at 12:13 PM, Russell Bryant rbry...@redhat.com wrote: IMO, to be the healthiest project we can be, we must focus on what code is actually a part of Nova. If you'd like to submit your changes for inclusion into Nova, then we can talk. That's ultimately what we're trying to accomplish here. What you are seeing here is a part of the pain of maintaining a fork. I am not OK with shifting part of that burden on to the upstream project when it doesn't help the upstream project *at all*. We're not maintaining a fork, nor trying to shift burden. That's unfair. We have a long term interest in the success of OpenStack, like everyone here. We're not asking upstream to do anything. This isn't adversarial. There are plenty of things that don't help nova directly, but certainly enable a vibrant ecosystem. For example, having extensible APIs and pluggable backends is critical to the philosophy and success of OpenStack as a whole. We absolutely understand that having solid, in-tree implementations is also important. That's why as a part of the blueprint, there was a pan-community effort made to create a libvirt implementation. Although that particular effort has hit some speed bumps, merging the API extension would still benefit members of the community by simplifying deployments (i.e. for us) and Havana backports (i.e. needing to provide only an updated compute driver w/ config change). Other hypervisor driver maintainers have also expressed the desire to see it merged to speed and simplify development of in-tree implementations moving forward. It's not just going to get dropped on the floor. In the end, I think that the need to have the reference implementation land at the same moment should be balanced against community interests. Yes, we really want to see the API in Havana, but it seems we're not alone. I would understand holding off if there were substantial downsides to merging, or if multiple hypervisor vendors had expressed a desire to see it not merged. But that doesn't seem to be the case. Six months is a long time, and ultimately the evolution of an open source ecosystem is just as important as the code itself. Thanks, -Adin ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] live-snapshot/cloning of virtual machines
Hi all, On Mon, Aug 19, 2013 at 11:49 PM, Bob Ball bob.b...@citrix.com wrote: I agree with the below from a XenServer perspective. As with vmware, XenServer supports live snapshotting and creating multiple clones from that live snapshot. I understand that there is a XenAPI equivalent in the works and therefore would argue the API changes need to be accepted as a minimum. Can nova technical leadership provide clarification on the current standing of this blueprint? Two hypervisor vendors have expressed plans for supporting this feature, and one has specifically requested that the API changes be merged, but it appears that both the API changeset [1] and novaclient support [2] have both been rejected pending libvirt support (which has assumedly been ruled out for the Havana release). [1] https://review.openstack.org/#/c/34036/ [2] https://review.openstack.org/#/c/43777/ In order to minimize the feature divergence between hypervisors, I'd also argue that we should accept the libvirt implementation even if it uses unsupported APIs - perhaps disabled by default with a suitable warning that it isn't considered safe by libvirt/QEmu. It's understandable that changes to the libvirt driver would be held back until libvirt/qemu-upstream support for live snapshotting is established (if ever), but given that other vendors whose release cadences don't necessarily align with the nova release schedule have expressed plans to support the interface it's unclear why lack of libvirt driver support would block the entire blueprint. Thanks, Tim Bob From: Shawn Hartsock [hartso...@vmware.com] Sent: 19 August 2013 20:59 To: Daniel P. Berrange; OpenStack Development Mailing List Subject: Re: [openstack-dev] [nova] live-snapshot/cloning of virtual machines For what it's worth... this doesn't seem too bad to me... I was planning on using this part of the vSphere API: * https://www.vmware.com/support/developer/vc-sdk/visdk400pubs/ReferenceGuide/vim.vm.CloneSpec.html ...to accomplish the clone part of the BP. The API contains a spec section where you tell the ESX hypervisor how to handle things like network identity... * https://www.vmware.com/support/developer/vc-sdk/visdk400pubs/ReferenceGuide/vim.vm.customization.IPSettings.html ... I was going to use this to plumb together the live-snapshot bit ... * https://www.vmware.com/support/developer/vc-sdk/visdk400pubs/ReferenceGuide/vim.VirtualMachine.html#createSnapshot ... which includes stuff about how to handle memory. So, I didn't read this blueprint as especially hard to accomplish in the vmwareapi driver. It just would eat more time than I have right now and would require a deeper level of understanding of how the vSphere hypervisor suite works than most of the other features currently use. I'm fully planning on playing with this in IceHouse just to see how it would go, it's probably one of the more nifty new features we could add. Note: these are old features for the API and they are a tad complicated, but it's all well within the realm of supported! In fact, it's standard operating procedure to use the clone feature to scale-out an application in some vSphere shops. (albeit, in production the admins I know personally, use clone with power-off from a 'template' and the system comes up with a new MAC/etc on first boot... cloning from a running system is possible, but I would recommend cloning from a power-off state unless you've got a hot-plug feature in your guest OS) # Shawn Hartsock - Original Message - From: Daniel P. Berrange berra...@redhat.com To: OpenStack Development Mailing List openstack-dev@lists.openstack.org Sent: Monday, August 19, 2013 5:24:59 AM Subject: Re: [openstack-dev] [nova] live-snapshot/cloning of virtual machines On Mon, Aug 19, 2013 at 08:28:58AM +1200, Robert Collins wrote: On 17 August 2013 07:01, Russell Bryant rbry...@redhat.com wrote: Maybe we've grown up to the point where we have to be more careful and not introduce these kind of features and the maintenance cost of introducing experimental features is too great. If that is the community consensus, then I'm happy keep the live snapshot stuff in a branch on github for people to experiment with. My feeling after following this discussion is that it's probably best to keep baking in another branch (github or whatever). The biggest reason is because of the last comment quoted from Daniel Berrange above. I feel that like that is a pretty big deal. So, reading between the lines here, I guess you're worried that we'd let code paths that violate what upstream will support leak into the main codepaths for libvirt - and thus we'd end up with a situation where we aren't supported by upstream for all regular operations. Yes, if you perform a live clone of a VM, then you have effectively tainted
Re: [openstack-dev] live-snapshot/cloning of virtual machines
I agree with the below from a XenServer perspective. As with vmware, XenServer supports live snapshotting and creating multiple clones from that live snapshot. I understand that there is a XenAPI equivalent in the works and therefore would argue the API changes need to be accepted as a minimum. In order to minimize the feature divergence between hypervisors, I'd also argue that we should accept the libvirt implementation even if it uses unsupported APIs - perhaps disabled by default with a suitable warning that it isn't considered safe by libvirt/QEmu. Bob From: Shawn Hartsock [hartso...@vmware.com] Sent: 19 August 2013 20:59 To: Daniel P. Berrange; OpenStack Development Mailing List Subject: Re: [openstack-dev] [nova] live-snapshot/cloning of virtual machines For what it's worth... this doesn't seem too bad to me... I was planning on using this part of the vSphere API: * https://www.vmware.com/support/developer/vc-sdk/visdk400pubs/ReferenceGuide/vim.vm.CloneSpec.html ...to accomplish the clone part of the BP. The API contains a spec section where you tell the ESX hypervisor how to handle things like network identity... * https://www.vmware.com/support/developer/vc-sdk/visdk400pubs/ReferenceGuide/vim.vm.customization.IPSettings.html ... I was going to use this to plumb together the live-snapshot bit ... * https://www.vmware.com/support/developer/vc-sdk/visdk400pubs/ReferenceGuide/vim.VirtualMachine.html#createSnapshot ... which includes stuff about how to handle memory. So, I didn't read this blueprint as especially hard to accomplish in the vmwareapi driver. It just would eat more time than I have right now and would require a deeper level of understanding of how the vSphere hypervisor suite works than most of the other features currently use. I'm fully planning on playing with this in IceHouse just to see how it would go, it's probably one of the more nifty new features we could add. Note: these are old features for the API and they are a tad complicated, but it's all well within the realm of supported! In fact, it's standard operating procedure to use the clone feature to scale-out an application in some vSphere shops. (albeit, in production the admins I know personally, use clone with power-off from a 'template' and the system comes up with a new MAC/etc on first boot... cloning from a running system is possible, but I would recommend cloning from a power-off state unless you've got a hot-plug feature in your guest OS) # Shawn Hartsock - Original Message - From: Daniel P. Berrange berra...@redhat.com To: OpenStack Development Mailing List openstack-dev@lists.openstack.org Sent: Monday, August 19, 2013 5:24:59 AM Subject: Re: [openstack-dev] [nova] live-snapshot/cloning of virtual machines On Mon, Aug 19, 2013 at 08:28:58AM +1200, Robert Collins wrote: On 17 August 2013 07:01, Russell Bryant rbry...@redhat.com wrote: Maybe we've grown up to the point where we have to be more careful and not introduce these kind of features and the maintenance cost of introducing experimental features is too great. If that is the community consensus, then I'm happy keep the live snapshot stuff in a branch on github for people to experiment with. My feeling after following this discussion is that it's probably best to keep baking in another branch (github or whatever). The biggest reason is because of the last comment quoted from Daniel Berrange above. I feel that like that is a pretty big deal. So, reading between the lines here, I guess you're worried that we'd let code paths that violate what upstream will support leak into the main codepaths for libvirt - and thus we'd end up with a situation where we aren't supported by upstream for all regular operations. Yes, if you perform a live clone of a VM, then you have effectively tainted that VM for the rest of its lifetime. From the virt host vendors' POV, any unexpected or problematic behaviour you get from that VM thereafter will be outside scope of support. The onus would be on the openstack sysadmin to demonstrate that the same problem occurs without the use of live cloning. Running a production cloud using a feature that your virt host vendor considers unsupported would be somewhat reckless IMHO, unless you think your sysadmins think they have the skills to solve all possible problems in that area themselves, which is unlikely for most cloud vendors. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http
