Re: [Openstack] Global deployment of Glance
On Tue, May 17, 2011 at 11:59 AM, Glen Campbell glen.campb...@rackspace.com wrote: If we are going to deploy Glance to support a global deployment of Nova, would it make sense to have replicas in different regions for better performance? Or, to put it another way, is there a recommended way to keep multiple Glance installations in sync? Hi Glen! I think a better idea than having multiple copies of an image in different regions is to do two things: a) Use a proxy caching server like Varnish or Squid to cache pieces or all of an image in various zones b) Use a highly-available storage system like Swift behind the global Glance server For a) we need to complete the HTTP 1.1 Cache headers blueprint (https://blueprints.launchpad.net/glance/+spec/caching) and for b) you would simply use the Swift backend, configured appropriately for a large Swift cluster. Users doing snapshots/backups, etc., would presumably get better performance if Glance was local, but how would we keep the base/shared images in sync? This is actually something that Rick H and Chris McG are working on. The basic strategy that they came up with was to add a parent ID attribute to the image and for any snapshot image, simply refer to the base image as the snapshot image's parent. The glance client would check for a parent_id that wasn't null and continue streaming the image while it found a parent URI/ID. For example, let's say you have a golden image with the URI: http://glance.example.com/12345. A user creates an instance with this image and some time later, decides to do a snapshot or backup of their running instance. The snapshotting code in the virtualization layer produces what is essentially a differential snapshot, containing only the differing bits of the existing image with the base golden image. This snapshot (typically much smaller than the original image) could be stored in the local (zone-local) Glance server with a call to POST /images. When pushing this snapshot image to the local Glance server, we would set the parent ID to http://glance.example.com/12345. Let's say at some later time, the user wanted to restore from this backup. The virtualization layer that implemented the restore call would need to stream the backup image from the local Glance server. In doing so, it would use the glance client class' get_image() method. When calling this method, the glance client would first return the snapshot image piece. Noticing the image had a parent ID, it would continue to stream the golden image from the global image Glance server in-line, essentially enabling us to store only the small diff of the snapshot locally while streaming the bulk of the image master from the global Glance server. I'll let Rick elaborate on the above and correct any mistakes I made in my description. :) -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Global deployment of Glance
Each zone should definitely have glance instances, IMO. At least two per zone for redundancy and networking reasons in large OpenStack installations. There's some work to do to support this, though. - Chris On May 17, 2011, at 8:59 AM, Glen Campbell wrote: If we are going to deploy Glance to support a global deployment of Nova, would it make sense to have replicas in different regions for better performance? Or, to put it another way, is there a recommended way to keep multiple Glance installations in sync? Users doing snapshots/backups, etc., would presumably get better performance if Glance was local, but how would we keep the base/shared images in sync? signature[4].png Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.com, and delete the original message. Your cooperation is appreciated. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Global deployment of Glance
Ignoring how it is actually implemented, I think we do want copies of base images in reach region. We don't want any sort of outage in one region to adversely affect another region. - Chris On May 17, 2011, at 9:36 AM, Jay Pipes wrote: On Tue, May 17, 2011 at 11:59 AM, Glen Campbell glen.campb...@rackspace.com wrote: If we are going to deploy Glance to support a global deployment of Nova, would it make sense to have replicas in different regions for better performance? Or, to put it another way, is there a recommended way to keep multiple Glance installations in sync? Hi Glen! I think a better idea than having multiple copies of an image in different regions is to do two things: a) Use a proxy caching server like Varnish or Squid to cache pieces or all of an image in various zones b) Use a highly-available storage system like Swift behind the global Glance server For a) we need to complete the HTTP 1.1 Cache headers blueprint (https://blueprints.launchpad.net/glance/+spec/caching) and for b) you would simply use the Swift backend, configured appropriately for a large Swift cluster. Users doing snapshots/backups, etc., would presumably get better performance if Glance was local, but how would we keep the base/shared images in sync? This is actually something that Rick H and Chris McG are working on. The basic strategy that they came up with was to add a parent ID attribute to the image and for any snapshot image, simply refer to the base image as the snapshot image's parent. The glance client would check for a parent_id that wasn't null and continue streaming the image while it found a parent URI/ID. For example, let's say you have a golden image with the URI: http://glance.example.com/12345. A user creates an instance with this image and some time later, decides to do a snapshot or backup of their running instance. The snapshotting code in the virtualization layer produces what is essentially a differential snapshot, containing only the differing bits of the existing image with the base golden image. This snapshot (typically much smaller than the original image) could be stored in the local (zone-local) Glance server with a call to POST /images. When pushing this snapshot image to the local Glance server, we would set the parent ID to http://glance.example.com/12345. Let's say at some later time, the user wanted to restore from this backup. The virtualization layer that implemented the restore call would need to stream the backup image from the local Glance server. In doing so, it would use the glance client class' get_image() method. When calling this method, the glance client would first return the snapshot image piece. Noticing the image had a parent ID, it would continue to stream the golden image from the global image Glance server in-line, essentially enabling us to store only the small diff of the snapshot locally while streaming the bulk of the image master from the global Glance server. I'll let Rick elaborate on the above and correct any mistakes I made in my description. :) -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Global deployment of Glance
Assuming you are using Swift for storage, the Swift ring configuration can specify zones and minimum number of replicas, which could handle all this logic and bit pushing for you. -Eric On Tue, May 17, 2011 at 06:36:38PM +, Glen Campbell wrote: That's probably the easiest to implement. This would mean that each deployment of new images would need to be installed in each region. On 5/17/11 12:47 PM, Chris Behrens chris.behr...@rackspace.com wrote: Ignoring how it is actually implemented, I think we do want copies of base images in reach region. We don't want any sort of outage in one region to adversely affect another region. - Chris On May 17, 2011, at 9:36 AM, Jay Pipes wrote: On Tue, May 17, 2011 at 11:59 AM, Glen Campbell glen.campb...@rackspace.com wrote: If we are going to deploy Glance to support a global deployment of Nova, would it make sense to have replicas in different regions for better performance? Or, to put it another way, is there a recommended way to keep multiple Glance installations in sync? Hi Glen! I think a better idea than having multiple copies of an image in different regions is to do two things: a) Use a proxy caching server like Varnish or Squid to cache pieces or all of an image in various zones b) Use a highly-available storage system like Swift behind the global Glance server For a) we need to complete the HTTP 1.1 Cache headers blueprint (https://blueprints.launchpad.net/glance/+spec/caching) and for b) you would simply use the Swift backend, configured appropriately for a large Swift cluster. Users doing snapshots/backups, etc., would presumably get better performance if Glance was local, but how would we keep the base/shared images in sync? This is actually something that Rick H and Chris McG are working on. The basic strategy that they came up with was to add a parent ID attribute to the image and for any snapshot image, simply refer to the base image as the snapshot image's parent. The glance client would check for a parent_id that wasn't null and continue streaming the image while it found a parent URI/ID. For example, let's say you have a golden image with the URI: http://glance.example.com/12345. A user creates an instance with this image and some time later, decides to do a snapshot or backup of their running instance. The snapshotting code in the virtualization layer produces what is essentially a differential snapshot, containing only the differing bits of the existing image with the base golden image. This snapshot (typically much smaller than the original image) could be stored in the local (zone-local) Glance server with a call to POST /images. When pushing this snapshot image to the local Glance server, we would set the parent ID to http://glance.example.com/12345. Let's say at some later time, the user wanted to restore from this backup. The virtualization layer that implemented the restore call would need to stream the backup image from the local Glance server. In doing so, it would use the glance client class' get_image() method. When calling this method, the glance client would first return the snapshot image piece. Noticing the image had a parent ID, it would continue to stream the golden image from the global image Glance server in-line, essentially enabling us to store only the small diff of the snapshot locally while streaming the bulk of the image master from the global Glance server. I'll let Rick elaborate on the above and correct any mistakes I made in my description. :) -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
