Re: [Openstack] Share Glance between cells or regions
Hi Jay, We are doing something similar. We have a single glance registry which is backed by galera DB replication. Then we have multiple glance-apis around the place. Currently they are all backed onto the same swift but I'd like to have it so each glance-api can talk to it's own swift. The issue I see is that the location of the image as stored in the glance-registry is a keystone url. So yes you could get a glance api to store data in a specific swift region (using the swift_store_region) but it has no way of knowing which region to pull an image out of. I think the location value stored when using swift needs to be the swift URL or else it needs to store the region in the DB too. Have you thought about this? Have a solution? Cheers, Sam On 16/05/2013, at 6:49 AM, Jay Pipes jaypi...@gmail.com wrote: On 05/15/2013 02:46 PM, John Paul Walters wrote: Hi, We're looking at setting up a geographically distributed OpenStack installation, and we're considering either cells or regions. We'd like to share a single Glance install between our regions (or cells), so the same images can be spawned anywhere. From here: http://docs.openstack.org/trunk/openstack-ops/content/segregate_cloud.html it's not clear whether that's possible. Can anyone shed some light on this? Is it possible in regions OR cells (or both)? Is there a better solution that I'm not thinking of? We will be sharing both the Keystone identity (note: not token/catalog) and Glance registry databases in a synchronously-replicated Galera MySQL cluster. Databases like the above, which have extremely low write to read ratios are ideal for this kind of replication. We are replicating working sets over the WAN using rsync replication in the WSREP clustering software. What this enables us to do is have a single set of account records and a single set of image (base and snapshot) records. Note that we back Glance in each zone with a zone-local Swift cluster. But what this allows us to do is have a user in zone A make a snapshot and then immediately (once the snapshot goes from the SAVING state to ACTIVE), the user is able to launch their snapshot in zone B. The Glance registry database has the location of the snapshot in zone A's Swift cluster and when Nova in zone B launches the image, the Glance API server in zone B simply pulls the image bits from Swift in zone A. Best, -jay p.s. I say will be sharing because we are currently updating our deployment to use this single Glance registry database. Originally we went down the route of each zone having its own Glance registry database and realized that since the pattern of write activity to the Glance registry is so low, it made sense to replicate it across our zones and give the users the ability to launch instances from snapshots in any zone. The single identity database is already in use across our zones. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Share Glance between cells or regions
You could have the same URL resolve in DNS to the local swift in each region or you could use anycast. Original message From: Sam Morrison sorri...@gmail.com Date: 05/19/2013 8:39 PM (GMT-05:00) To: Jay Pipes jaypi...@gmail.com Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Share Glance between cells or regions Hi Jay, We are doing something similar. We have a single glance registry which is backed by galera DB replication. Then we have multiple glance-apis around the place. Currently they are all backed onto the same swift but I'd like to have it so each glance-api can talk to it's own swift. The issue I see is that the location of the image as stored in the glance-registry is a keystone url. So yes you could get a glance api to store data in a specific swift region (using the swift_store_region) but it has no way of knowing which region to pull an image out of. I think the location value stored when using swift needs to be the swift URL or else it needs to store the region in the DB too. Have you thought about this? Have a solution? Cheers, Sam On 16/05/2013, at 6:49 AM, Jay Pipes jaypi...@gmail.com wrote: On 05/15/2013 02:46 PM, John Paul Walters wrote: Hi, We're looking at setting up a geographically distributed OpenStack installation, and we're considering either cells or regions. We'd like to share a single Glance install between our regions (or cells), so the same images can be spawned anywhere. From here: http://docs.openstack.org/trunk/openstack-ops/content/segregate_cloud.html it's not clear whether that's possible. Can anyone shed some light on this? Is it possible in regions OR cells (or both)? Is there a better solution that I'm not thinking of? We will be sharing both the Keystone identity (note: not token/catalog) and Glance registry databases in a synchronously-replicated Galera MySQL cluster. Databases like the above, which have extremely low write to read ratios are ideal for this kind of replication. We are replicating working sets over the WAN using rsync replication in the WSREP clustering software. What this enables us to do is have a single set of account records and a single set of image (base and snapshot) records. Note that we back Glance in each zone with a zone-local Swift cluster. But what this allows us to do is have a user in zone A make a snapshot and then immediately (once the snapshot goes from the SAVING state to ACTIVE), the user is able to launch their snapshot in zone B. The Glance registry database has the location of the snapshot in zone A's Swift cluster and when Nova in zone B launches the image, the Glance API server in zone B simply pulls the image bits from Swift in zone A. Best, -jay p.s. I say will be sharing because we are currently updating our deployment to use this single Glance registry database. Originally we went down the route of each zone having its own Glance registry database and realized that since the pattern of write activity to the Glance registry is so low, it made sense to replicate it across our zones and give the users the ability to launch instances from snapshots in any zone. The single identity database is already in use across our zones. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Share Glance between cells or regions
The problem I'm talking about is: Glance-API A get's an image create request for image X It stores image X in swift A It stores the location in glance-registry as swift+http://tenant%3Ausername:password@keystone:5000/v2.0/images/X Glance-API B gets a request to get that image Glance-API is backed by Swift B The location url from glance registry doesn't allow it to find out that the image is in Swift A (it will look in swift B to find it and get a 404) Does that make sense? Sam On 20/05/2013, at 1:24 PM, Drew Weaver drew.wea...@thenap.com wrote: You could have the same URL resolve in DNS to the local swift in each region or you could use anycast. Original message From: Sam Morrison sorri...@gmail.com Date: 05/19/2013 8:39 PM (GMT-05:00) To: Jay Pipes jaypi...@gmail.com Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Share Glance between cells or regions Hi Jay, We are doing something similar. We have a single glance registry which is backed by galera DB replication. Then we have multiple glance-apis around the place. Currently they are all backed onto the same swift but I'd like to have it so each glance-api can talk to it's own swift. The issue I see is that the location of the image as stored in the glance-registry is a keystone url. So yes you could get a glance api to store data in a specific swift region (using the swift_store_region) but it has no way of knowing which region to pull an image out of. I think the location value stored when using swift needs to be the swift URL or else it needs to store the region in the DB too. Have you thought about this? Have a solution? Cheers, Sam On 16/05/2013, at 6:49 AM, Jay Pipes jaypi...@gmail.com wrote: On 05/15/2013 02:46 PM, John Paul Walters wrote: Hi, We're looking at setting up a geographically distributed OpenStack installation, and we're considering either cells or regions. We'd like to share a single Glance install between our regions (or cells), so the same images can be spawned anywhere. From here: http://docs.openstack.org/trunk/openstack-ops/content/segregate_cloud.html it's not clear whether that's possible. Can anyone shed some light on this? Is it possible in regions OR cells (or both)? Is there a better solution that I'm not thinking of? We will be sharing both the Keystone identity (note: not token/catalog) and Glance registry databases in a synchronously-replicated Galera MySQL cluster. Databases like the above, which have extremely low write to read ratios are ideal for this kind of replication. We are replicating working sets over the WAN using rsync replication in the WSREP clustering software. What this enables us to do is have a single set of account records and a single set of image (base and snapshot) records. Note that we back Glance in each zone with a zone-local Swift cluster. But what this allows us to do is have a user in zone A make a snapshot and then immediately (once the snapshot goes from the SAVING state to ACTIVE), the user is able to launch their snapshot in zone B. The Glance registry database has the location of the snapshot in zone A's Swift cluster and when Nova in zone B launches the image, the Glance API server in zone B simply pulls the image bits from Swift in zone A. Best, -jay p.s. I say will be sharing because we are currently updating our deployment to use this single Glance registry database. Originally we went down the route of each zone having its own Glance registry database and realized that since the pattern of write activity to the Glance registry is so low, it made sense to replicate it across our zones and give the users the ability to launch instances from snapshots in any zone. The single identity database is already in use across our zones. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Share Glance between cells or regions
Hi, We're looking at setting up a geographically distributed OpenStack installation, and we're considering either cells or regions. We'd like to share a single Glance install between our regions (or cells), so the same images can be spawned anywhere. From here: http://docs.openstack.org/trunk/openstack-ops/content/segregate_cloud.html it's not clear whether that's possible. Can anyone shed some light on this? Is it possible in regions OR cells (or both)? Is there a better solution that I'm not thinking of? thanks, JP ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Share Glance between cells or regions
+1. We're going to be running a bunch of parallel deployments of openstack for the purpose of experimentation in system design. it would be nice to be able to share glance and keystone between instances. -nld On Wed, May 15, 2013 at 1:46 PM, John Paul Walters jwalt...@isi.edu wrote: Hi, We're looking at setting up a geographically distributed OpenStack installation, and we're considering either cells or regions. We'd like to share a single Glance install between our regions (or cells), so the same images can be spawned anywhere. From here: http://docs.openstack.org/trunk/openstack-ops/content/segregate_cloud.html it's not clear whether that's possible. Can anyone shed some light on this? Is it possible in regions OR cells (or both)? Is there a better solution that I'm not thinking of? thanks, JP ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Share Glance between cells or regions
+2 I feel there is a difference between - Is it possible ? - Is it a good approach for use case X ? Multi-site varies hugely depending on whether the aim is sharing the work or recovery scenarios. Defining a good model for multi-site, resilient deployments and multi-site, maximum throughput/utilisation would be very interesting. Tim From: Openstack [mailto:openstack-bounces+tim.bell=cern...@lists.launchpad.net] On Behalf Of Narayan Desai Sent: 15 May 2013 21:14 To: John Paul Walters Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Share Glance between cells or regions +1. We're going to be running a bunch of parallel deployments of openstack for the purpose of experimentation in system design. it would be nice to be able to share glance and keystone between instances. -nld On Wed, May 15, 2013 at 1:46 PM, John Paul Walters jwalt...@isi.edu mailto:jwalt...@isi.edu wrote: Hi, We're looking at setting up a geographically distributed OpenStack installation, and we're considering either cells or regions. We'd like to share a single Glance install between our regions (or cells), so the same images can be spawned anywhere. From here: http://docs.openstack.org/trunk/openstack-ops/content/segregate_cloud.html it's not clear whether that's possible. Can anyone shed some light on this? Is it possible in regions OR cells (or both)? Is there a better solution that I'm not thinking of? thanks, JP ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp smime.p7s Description: S/MIME cryptographic signature ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Share Glance between cells or regions
All, We've been working on a Virtual Organization (VO) capability to allow federated OpenStacks. So far, we've concentrated on just federated data access, having modified Keystone and Swift on both the server and client sides. The are a number of further distributed/federated capabilities that one can imagine, e.g., authorizing one Keystone to instantiate VM through a remote Nova. Obviously, any type of resource federation requires security federation. In the general case, this requires federated identity management, as David Chadwick at Kent has been working on. This type of federation is somewhat different than just sharing a Keystone or Glance among multiple instances, but is arguably more general. I would really like the OS community to achieve some rough consensus about what kinds of federation models are desirable and how to build them. --Craig On 5/15/13 12:21 PM, Tim Bell wrote: +2 I feel there is a difference between -Is it possible ? -Is it a good approach for use case X ? Multi-site varies hugely depending on whether the aim is sharing the work or recovery scenarios. Defining a good model for multi-site, resilient deployments and multi-site, maximum throughput/utilisation would be very interesting. Tim *From:*Openstack [mailto:openstack-bounces+tim.bell=cern...@lists.launchpad.net] *On Behalf Of *Narayan Desai *Sent:* 15 May 2013 21:14 *To:* John Paul Walters *Cc:* openstack@lists.launchpad.net *Subject:* Re: [Openstack] Share Glance between cells or regions +1. We're going to be running a bunch of parallel deployments of openstack for the purpose of experimentation in system design. it would be nice to be able to share glance and keystone between instances. -nld On Wed, May 15, 2013 at 1:46 PM, John Paul Walters jwalt...@isi.edu mailto:jwalt...@isi.edu wrote: Hi, We're looking at setting up a geographically distributed OpenStack installation, and we're considering either cells or regions. We'd like to share a single Glance install between our regions (or cells), so the same images can be spawned anywhere. From here: http://docs.openstack.org/trunk/openstack-ops/content/segregate_cloud.html it's not clear whether that's possible. Can anyone shed some light on this? Is it possible in regions OR cells (or both)? Is there a better solution that I'm not thinking of? thanks, JP ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Share Glance between cells or regions
On 05/15/2013 08:46 AM, John Paul Walters wrote: Hi, We're looking at setting up a geographically distributed OpenStack installation, and we're considering either cells or regions. We'd like to share a single Glance install between our regions (or cells), so the same images can be spawned anywhere. From here: In his keynote speech at the Portland Summit Randy Sobie mentioned a similar wish. The relevant part of the speech goes for about 30 seconds starting here: http://www.youtube.com/watch?feature=player_detailpagev=Oyu8uwSRlms#t=1120s ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
