Re: [openstack-dev] [nova][cinder] Disabling nova volume-update (aka swap volume; aka cinder live migration)
On 24/08, Matt Riedemann wrote: > On 8/22/2018 4:46 AM, Gorka Eguileor wrote: > > The solution is conceptually simple. We add a new API microversion in > > Cinder that adds and optional parameter called "generic_keep_source" > > (defaults to False) to both migrate and retype operations. > > But if the problem is that users are not using the retype API and instead > are hitting the compute swap volume API instead, they won't use this new > parameter anyway. Again, retype is admin-or-owner but volume migration (in > cinder) and swap volume (in nova) are both admin-only, so are admins calling > swap volume directly or are people easing up the policy restrictions so > non-admins can use these migration APIs? > > -- > > Thanks, > > Matt Hi, These are two different topics, and I thought we had already closed that part of the discussion. Nova needs to fix that issue, a good option could be not allowing non Cinder callers to do that operation. As mbooth mentioned, these issues come from real user needs, and this is a different topic, and the topic I was talking about with the proposed solution. I agree with mbooth that we should find a way to address these customer needs, even if it's in a limited way. Cheers, Gorka. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder] Disabling nova volume-update (aka swap volume; aka cinder live migration)
On 25/08, Sean McGinnis wrote: > On Fri, Aug 24, 2018 at 04:20:21PM -0500, Matt Riedemann wrote: > > On 8/20/2018 10:29 AM, Matthew Booth wrote: > > > Secondly, is there any reason why we shouldn't just document then you > > > have to delete snapshots before doing a volume migration? Hopefully > > > some cinder folks or operators can chime in to let me know how to back > > > them up or somehow make them independent before doing this, at which > > > point the volume itself should be migratable? > > > > Coincidentally the volume migration API never had API reference > > documentation. I have that here now [1]. It clearly states the preconditions > > to migrate a volume based on code in the volume API. However, volume > > migration is admin-only by default and retype (essentially like resize) is > > admin-or-owner so non-admins can do it and specify to migrate. In general I > > think it's best to have preconditions for *any* API documented, so anything > > needed to perform a retype should be documented in the API, like that the > > volume can't have snapshots. > > That's where things get tricky though. There aren't really reconditions we can > have as a blanket statement with the retype API. > > A retype can do a lot of different things, all dependent on what type you are > coming from and trying to go to. There are some retypes where all it does is > enable vendor flag ``foo`` on the volume with no change in any other state. > Then there are other retypes (using --migrate-policy on-demand) that > completely > move the volume from one backend to another one, copying every block along the > way from the original to the new volume. It really depends on what types you > are trying to retype to. > We can say that retypes that require migration between different vendor backends cannot be performed with snapshots, and between arrays from the same vendor will depend on the driver (though I don't know if any driver can actually pull this off). Cheers, Gorka. > > > > [1] https://review.openstack.org/#/c/595379/ > > > > -- > > > > Thanks, > > > > Matt > > > > __ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder] Disabling nova volume-update (aka swap volume; aka cinder live migration)
On Fri, Aug 24, 2018 at 04:20:21PM -0500, Matt Riedemann wrote: > On 8/20/2018 10:29 AM, Matthew Booth wrote: > > Secondly, is there any reason why we shouldn't just document then you > > have to delete snapshots before doing a volume migration? Hopefully > > some cinder folks or operators can chime in to let me know how to back > > them up or somehow make them independent before doing this, at which > > point the volume itself should be migratable? > > Coincidentally the volume migration API never had API reference > documentation. I have that here now [1]. It clearly states the preconditions > to migrate a volume based on code in the volume API. However, volume > migration is admin-only by default and retype (essentially like resize) is > admin-or-owner so non-admins can do it and specify to migrate. In general I > think it's best to have preconditions for *any* API documented, so anything > needed to perform a retype should be documented in the API, like that the > volume can't have snapshots. That's where things get tricky though. There aren't really reconditions we can have as a blanket statement with the retype API. A retype can do a lot of different things, all dependent on what type you are coming from and trying to go to. There are some retypes where all it does is enable vendor flag ``foo`` on the volume with no change in any other state. Then there are other retypes (using --migrate-policy on-demand) that completely move the volume from one backend to another one, copying every block along the way from the original to the new volume. It really depends on what types you are trying to retype to. > > [1] https://review.openstack.org/#/c/595379/ > > -- > > Thanks, > > Matt > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder] Disabling nova volume-update (aka swap volume; aka cinder live migration)
On 8/22/2018 4:46 AM, Gorka Eguileor wrote: The solution is conceptually simple. We add a new API microversion in Cinder that adds and optional parameter called "generic_keep_source" (defaults to False) to both migrate and retype operations. But if the problem is that users are not using the retype API and instead are hitting the compute swap volume API instead, they won't use this new parameter anyway. Again, retype is admin-or-owner but volume migration (in cinder) and swap volume (in nova) are both admin-only, so are admins calling swap volume directly or are people easing up the policy restrictions so non-admins can use these migration APIs? -- Thanks, Matt __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder] Disabling nova volume-update (aka swap volume; aka cinder live migration)
On 8/20/2018 10:29 AM, Matthew Booth wrote: Secondly, is there any reason why we shouldn't just document then you have to delete snapshots before doing a volume migration? Hopefully some cinder folks or operators can chime in to let me know how to back them up or somehow make them independent before doing this, at which point the volume itself should be migratable? Coincidentally the volume migration API never had API reference documentation. I have that here now [1]. It clearly states the preconditions to migrate a volume based on code in the volume API. However, volume migration is admin-only by default and retype (essentially like resize) is admin-or-owner so non-admins can do it and specify to migrate. In general I think it's best to have preconditions for *any* API documented, so anything needed to perform a retype should be documented in the API, like that the volume can't have snapshots. [1] https://review.openstack.org/#/c/595379/ -- Thanks, Matt __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder] Disabling nova volume-update (aka swap volume; aka cinder live migration)
On 22/08, Matthew Booth wrote: > On Wed, 22 Aug 2018 at 10:47, Gorka Eguileor wrote: > > > > On 20/08, Matthew Booth wrote: > > > For those who aren't familiar with it, nova's volume-update (also > > > called swap volume by nova devs) is the nova part of the > > > implementation of cinder's live migration (also called retype). > > > Volume-update is essentially an internal cinder<->nova api, but as > > > that's not a thing it's also unfortunately exposed to users. Some > > > users have found it and are using it, but because it's essentially an > > > internal cinder<->nova api it breaks pretty easily if you don't treat > > > it like a special snowflake. It looks like we've finally found a way > > > it's broken for non-cinder callers that we can't fix, even with a > > > dirty hack. > > > > > > volume-updateessentially does a live copy of the > > > data on volume to volume, then seamlessly swaps the > > > attachment to from to . The guest OS on > > > will not notice anything at all as the hypervisor swaps the storage > > > backing an attached volume underneath it. > > > > > > When called by cinder, as intended, cinder does some post-operation > > > cleanup such that is deleted and inherits the same > > > volume_id; that is effectively becomes . When called any > > > other way, however, this cleanup doesn't happen, which breaks a bunch > > > of assumptions. One of these is that a disk's serial number is the > > > same as the attached volume_id. Disk serial number, in KVM at least, > > > is immutable, so can't be updated during volume-update. This is fine > > > if we were called via cinder, because the cinder cleanup means the > > > volume_id stays the same. If called any other way, however, they no > > > longer match, at least until a hard reboot when it will be reset to > > > the new volume_id. It turns out this breaks live migration, but > > > probably other things too. We can't think of a workaround. > > > > > > I wondered why users would want to do this anyway. It turns out that > > > sometimes cinder won't let you migrate a volume, but nova > > > volume-update doesn't do those checks (as they're specific to cinder > > > internals, none of nova's business, and duplicating them would be > > > fragile, so we're not adding them!). Specifically we know that cinder > > > won't let you migrate a volume with snapshots. There may be other > > > reasons. If cinder won't let you migrate your volume, you can still > > > move your data by using nova's volume-update, even though you'll end > > > up with a new volume on the destination, and a slightly broken > > > instance. Apparently the former is a trade-off worth making, but the > > > latter has been reported as a bug. > > > > > > > Hi Matt, > > > > As you know, I'm in favor of making this REST API call only authorized > > for Cinder to avoid messing the cloud. > > > > I know you wanted Cinder to have a solution to do live migrations of > > volumes with snapshots, and while this is not possible to do in a > > reasonable fashion, I kept thinking about it given your strong feelings > > to provide a solution for users that really need this, and I think we > > may have a "reasonable" compromise. > > > > The solution is conceptually simple. We add a new API microversion in > > Cinder that adds and optional parameter called "generic_keep_source" > > (defaults to False) to both migrate and retype operations. > > > > This means that if the driver optimized migration cannot do the > > migration and the generic migration code is the one doing the migration, > > then, instead of our final step being to swap the volume id's and > > deleting the source volume, what we would do is to swap the volume id's > > and move all the snapshots to reference the new volume. Then we would > > create a user message with the new ID of the volume. > > > > This way we can preserve the old volume with all its snapshots and do > > the live migration. > > > > The implementation is a little bit tricky, as we'll have to add anew > > "update_migrated_volume" mechanism to support the renaming of both > > volumes, since the old one wouldn't work with this among other things, > > but it's doable. > > > > Unfortunately I don't have the time right now to work on this... > > Sounds promising, and honestly more than I'd have hoped for. > > Matt > Hi Matt, Reading Sean's reply I notice that I phrased that wrong. The volume on the new storage backend wouldn't have any snapshots. The result of the operation would be a new volume with the old ID and no snapshots (this would be the one in use by Nova), and the old volume with all the snapshots having a new ID on the DB. Due to Cinder's mechanism to create this new volume we wouldn't be returning it on the REST API call, but as a user message instead. Sorry for the confusion. Cheers, Gorka. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe:
Re: [openstack-dev] [nova][cinder] Disabling nova volume-update (aka swap volume; aka cinder live migration)
On 22/08, Sean McGinnis wrote: > > > > The solution is conceptually simple. We add a new API microversion in > > Cinder that adds and optional parameter called "generic_keep_source" > > (defaults to False) to both migrate and retype operations. > > > > This means that if the driver optimized migration cannot do the > > migration and the generic migration code is the one doing the migration, > > then, instead of our final step being to swap the volume id's and > > deleting the source volume, what we would do is to swap the volume id's > > and move all the snapshots to reference the new volume. Then we would > > create a user message with the new ID of the volume. > > > > How would you propose to "move all the snapshots to reference the new volume"? > Most storage does not allow a snapshot to be moved from one volume to another. > really the only way a migration of a snapshot can work across all storage > types > would be to incrementally copy the data from a source to a destination up to > the point of the oldest snapshot, create a new snapshot on the new volume, > then > proceed through until all snapshots have been rebuilt on the new volume. > Hi Sean, Sorry, I phrased that wrong. When I say move the snapshots to the new volume I mean to the "New Volume DB entry", which is now pointing to the old volume. So we wouldn't really be moving the snapshots, we would just be leaving the old volume with its snapshots under a new UUID, and the old UUID that the user had attached to Nova will be referencing the new volume. Again, sorry for the confusion. Cheers, Gorka. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder] Disabling nova volume-update (aka swap volume; aka cinder live migration)
> > The solution is conceptually simple. We add a new API microversion in > Cinder that adds and optional parameter called "generic_keep_source" > (defaults to False) to both migrate and retype operations. > > This means that if the driver optimized migration cannot do the > migration and the generic migration code is the one doing the migration, > then, instead of our final step being to swap the volume id's and > deleting the source volume, what we would do is to swap the volume id's > and move all the snapshots to reference the new volume. Then we would > create a user message with the new ID of the volume. > How would you propose to "move all the snapshots to reference the new volume"? Most storage does not allow a snapshot to be moved from one volume to another. really the only way a migration of a snapshot can work across all storage types would be to incrementally copy the data from a source to a destination up to the point of the oldest snapshot, create a new snapshot on the new volume, then proceed through until all snapshots have been rebuilt on the new volume. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder] Disabling nova volume-update (aka swap volume; aka cinder live migration)
On Wed, 22 Aug 2018 at 10:47, Gorka Eguileor wrote: > > On 20/08, Matthew Booth wrote: > > For those who aren't familiar with it, nova's volume-update (also > > called swap volume by nova devs) is the nova part of the > > implementation of cinder's live migration (also called retype). > > Volume-update is essentially an internal cinder<->nova api, but as > > that's not a thing it's also unfortunately exposed to users. Some > > users have found it and are using it, but because it's essentially an > > internal cinder<->nova api it breaks pretty easily if you don't treat > > it like a special snowflake. It looks like we've finally found a way > > it's broken for non-cinder callers that we can't fix, even with a > > dirty hack. > > > > volume-updateessentially does a live copy of the > > data on volume to volume, then seamlessly swaps the > > attachment to from to . The guest OS on > > will not notice anything at all as the hypervisor swaps the storage > > backing an attached volume underneath it. > > > > When called by cinder, as intended, cinder does some post-operation > > cleanup such that is deleted and inherits the same > > volume_id; that is effectively becomes . When called any > > other way, however, this cleanup doesn't happen, which breaks a bunch > > of assumptions. One of these is that a disk's serial number is the > > same as the attached volume_id. Disk serial number, in KVM at least, > > is immutable, so can't be updated during volume-update. This is fine > > if we were called via cinder, because the cinder cleanup means the > > volume_id stays the same. If called any other way, however, they no > > longer match, at least until a hard reboot when it will be reset to > > the new volume_id. It turns out this breaks live migration, but > > probably other things too. We can't think of a workaround. > > > > I wondered why users would want to do this anyway. It turns out that > > sometimes cinder won't let you migrate a volume, but nova > > volume-update doesn't do those checks (as they're specific to cinder > > internals, none of nova's business, and duplicating them would be > > fragile, so we're not adding them!). Specifically we know that cinder > > won't let you migrate a volume with snapshots. There may be other > > reasons. If cinder won't let you migrate your volume, you can still > > move your data by using nova's volume-update, even though you'll end > > up with a new volume on the destination, and a slightly broken > > instance. Apparently the former is a trade-off worth making, but the > > latter has been reported as a bug. > > > > Hi Matt, > > As you know, I'm in favor of making this REST API call only authorized > for Cinder to avoid messing the cloud. > > I know you wanted Cinder to have a solution to do live migrations of > volumes with snapshots, and while this is not possible to do in a > reasonable fashion, I kept thinking about it given your strong feelings > to provide a solution for users that really need this, and I think we > may have a "reasonable" compromise. > > The solution is conceptually simple. We add a new API microversion in > Cinder that adds and optional parameter called "generic_keep_source" > (defaults to False) to both migrate and retype operations. > > This means that if the driver optimized migration cannot do the > migration and the generic migration code is the one doing the migration, > then, instead of our final step being to swap the volume id's and > deleting the source volume, what we would do is to swap the volume id's > and move all the snapshots to reference the new volume. Then we would > create a user message with the new ID of the volume. > > This way we can preserve the old volume with all its snapshots and do > the live migration. > > The implementation is a little bit tricky, as we'll have to add anew > "update_migrated_volume" mechanism to support the renaming of both > volumes, since the old one wouldn't work with this among other things, > but it's doable. > > Unfortunately I don't have the time right now to work on this... Sounds promising, and honestly more than I'd have hoped for. Matt -- Matthew Booth Red Hat OpenStack Engineer, Compute DFG Phone: +442070094448 (UK) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder] Disabling nova volume-update (aka swap volume; aka cinder live migration)
On 20/08, Matthew Booth wrote: > For those who aren't familiar with it, nova's volume-update (also > called swap volume by nova devs) is the nova part of the > implementation of cinder's live migration (also called retype). > Volume-update is essentially an internal cinder<->nova api, but as > that's not a thing it's also unfortunately exposed to users. Some > users have found it and are using it, but because it's essentially an > internal cinder<->nova api it breaks pretty easily if you don't treat > it like a special snowflake. It looks like we've finally found a way > it's broken for non-cinder callers that we can't fix, even with a > dirty hack. > > volume-updateessentially does a live copy of the > data on volume to volume, then seamlessly swaps the > attachment to from to . The guest OS on > will not notice anything at all as the hypervisor swaps the storage > backing an attached volume underneath it. > > When called by cinder, as intended, cinder does some post-operation > cleanup such that is deleted and inherits the same > volume_id; that is effectively becomes . When called any > other way, however, this cleanup doesn't happen, which breaks a bunch > of assumptions. One of these is that a disk's serial number is the > same as the attached volume_id. Disk serial number, in KVM at least, > is immutable, so can't be updated during volume-update. This is fine > if we were called via cinder, because the cinder cleanup means the > volume_id stays the same. If called any other way, however, they no > longer match, at least until a hard reboot when it will be reset to > the new volume_id. It turns out this breaks live migration, but > probably other things too. We can't think of a workaround. > > I wondered why users would want to do this anyway. It turns out that > sometimes cinder won't let you migrate a volume, but nova > volume-update doesn't do those checks (as they're specific to cinder > internals, none of nova's business, and duplicating them would be > fragile, so we're not adding them!). Specifically we know that cinder > won't let you migrate a volume with snapshots. There may be other > reasons. If cinder won't let you migrate your volume, you can still > move your data by using nova's volume-update, even though you'll end > up with a new volume on the destination, and a slightly broken > instance. Apparently the former is a trade-off worth making, but the > latter has been reported as a bug. > Hi Matt, As you know, I'm in favor of making this REST API call only authorized for Cinder to avoid messing the cloud. I know you wanted Cinder to have a solution to do live migrations of volumes with snapshots, and while this is not possible to do in a reasonable fashion, I kept thinking about it given your strong feelings to provide a solution for users that really need this, and I think we may have a "reasonable" compromise. The solution is conceptually simple. We add a new API microversion in Cinder that adds and optional parameter called "generic_keep_source" (defaults to False) to both migrate and retype operations. This means that if the driver optimized migration cannot do the migration and the generic migration code is the one doing the migration, then, instead of our final step being to swap the volume id's and deleting the source volume, what we would do is to swap the volume id's and move all the snapshots to reference the new volume. Then we would create a user message with the new ID of the volume. This way we can preserve the old volume with all its snapshots and do the live migration. The implementation is a little bit tricky, as we'll have to add anew "update_migrated_volume" mechanism to support the renaming of both volumes, since the old one wouldn't work with this among other things, but it's doable. Unfortunately I don't have the time right now to work on this... Cheers, Gorka. > I'd like to make it very clear that nova's volume-update, isn't > expected to work correctly except when called by cinder. Specifically > there was a proposal that we disable volume-update from non-cinder > callers in some way, possibly by asserting volume state that can only > be set by cinder. However, I'm also very aware that users are calling > volume-update because it fills a need, and we don't want to trap data > that wasn't previously trapped. > > Firstly, is anybody aware of any other reasons to use nova's > volume-update directly? > > Secondly, is there any reason why we shouldn't just document then you > have to delete snapshots before doing a volume migration? Hopefully > some cinder folks or operators can chime in to let me know how to back > them up or somehow make them independent before doing this, at which > point the volume itself should be migratable? > > If we can establish that there's an acceptable alternative to calling > volume-update directly for all use-cases we're aware of, I'm going to > propose heading off this class of bug by disabling it for
[openstack-dev] [nova][cinder] Disabling nova volume-update (aka swap volume; aka cinder live migration)
For those who aren't familiar with it, nova's volume-update (also called swap volume by nova devs) is the nova part of the implementation of cinder's live migration (also called retype). Volume-update is essentially an internal cinder<->nova api, but as that's not a thing it's also unfortunately exposed to users. Some users have found it and are using it, but because it's essentially an internal cinder<->nova api it breaks pretty easily if you don't treat it like a special snowflake. It looks like we've finally found a way it's broken for non-cinder callers that we can't fix, even with a dirty hack. volume-updateessentially does a live copy of the data on volume to volume, then seamlessly swaps the attachment to from to . The guest OS on will not notice anything at all as the hypervisor swaps the storage backing an attached volume underneath it. When called by cinder, as intended, cinder does some post-operation cleanup such that is deleted and inherits the same volume_id; that is effectively becomes . When called any other way, however, this cleanup doesn't happen, which breaks a bunch of assumptions. One of these is that a disk's serial number is the same as the attached volume_id. Disk serial number, in KVM at least, is immutable, so can't be updated during volume-update. This is fine if we were called via cinder, because the cinder cleanup means the volume_id stays the same. If called any other way, however, they no longer match, at least until a hard reboot when it will be reset to the new volume_id. It turns out this breaks live migration, but probably other things too. We can't think of a workaround. I wondered why users would want to do this anyway. It turns out that sometimes cinder won't let you migrate a volume, but nova volume-update doesn't do those checks (as they're specific to cinder internals, none of nova's business, and duplicating them would be fragile, so we're not adding them!). Specifically we know that cinder won't let you migrate a volume with snapshots. There may be other reasons. If cinder won't let you migrate your volume, you can still move your data by using nova's volume-update, even though you'll end up with a new volume on the destination, and a slightly broken instance. Apparently the former is a trade-off worth making, but the latter has been reported as a bug. I'd like to make it very clear that nova's volume-update, isn't expected to work correctly except when called by cinder. Specifically there was a proposal that we disable volume-update from non-cinder callers in some way, possibly by asserting volume state that can only be set by cinder. However, I'm also very aware that users are calling volume-update because it fills a need, and we don't want to trap data that wasn't previously trapped. Firstly, is anybody aware of any other reasons to use nova's volume-update directly? Secondly, is there any reason why we shouldn't just document then you have to delete snapshots before doing a volume migration? Hopefully some cinder folks or operators can chime in to let me know how to back them up or somehow make them independent before doing this, at which point the volume itself should be migratable? If we can establish that there's an acceptable alternative to calling volume-update directly for all use-cases we're aware of, I'm going to propose heading off this class of bug by disabling it for non-cinder callers. Matt -- Matthew Booth Red Hat OpenStack Engineer, Compute DFG Phone: +442070094448 (UK) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
