On 3/1/2016 11:36 PM, John Griffith wrote:
On Tue, Mar 1, 2016 at 3:48 PM, Murray, Paul (HP Cloud) <pmur...@hpe.com <mailto:pmur...@hpe.com>> wrote: > -----Original Message----- > From: D'Angelo, Scott > > Matt, changing Nova to store the connector info at volume attach time does > help. Where the gap will remain is after Nova evacuation or live migration, This will happen with shelve as well I think. Volumes are not detached in shelve IIRC. > when that info will need to be updated in Cinder. We need to change the > Cinder API to have some mechanism to allow this. > We'd also like Cinder to store the appropriate info to allow a force-detach for > the cases where Nova cannot make the call to Cinder. > Ongoing work for this and related issues is tracked and discussed here: > https://etherpad.openstack.org/p/cinder-nova-api-changes > > Scott D'Angelo (scottda) > ________________________________________ > From: Matt Riedemann [mrie...@linux.vnet.ibm.com <mailto:mrie...@linux.vnet.ibm.com>] > Sent: Monday, February 29, 2016 7:48 AM > To: openstack-dev@lists.openstack.org <mailto:openstack-dev@lists.openstack.org> > Subject: Re: [openstack-dev] [nova][cinder] volumes stuck detaching > attaching and force detach > > On 2/22/2016 4:08 PM, Walter A. Boring IV wrote: > > On 02/22/2016 11:24 AM, John Garbutt wrote: > >> Hi, > >> > >> Just came up on IRC, when nova-compute gets killed half way through a > >> volume attach (i.e. no graceful shutdown), things get stuck in a bad > >> state, like volumes stuck in the attaching state. > >> > >> This looks like a new addition to this conversation: > >> http://lists.openstack.org/pipermail/openstack-dev/2015- > December/0826 > >> 83.html > >> > >> And brings us back to this discussion: > >> https://blueprints.launchpad.net/nova/+spec/add-force-detach-to-nova > >> > >> What if we move our attention towards automatically recovering from > >> the above issue? I am wondering if we can look at making our usually > >> recovery code deal with the above situation: > >> > https://github.com/openstack/nova/blob/834b5a9e3a4f8c6ee2e3387845fc24 > >> c79f4bf615/nova/compute/manager.py#L934 > >> > >> > >> Did we get the Cinder APIs in place that enable the force-detach? I > >> think we did and it was this one? > >> https://blueprints.launchpad.net/python-cinderclient/+spec/nova-force > >> -detach-needs-cinderclient-api > >> > >> > >> I think diablo_rojo might be able to help dig for any bugs we have > >> related to this. I just wanted to get this idea out there before I > >> head out. > >> > >> Thanks, > >> John > >> > >> > __________________________________________________________ > ___________ > >> _____ > >> > >> OpenStack Development Mailing List (not for usage questions) > >> Unsubscribe: > >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> . > >> > > The problem is a little more complicated. > > > > In order for cinder backends to be able to do a force detach > > correctly, the Cinder driver needs to have the correct 'connector' > > dictionary passed in to terminate_connection. That connector > > dictionary is the collection of initiator side information which is gleaned > here: > > https://github.com/openstack/os-brick/blob/master/os_brick/initiator/c > > onnector.py#L99-L144 > > > > > > The plan was to save that connector information in the Cinder > > volume_attachment table. When a force detach is called, Cinder has > > the existing connector saved if Nova doesn't have it. The problem was > > live migration. When you migrate to the destination n-cpu host, the > > connector that Cinder had is now out of date. There is no API in > > Cinder today to allow updating an existing attachment. > > > > So, the plan at the Mitaka summit was to add this new API, but it > > required microversions to land, which we still don't have in Cinder's > > API today. > > > > > > Walt > > > > > __________________________________________________________ > ____________ > > ____ OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: > >openstack-dev-requ...@lists.openstack.org?subject:unsubscribe <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> > >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > Regarding storing off the initial connector information from the attach, does > this [1] help bridge the gap? That adds the connector dict to the > connection_info dict that is serialized and stored in the nova > block_device_mappings table, and then in that patch is used to pass it to > terminate_connection in the case that the host has changed. > > [1]https://review.openstack.org/#/c/266095/ > > -- > > Thanks, > > Matt Riedemann > > > __________________________________________________________ > ________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: OpenStack-dev- >requ...@lists.openstack.org?subject:unsubscribe <http://requ...@lists.openstack.org?subject:unsubscribe> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __________________________________________________________ > ________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: OpenStack-dev- > requ...@lists.openstack.org?subject:unsubscribe <http://requ...@lists.openstack.org?subject:unsubscribe> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev I think Matt's patch is closer to where we need to be than some of the other paths that folks are on actually. There's a lot of complexity that's being proposed in my opinion that might not be needed. I could be wrong, but I've been looking at what got things sort of messy to begin with and it mostly revolves around diverging on what cinders initialize_connection and attach calls do. I worked on a couple of things that hopefully might help: [1] is a simple add to the devref in cinder just to help new driver writers get a better idea of what the expectations are here. I realize that there are a couple of drivers in Cinder that have already diverged here, but it's at least a guideline. I also realize that the workflow may be a bit different for devices like Ceph. The other thing that I started (haven't spent the time fixing up the horrid unit tests because I wanted feedback on the proposal first) is this patch here: [2]. The idea for the first step is to clean up the attach code that we have now and actually use some of the tables that Cinder provides. Particularly we have a volume_attachments table that could persist all of the connector info rather easily and more importantly return an attachment-id to the caller. We're already passing this info in, but sadly we're just not *doing* much with it. The idea here is that the caller (Nova or otherwise) doesn't need to mess with all this tracking of references, connectors etc. Instead, it just uses the returned attachment-id and that's it. Note that we ALREADY have all of the connector info etc during the initialize and then the additional info on the attach calls. We just weren't *doing* anything with it. Nova is done with a volume and wants to detach it... just call "terminate_connection" with the volume-id and the attachment-id it received during initialize_connection. Cinder can/should figure out what to do if it's a multi-attached volume on the same node or whatever. This makes detach a much simpler endeavor I think, because now you just provide the attachment-id, Cinder looks it up in it's DB and it has all of the info from the connector etc. I realize this would be a change in the API in the future, but adding the attachment ID is backward compatible and won't break any existing callers. So what about multi-attach? I mentioned that earlier and that would be the next piece of follow on work on the Cinder side if this idea is viewed as worth pursuing, but would require we agree to go this route on the Nova side as well. Currently there are some challenges with the current Cinder implementation of multi-attach but I think they're pretty easy to change around, and then we could get back to just using a unique attachment-id for every attach call and handle the business of sorting and tracking those things on the Cinder side. Again, trying to keep things simple and non-obtrusive for the consumer while maintaining backward compatibility. Honestly, I don't think that any consumer of Cinder (Nova) should have to know anything about all of these details. It should just simply be "Hey... Cinder; I'm attaching this volume *here*". This proposal covers the *special* drivers that can't reuse a Tgt IQN, or that work with things like Access-Groups. More importantly it provides all of the Cinder drivers more flexibility to do things in their own driver code if they need to while persisting all of the various data that might be desired. Anyway, I would really love to keep working on this if there's interest, but if folks feel the right direction is to build in shared logic between the two sides and pass connector objects with every call, that's fair enough. I do worry about complexity in some of what's being discussed though. [1]: https://review.openstack.org/#/c/285471/ [2]: https://review.openstack.org/#/c/284867/ Thanks, John PS I know, the devref doc isn't publishing, I've fixed that up and submitted a patch to get that addressed __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
I don't know the specifics of the cinder side changes but I like the idea of cinder storing the connector information with the volume attachment so that nova doesn't have to track that in the block_device_mappings.connection_info serialized dict-o-doom.
Nova doesn't currently pass the attachment_id to the terminate_connection API, but it does use it when calling os-detach, so I don't really see a problem with nova passing the same attachment_id to the terminate_connection API. If anything, nova would persist the attachment_id in the block_device_mappings table which would make more sense to me, although we can also look it up via volume ID and instance uuid.
-- Thanks, Matt Riedemann __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
