Here’s the way we’ve approached this:
- A user can always send a delete request for a VM in any state (this is the only action that is always allowed). - Once a VM has a task_state of “Deleting” (set in the API server) the only action they can perform is delete o Hence at this point we can stop billing for it, and the user shouldn’t have it counted in their quota - A common reason for VMs getting stuck in Deleting is that the compute manager is restarted (or fails) �C so we have added code to the computer manager start-up to check for instances with a task_state of deleting and delete them (this needs to be able to cope with various exceptions if the delete was part way thought). Since the manage is restarting we can be sure that the eventlet that was handling the delete isn’t doing it anymore ;-) So from the user perspective we honour their request to delete VMs, make sure they can’t change their mind, and try to cleanup eventually as part of compute_manager restart. - By the same logic we also reset the “Image_Snapshot” and “Image_backup” task_state, as we know they aren’t true anymore. - It would be possible to also handle other task_states such as “rebuilding” and “rebooting”, but we haven’t tried that. Phil From: [email protected] [mailto:[email protected]] On Behalf Of Guilherme Birk Sent: 22 March 2012 13:24 To: Openstack Mail List Subject: Re: [Openstack] Can't delete instances with "error" status. Gabe, responding to your question "Do you know how to reliably reproduce an instance in ERROR state that cannot be deleted?": In my case, I'm updating the status of the VM to "error" directly on the database. This is just for testing. But even when my VM is running and working fine, when I update it to "error" in the database I can't delete it. > From: [email protected]<mailto:[email protected]> > To: [email protected]<mailto:[email protected]> > Date: Thu, 22 Mar 2012 06:49:15 +0000 > CC: [email protected]<mailto:[email protected]> > Subject: Re: [Openstack] Can't delete instances with "error" status. > > There are definitely lots of cases where deleting an instance in error state > works fine, and I’d like to know about the cases where it doesn’t. They do > count against quota as well, so that’s a problem! > > I can see value in keeping an instance around �C if the operations team has > configured it to do so. However, it seems like the end user has asked for it > to be deleted and to them it should appear to be deleted. Johannes had an > idea a while ago to allow the operations team to specify a project ID that > deleted servers that match certain criteria should be moved to. If the delete > finishes up fine, then its no problem, the delete is done, the customer is > happy and the ops account is empty. If it fails for some reason, there is > manual cleanup to be done, but that should be on the operator of the > deployment, not the user. I think its critical for anything like this to be > configurable, as public clouds and private clouds have different privacy and > retention concerns, I would guess. > > Do we have cases where we can reliably reproduce this issue? If its happening > every time on some deployments there is a very serious problem! > > Gabe > > > From: Craig Vyvial > [mailto:[email protected]]<mailto:[mailto:[email protected]]> > Sent: Thursday, March 22, 2012 2:20 AM > To: Ga be Westmaas > Cc: Yong Sheng Gong; Openstack Mail List > Subject: Re: [Openstack] Can't delete instances with "error" status. > > My understanding is that you would not always want a user to delete an > instance in an error state. So an operations person can figure out what went > wrong. I think the instances that are in error state do not count against the > quota but i agree that they clutter up the API calls to list servers. > > I have noticed this with my team and written code around this case to force > the instance into an 'active' state before sending nova the delete call if > the instance was in an 'error' or 'suspended' state. > > -Craig > > On Thu, Mar 22, 2012 at 1:02 AM, Gabe Westmaas > <[email protected]<mailto:[email protected]>> wrote: > Instances in deleted status can normally be deleted, but there is definitely > a bug to file here somewhere �C possibly more than one. A common reason I > have seen is that the node the instance lives on is no longer operating > correctly, so the compute manager never gets the delete request, so it > doesn’t finish. If we can narrow the cases where this happens, we can file > bugs and decide how to resolve them �C although there may be some additional > work beyond just a developer picking up the bug and working on it to decide > what should happen! > > Do you know how to reliably reproduce an instance in ERROR state that cannot > be deleted? > > Gabe > > From: > openstack-bounces+gabe.westmaas=rackspace....@lists.launchpad.net<mailto:openstack-bounces+gabe.westmaas=rackspace....@lists.launchpad.net> > > [mailto:openstack-bounces+gabe.westmaas=rackspace....@lists.launchpad.net]<mailto:[mailto:openstack-bounces+gabe.westmaas=rackspace....@lists.launchpad.net]> > On Behalf Of Yong Sheng Gong > Sent: Thursday, March 22, 2012 12:58 AM > > To: Openstack Mail List > Subject: Re: [Openstack] Can't delete instances with "error" status. > > why not allow "nova delete" and "euca-terminate " to delete the inst ance > with "error" status? > > Yong Sheng Gong > > [email protected]<mailto:[email protected]> > wrote: ----- > To: Guilherme Birk <[email protected]<mailto:[email protected]>>, > Openstack Mail List > <[email protected]<mailto:[email protected]>> > From: Mandar Vaze <[email protected]<mailto:[email protected]>> > Sent by: > [email protected]<mailto:[email protected]> > Date: 03/22/2012 12:26PM > Subject: Re: [Openstack] Can't delete instances with "error" status. > I agree user shouldn’t have to update DB. > There needs to be some periodic cleanup task that “fixes” the > vm_state/task_state for “stuck” instances. > > -Mandar > > From: > [email protected]<mailto:[email protected]> > > [mailto:[email protected]]<mailto:[mailto:[email protected]]> > On Behalf Of Guilherme Birk > Sent: Wednesday, March 21, 2012 6:05 PM > To: Openstack Mail List > Subject: Re: [Openstack] Can't delete instances with "error" status. > > This is the only option? I've already done that, but it's kind "strange" I > update the database row everytime the script identifies a VM with error. > ________________________________________ > Date: Wed, 21 Mar 2012 09:39:03 +0800 > Subject: Re: [Openstack] Can't delete instances with "error" status. > From: [email protected]<mailto:[email protected]> > To: [email protected]<mailto:[email protected]> > update "instances" table in "nova" db, set the vm_status of the instance > which you want to delete "active" and set the task_status=NULL. > After that, try to use euca-terminate > > On Wed, Mar 21, 2012 at 1:24 AM, Guilherme Birk > <[email protected]<mailto:[email protected]>> wrote: > I'm attempting to make a python script that controls all my virtual machines. > Sometimes, when the script identifies that exists an instance with status of > "error", he creates a new instance and tries to delete the old one with curl > commands, but I'm not getting any response and the VM isn't deleted. When I > execute euca-terminate instance <i-name> I got nothing too. How I should > delete instances with error status ? I didn't found any way using nova-manage > too. > > _______________________________________________ > Mailing list: https://launchpad.net/~openstack > Post to : > [email protected]<mailto:[email protected]> > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp > > > > -- > 非淡薄无以明志，非宁静无以致远 > _______________________________________________ > Mailing list: https://launchpad.net/~openstack > Post to : > [email protected]<mailto:[email protected]> > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp > > > > _______________________________________________ > Mailing list: https://launchpad.net/~openstack > Post to : > [email protected]<mailto:[email protected]> > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp > > _______________________________________________ > Mailing list: https://launchpad.net/~openstack > Post to : [email protected]<mailto:[email protected]> > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp
_______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : [email protected] Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
