A scenario that I've seen: Take 'nova-compute' down for software upgrade, API still accessible since you want to provide API uptime (aka not taking the whole cluster offline).
User Y deletes VM on that hypervisor where nova-compute is currently down, DB locally deletes, at this point VM 'A' is still active but nova thinks its not. User X requests VM, gets allocated 'A' hostname, ip (or other resource). Uh oh, u now have two of the same IP/hostname in the same network (second 'A' is in damaged state since it likely can't even 'ifup' its network). Upgrade software on that hypervisor (yum install xyz...), service nova-compute restart (back to normal). Now the first 'A' could get deleted, but u still have second broken 'A'. Now what? On 10/7/13 4:21 PM, "Vishvananda Ishaya" <vishvana...@gmail.com> wrote: > >On Oct 7, 2013, at 3:49 PM, Joshua Harlow <harlo...@yahoo-inc.com> wrote: > >> This brings up another question, do people actually like/use the >> 'local_delete' feature in nova? >> >> In general it seems to free resources that are not actually capable of >> being freed and has been problematic for y! usage. >> >> Deleting from the DB allows another request to actually take those >> resources over, yet the previous VM (+network,volumes...) that wasn't >> deleted still has those resources (likely attached to it in the case of >>a >> volume, or in the case of a hypervisor the VM resource is still active, >> but maybe nova-compute is down) so u end up in a conflict. How are >>others >> using this code? Has it been working out? > >We haven't had any trouble with the two settings set as below. Users seem >to get far more frustrated when they have things that they cannot delete, >especially when it is using up their precious quota. > >Vish > >> >> -Josh >> >> On 10/7/13 3:34 PM, "Vishvananda Ishaya" <vishvana...@gmail.com> wrote: >> >>> There is a configuration option stating what to do with instances that >>> are still in the hypervisor but have been deleted from the database. I >>> think you want: >>> >>> running_deleted_instance_action=reap >>> >>> You probably also want >>> >>> resume_guests_state_on_host_boot=true >>> >>> to bring back the instances that were running before the node was >>>powered >>> off. We should definitely consider changing the default of these two >>> values since I think the default values are probably not what most >>>people >>> would want. >>> >>> Vish >>> On Oct 7, 2013, at 1:24 PM, Chris Friesen <chris.frie...@windriver.com> >>> wrote: >>> >>>> On 10/07/2013 12:44 PM, Russell Bryant wrote: >>>>> On 10/07/2013 02:28 PM, Chris Friesen wrote: >>>>>> >>>>>> I've been doing a lot of instance creation/deletion/evacuate and >>>>>>I've >>>>>> noticed that if I >>>>>> >>>>>> 1)create an instance >>>>>> 2) power off the compute node it was running on >>>>>> 3) delete the instance >>>>>> 4) boot up the compute node >>>>>> >>>>>> then the instance rootfs stays around in /var/lib/nova/instances/. >>>>>> Eventually this could add up to significant amounts of space. >>>>>> >>>>>> >>>>>> Is this expected behaviour? (This is on grizzly, so maybe havana is >>>>>> different.) If not, should I file a bug for it? >>>>>> >>>>>> I think it would make sense for the compute node to come up, query >>>>>>all >>>>>> the instances in /var/lib/nova/instances/, and delete the ones for >>>>>> instances that aren't in the database. >>>>> >>>>> How long are you waiting after starting up the compute node? I would >>>>> expect it to get cleaned up by a periodic task, so you might have to >>>>> wait roughly 10 minutes (by default). >>>> >>>> This is nearly 50 minutes after booting up the compute node: >>>> >>>> cfriesen@compute2:/var/lib/nova/instances$ ls -1 >>>> 39e459b1-3878-41db-aaaf-7c7d0dfa2b19 >>>> 41a60975-d6b8-468e-90bc-d7de58c2124d >>>> 46aec2ae-b6de-4503-a238-af736f81f1a4 >>>> 50ec3d89-1c9d-4c28-adaf-26c924dfa3ed >>>> _base >>>> c6ec71a3-658c-4c7c-aa42-cc26296ce7fb >>>> c72845e9-0d34-459f-b602-bb2ee409728b >>>> compute_nodes >>>> locks >>>> >>>> Of these, only two show up in "nova list". >>>> >>>> Chris >>>> >>>> >>>> _______________________________________________ >>>> OpenStack-dev mailing list >>>> OpenStack-dev@lists.openstack.org >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >>> _______________________________________________ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
