Alex, You are absolutely right. We need multiple confirmations that host is actually failed on physical level and must be evacuated before we proceed with auto-evacuate. Fencing is also requirement for such kind of action.
This etherpad [1] has some great notes and suggestions on the topic. Hope it will be incorporated in Tim's session, or any other session on this topic. [1] https://etherpad.openstack.org/openstack-instance-high-availability -- Best regards, Oleg Gelbukh Mirantis Labs On Wed, Oct 9, 2013 at 3:59 PM, Alex Glikson <glik...@il.ibm.com> wrote: > *> Hypervisor failure detection is also more or less solved problem in > Nova [2]. There are other candidates for that task as well, like > Ceilometer's hardware agent [3] (still WIP to my knowledge).* > > The problem is that in some cases you want to be *really* sure that the > hypervisor is down before running 'evacuate' (otherwise it could lead to an > application crash). And you want to do it on scale. So, polling and > traditional monitoring might not be good enough for a fully-automated > service (e.g., you may need to do 'fencing' to ensure that the node will > not suddenly come back with all the VMs still running). > > Regards, > Alex > > > > > From: Oleg Gelbukh <ogelb...@mirantis.com> > To: OpenStack Development Mailing List < > openstack-dev@lists.openstack.org>, > Date: 09/10/2013 02:09 PM > Subject: Re: [openstack-dev] [nova] automatically evacuate > instances on compute failure > ------------------------------ > > > > Hello, > > We have much interest in this discussion (with focus on second scenario > outlined by Tim), and working on its design at the moment. Thanks to > everyone for valuable insights in this thread. > > It looks like external orchestration daemon problem is partially solved > already by Heat with HARestarter resource [1]. > > Hypervisor failure detection is also more or less solved problem in Nova > [2]. There are other candidates for that task as well, like Ceilometer's > hardware agent [3] (still WIP to my knowledge). > > [1] * > https://github.com/openstack/heat/blob/stable/grizzly/heat/engine/resources/instance.py#L35 > *<https://github.com/openstack/heat/blob/stable/grizzly/heat/engine/resources/instance.py#L35> > [2] * > http://docs.openstack.org/developer/nova/api/nova.api.openstack.compute.contrib.hypervisors.html#module-nova.api.openstack.compute.contrib.hypervisors > *<http://docs.openstack.org/developer/nova/api/nova.api.openstack.compute.contrib.hypervisors.html#module-nova.api.openstack.compute.contrib.hypervisors> > [3] * > https://blueprints.launchpad.net/ceilometer/+spec/monitoring-physical-devices > *<https://blueprints.launchpad.net/ceilometer/+spec/monitoring-physical-devices> > -- > Best regards, > Oleg Gelbukh > Mirantis Labs > > > On Wed, Oct 9, 2013 at 9:26 AM, Tim Bell > <*tim.b...@cern.ch*<tim.b...@cern.ch>> > wrote: > I have proposed the summit design session for Hong Kong (* > http://summit.openstack.org/cfp/details/103*<http://summit.openstack.org/cfp/details/103>) > to discuss exactly these sort of points. We have the low level Nova > commands but need a service to automate the process. > > I see two scenarios > > - A hardware intervention needs to be scheduled, please rebalance this > workload elsewhere before it fails completely > - A hypervisor has failed, please recover what you can using shared > storage and give me a policy on what to do with the other VMs (restart, > leave down till repair etc.) > > Most OpenStack production sites have some sort of script doing this sort > of thing now. However, each one will be implementing the logic for > migration differently so there is no agreed best practise approach. > > Tim > > > -----Original Message----- > > From: Chris Friesen > > [mailto:*chris.frie...@windriver.com*<chris.frie...@windriver.com> > ] > > Sent: 09 October 2013 00:48 > > To: *openstack-dev@lists.openstack.org*<openstack-dev@lists.openstack.org> > > Subject: Re: [openstack-dev] [nova] automatically evacuate instances on > compute failure > > > > On 10/08/2013 03:20 PM, Alex Glikson wrote: > > > Seems that this can be broken into 3 incremental pieces. First, would > > > be great if the ability to schedule a single 'evacuate' would be > > > finally merged > > > (_* > https://blueprints.launchpad.net/nova/+spec/find-host-and-evacuate-instance_ > *<https://blueprints.launchpad.net/nova/+spec/find-host-and-evacuate-instance_> > ). > > > > Agreed. > > > > > Then, it would make sense to have the logic that evacuates an entire > > > host > > > (_* > https://blueprints.launchpad.net/python-novaclient/+spec/find-and-evacuate-host_ > *<https://blueprints.launchpad.net/python-novaclient/+spec/find-and-evacuate-host_> > ). > > > The reasoning behind suggesting that this should not necessarily be in > > > Nova is, perhaps, that it *can* be implemented outside Nova using the > > > indvidual 'evacuate' API. > > > > This actually more-or-less exists already in the existing "nova > host-evacuate" command. One major issue with this however is that it > > requires the caller to specify whether all the instances are on shared > or local storage, and so it can't handle a mix of local and shared > > storage for the instances. If any of them boot off block storage for > > instance you need to move them first and then do the remaining ones as a > group. > > > > It would be nice to embed the knowledge of whether or not an instance is > on shared storage in the instance itself at creation time. I > > envision specifying this in the config file for the compute manager > along with the instance storage location, and the compute manager > > could set the field in the instance at creation time. > > > > > Finally, it should be possible to close the loop and invoke the > > > evacuation automatically as a result of a failure detection (not clear > > > how exactly this would work, though). Hopefully we will have at least > > > the first part merged soon (not sure if anyone is actively working on > > > a rebase). > > > > My interpretation of the discussion so far is that the nova maintainers > would prefer this to be driven by an outside orchestration daemon. > > > > Currently the only way a service is recognized to be "down" is if > someone calls is_up() and it notices that the service hasn't sent an update > > in the last minute. There's nothing in nova actively scanning for > compute node failures, which is where the outside daemon comes in. > > > > Also, there is some complexity involved in dealing with auto-evacuate: > > What do you do if an evacuate fails? How do you recover intelligently > if there is no admin involved? > > > > Chris > > > > _______________________________________________ > > OpenStack-dev mailing list > > *OpenStack-dev@lists.openstack.org* <OpenStack-dev@lists.openstack.org> > > *http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev*<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev> > > _______________________________________________ > OpenStack-dev mailing list* > **OpenStack-dev@lists.openstack.org* <OpenStack-dev@lists.openstack.org>* > **http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev*<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev> > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
