Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-25 Thread Géza Gémes
Subject: Re: [openstack-dev] [Nova] Automatic evacuate On 10/21/2014 06:44 AM, Balázs Gibizer wrote: Hi, Sorry for the top posting but it was hard to fit my complete view inline. I'm also thinking about a possible solution for automatic server evacuation. I see two separate sub problems

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-22 Thread David Vossel
- Original Message - On 10/21/2014 07:53 PM, David Vossel wrote: - Original Message - -Original Message- From: Russell Bryant [mailto:rbry...@redhat.com] Sent: October 21, 2014 15:07 To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Nova

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-21 Thread Balázs Gibizer
: [openstack-dev] [Nova] Automatic evacuate -Original Message- From: Florian Haas [mailto:flor...@hastexo.com] Sent: Friday, October 17, 2014 1:49 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Nova] Automatic evacuate On Fri, Oct

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-21 Thread Russell Bryant
On 10/21/2014 06:44 AM, Balázs Gibizer wrote: Hi, Sorry for the top posting but it was hard to fit my complete view inline. I'm also thinking about a possible solution for automatic server evacuation. I see two separate sub problems of this problem: 1)compute node monitoring and fencing,

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-21 Thread Balázs Gibizer
-Original Message- From: Russell Bryant [mailto:rbry...@redhat.com] Sent: October 21, 2014 15:07 To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Nova] Automatic evacuate On 10/21/2014 06:44 AM, Balázs Gibizer wrote: Hi, Sorry for the top posting

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-21 Thread David Vossel
- Original Message - -Original Message- From: Russell Bryant [mailto:rbry...@redhat.com] Sent: October 21, 2014 15:07 To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Nova] Automatic evacuate On 10/21/2014 06:44 AM, Balázs Gibizer wrote: Hi

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-21 Thread Géza Gémes
On 10/21/2014 07:53 PM, David Vossel wrote: - Original Message - -Original Message- From: Russell Bryant [mailto:rbry...@redhat.com] Sent: October 21, 2014 15:07 To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Nova] Automatic evacuate On 10/21/2014 06:44 AM

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-21 Thread David Vossel
- Original Message - On Thu, Oct 16, 2014 at 7:48 PM, Jay Pipes jaypi...@gmail.com wrote: While one of us (Jay or me) speaking for the other and saying we agree is a distributed consensus problem that dwarfs the complexity of Paxos You've always had a way with words, Florian

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-17 Thread Jastrzebski, Michal
-Original Message- From: Florian Haas [mailto:flor...@hastexo.com] Sent: Thursday, October 16, 2014 10:53 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Nova] Automatic evacuate On Thu, Oct 16, 2014 at 9:25 AM, Jastrzebski

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-17 Thread Florian Haas
: [openstack-dev] [Nova] Automatic evacuate On Thu, Oct 16, 2014 at 9:25 AM, Jastrzebski, Michal michal.jastrzeb...@intel.com wrote: In my opinion flavor defining is a bit hacky. Sure, it will provide us functionality fairly quickly, but also will strip us from flexibility Heat would give. Healing

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-16 Thread Jastrzebski, Michal
-Original Message- From: Russell Bryant [mailto:rbry...@redhat.com] Sent: Thursday, October 16, 2014 5:04 AM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Nova] Automatic evacuate On 10/15/2014 05:07 PM, Florian Haas wrote: On Wed, Oct 15, 2014 at 10:03 PM

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-16 Thread Sylvain Bauza
Le 16/10/2014 05:04, Russell Bryant a écrit : On 10/15/2014 05:07 PM, Florian Haas wrote: On Wed, Oct 15, 2014 at 10:03 PM, Russell Bryant rbry...@redhat.com wrote: Am I making sense? Yep, the downside is just that you need to provide a new set of flavors for ha vs non-ha. A benefit though

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-16 Thread Florian Haas
(5) Let monitoring and orchestration services deal with these use cases and have Nova simply provide the primitive API calls that it already does (i.e. host evacuate). That would arguably lead to an incredible amount of wheel reinvention for node failure detection, service failure

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-16 Thread Florian Haas
On Thu, Oct 16, 2014 at 5:04 AM, Russell Bryant rbry...@redhat.com wrote: On 10/15/2014 05:07 PM, Florian Haas wrote: On Wed, Oct 15, 2014 at 10:03 PM, Russell Bryant rbry...@redhat.com wrote: Am I making sense? Yep, the downside is just that you need to provide a new set of flavors for ha

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-16 Thread Florian Haas
On Thu, Oct 16, 2014 at 9:25 AM, Jastrzebski, Michal michal.jastrzeb...@intel.com wrote: In my opinion flavor defining is a bit hacky. Sure, it will provide us functionality fairly quickly, but also will strip us from flexibility Heat would give. Healing can be done in several ways, simple

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-16 Thread Thomas Herve
This still doesn't do away with the requirement to reliably detect node failure, and to fence misbehaving nodes. Detecting that a node has failed, and fencing it if unsure, is a prerequisite for any recovery action. So you need Corosync/Pacemaker anyway. Obviously, yes. My post

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-16 Thread Florian Haas
On Thu, Oct 16, 2014 at 11:01 AM, Thomas Herve thomas.he...@enovance.com wrote: This still doesn't do away with the requirement to reliably detect node failure, and to fence misbehaving nodes. Detecting that a node has failed, and fencing it if unsure, is a prerequisite for any recovery

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-16 Thread Russell Bryant
On 10/16/2014 04:29 AM, Florian Haas wrote: (5) Let monitoring and orchestration services deal with these use cases and have Nova simply provide the primitive API calls that it already does (i.e. host evacuate). That would arguably lead to an incredible amount of wheel reinvention for node

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-16 Thread Russell Bryant
On 10/16/2014 05:01 AM, Thomas Herve wrote: This still doesn't do away with the requirement to reliably detect node failure, and to fence misbehaving nodes. Detecting that a node has failed, and fencing it if unsure, is a prerequisite for any recovery action. So you need Corosync/Pacemaker

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-16 Thread Florian Haas
On Thu, Oct 16, 2014 at 1:59 PM, Russell Bryant rbry...@redhat.com wrote: On 10/16/2014 04:29 AM, Florian Haas wrote: (5) Let monitoring and orchestration services deal with these use cases and have Nova simply provide the primitive API calls that it already does (i.e. host evacuate). That

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-16 Thread Steve Gordon
- Original Message - From: Florian Haas flor...@hastexo.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org On Thu, Oct 16, 2014 at 1:59 PM, Russell Bryant rbry...@redhat.com wrote: On 10/16/2014 04:29 AM, Florian Haas wrote:

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-16 Thread Florian Haas
On Thu, Oct 16, 2014 at 4:31 PM, Steve Gordon sgor...@redhat.com wrote: Running compute nodes as baremetal extensions of a different Corosync/Pacemaker cluster (presumably the one that manages the other Nova services) would potentially be an option, although vendors would need to buy into

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-16 Thread Russell Bryant
On 10/16/2014 09:00 AM, Florian Haas wrote: On Thu, Oct 16, 2014 at 1:59 PM, Russell Bryant rbry...@redhat.com wrote: On 10/16/2014 04:29 AM, Florian Haas wrote: (5) Let monitoring and orchestration services deal with these use cases and have Nova simply provide the primitive API calls that

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-16 Thread Adam Lawson
Be forewarned; here's my two cents before I've had my morning coffee. It would seem to me that if we were seeking some level of resiliency against host failures (if a host fails, evacuate the instances that were hosted on it to a host that isn't broken), it would seem that host HA is a good

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-16 Thread Russell Bryant
On 10/16/2014 01:03 PM, Adam Lawson wrote: Be forewarned; here's my two cents before I've had my morning coffee. It would seem to me that if we were seeking some level of resiliency against host failures (if a host fails, evacuate the instances that were hosted on it to a host that isn't

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-16 Thread Florian Haas
On Thu, Oct 16, 2014 at 7:03 PM, Adam Lawson alaw...@aqorn.com wrote: Be forewarned; here's my two cents before I've had my morning coffee. It would seem to me that if we were seeking some level of resiliency against host failures (if a host fails, evacuate the instances that were hosted on

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-16 Thread Florian Haas
On Thu, Oct 16, 2014 at 7:48 PM, Jay Pipes jaypi...@gmail.com wrote: While one of us (Jay or me) speaking for the other and saying we agree is a distributed consensus problem that dwarfs the complexity of Paxos You've always had a way with words, Florian :) I knew you'd like that one. :) ,

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-16 Thread Adam Lawson
Okay the coffee kicked in. I can see how my comment could be interpreted that way so let's take a step backward so I can explain my perspective here. Amazon was the first to implement a commercial-grade cloud IaaS, Openstack was developed as an alternative. If we avoided wheel re-invention as a

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-16 Thread Russell Bryant
On 10/16/2014 02:40 PM, Adam Lawson wrote: Question: is host HA not achievable using the programs we have in place now (with modification of course)? If not, I'm still a champion to see it done within our four walls. Yes, it is achievable (without modification, even). That was the primary

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-16 Thread Florian Haas
On Thu, Oct 16, 2014 at 9:40 PM, Russell Bryant rbry...@redhat.com wrote: On 10/16/2014 02:40 PM, Adam Lawson wrote: Question: is host HA not achievable using the programs we have in place now (with modification of course)? If not, I'm still a champion to see it done within our four walls.

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-15 Thread Jastrzebski, Michal
To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Nova] Automatic evacuate On 10/14/2014 01:01 PM, Jay Pipes wrote: 2) Looking forward, there is a lot of demand for doing this on a per instance basis. We should decide on a best practice for allowing end users to indicate

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-15 Thread Russell Bryant
On 10/13/2014 05:59 PM, Russell Bryant wrote: Nice timing. I was working on a blog post on this topic. which is now here: http://blog.russellbryant.net/2014/10/15/openstack-instance-ha-proposal/ -- Russell Bryant ___ OpenStack-dev mailing list

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-15 Thread Florian Haas
On Wed, Oct 15, 2014 at 7:20 PM, Russell Bryant rbry...@redhat.com wrote: On 10/13/2014 05:59 PM, Russell Bryant wrote: Nice timing. I was working on a blog post on this topic. which is now here: http://blog.russellbryant.net/2014/10/15/openstack-instance-ha-proposal/ I am absolutely

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-15 Thread Jay Pipes
On 10/15/2014 03:16 PM, Florian Haas wrote: On Wed, Oct 15, 2014 at 7:20 PM, Russell Bryant rbry...@redhat.com wrote: On 10/13/2014 05:59 PM, Russell Bryant wrote: Nice timing. I was working on a blog post on this topic. which is now here:

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-15 Thread Florian Haas
On Wed, Oct 15, 2014 at 9:58 PM, Jay Pipes jaypi...@gmail.com wrote: On 10/15/2014 03:16 PM, Florian Haas wrote: On Wed, Oct 15, 2014 at 7:20 PM, Russell Bryant rbry...@redhat.com wrote: On 10/13/2014 05:59 PM, Russell Bryant wrote: Nice timing. I was working on a blog post on this topic.

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-15 Thread Florian Haas
On Wed, Oct 15, 2014 at 10:03 PM, Russell Bryant rbry...@redhat.com wrote: Am I making sense? Yep, the downside is just that you need to provide a new set of flavors for ha vs non-ha. A benefit though is that it's a way to support it today without *any* changes to OpenStack. Users are

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-15 Thread Adam Lawson
It would seem to me that if guest HA is highly-desired, and it is, requiring multiple flavors for multiple SLA requirements (and that's what we're really talking about) introduces a trade-off that conceivably isn't needed - double the flavor requirement for the same spec (512/1/10 and another for

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-15 Thread Jay Pipes
On 10/15/2014 04:50 PM, Florian Haas wrote: On Wed, Oct 15, 2014 at 9:58 PM, Jay Pipes jaypi...@gmail.com wrote: On 10/15/2014 03:16 PM, Florian Haas wrote: On Wed, Oct 15, 2014 at 7:20 PM, Russell Bryant rbry...@redhat.com wrote: On 10/13/2014 05:59 PM, Russell Bryant wrote: Nice

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-15 Thread Russell Bryant
On 10/15/2014 06:30 PM, Jay Pipes wrote: On 10/15/2014 04:50 PM, Florian Haas wrote: On Wed, Oct 15, 2014 at 9:58 PM, Jay Pipes jaypi...@gmail.com wrote: On 10/15/2014 03:16 PM, Florian Haas wrote: On Wed, Oct 15, 2014 at 7:20 PM, Russell Bryant rbry...@redhat.com wrote: On 10/13/2014

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-15 Thread Russell Bryant
On 10/15/2014 05:07 PM, Florian Haas wrote: On Wed, Oct 15, 2014 at 10:03 PM, Russell Bryant rbry...@redhat.com wrote: Am I making sense? Yep, the downside is just that you need to provide a new set of flavors for ha vs non-ha. A benefit though is that it's a way to support it today without

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-14 Thread Sylvain Bauza
Le 14/10/2014 01:46, Adam Lawson a écrit : /I think Adam is talking about this bp: https://blueprints.launchpad.net/nova/+spec/evacuate-instance-automatically/ Correct - yes. Sorry about that. ; ) So it would seem the question is not whether to support auto-evac but how it should

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-14 Thread Jay Pipes
On 10/13/2014 05:59 PM, Russell Bryant wrote: Nice timing. I was working on a blog post on this topic. On 10/13/2014 05:40 PM, Fei Long Wang wrote: I think Adam is talking about this bp: https://blueprints.launchpad.net/nova/+spec/evacuate-instance-automatically For now, we're using Nagios

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-14 Thread Tim Bell
-Original Message- From: Jay Pipes [mailto:jaypi...@gmail.com] Sent: 14 October 2014 19:01 To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Nova] Automatic evacuate On 10/13/2014 05:59 PM, Russell Bryant wrote: Nice timing. I was working on a blog post

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-14 Thread Russell Bryant
On 10/14/2014 01:01 PM, Jay Pipes wrote: 2) Looking forward, there is a lot of demand for doing this on a per instance basis. We should decide on a best practice for allowing end users to indicate whether they would like their VMs automatically rescued by the infrastructure, or just left down

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-14 Thread Mathieu Gagné
On 2014-10-14 2:49 PM, Tim Bell wrote: Nova is also not the right place to do the generic solution as many other parts could be involved... neutron and cinder come to mind. Nova needs to provide the basic functions but it needs something outside to make it all happen transparently. I would

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-14 Thread Adam Lawson
Nova is also not the right place to do the generic solution as many other parts could be involved... neutron and cinder come to mind. Nova needs to provide the basic functions but it needs something outside to make it all happen transparently. I would really like a shared solution rather

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-13 Thread Adam Lawson
Looks like this was proposed and denied to be part of Nova for some reason last year. Thoughts on why and is the reasoning (whatever it was) still applicable? *Adam Lawson* AQORN, Inc. 427 North Tatnall Street Ste. 58461 Wilmington, Delaware 19801-2230 Toll-free: (844) 4-AQORN-NOW ext. 101

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-13 Thread Joe Gordon
On Mon, Oct 13, 2014 at 1:32 PM, Adam Lawson alaw...@aqorn.com wrote: Looks like this was proposed and denied to be part of Nova for some reason last year. Thoughts on why and is the reasoning (whatever it was) still applicable? Link? *Adam Lawson* AQORN, Inc. 427 North Tatnall

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-13 Thread Fei Long Wang
I think Adam is talking about this bp: https://blueprints.launchpad.net/nova/+spec/evacuate-instance-automatically For now, we're using Nagios probe/event to trigger the Nova evacuate command, but I think it's possible to do that in Nova if we can find a good way to define the trigger policy.

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-13 Thread Russell Bryant
Nice timing. I was working on a blog post on this topic. On 10/13/2014 05:40 PM, Fei Long Wang wrote: I think Adam is talking about this bp: https://blueprints.launchpad.net/nova/+spec/evacuate-instance-automatically For now, we're using Nagios probe/event to trigger the Nova evacuate

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-13 Thread Jay Lau
This is also a use case for Congress, please check use case 3 in the following link. https://docs.google.com/document/d/1ExDmT06vDZjzOPePYBqojMRfXodvsk0R8nRkX-zrkSw/edit# 2014-10-14 5:59 GMT+08:00 Russell Bryant rbry...@redhat.com: Nice timing. I was working on a blog post on this topic. On

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-13 Thread Russell Bryant
On 10/13/2014 06:18 PM, Jay Lau wrote: This is also a use case for Congress, please check use case 3 in the following link. https://docs.google.com/document/d/1ExDmT06vDZjzOPePYBqojMRfXodvsk0R8nRkX-zrkSw/edit# Wow, really? That honestly makes me very worried about the scope of Congress

Re: [openstack-dev] [Nova] Automatic evacuate

2014-10-13 Thread Adam Lawson
*I think Adam is talking about this bp: https://blueprints.launchpad.net/nova/+spec/evacuate-instance-automatically https://blueprints.launchpad.net/nova/+spec/evacuate-instance-automatically * Correct - yes. Sorry about that. ; ) So it would seem the question is not whether to support