Re: [Openstack] Hardware HA

[Caitlin Bestler]

Ryan Lane Wrote in response to Soren Hansen:

>> That's the whole point. For most interesting applications, "fast"
>> automatic migration isn't anywhere near fast enough. Don't try to 
>> avoid failure. Expect it and design around it.
>>

>This assumes all application designers are doing this. Most web applications do
>this fairly well, but most enterprise applications do this very poorly.

>Hardware HA is useful for more than just poorly designed applications though.
> I have a cloud instance that runs my personal website. I don't want to pay for
>two (or more, realistically) instances just to ensure that if my host dies 
>that my
>site will continue to run. My provider should automatically detect the hardware
>failure and re-launch my instance on another piece of hardware; it should also
>notify me that it happened, but that's a different story ;).

There are techniques to migrate VMs between non-HA hosts, and there are
techniques that allow applications to be written so that any instance of the 
server
can be lost without impairing the application (you just start a new instance of 
the
server, rather than migrating the server).

But neither of those solve the problem as well has hardware High Availability.
Whether Hardware HA is a cost effective solution is something that customers
will ultimately have to determine. 

A successful proposal would need to include identifying when a VM wants/needs
to be hosted on a Hardware-HA enhanced host, a method of identifying the 
Hardware-HA enhanced hosts, and the ability to track when a Hardware-HA
Host is in degraded mode (i.e., it currently is one resource failure away from
an absolute failure).

I think those features can be designed in a way that does not impose too strong
of a burden on the core scheduling algorithm, as long as it isn't required to
evaluate a long list of "Hardware HA QoS metrics" to do optimal guest to host
assignments.

This is actually virtually the same issue as Object Storage support for 
self-healing
Mirroring (via ZFS) that we have proposed for Swift. It defines an enhanced 
capability
For specific servers that can be characterized in a way that the generic  
control and
Management plane algorithms can understand. The hardest part of that 
understanding
In both cases is the addition of a "degraded" status for a server.

Without Hardware HA or self-healing mirroring a host/data server is either "up" 
or "down".
With Hardware HA and self-healing mirroring they can be "degraded". The 
Hardware HA
Host can be down to a single hardware node. The self-healing mirror could be 
done to a
Single working storage device. In either case the remaining copy is still 
functional, but you
Probably want to begin migrating the VMs/Swift Partitions elsewhere (unless 
your mean
Time to repair is really good).


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Hardware HA

[Ryan Lane]

> I know. That's what makes them a poor fit for "the cloud".
>

Meh. Private clouds will still use applications like this. I think
"the cloud" is great for cloud providers, but why limit nova's
usefulness to just cloud providers?

"The cloud" way of doing things pushes the responsibility of keeping
applications alive to the client. There's a lot of clients that don't
have this level of sophistication.

>> Hardware HA is useful for more than just poorly designed applications
>> though. I have a cloud instance that runs my personal website. I don't
>> want to pay for two (or more, realistically) instances just to ensure
>> that if my host dies that my site will continue to run. My provider
>> should automatically detect the hardware failure and re-launch my
>> instance on another piece of hardware; it should also notify me that
>> it happened, but that's a different story ;).
>
> I'm not sure I count that as High Availability. It's more like
> Eventual Availability. :)
>

So, this is one HA mode for VMware. There is also a newer HA mode that
is much more expensive (from the resources perspective) that keeps a
shadow copy of a virtual machine on another piece of hardware, and if
the primary instance's hardware dies, it automatically switches over
to the shadow copy.

Both modes are really useful. There's a huge level of automation
needed for doing things "the cloud way" that is completely
unnecessary. I don't want to have to monitor my instances to see if
one died due to a hardware failure, then start new ones, then pool
them, then depool the dead ones. I want my provider to handle hardware
deaths for me. If I have 200 web servers instances, and 40 of them die
because they are on nodes that die, I want them to restart somewhere
else. It removes all the bullshit automation I'd need to do otherwise.

- Ryan Lane

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Hardware HA

[Ryan Lane]

> That's the whole point. For most interesting applications, "fast"
> automatic migration isn't anywhere near fast enough. Don't try to
> avoid failure. Expect it and design around it.
>

This assumes all application designers are doing this. Most web
applications do this fairly well, but most enterprise applications do
this very poorly.

Hardware HA is useful for more than just poorly designed applications
though. I have a cloud instance that runs my personal website. I don't
want to pay for two (or more, realistically) instances just to ensure
that if my host dies that my site will continue to run. My provider
should automatically detect the hardware failure and re-launch my
instance on another piece of hardware; it should also notify me that
it happened, but that's a different story ;).

- Ryan Lane

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Hardware HA

[Soren Hansen]

2011/11/10 Ryan Lane :
>> That's the whole point. For most interesting applications, "fast"
>> automatic migration isn't anywhere near fast enough. Don't try to
>> avoid failure. Expect it and design around it.
> This assumes all application designers are doing this. Most web
> applications do this fairly well, but most enterprise applications do
> this very poorly.

I know. That's what makes them a poor fit for "the cloud".

> Hardware HA is useful for more than just poorly designed applications
> though. I have a cloud instance that runs my personal website. I don't
> want to pay for two (or more, realistically) instances just to ensure
> that if my host dies that my site will continue to run. My provider
> should automatically detect the hardware failure and re-launch my
> instance on another piece of hardware; it should also notify me that
> it happened, but that's a different story ;).

I'm not sure I count that as High Availability. It's more like
Eventual Availability. :)

-- 
Soren Hansen        | http://linux2go.dk/
Ubuntu Developer    | http://www.ubuntu.com/
OpenStack Developer | http://www.openstack.org/

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Hardware HA

[Soren Hansen]

2011/11/10 Viacheslav Biriukov :
> Hm
> If we planning vm hosting we work on the other level. So if hw node fails we
> need fast automatic migration to other node.

That's the whole point. For most interesting applications, "fast"
automatic migration isn't anywhere near fast enough. Don't try to
avoid failure. Expect it and design around it.

-- 
Soren Hansen        | http://linux2go.dk/
Ubuntu Developer    | http://www.ubuntu.com/
OpenStack Developer | http://www.openstack.org/

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Hardware HA

[Ilya Alekseyev]

Hi Armando!

It is very interesting feature. Are you already have specs for this
blueprints or may be etherpad?

Regards,
Ilya

10 ноября 2011 г. 20:16 пользователь Armando Migliaccio <
armando.migliac...@eu.citrix.com> написал:

> There is a blueprint that touches these aspects:
>
> https://blueprints.launchpad.net/nova/+spec/guest-ha
>
> This is tailored at use cases where you cannot redesign an existing app.
>
> The work is at the early stages, but you are more than welcome to join the
> effort!
>
> Cheers,
> Armando
>
> > -Original Message-
> > From: openstack-bounces+armando.migliaccio=
> eu.citrix@lists.launchpad.net
> > [mailto:openstack-
> > bounces+armando.migliaccio=eu.citrix@lists.launchpad.net] On Behalf
> Of
> > Soren Hansen
> > Sent: 10 November 2011 15:51
> > To: Viacheslav Biriukov
> > Cc: openstack@lists.launchpad.net
> > Subject: Re: [Openstack] Hardware HA
> >
> > 2011/11/10 Viacheslav Biriukov :
> > > Hi all.
> > > What are the best practices for HA of the hardware compute-node, and
> virtual
> > > machines.
> > > After googling I found matahari, pacemaker-cloud, but nothing about
> > > build-in fiches  openstack.
> > > 1) How do you create such environments?
> > > 2) Does it is right way to use pacemaker-cloud with openstack? Is it
> stable?
> >
> > I'd avoid depending on anything like that altogether. Try to design
> > your application so that it doesn't depend on any one instance being
> > up. It'll work out better in the long run.
> >
> > --
> > Soren Hansen| http://linux2go.dk/
> > Ubuntu Developer| http://www.ubuntu.com/
> > OpenStack Developer | http://www.openstack.org/
> >
> > ___
> > Mailing list: https://launchpad.net/~openstack
> > Post to : openstack@lists.launchpad.net
> > Unsubscribe : https://launchpad.net/~openstack
> > More help   : https://help.launchpad.net/ListHelp
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Hardware HA

[Razique Mahroua]

Funny you bring that up today;I spent the day working on that. I've implemented Gluster FS on my openstack running installation and written a script along that.Here is the implementationnode1- 1 instance runningthe node 1 crashes (could be anything atm)the script detect the node is gone (to be defined : heartbeat, monitoring, etc...) and relaunch the instance in the specified node.I've tested it and successfully works. Just need to write the final touchesRegards
Razique Mahrouarazique.mahr...@gmail.com

Le 10 nov. 2011 à 17:15, Viacheslav Biriukov a écrit :HmIf we planning vm hosting we work on the other level. So if hw node fails we need fast automatic migration to other node.2011/11/10 Soren Hansen 
2011/11/10 Viacheslav Biriukov :
> Hi all.
> What are the best practices for HA of the hardware compute-node, and virtual
> machines.
> After googling I found matahari, pacemaker-cloud, but nothing about
> build-in fiches  openstack.
> 1) How do you create such environments?
> 2) Does it is right way to use pacemaker-cloud with openstack? Is it stable?

I'd avoid depending on anything like that altogether. Try to design
your application so that it doesn't depend on any one instance being
up. It'll work out better in the long run.

--
Soren Hansen        | http://linux2go.dk/
Ubuntu Developer    | http://www.ubuntu.com/
OpenStack Developer | http://www.openstack.org/
-- Viacheslav BiriukovBRhttp://biriukov.com

___Mailing list: https://launchpad.net/~openstackPost to : openstack@lists.launchpad.netUnsubscribe : https://launchpad.net/~openstackMore help   : https://help.launchpad.net/ListHelp___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Hardware HA

[Viacheslav Biriukov]

Hm
If we planning vm hosting we work on the other level. So if hw node fails
we need fast automatic migration to other node.

2011/11/10 Soren Hansen 

> 2011/11/10 Viacheslav Biriukov :
> > Hi all.
> > What are the best practices for HA of the hardware compute-node, and
> virtual
> > machines.
> > After googling I found matahari, pacemaker-cloud, but nothing about
> > build-in fiches  openstack.
> > 1) How do you create such environments?
> > 2) Does it is right way to use pacemaker-cloud with openstack? Is it
> stable?
>
> I'd avoid depending on anything like that altogether. Try to design
> your application so that it doesn't depend on any one instance being
> up. It'll work out better in the long run.
>
> --
> Soren Hansen| http://linux2go.dk/
> Ubuntu Developer| http://www.ubuntu.com/
> OpenStack Developer | http://www.openstack.org/
>



-- 
Viacheslav Biriukov
BR
http://biriukov.com
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Hardware HA

[Armando Migliaccio]

There is a blueprint that touches these aspects:

https://blueprints.launchpad.net/nova/+spec/guest-ha

This is tailored at use cases where you cannot redesign an existing app. 

The work is at the early stages, but you are more than welcome to join the 
effort!

Cheers,
Armando

> -Original Message-
> From: openstack-bounces+armando.migliaccio=eu.citrix@lists.launchpad.net
> [mailto:openstack-
> bounces+armando.migliaccio=eu.citrix@lists.launchpad.net] On Behalf Of
> Soren Hansen
> Sent: 10 November 2011 15:51
> To: Viacheslav Biriukov
> Cc: openstack@lists.launchpad.net
> Subject: Re: [Openstack] Hardware HA
> 
> 2011/11/10 Viacheslav Biriukov :
> > Hi all.
> > What are the best practices for HA of the hardware compute-node, and virtual
> > machines.
> > After googling I found matahari, pacemaker-cloud, but nothing about
> > build-in fiches  openstack.
> > 1) How do you create such environments?
> > 2) Does it is right way to use pacemaker-cloud with openstack? Is it stable?
> 
> I'd avoid depending on anything like that altogether. Try to design
> your application so that it doesn't depend on any one instance being
> up. It'll work out better in the long run.
> 
> --
> Soren Hansen        | http://linux2go.dk/
> Ubuntu Developer    | http://www.ubuntu.com/
> OpenStack Developer | http://www.openstack.org/
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Hardware HA

[Soren Hansen]

2011/11/10 Viacheslav Biriukov :
> Hi all.
> What are the best practices for HA of the hardware compute-node, and virtual
> machines.
> After googling I found matahari, pacemaker-cloud, but nothing about
> build-in fiches  openstack.
> 1) How do you create such environments?
> 2) Does it is right way to use pacemaker-cloud with openstack? Is it stable?

I'd avoid depending on anything like that altogether. Try to design
your application so that it doesn't depend on any one instance being
up. It'll work out better in the long run.

-- 
Soren Hansen        | http://linux2go.dk/
Ubuntu Developer    | http://www.ubuntu.com/
OpenStack Developer | http://www.openstack.org/

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Hardware HA

[Pádraig Brady]

On 11/10/2011 02:38 PM, Viacheslav Biriukov wrote:
> Hi all.
> 
> What are the best practices for HA of the hardware compute-node, and virtual 
> machines.
> After googling I found matahari, pacemaker-cloud, but nothing about build-in 
> fiches  openstack. 
> 
> 1) How do you create such environments?  
> 2) Does it is right way to use pacemaker-cloud with openstack? Is it stable?

About pacemaker-cloud, one can currently use it with openstack to some extent:
http://ahsalkeld.wordpress.com/a-mash-up-of-openstack-and-pacemaker-cloud/
However there are lots of manual steps and it's still in development.
It would be cool if you wanted to give that a shot and reported any issues or 
thoughts.

cheers,
Pádraig.

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] Hardware HA

[Viacheslav Biriukov]

Hi all.

What are the best practices for HA of the hardware compute-node, and
virtual machines.
After googling I found matahari, pacemaker-cloud, but nothing about
build-in fiches  openstack.

1) How do you create such environments?
2) Does it is right way to use pacemaker-cloud with openstack? Is it stable?

Tnx
-- 
Viacheslav Biriukov
BR
http://biriukov.com
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp