Re: [ovirt-users] Recovering hosted-engine

2017-07-03 Thread Maton, Brett 
It depends on how 'toasted' the engine is.

It is possible to mount the disk, if for example it's a kernel upgrade
that's gone wrong and stopping the machine form booting

On 3 July 2017 at 15:23, Yedidyah Bar David <d...@redhat.com> wrote:

> On Mon, Jul 3, 2017 at 4:40 PM, Andrew Dent <ad...@ctcroydon.com.au>
> wrote:
> > Hi Didi
> >
> > Fair enough.
> > If I'm in this situation.
> > I have 3 hosts with 6 production VMs.
> > The hosted-engine VM is completely toast and not recoverable.
>
> Meaning? It does not even boot? If so, then I am afraid that you
> need to follow the linked procedure, including deploying hosted-engine
> again on new (or fully cleaned) storage.
>
> Are all 3 hosts also hosted-engine hosts? And you want to keep them
> this way? If so, you'll have to reinstall them.
>
> I think, didn't try, that after you deploy a new hosted-engine host,
> and restore the engine, the new engine will see your existing hosts,
> but hosted-engine-ha will not work.
>
> If you want to try and keep your VMs running, I guess your best bet
> is to try to add a new non-hosted-engine host (one or more, as needed),
> migrate your VMs to it (them), then remove the hosted-engine hosts and
> deploy them again as described there.
>
> > However I have a backup of the hosted-engine database (do I need anything
> > else).
>
> Only database? Or a backup taken by engine-backup? You also need various
> configuration files. To see the full list, in case you didn't use
> engine-backup but can restore individual files, check BACKUP_PATHS inside
> the script. But this is definitely not supported - you should first try
> verifying by restoring on a test VM and see what happens.
>
> >
> > Is it possible to build a new VM, import the backup of the previous
> > hosted-engine database and reconnect the storage domains and VMs in their
> > running state without any VMs experiencing an outage?
>
> As I wrote above, I think it is, but hosted-engine-ha will not work.
>
> If at all possible, I strongly suggest to try to simulate this in a
> test env. Can be done using nested-kvm VMs for hosts, if you do not have
> enough real hardware.
>
> >
> > The URL
> > http://www.ovirt.org/documentation/self-hosted/
> chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environment/
> > looks to be longer now.
> > I'll review, test and see if this will give me what I'm looking for.
> >
> > The broken link still seems to be broken.
> > When I click the link, the browser ends up at this
> > http://www.ovirt.org/documentation/self-hosted/
> chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_
> Environment/chap-Installing_Additional_Hosts_to_a_Self-Hosted_Environment#
> documentation%20self%20hosted%20chap%20Backing%20up%20and%
> 20Restoring%20an%20EL%20Based%20Self%20Hosted%20Environment%
> 20chap%20Installing%20Additional%20Hosts%20to%20a%
> 20Self%20Hosted%20Environment
> >
> > But I suspect it should be
> > http://www.ovirt.org/documentation/self-hosted/
> chap-Installing_Additional_Hosts_to_a_Self-Hosted_Environment/
>
> In my previous mail I just wrote I sent a pull request, not that
> the site is updated already. I updated it since and it now seems
> ok to me.
>
> Best,
>
> >
> > Kind regards
> >
> > Andrew
> >
> > -- Original Message --
> > From: "Yedidyah Bar David" <d...@redhat.com>
> > To: "Andrew Dent" <ad...@ctcroydon.com.au>
> > Cc: "users" <users@ovirt.org>
> > Sent: 3/07/2017 11:12:05 PM
> > Subject: Re: [ovirt-users] Recovering hosted-engine
> >
> >> On Mon, Jul 3, 2017 at 3:46 PM, Andrew Dent <ad...@ctcroydon.com.au>
> >> wrote:
> >>>
> >>>  Has anyone successfully completed a hosted-engine recovery on a
> multiple
> >>>  host setup with production VMs?
> >>
> >>
> >> I'd like to clarify that "recovery" can span a large spectrum of
> >> flows, from a trivial "I did some change to the engine database
> >> that broke stuff and I want to restore a backup I took prior to
> >> this change" to a full system restoration including purchasing
> >> and deploying new (perhaps different) hosts/network/storage
> >> hardware, including many other flows in between.
> >>
> >> So when you plan for recovery, you should define very well what
> >> flows you plan to handle, and how you handle each.
> >>
> >> The linked procedure correctly says it's "providing an example".
> >>
> >>>
> >&g

Re: [ovirt-users] Recovering hosted-engine

2017-07-03 Thread Yedidyah Bar David 
On Mon, Jul 3, 2017 at 4:40 PM, Andrew Dent <ad...@ctcroydon.com.au> wrote:
> Hi Didi
>
> Fair enough.
> If I'm in this situation.
> I have 3 hosts with 6 production VMs.
> The hosted-engine VM is completely toast and not recoverable.

Meaning? It does not even boot? If so, then I am afraid that you
need to follow the linked procedure, including deploying hosted-engine
again on new (or fully cleaned) storage.

Are all 3 hosts also hosted-engine hosts? And you want to keep them
this way? If so, you'll have to reinstall them.

I think, didn't try, that after you deploy a new hosted-engine host,
and restore the engine, the new engine will see your existing hosts,
but hosted-engine-ha will not work.

If you want to try and keep your VMs running, I guess your best bet
is to try to add a new non-hosted-engine host (one or more, as needed),
migrate your VMs to it (them), then remove the hosted-engine hosts and
deploy them again as described there.

> However I have a backup of the hosted-engine database (do I need anything
> else).

Only database? Or a backup taken by engine-backup? You also need various
configuration files. To see the full list, in case you didn't use
engine-backup but can restore individual files, check BACKUP_PATHS inside
the script. But this is definitely not supported - you should first try
verifying by restoring on a test VM and see what happens.

>
> Is it possible to build a new VM, import the backup of the previous
> hosted-engine database and reconnect the storage domains and VMs in their
> running state without any VMs experiencing an outage?

As I wrote above, I think it is, but hosted-engine-ha will not work.

If at all possible, I strongly suggest to try to simulate this in a
test env. Can be done using nested-kvm VMs for hosts, if you do not have
enough real hardware.

>
> The URL
> http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environment/
> looks to be longer now.
> I'll review, test and see if this will give me what I'm looking for.
>
> The broken link still seems to be broken.
> When I click the link, the browser ends up at this
> http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environment/chap-Installing_Additional_Hosts_to_a_Self-Hosted_Environment#documentation%20self%20hosted%20chap%20Backing%20up%20and%20Restoring%20an%20EL%20Based%20Self%20Hosted%20Environment%20chap%20Installing%20Additional%20Hosts%20to%20a%20Self%20Hosted%20Environment
>
> But I suspect it should be
> http://www.ovirt.org/documentation/self-hosted/chap-Installing_Additional_Hosts_to_a_Self-Hosted_Environment/

In my previous mail I just wrote I sent a pull request, not that
the site is updated already. I updated it since and it now seems
ok to me.

Best,

>
> Kind regards
>
> Andrew
>
> -- Original Message --
> From: "Yedidyah Bar David" <d...@redhat.com>
> To: "Andrew Dent" <ad...@ctcroydon.com.au>
> Cc: "users" <users@ovirt.org>
> Sent: 3/07/2017 11:12:05 PM
> Subject: Re: [ovirt-users] Recovering hosted-engine
>
>> On Mon, Jul 3, 2017 at 3:46 PM, Andrew Dent <ad...@ctcroydon.com.au>
>> wrote:
>>>
>>>  Has anyone successfully completed a hosted-engine recovery on a multiple
>>>  host setup with production VMs?
>>
>>
>> I'd like to clarify that "recovery" can span a large spectrum of
>> flows, from a trivial "I did some change to the engine database
>> that broke stuff and I want to restore a backup I took prior to
>> this change" to a full system restoration including purchasing
>> and deploying new (perhaps different) hosts/network/storage
>> hardware, including many other flows in between.
>>
>> So when you plan for recovery, you should define very well what
>> flows you plan to handle, and how you handle each.
>>
>> The linked procedure correctly says it's "providing an example".
>>
>>>
>>>  Kind regards
>>>
>>>
>>>  Andrew
>>>
>>>
>>>
>>>  -- Original Message --
>>>  From: "Andrew Dent" <ad...@ctcroydon.com.au>
>>>  To: "users" <users@ovirt.org>
>>>  Sent: 2/07/2017 2:22:16 PM
>>>  Subject: [ovirt-users] Recovering hosted-engine
>>>
>>>  Hi
>>>
>>>  A couple of questions about hosted-engine recovery.
>>>  Part way through this URL, in the section "Workflow for Restoring the
>>>  Self-Hosted Engine Environment"
>>>
>>> http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environmen

Re: [ovirt-users] Recovering hosted-engine

2017-07-03 Thread Nelson Lameiras 
Hi andrew,

This is my personal experience on recovery.

Yes you can recover the hosted-engine from a backup (almost every action you 
did on on oVirt after backup will have to be manually reproduced)
This can be tricky depending on your conditions (and there are some subtleties 
which can have significant impact later on)

- Do you have still SSH access to engine ?
- If so, do you trust engine VM OS, or should you reinstall it ?

If both questions are positive, then you just need to do a engine restore as 
explained here :

http://www.ovirt.org/documentation/admin-guide/chap-Backups_and_Migration/

cordialement, regards, 


Nelson LAMEIRAS 
Ingénieur Systèmes et Réseaux / Systems and Networks engineer 
Tel: +33 5 32 09 09 70 
nelson.lamei...@lyra-network.com 

www.lyra-network.com | www.payzen.eu 





Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE

- Original Message -
From: "Andrew Dent" <ad...@ctcroydon.com.au>
To: "Yedidyah Bar David" <d...@redhat.com>, "users" <users@ovirt.org>
Sent: Monday, July 3, 2017 3:40:38 PM
Subject: Re: [ovirt-users] Recovering hosted-engine

Hi Didi

Fair enough.
If I'm in this situation.
I have 3 hosts with 6 production VMs.
The hosted-engine VM is completely toast and not recoverable.
However I have a backup of the hosted-engine database (do I need 
anything else).

Is it possible to build a new VM, import the backup of the previous 
hosted-engine database and reconnect the storage domains and VMs in 
their running state without any VMs experiencing an outage?

The URL  
http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environment/
looks to be longer now.
I'll review, test and see if this will give me what I'm looking for.

The broken link still seems to be broken.
When I click the link, the browser ends up at this
http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environment/chap-Installing_Additional_Hosts_to_a_Self-Hosted_Environment#documentation%20self%20hosted%20chap%20Backing%20up%20and%20Restoring%20an%20EL%20Based%20Self%20Hosted%20Environment%20chap%20Installing%20Additional%20Hosts%20to%20a%20Self%20Hosted%20Environment

But I suspect it should be
http://www.ovirt.org/documentation/self-hosted/chap-Installing_Additional_Hosts_to_a_Self-Hosted_Environment/

Kind regards

Andrew

-- Original Message --
From: "Yedidyah Bar David" <d...@redhat.com>
To: "Andrew Dent" <ad...@ctcroydon.com.au>
Cc: "users" <users@ovirt.org>
Sent: 3/07/2017 11:12:05 PM
Subject: Re: [ovirt-users] Recovering hosted-engine

>On Mon, Jul 3, 2017 at 3:46 PM, Andrew Dent <ad...@ctcroydon.com.au> 
>wrote:
>>  Has anyone successfully completed a hosted-engine recovery on a 
>>multiple
>>  host setup with production VMs?
>
>I'd like to clarify that "recovery" can span a large spectrum of
>flows, from a trivial "I did some change to the engine database
>that broke stuff and I want to restore a backup I took prior to
>this change" to a full system restoration including purchasing
>and deploying new (perhaps different) hosts/network/storage
>hardware, including many other flows in between.
>
>So when you plan for recovery, you should define very well what
>flows you plan to handle, and how you handle each.
>
>The linked procedure correctly says it's "providing an example".
>
>>
>>  Kind regards
>>
>>
>>  Andrew
>>
>>
>>
>>  -- Original Message --
>>  From: "Andrew Dent" <ad...@ctcroydon.com.au>
>>  To: "users" <users@ovirt.org>
>>  Sent: 2/07/2017 2:22:16 PM
>>  Subject: [ovirt-users] Recovering hosted-engine
>>
>>  Hi
>>
>>  A couple of questions about hosted-engine recovery.
>>  Part way through this URL, in the section "Workflow for Restoring the
>>  Self-Hosted Engine Environment"
>>  
>>http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environment/
>>  it looks like once the hosted-engine is recovered on Host 1, the VMs 
>>on Host
>>  2 and 3 will be running, but not accessible to the recovered Hosted 
>>Engine.
>>  Is that correct?
>
>I am pretty certain that the procedure assumed that all hosts need 
>restoration,
>not that some are still up-and-running.
>
>>  If so, how to you remove host 2 and host 3 from the environment, then 
>>add
>>  back in again while keeping the VMs running?
>
>That's a good question.
>
>Please try to describe the exact flow you have in mind. What's broken 
>and
>needs restoration, and how do you plan to do

Re: [ovirt-users] Recovering hosted-engine

2017-07-03 Thread Andrew Dent 

Hi Didi

Fair enough.
If I'm in this situation.
I have 3 hosts with 6 production VMs.
The hosted-engine VM is completely toast and not recoverable.
However I have a backup of the hosted-engine database (do I need 
anything else).


Is it possible to build a new VM, import the backup of the previous 
hosted-engine database and reconnect the storage domains and VMs in 
their running state without any VMs experiencing an outage?


The URL  
http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environment/

looks to be longer now.
I'll review, test and see if this will give me what I'm looking for.

The broken link still seems to be broken.
When I click the link, the browser ends up at this
http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environment/chap-Installing_Additional_Hosts_to_a_Self-Hosted_Environment#documentation%20self%20hosted%20chap%20Backing%20up%20and%20Restoring%20an%20EL%20Based%20Self%20Hosted%20Environment%20chap%20Installing%20Additional%20Hosts%20to%20a%20Self%20Hosted%20Environment

But I suspect it should be
http://www.ovirt.org/documentation/self-hosted/chap-Installing_Additional_Hosts_to_a_Self-Hosted_Environment/

Kind regards

Andrew

-- Original Message --
From: "Yedidyah Bar David" <d...@redhat.com>
To: "Andrew Dent" <ad...@ctcroydon.com.au>
Cc: "users" <users@ovirt.org>
Sent: 3/07/2017 11:12:05 PM
Subject: Re: [ovirt-users] Recovering hosted-engine

On Mon, Jul 3, 2017 at 3:46 PM, Andrew Dent <ad...@ctcroydon.com.au> 
wrote:
 Has anyone successfully completed a hosted-engine recovery on a 
multiple

 host setup with production VMs?


I'd like to clarify that "recovery" can span a large spectrum of
flows, from a trivial "I did some change to the engine database
that broke stuff and I want to restore a backup I took prior to
this change" to a full system restoration including purchasing
and deploying new (perhaps different) hosts/network/storage
hardware, including many other flows in between.

So when you plan for recovery, you should define very well what
flows you plan to handle, and how you handle each.

The linked procedure correctly says it's "providing an example".



 Kind regards


 Andrew



 -- Original Message --
 From: "Andrew Dent" <ad...@ctcroydon.com.au>
 To: "users" <users@ovirt.org>
 Sent: 2/07/2017 2:22:16 PM
 Subject: [ovirt-users] Recovering hosted-engine

 Hi

 A couple of questions about hosted-engine recovery.
 Part way through this URL, in the section "Workflow for Restoring the
 Self-Hosted Engine Environment"
 
http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environment/
 it looks like once the hosted-engine is recovered on Host 1, the VMs 
on Host
 2 and 3 will be running, but not accessible to the recovered Hosted 
Engine.

 Is that correct?


I am pretty certain that the procedure assumed that all hosts need 
restoration,

not that some are still up-and-running.

 If so, how to you remove host 2 and host 3 from the environment, then 
add

 back in again while keeping the VMs running?


That's a good question.

Please try to describe the exact flow you have in mind. What's broken 
and

needs restoration, and how do you plan to do that?



 Host 2 and Host 3 are not recoverable in their current state. These 
hosts

 need to be removed from the environment, and then added again to the
 environment using the hosted-engine deployment script. For more 
information
 on these actions, see the Removing Non-Operational Hosts from a 
Restored
 Self-Hosted Engine Environment section below and Chapter 7: 
Installing

 Additional Hosts to a Self-Hosted Environment.

 BTW: The link referring to chapter 7 is broken.


You are right. The link in the bottom of the page seems working.
Now pushed [1] to fix. Thanks for the report!

[1]

Best,
--
Didi


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Recovering hosted-engine

2017-07-03 Thread Yedidyah Bar David 
On Mon, Jul 3, 2017 at 3:46 PM, Andrew Dent  wrote:
> Has anyone successfully completed a hosted-engine recovery on a multiple
> host setup with production VMs?

I'd like to clarify that "recovery" can span a large spectrum of
flows, from a trivial "I did some change to the engine database
that broke stuff and I want to restore a backup I took prior to
this change" to a full system restoration including purchasing
and deploying new (perhaps different) hosts/network/storage
hardware, including many other flows in between.

So when you plan for recovery, you should define very well what
flows you plan to handle, and how you handle each.

The linked procedure correctly says it's "providing an example".

>
> Kind regards
>
>
> Andrew
>
>
>
> -- Original Message --
> From: "Andrew Dent" 
> To: "users" 
> Sent: 2/07/2017 2:22:16 PM
> Subject: [ovirt-users] Recovering hosted-engine
>
> Hi
>
> A couple of questions about hosted-engine recovery.
> Part way through this URL, in the section "Workflow for Restoring the
> Self-Hosted Engine Environment"
> http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environment/
> it looks like once the hosted-engine is recovered on Host 1, the VMs on Host
> 2 and 3 will be running, but not accessible to the recovered Hosted Engine.
> Is that correct?

I am pretty certain that the procedure assumed that all hosts need restoration,
not that some are still up-and-running.

> If so, how to you remove host 2 and host 3 from the environment, then add
> back in again while keeping the VMs running?

That's a good question.

Please try to describe the exact flow you have in mind. What's broken and
needs restoration, and how do you plan to do that?

>
> Host 2 and Host 3 are not recoverable in their current state. These hosts
> need to be removed from the environment, and then added again to the
> environment using the hosted-engine deployment script. For more information
> on these actions, see the Removing Non-Operational Hosts from a Restored
> Self-Hosted Engine Environment section below and Chapter 7: Installing
> Additional Hosts to a Self-Hosted Environment.
>
> BTW: The link referring to chapter 7 is broken.

You are right. The link in the bottom of the page seems working.
Now pushed [1] to fix. Thanks for the report!

[1] https://github.com/oVirt/ovirt-site/pull/1056

Best,
-- 
Didi
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Recovering hosted-engine

2017-07-03 Thread Andrew Dent 
Has anyone successfully completed a hosted-engine recovery on a multiple 
host setup with production VMs?


Kind regards



Andrew



-- Original Message --
From: "Andrew Dent" 
To: "users" 
Sent: 2/07/2017 2:22:16 PM
Subject: [ovirt-users] Recovering hosted-engine


Hi

A couple of questions about hosted-engine recovery.
Part way through this URL, in the section "Workflow for Restoring the 
Self-Hosted Engine Environment"

http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environment/
it looks like once the hosted-engine is recovered on Host 1, the VMs on 
Host 2 and 3 will be running, but not accessible to the recovered 
Hosted Engine.

Is that correct?
If so, how to you remove host 2 and host 3 from the environment, then 
add back in again while keeping the VMs running?


Host 2 and Host 3 are not recoverable in their current state. These 
hosts need to be removed from the environment, and then added again to 
the environment using the hosted-engine deployment script. For more 
information on these actions, see the Removing Non-Operational Hosts 
from a Restored Self-Hosted Engine Environment section below and 
Chapter 7: Installing Additional Hosts to a Self-Hosted Environment 
.


BTW: The link referring to chapter 7 is broken.

Kind regards



Andrew

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users