Re: [ovirt-users] Recovering oVirt-Engine with a backup before upgrading to 4.2

[Sven Achtelik]

Hi All,

I'm still failing on this one. I tried setting the locale different, but that 
doesn't seem to do the trick. When looking at the logfile 
/var/lib/pgsql/initdb_rh-postgresql95-postgresql.log I can see that the setup 
process is somehow getting the information and setting the Encoding to UTF8.


/var/lib/pgsql/initdb_rh-postgresql95-postgresql.log

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".


 This leads into the issue that is showing up in 
/var/lib/pgsql/upgrade_rh-postgresql95-postgresql.log

Checking cluster versions   ok
Checking database user is the install user  ok
Checking database connection settings   ok
Checking for prepared transactions  ok
Checking for reg* system OID user data typesok
Checking for contrib/isn with bigint-passing mismatch   ok
Checking for invalid "line" user columnsok
Creating dump of global objects ok
Creating dump of database schemas
  engine
  ovirt_engine_history
  postgres
  template1
ok

encodings for database "postgres" do not match:  old "SQL_ASCII", new "UTF8"
Failure, exiting
---

Even changing the locale.conf to en_US without the UTF8 doesn't change 
anything. Is this information cached somewhere and needs to be reread before 
this will work ?

Any advice on moving forward is appreciated. 

Thank you, 
Sven 



> -Ursprüngliche Nachricht-
> Von: Yedidyah Bar David [mailto:d...@redhat.com]
> Gesendet: Freitag, 20. April 2018 08:55
> An: Sven Achtelik <sven.achte...@eps.aero>; Simone Tiraboschi
> <stira...@redhat.com>
> Cc: users@ovirt.org
> Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before
> upgrading to 4.2
> 
> On Fri, Apr 13, 2018 at 12:00 PM, Sven Achtelik <sven.achte...@eps.aero>
> wrote:
> > Hi All,
> >
> > I got my stuff up and running again. I works like described in the manual 
> > and I
> used some extra hardware to jumpstart this. I'm now back on my hosted Engine
> 4.1.9 with 3 Hosts running it. The Engine is running on the appliance that is
> pulled by the deployment tool and after having everything stable again I
> thought of upgrading to 4.2. Thing is that this is just not working with the
> appliance because of some issue when upgrading Postgres inside. Looking at
> the logs I found this:
> > 
> > Creating dump of database schemas
> >   engine
> >   ovirt_engine_history
> >   postgres
> >   template1
> > ok
> >
> > encodings for database "postgres" do not match:  old "SQL_ASCII", new
> "UTF8"
> > Failure, exiting
> > --
> >
> > After some research I found something here
> https://bugzilla.redhat.com/show_bug.cgi?id=1525976, but I'm not sure what
> to do with that Information. I used the appliance and didn't do anything 
> manual
> in the complete process and I'm wondering why I'm getting this issue now ?
> Could someone advice on how to proceed ?
> 
> Looks like:
> 
> https://bugzilla.redhat.com/1528371
> 
> Which version do you upgrade to?
> 
> If to one that should be covered by above bug, please attach your setup log 
> to it.
> Thanks.
> 
> I was on vacation last week and will be in next one too. Adding Simone.
> 
> Best regards,
> 
> >
> > Thank you,
> > Sven
> >
> >> -Ursprüngliche Nachricht-
> >> Von: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] Im
> >> Auftrag von Sven Achtelik
> >> Gesendet: Mittwoch, 28. März 2018 18:28
> >> An: Yedidyah Bar David
> >> Cc: users@ovirt.org
> >> Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup
> >> before upgrading to 4.2
> >>
> >>
> >>
> >> > -Ursprüngliche Nachricht-
> >> > Von: Yedidyah Bar David [mailto:d...@redhat.com]
> >> > Gesendet: Mittwoch, 28. März 2018 10:06
> >> > An: Sven Achtelik
> >> > Cc: users@ovirt.org
> >> > Betreff: Re:

Re: [ovirt-users] Recovering oVirt-Engine with a backup before upgrading to 4.2

[Sven Achtelik]

I added the 4.2 Repos and this what the yum update installed the following 
packages

[root@ovirt-engine ~]# yum list "ovirt-*-setup*"  
Loaded plugins: fastestmirror, versionlock
Determining fastest mirrors
 * base: centos.host-engine.com
 * extras: mirror.riverfrontnetworks.com
 * ovirt-4.1-epel: mirrors.lug.mtu.edu
 * ovirt-4.2: resources.ovirt.org
 * ovirt-4.2-epel: mirrors.lug.mtu.edu
 * updates: mirrors.advancedhosters.com
Installed Packages
ovirt-engine-dwh-setup.noarch   
   4.2.2.2-1.el7.centos 
@ovirt-4.2
ovirt-engine-setup.noarch   
   4.2.2.6-1.el7.centos 
@ovirt-4.2
ovirt-engine-setup-base.noarch  
   4.2.2.6-1.el7.centos 
@ovirt-4.2
ovirt-engine-setup-plugin-ovirt-engine.noarch   
   4.2.2.6-1.el7.centos 
@ovirt-4.2
ovirt-engine-setup-plugin-ovirt-engine-common.noarch
   4.2.2.6-1.el7.centos 
@ovirt-4.2
ovirt-engine-setup-plugin-vmconsole-proxy-helper.noarch 
   4.2.2.6-1.el7.centos 
@ovirt-4.2
ovirt-engine-setup-plugin-websocket-proxy.noarch
   4.2.2.6-1.el7.centos 
@ovirt-4.2
ovirt-imageio-proxy-setup.noarch
   1.2.2-0.el7.centos   
@ovirt-4.2
Available Packages
ovirt-ansible-engine-setup.noarch   
   1.1.0-1.el7.centos   
ovirt-4.2 
ovirt-engine-extension-aaa-ldap-setup.noarch
   1.3.7-1.el7.centos   
ovirt-4.2 
ovirt-engine-setup-plugin-dockerc.noarch
   4.1.9.1-1.el7.centos 
ovirt-4.1 
ovirt-engine-setup-plugin-live.noarch   
   4.1.0-1.el7.centos   
ovirt-4.1 
ovirt-hosted-engine-setup.noarch


> -Ursprüngliche Nachricht-
> Von: Yedidyah Bar David [mailto:d...@redhat.com]
> Gesendet: Freitag, 20. April 2018 08:55
> An: Sven Achtelik; Simone Tiraboschi
> Cc: users@ovirt.org
> Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before
> upgrading to 4.2
> 
> On Fri, Apr 13, 2018 at 12:00 PM, Sven Achtelik <sven.achte...@eps.aero>
> wrote:
> > Hi All,
> >
> > I got my stuff up and running again. I works like described in the manual 
> > and I
> used some extra hardware to jumpstart this. I'm now back on my hosted Engine
> 4.1.9 with 3 Hosts running it. The Engine is running on the appliance that is
> pulled by the deployment tool and after having everything stable again I
> thought of upgrading to 4.2. Thing is that this is just not working with the
> appliance because of some issue when upgrading Postgres inside. Looking at
> the logs I found this:
> > 
> > Creating dump of database schemas
> >   engine
> >   ovirt_engine_history
> >   postgres
> >   template1
> > ok
> >
> > encodings for database "postgres" do not match:  old "SQL_ASCII", new
> "UTF8"
> > Failure, exiting
> > --
> >
> > After some research I found something here
> https://bugzilla.redhat.com/show_bug.cgi?id=1525976, but I'm not sure what
> to do with that Information. I used the appliance and didn't do anything 
> manual
> in the complete p

Re: [ovirt-users] Recovering oVirt-Engine with a backup before upgrading to 4.2

[Yedidyah Bar David]

On Fri, Apr 13, 2018 at 12:00 PM, Sven Achtelik <sven.achte...@eps.aero> wrote:
> Hi All,
>
> I got my stuff up and running again. I works like described in the manual and 
> I used some extra hardware to jumpstart this. I'm now back on my hosted 
> Engine 4.1.9 with 3 Hosts running it. The Engine is running on the appliance 
> that is pulled by the deployment tool and after having everything stable 
> again I thought of upgrading to 4.2. Thing is that this is just not working 
> with the appliance because of some issue when upgrading Postgres inside. 
> Looking at the logs I found this:
> 
> Creating dump of database schemas
>   engine
>   ovirt_engine_history
>   postgres
>   template1
> ok
>
> encodings for database "postgres" do not match:  old "SQL_ASCII", new "UTF8"
> Failure, exiting
> --
>
> After some research I found something here 
> https://bugzilla.redhat.com/show_bug.cgi?id=1525976, but I'm not sure what to 
> do with that Information. I used the appliance and didn't do anything manual 
> in the complete process and I'm wondering why I'm getting this issue now ? 
> Could someone advice on how to proceed ?

Looks like:

https://bugzilla.redhat.com/1528371

Which version do you upgrade to?

If to one that should be covered by above bug, please attach your
setup log to it. Thanks.

I was on vacation last week and will be in next one too. Adding Simone.

Best regards,

>
> Thank you,
> Sven
>
>> -Ursprüngliche Nachricht-
>> Von: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] Im Auftrag
>> von Sven Achtelik
>> Gesendet: Mittwoch, 28. März 2018 18:28
>> An: Yedidyah Bar David
>> Cc: users@ovirt.org
>> Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before
>> upgrading to 4.2
>>
>>
>>
>> > -Ursprüngliche Nachricht-
>> > Von: Yedidyah Bar David [mailto:d...@redhat.com]
>> > Gesendet: Mittwoch, 28. März 2018 10:06
>> > An: Sven Achtelik
>> > Cc: users@ovirt.org
>> > Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup
>> > before upgrading to 4.2
>> >
>> > On Tue, Mar 27, 2018 at 9:14 PM, Sven Achtelik
>> > <sven.achte...@eps.aero>
>> > wrote:
>> > > Hi All,
>> > >
>> > >
>> > >
>> > > I’m still facing issues with my HE engine. Here are the steps that I
>> > > took to end up in this situation:
>> > >
>> > >
>> > >
>> > > - Update Engine from 4.1.7 to 4.1.9
>> > >
>> > > o   That worked as expected
>> > >
>> > > - Automatic Backup of Engine DB in the night
>> > >
>> > > - Upgraded Engine from 4.1.9 to 4.2.1
>> > >
>> > > o   That worked fine
>> > >
>> > > - Noticed Issues with the HA support for HE
>> > >
>> > > o   Cause was not having the latest ovirt-ha agent/broker version on 
>> > > hosts
>> > >
>> > > - After updating the first host with the latest packages for the
>> > > Agent/Broker engine was started twice
>> > >
>> > > o   As a result the Engine VM Disk was corrupted and there is no Backup 
>> > > of
>> > > the Disk
>> > >
>> > > o   There is also no Backup of the Engine DB with version 4.2
>> > >
>> > > - VM disk was repaired with fsck.ext4, but DB is corrupt
>> > >
>> > > o   Can’t restore the Engine DB because the Backup DB from Engine V 4.1
>> > >
>> > > - Rolled back all changes on Engine VM to 4.1.9 and imported 
>> > > Backup
>> > >
>> > > o   Checked for HA VMs to set as disabled and started the Engine
>> > >
>> > > - Login is fine but the Engine is having trouble picking up and
>> > > information from the Hosts
>> > >
>> > > o   No information on running VMs or hosts status
>> > >
>> > > - Final Situation
>> > >
>> > > o   2 Hosts have VMs still running and I can’t stop those
>> > >
>> > > o   I still have the image of my corrupted Engine VM (v4.2)
>> > >
>> > >
>> > >
>> > > Since there were no major changes after upgrading from 4.1 to 4.2,
>> > > would it be possible to manually r

Re: [ovirt-users] Recovering oVirt-Engine with a backup before upgrading to 4.2

[Shani Leviim]

Hi Sven,

I've found this one: https://gist.github.com/turboladen/6790847

I think it should help :)


*Regards,*

*Shani Leviim*

On Fri, Apr 13, 2018 at 12:00 PM, Sven Achtelik <sven.achte...@eps.aero>
wrote:

> Hi All,
>
> I got my stuff up and running again. I works like described in the manual
> and I used some extra hardware to jumpstart this. I'm now back on my hosted
> Engine 4.1.9 with 3 Hosts running it. The Engine is running on the
> appliance that is pulled by the deployment tool and after having everything
> stable again I thought of upgrading to 4.2. Thing is that this is just not
> working with the appliance because of some issue when upgrading Postgres
> inside. Looking at the logs I found this:
> 
> Creating dump of database schemas
>   engine
>   ovirt_engine_history
>   postgres
>   template1
> ok
>
> encodings for database "postgres" do not match:  old "SQL_ASCII", new
> "UTF8"
> Failure, exiting
> --
>
> After some research I found something here https://bugzilla.redhat.com/
> show_bug.cgi?id=1525976, but I'm not sure what to do with that
> Information. I used the appliance and didn't do anything manual in the
> complete process and I'm wondering why I'm getting this issue now ? Could
> someone advice on how to proceed ?
>
> Thank you,
> Sven
>
> > -Ursprüngliche Nachricht-
> > Von: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] Im Auftrag
> > von Sven Achtelik
> > Gesendet: Mittwoch, 28. März 2018 18:28
> > An: Yedidyah Bar David
> > Cc: users@ovirt.org
> > Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before
> > upgrading to 4.2
> >
> >
> >
> > > -Ursprüngliche Nachricht-----
> > > Von: Yedidyah Bar David [mailto:d...@redhat.com]
> > > Gesendet: Mittwoch, 28. März 2018 10:06
> > > An: Sven Achtelik
> > > Cc: users@ovirt.org
> > > Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup
> > > before upgrading to 4.2
> > >
> > > On Tue, Mar 27, 2018 at 9:14 PM, Sven Achtelik
> > > <sven.achte...@eps.aero>
> > > wrote:
> > > > Hi All,
> > > >
> > > >
> > > >
> > > > I’m still facing issues with my HE engine. Here are the steps that I
> > > > took to end up in this situation:
> > > >
> > > >
> > > >
> > > > - Update Engine from 4.1.7 to 4.1.9
> > > >
> > > > o   That worked as expected
> > > >
> > > > - Automatic Backup of Engine DB in the night
> > > >
> > > > - Upgraded Engine from 4.1.9 to 4.2.1
> > > >
> > > > o   That worked fine
> > > >
> > > > - Noticed Issues with the HA support for HE
> > > >
> > > > o   Cause was not having the latest ovirt-ha agent/broker version on
> hosts
> > > >
> > > > - After updating the first host with the latest packages for
> the
> > > > Agent/Broker engine was started twice
> > > >
> > > > o   As a result the Engine VM Disk was corrupted and there is no
> Backup of
> > > > the Disk
> > > >
> > > > o   There is also no Backup of the Engine DB with version 4.2
> > > >
> > > > - VM disk was repaired with fsck.ext4, but DB is corrupt
> > > >
> > > > o   Can’t restore the Engine DB because the Backup DB from Engine V
> 4.1
> > > >
> > > > - Rolled back all changes on Engine VM to 4.1.9 and imported
> Backup
> > > >
> > > > o   Checked for HA VMs to set as disabled and started the Engine
> > > >
> > > > - Login is fine but the Engine is having trouble picking up
> and
> > > > information from the Hosts
> > > >
> > > > o   No information on running VMs or hosts status
> > > >
> > > > - Final Situation
> > > >
> > > > o   2 Hosts have VMs still running and I can’t stop those
> > > >
> > > > o   I still have the image of my corrupted Engine VM (v4.2)
> > > >
> > > >
> > > >
> > > > Since there were no major changes after upgrading from 4.1 to 4.2,
> > > > would it be possible to manually restore the 4.1 DB to the 4.2
> > > > Engine VM to this up and

Re: [ovirt-users] Recovering oVirt-Engine with a backup before upgrading to 4.2

[Sven Achtelik]

Hi All, 

I got my stuff up and running again. I works like described in the manual and I 
used some extra hardware to jumpstart this. I'm now back on my hosted Engine 
4.1.9 with 3 Hosts running it. The Engine is running on the appliance that is 
pulled by the deployment tool and after having everything stable again I 
thought of upgrading to 4.2. Thing is that this is just not working with the 
appliance because of some issue when upgrading Postgres inside. Looking at the 
logs I found this: 

Creating dump of database schemas
  engine
  ovirt_engine_history
  postgres
  template1
ok

encodings for database "postgres" do not match:  old "SQL_ASCII", new "UTF8"
Failure, exiting
--

After some research I found something here 
https://bugzilla.redhat.com/show_bug.cgi?id=1525976, but I'm not sure what to 
do with that Information. I used the appliance and didn't do anything manual in 
the complete process and I'm wondering why I'm getting this issue now ? Could 
someone advice on how to proceed ? 

Thank you, 
Sven 

> -Ursprüngliche Nachricht-
> Von: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] Im Auftrag
> von Sven Achtelik
> Gesendet: Mittwoch, 28. März 2018 18:28
> An: Yedidyah Bar David
> Cc: users@ovirt.org
> Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before
> upgrading to 4.2
> 
> 
> 
> > -Ursprüngliche Nachricht-
> > Von: Yedidyah Bar David [mailto:d...@redhat.com]
> > Gesendet: Mittwoch, 28. März 2018 10:06
> > An: Sven Achtelik
> > Cc: users@ovirt.org
> > Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup
> > before upgrading to 4.2
> >
> > On Tue, Mar 27, 2018 at 9:14 PM, Sven Achtelik
> > <sven.achte...@eps.aero>
> > wrote:
> > > Hi All,
> > >
> > >
> > >
> > > I’m still facing issues with my HE engine. Here are the steps that I
> > > took to end up in this situation:
> > >
> > >
> > >
> > > - Update Engine from 4.1.7 to 4.1.9
> > >
> > > o   That worked as expected
> > >
> > > - Automatic Backup of Engine DB in the night
> > >
> > > - Upgraded Engine from 4.1.9 to 4.2.1
> > >
> > > o   That worked fine
> > >
> > > - Noticed Issues with the HA support for HE
> > >
> > > o   Cause was not having the latest ovirt-ha agent/broker version on hosts
> > >
> > > - After updating the first host with the latest packages for the
> > > Agent/Broker engine was started twice
> > >
> > > o   As a result the Engine VM Disk was corrupted and there is no Backup of
> > > the Disk
> > >
> > > o   There is also no Backup of the Engine DB with version 4.2
> > >
> > > - VM disk was repaired with fsck.ext4, but DB is corrupt
> > >
> > > o   Can’t restore the Engine DB because the Backup DB from Engine V 4.1
> > >
> > > - Rolled back all changes on Engine VM to 4.1.9 and imported 
> > > Backup
> > >
> > > o   Checked for HA VMs to set as disabled and started the Engine
> > >
> > > - Login is fine but the Engine is having trouble picking up and
> > > information from the Hosts
> > >
> > > o   No information on running VMs or hosts status
> > >
> > > - Final Situation
> > >
> > > o   2 Hosts have VMs still running and I can’t stop those
> > >
> > > o   I still have the image of my corrupted Engine VM (v4.2)
> > >
> > >
> > >
> > > Since there were no major changes after upgrading from 4.1 to 4.2,
> > > would it be possible to manually restore the 4.1 DB to the 4.2
> > > Engine VM to this up and running again or are there modifications
> > > made to the DB on upgrading that are relevant for this ?
> >
> > engine-backup requires restoring to the same version used to take the
> > backup, with a single exception - on 4.0, it can restore 3.6.
> >
> > It's very easy to patch it to allow also 4.1->4.2, search inside it
> > for "VALID_BACKUP_RESTORE_PAIRS". However, I do not think anyone ever
> > tested this, so no idea might break. In 3.6->4.0 days, we did have to
> > fix a few other things, notably apache httpd and iptables->firewalld:
> >
> > https://bugzilla.redhat.com/show_bug.cgi?id=1318580
> >
> > > All my work on rolling back to 4.1.9 with the

Re: [ovirt-users] Recovering oVirt-Engine with a backup before upgrading to 4.2

[Sven Achtelik]

> -Ursprüngliche Nachricht-
> Von: Yedidyah Bar David [mailto:d...@redhat.com]
> Gesendet: Mittwoch, 28. März 2018 10:06
> An: Sven Achtelik
> Cc: users@ovirt.org
> Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before
> upgrading to 4.2
> 
> On Tue, Mar 27, 2018 at 9:14 PM, Sven Achtelik <sven.achte...@eps.aero>
> wrote:
> > Hi All,
> >
> >
> >
> > I’m still facing issues with my HE engine. Here are the steps that I
> > took to end up in this situation:
> >
> >
> >
> > - Update Engine from 4.1.7 to 4.1.9
> >
> > o   That worked as expected
> >
> > - Automatic Backup of Engine DB in the night
> >
> > - Upgraded Engine from 4.1.9 to 4.2.1
> >
> > o   That worked fine
> >
> > - Noticed Issues with the HA support for HE
> >
> > o   Cause was not having the latest ovirt-ha agent/broker version on hosts
> >
> > - After updating the first host with the latest packages for the
> > Agent/Broker engine was started twice
> >
> > o   As a result the Engine VM Disk was corrupted and there is no Backup of
> > the Disk
> >
> > o   There is also no Backup of the Engine DB with version 4.2
> >
> > - VM disk was repaired with fsck.ext4, but DB is corrupt
> >
> > o   Can’t restore the Engine DB because the Backup DB from Engine V 4.1
> >
> > - Rolled back all changes on Engine VM to 4.1.9 and imported Backup
> >
> > o   Checked for HA VMs to set as disabled and started the Engine
> >
> > - Login is fine but the Engine is having trouble picking up and
> > information from the Hosts
> >
> > o   No information on running VMs or hosts status
> >
> > - Final Situation
> >
> > o   2 Hosts have VMs still running and I can’t stop those
> >
> > o   I still have the image of my corrupted Engine VM (v4.2)
> >
> >
> >
> > Since there were no major changes after upgrading from 4.1 to 4.2,
> > would it be possible to manually restore the 4.1 DB to the 4.2 Engine
> > VM to this up and running again or are there modifications made to the
> > DB on upgrading that are relevant for this ?
> 
> engine-backup requires restoring to the same version used to take the backup,
> with a single exception - on 4.0, it can restore 3.6.
> 
> It's very easy to patch it to allow also 4.1->4.2, search inside it for
> "VALID_BACKUP_RESTORE_PAIRS". However, I do not think anyone ever
> tested this, so no idea might break. In 3.6->4.0 days, we did have to fix a 
> few
> other things, notably apache httpd and iptables->firewalld:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1318580
> 
> > All my work on rolling back to 4.1.9 with the DB restore failed as the
> > Engine is not capable of picking up information from the hosts.
> 
> No idea why, but not sure it's related to your restore flow.
> 
> > Lessons learned is to always make a copy/snapshot of the engine VM
> > disk before upgrading anything.
> 
> If it's a hosted-engine, this isn't supported - see my reply on the list ~ 1 
> hour
> ago...
> 
> > What are my options on getting
> > back to a working environment ? Any help or hint is greatly appreciated.
> 
> Restore again with either methods - what you tried, or patching engine-
> backup and restore directly into 4.2 - and if the engine fails to talk to the 
> hosts,
> try to debug/fix this.
> 
> If you suspect corruption more severe that just the db, you can install a 
> fresh
> engine machine from scratch and restore to it. If it's a hosted-engine, you'll
> need to deploy hosted-engine from scratch, check docs about hosted-engine
> backup/restore.

I read through those documents and it seems that I would need an extra 
Host/Hardware which I don't have. 
https://ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environment/

So how would I be able to get a new setup working when I would like to use the 
Engine-VM-Image ? At this point it sounds like I would have to manually 
reinstall the machine that is left over and running. I'm lost at this point. 
> 
> Best regards,
> --
> Didi
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Recovering oVirt-Engine with a backup before upgrading to 4.2

[Sven Achtelik]

> -Ursprüngliche Nachricht-
> Von: Yedidyah Bar David [mailto:d...@redhat.com]
> Gesendet: Mittwoch, 28. März 2018 10:06
> An: Sven Achtelik
> Cc: users@ovirt.org
> Betreff: Re: [ovirt-users] Recovering oVirt-Engine with a backup before
> upgrading to 4.2
> 
> On Tue, Mar 27, 2018 at 9:14 PM, Sven Achtelik <sven.achte...@eps.aero>
> wrote:
> > Hi All,
> >
> >
> >
> > I’m still facing issues with my HE engine. Here are the steps that I
> > took to end up in this situation:
> >
> >
> >
> > - Update Engine from 4.1.7 to 4.1.9
> >
> > o   That worked as expected
> >
> > - Automatic Backup of Engine DB in the night
> >
> > - Upgraded Engine from 4.1.9 to 4.2.1
> >
> > o   That worked fine
> >
> > - Noticed Issues with the HA support for HE
> >
> > o   Cause was not having the latest ovirt-ha agent/broker version on hosts
> >
> > - After updating the first host with the latest packages for the
> > Agent/Broker engine was started twice
> >
> > o   As a result the Engine VM Disk was corrupted and there is no Backup of
> > the Disk
> >
> > o   There is also no Backup of the Engine DB with version 4.2
> >
> > - VM disk was repaired with fsck.ext4, but DB is corrupt
> >
> > o   Can’t restore the Engine DB because the Backup DB from Engine V 4.1
> >
> > - Rolled back all changes on Engine VM to 4.1.9 and imported Backup
> >
> > o   Checked for HA VMs to set as disabled and started the Engine
> >
> > - Login is fine but the Engine is having trouble picking up and
> > information from the Hosts
> >
> > o   No information on running VMs or hosts status
> >
> > - Final Situation
> >
> > o   2 Hosts have VMs still running and I can’t stop those
> >
> > o   I still have the image of my corrupted Engine VM (v4.2)
> >
> >
> >
> > Since there were no major changes after upgrading from 4.1 to 4.2,
> > would it be possible to manually restore the 4.1 DB to the 4.2 Engine
> > VM to this up and running again or are there modifications made to the
> > DB on upgrading that are relevant for this ?
> 
> engine-backup requires restoring to the same version used to take the backup,
> with a single exception - on 4.0, it can restore 3.6.
> 
> It's very easy to patch it to allow also 4.1->4.2, search inside it for
> "VALID_BACKUP_RESTORE_PAIRS". However, I do not think anyone ever
> tested this, so no idea might break. In 3.6->4.0 days, we did have to fix a 
> few
> other things, notably apache httpd and iptables->firewalld:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1318580
> 
> > All my work on rolling back to 4.1.9 with the DB restore failed as the
> > Engine is not capable of picking up information from the hosts.
> 
> No idea why, but not sure it's related to your restore flow.
> 
> > Lessons learned is to always make a copy/snapshot of the engine VM
> > disk before upgrading anything.
> 
> If it's a hosted-engine, this isn't supported - see my reply on the list ~ 1 
> hour
> ago...
> 
> > What are my options on getting
> > back to a working environment ? Any help or hint is greatly appreciated.
> 
> Restore again with either methods - what you tried, or patching engine-
> backup and restore directly into 4.2 - and if the engine fails to talk to the 
> hosts,
> try to debug/fix this.
> 
> If you suspect corruption more severe that just the db, you can install a 
> fresh
> engine machine from scratch and restore to it. If it's a hosted-engine, you'll
> need to deploy hosted-engine from scratch, check docs about hosted-engine
> backup/restore.

Will the setup of hosted engine from scratch require a new storage domain for 
the new engine or can I use the one that is already there ? What about the VMs 
running on my hosts, will they be effected by that ? It might be best to start 
with a fresh VM.

> 
> Best regards,
> --
> Didi
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Recovering oVirt-Engine with a backup before upgrading to 4.2

[Yedidyah Bar David]

On Tue, Mar 27, 2018 at 9:14 PM, Sven Achtelik  wrote:
> Hi All,
>
>
>
> I’m still facing issues with my HE engine. Here are the steps that I took to
> end up in this situation:
>
>
>
> - Update Engine from 4.1.7 to 4.1.9
>
> o   That worked as expected
>
> - Automatic Backup of Engine DB in the night
>
> - Upgraded Engine from 4.1.9 to 4.2.1
>
> o   That worked fine
>
> - Noticed Issues with the HA support for HE
>
> o   Cause was not having the latest ovirt-ha agent/broker version on hosts
>
> - After updating the first host with the latest packages for the
> Agent/Broker engine was started twice
>
> o   As a result the Engine VM Disk was corrupted and there is no Backup of
> the Disk
>
> o   There is also no Backup of the Engine DB with version 4.2
>
> - VM disk was repaired with fsck.ext4, but DB is corrupt
>
> o   Can’t restore the Engine DB because the Backup DB from Engine V 4.1
>
> - Rolled back all changes on Engine VM to 4.1.9 and imported Backup
>
> o   Checked for HA VMs to set as disabled and started the Engine
>
> - Login is fine but the Engine is having trouble picking up and
> information from the Hosts
>
> o   No information on running VMs or hosts status
>
> - Final Situation
>
> o   2 Hosts have VMs still running and I can’t stop those
>
> o   I still have the image of my corrupted Engine VM (v4.2)
>
>
>
> Since there were no major changes after upgrading from 4.1 to 4.2, would it
> be possible to manually restore the 4.1 DB to the 4.2 Engine VM to this up
> and running again or are there modifications made to the DB on upgrading
> that are relevant for this ?

engine-backup requires restoring to the same version used to take the backup,
with a single exception - on 4.0, it can restore 3.6.

It's very easy to patch it to allow also 4.1->4.2, search inside it for
"VALID_BACKUP_RESTORE_PAIRS". However, I do not think anyone ever tested
this, so no idea might break. In 3.6->4.0 days, we did have to fix a few
other things, notably apache httpd and iptables->firewalld:

https://bugzilla.redhat.com/show_bug.cgi?id=1318580

> All my work on rolling back to 4.1.9 with the
> DB restore failed as the Engine is not capable of picking up information
> from the hosts.

No idea why, but not sure it's related to your restore flow.

> Lessons learned is to always make a copy/snapshot of the
> engine VM disk before upgrading anything.

If it's a hosted-engine, this isn't supported - see my reply on the
list ~ 1 hour ago...

> What are my options on getting
> back to a working environment ? Any help or hint is greatly appreciated.

Restore again with either methods - what you tried, or patching engine-backup
and restore directly into 4.2 - and if the engine fails to talk to the hosts,
try to debug/fix this.

If you suspect corruption more severe that just the db, you can install a
fresh engine machine from scratch and restore to it. If it's a hosted-engine,
you'll need to deploy hosted-engine from scratch, check docs about hosted-engine
backup/restore.

Best regards,
-- 
Didi
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[ovirt-users] Recovering oVirt-Engine with a backup before upgrading to 4.2

[Sven Achtelik]

Hi All,

I'm still facing issues with my HE engine. Here are the steps that I took to 
end up in this situation:


- Update Engine from 4.1.7 to 4.1.9

o   That worked as expected

- Automatic Backup of Engine DB in the night

- Upgraded Engine from 4.1.9 to 4.2.1

o   That worked fine

- Noticed Issues with the HA support for HE

o   Cause was not having the latest ovirt-ha agent/broker version on hosts

- After updating the first host with the latest packages for the 
Agent/Broker engine was started twice

o   As a result the Engine VM Disk was corrupted and there is no Backup of the 
Disk

o   There is also no Backup of the Engine DB with version 4.2

- VM disk was repaired with fsck.ext4, but DB is corrupt

o   Can't restore the Engine DB because the Backup DB from Engine V 4.1

- Rolled back all changes on Engine VM to 4.1.9 and imported Backup

o   Checked for HA VMs to set as disabled and started the Engine

- Login is fine but the Engine is having trouble picking up and 
information from the Hosts

o   No information on running VMs or hosts status

- Final Situation

o   2 Hosts have VMs still running and I can't stop those

o   I still have the image of my corrupted Engine VM (v4.2)

Since there were no major changes after upgrading from 4.1 to 4.2, would it be 
possible to manually restore the 4.1 DB to the 4.2 Engine VM to this up and 
running again or are there modifications made to the DB on upgrading that are 
relevant for this ? All my work on rolling back to 4.1.9 with the DB restore 
failed as the Engine is not capable of picking up information from the hosts. 
Lessons learned is to always make a copy/snapshot of the engine VM disk before 
upgrading anything. What are my options on getting back to a working 
environment ? Any help or hint is greatly appreciated.

Thank you,
Sven
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users