[ovirt-users] ovirt-engine service crash

[Andrew Dent]

Hi

I have a hosted-engine 4.1.2.2-1.el7.centos setup
The ovirt-engine service crashed and this is the result of querying the 
status of the service.
Restarting the service works, but then about 10 minutes later while 
using the GUI it will crash again.
I don't seem to need to restart apache or the webproxy services, though 
if I do restart those two, the ovirt-engine service will still crash ~10 
minutes later.
How can I fix this, and where to I look for more hints as to why this is 
happening?


Redirecting to /bin/systemctl status  ovirt-engine.service
● ovirt-engine.service - oVirt Engine
  Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service; enabled; 
vendor preset: disabled)
  Active: failed (Result: exit-code) since Sat 2017-08-12 21:01:52 AEST; 
2min 14s ago
 Process: 2028 
ExecStart=/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py 
--redirect-output --systemd=notify $EXTRA_ARGS start (code=exited, 
status=1/FAILURE)

Main PID: 2028 (code=exited, status=1/FAILURE)

Kind regards



Andrew Dent



___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Recovering hosted-engine

[Andrew Dent]

Hi Didi

Fair enough.
If I'm in this situation.
I have 3 hosts with 6 production VMs.
The hosted-engine VM is completely toast and not recoverable.
However I have a backup of the hosted-engine database (do I need 
anything else).


Is it possible to build a new VM, import the backup of the previous 
hosted-engine database and reconnect the storage domains and VMs in 
their running state without any VMs experiencing an outage?


The URL  
http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environment/

looks to be longer now.
I'll review, test and see if this will give me what I'm looking for.

The broken link still seems to be broken.
When I click the link, the browser ends up at this
http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environment/chap-Installing_Additional_Hosts_to_a_Self-Hosted_Environment#documentation%20self%20hosted%20chap%20Backing%20up%20and%20Restoring%20an%20EL%20Based%20Self%20Hosted%20Environment%20chap%20Installing%20Additional%20Hosts%20to%20a%20Self%20Hosted%20Environment

But I suspect it should be
http://www.ovirt.org/documentation/self-hosted/chap-Installing_Additional_Hosts_to_a_Self-Hosted_Environment/

Kind regards

Andrew

-- Original Message --
From: "Yedidyah Bar David" <d...@redhat.com>
To: "Andrew Dent" <ad...@ctcroydon.com.au>
Cc: "users" <users@ovirt.org>
Sent: 3/07/2017 11:12:05 PM
Subject: Re: [ovirt-users] Recovering hosted-engine

On Mon, Jul 3, 2017 at 3:46 PM, Andrew Dent <ad...@ctcroydon.com.au> 
wrote:
 Has anyone successfully completed a hosted-engine recovery on a 
multiple

 host setup with production VMs?


I'd like to clarify that "recovery" can span a large spectrum of
flows, from a trivial "I did some change to the engine database
that broke stuff and I want to restore a backup I took prior to
this change" to a full system restoration including purchasing
and deploying new (perhaps different) hosts/network/storage
hardware, including many other flows in between.

So when you plan for recovery, you should define very well what
flows you plan to handle, and how you handle each.

The linked procedure correctly says it's "providing an example".



 Kind regards


 Andrew



 -- Original Message --
 From: "Andrew Dent" <ad...@ctcroydon.com.au>
 To: "users" <users@ovirt.org>
 Sent: 2/07/2017 2:22:16 PM
 Subject: [ovirt-users] Recovering hosted-engine

 Hi

 A couple of questions about hosted-engine recovery.
 Part way through this URL, in the section "Workflow for Restoring the
 Self-Hosted Engine Environment"
 
http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environment/
 it looks like once the hosted-engine is recovered on Host 1, the VMs 
on Host
 2 and 3 will be running, but not accessible to the recovered Hosted 
Engine.

 Is that correct?


I am pretty certain that the procedure assumed that all hosts need 
restoration,

not that some are still up-and-running.

 If so, how to you remove host 2 and host 3 from the environment, then 
add

 back in again while keeping the VMs running?


That's a good question.

Please try to describe the exact flow you have in mind. What's broken 
and

needs restoration, and how do you plan to do that?



 Host 2 and Host 3 are not recoverable in their current state. These 
hosts

 need to be removed from the environment, and then added again to the
 environment using the hosted-engine deployment script. For more 
information
 on these actions, see the Removing Non-Operational Hosts from a 
Restored
 Self-Hosted Engine Environment section below and Chapter 7: 
Installing

 Additional Hosts to a Self-Hosted Environment.

 BTW: The link referring to chapter 7 is broken.


You are right. The link in the bottom of the page seems working.
Now pushed [1] to fix. Thanks for the report!

[1]

Best,
--
Didi


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Recovering hosted-engine

[Andrew Dent]

Has anyone successfully completed a hosted-engine recovery on a multiple 
host setup with production VMs?


Kind regards



Andrew



-- Original Message --
From: "Andrew Dent" <ad...@ctcroydon.com.au>
To: "users" <users@ovirt.org>
Sent: 2/07/2017 2:22:16 PM
Subject: [ovirt-users] Recovering hosted-engine


Hi

A couple of questions about hosted-engine recovery.
Part way through this URL, in the section "Workflow for Restoring the 
Self-Hosted Engine Environment"

http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environment/
it looks like once the hosted-engine is recovered on Host 1, the VMs on 
Host 2 and 3 will be running, but not accessible to the recovered 
Hosted Engine.

Is that correct?
If so, how to you remove host 2 and host 3 from the environment, then 
add back in again while keeping the VMs running?


Host 2 and Host 3 are not recoverable in their current state. These 
hosts need to be removed from the environment, and then added again to 
the environment using the hosted-engine deployment script. For more 
information on these actions, see the Removing Non-Operational Hosts 
from a Restored Self-Hosted Engine Environment section below and 
Chapter 7: Installing Additional Hosts to a Self-Hosted Environment 
<http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environment/chap-Installing_Additional_Hosts_to_a_Self-Hosted_Environment>.


BTW: The link referring to chapter 7 is broken.

Kind regards



Andrew

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[ovirt-users] Recovering hosted-engine

[Andrew Dent]

Hi

A couple of questions about hosted-engine recovery.
Part way through this URL, in the section "Workflow for Restoring the 
Self-Hosted Engine Environment"

http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environment/
it looks like once the hosted-engine is recovered on Host 1, the VMs on 
Host 2 and 3 will be running, but not accessible to the recovered Hosted 
Engine.

Is that correct?
If so, how to you remove host 2 and host 3 from the environment, then 
add back in again while keeping the VMs running?


Host 2 and Host 3 are not recoverable in their current state. These 
hosts need to be removed from the environment, and then added again to 
the environment using the hosted-engine deployment script. For more 
information on these actions, see the Removing Non-Operational Hosts 
from a Restored Self-Hosted Engine Environment section below and Chapter 
7: Installing Additional Hosts to a Self-Hosted Environment 
.


BTW: The link referring to chapter 7 is broken.

Kind regards



Andrew

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] hosted-engine VM and services not working

[Andrew Dent]

Hi Yaniv

I found a solution.
Our Ovirt 3.6 AIO box was still running and had those VMs still 
configured in their pre exported and switch off state.
I removed any snap shots I found from those pre exported VMs, then 
copied the disk image files and other bits from host01 (Ovirt v 4.1) 
back into the Ovirt 3.6 AIO box, and were needed fixing the relevent IDs 
to be what the Engine in the Ovirt 3.6 box expected.
The VMs then started up properly again without hassle and with the 
latest files on the Ovirt 3.6 AIO box.


So now in the progress of rebuilding host01 with hosted-engine v4.1

Kind regards



Andrew

-- Original Message --
From: "Yaniv Kaul" <yk...@redhat.com>
To: "Andrew Dent" <ad...@ctcroydon.com.au>
Cc: "users" <users@ovirt.org>
Sent: 18/06/2017 6:00:09 PM
Subject: Re: [ovirt-users] hosted-engine VM and services not working




On Sat, Jun 17, 2017 at 12:50 AM, <ad...@ctcroydon.com.au> wrote:
If I reinstall and the rerun the hosted-engine setup how do I get the 
VMs in their current running state back into and being recognised by 
the new hosted engine?


Current running state is again quite challenging. You'll need to fix 
the hosted-engine.


Can import the storage domain? (not for running VMs)
Y.



Kind regards

Andrew

On 17 Jun 2017, at 6:54 AM, Yaniv Kaul <yk...@redhat.com> wrote:




On Fri, Jun 16, 2017 at 9:11 AM, Andrew Dent <ad...@ctcroydon.com.au> 
wrote:

Hi

Well I've got myself into a fine mess.

host01 was setup with hosted-engine v4.1. This was successful.
Imported 3 VMs from a v3.6 OVirt AIO instance. (This OVirt 3.6 is 
still running with more VMs on it)
Tried to add host02 to the new Ovirt 4.1 setup. This partially 
succeeded but I couldn't add any storage domains to it. Cannot 
remember why.

In Ovirt engine UI I removed host02.
I reinstalled host02 with Centos7, tried to add it and Ovirt UI told 
me it was already there (but it wasn't listed in the UI).
Renamed the reinstalled host02 to host03, changed the ipaddress, 
reconfig the DNS server and added host03 into the Ovirt Engine UI.

All good, and I was able to import more VMs to it.
I was also able to shutdown a VM on host01 assign it to host03 and 
start the VM. Cool, everything working.

The above was all last couple of weeks.

This week I performed some yum updates on the Engine VM. No reboot.
Today noticed that the Ovirt services in the Engine VM were in a 
endless restart loop. They would be up for a 5 minutes and then die.
Looking into /var/log/ovirt-engine/engine.log and I could only see 
errors relating to host02. Ovirt was trying to find it and failing. 
Then falling over.
I ran "hosted-engine --clean-metadata" thinking it would cleanup and 
remove bad references to hosts, but now realise that was a really 
bad idea as it didn't do what I'd hoped.
At this point the sequence below worked, I could login to Ovirt UI 
but after 5 minutes the services would be off

service ovirt-engine restart
service ovirt-websocket-proxy restart
service httpd restart

I saw some reference to having to remove hosts from the database by 
hand in situations where under the hood of Ovirt a decommission host 
was still listed, but wasn't showing in the GUI.
So I removed reference to host02 (vds_id and host_id) in the 
following tables in this order.

vds_dynamic
vds_statistics
vds_static
host_device

Now when I try to start ovirt-websocket it will not start
service ovirt-websocket start
Redirecting to /bin/systemctl start  ovirt-websocket.service
Failed to start ovirt-websocket.service: Unit not found.

I'm now thinking that I need to do the following in the engine VM
# engine-cleanup # yum remove ovirt-engine # yum install 
ovirt-engine # engine-setup
But to run engine-cleanup I need to put the engine-vm into 
maintenance mode and because of the --clean-metadata that I ran 
earlier on host01 I cannot do that.


What is the best course of action from here?


To be honest, with all the steps taken above, I'd install everything 
(including OS) from scratch...

There's a bit too much mess to try to clean up properly here.
Y.



Cheers



Andrew


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users 
<http://lists.ovirt.org/mailman/listinfo/users>




___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[ovirt-users] hosted-engine VM and services not working

[Andrew Dent]

Hi

Well I've got myself into a fine mess.

host01 was setup with hosted-engine v4.1. This was successful.
Imported 3 VMs from a v3.6 OVirt AIO instance. (This OVirt 3.6 is still 
running with more VMs on it)
Tried to add host02 to the new Ovirt 4.1 setup. This partially succeeded 
but I couldn't add any storage domains to it. Cannot remember why.

In Ovirt engine UI I removed host02.
I reinstalled host02 with Centos7, tried to add it and Ovirt UI told me 
it was already there (but it wasn't listed in the UI).
Renamed the reinstalled host02 to host03, changed the ipaddress, 
reconfig the DNS server and added host03 into the Ovirt Engine UI.

All good, and I was able to import more VMs to it.
I was also able to shutdown a VM on host01 assign it to host03 and start 
the VM. Cool, everything working.

The above was all last couple of weeks.

This week I performed some yum updates on the Engine VM. No reboot.
Today noticed that the Ovirt services in the Engine VM were in a endless 
restart loop. They would be up for a 5 minutes and then die.
Looking into /var/log/ovirt-engine/engine.log and I could only see 
errors relating to host02. Ovirt was trying to find it and failing. Then 
falling over.
I ran "hosted-engine --clean-metadata" thinking it would cleanup and 
remove bad references to hosts, but now realise that was a really bad 
idea as it didn't do what I'd hoped.
At this point the sequence below worked, I could login to Ovirt UI but 
after 5 minutes the services would be off

service ovirt-engine restart
service ovirt-websocket-proxy restart
service httpd restart

I saw some reference to having to remove hosts from the database by hand 
in situations where under the hood of Ovirt a decommission host was 
still listed, but wasn't showing in the GUI.
So I removed reference to host02 (vds_id and host_id) in the following 
tables in this order.

vds_dynamic
vds_statistics
vds_static
host_device

Now when I try to start ovirt-websocket it will not start
service ovirt-websocket start
Redirecting to /bin/systemctl start  ovirt-websocket.service
Failed to start ovirt-websocket.service: Unit not found.

I'm now thinking that I need to do the following in the engine VM
# engine-cleanup # yum remove ovirt-engine # yum install ovirt-engine # 
engine-setup
But to run engine-cleanup I need to put the engine-vm into maintenance 
mode and because of the --clean-metadata that I ran earlier on host01 I 
cannot do that.


What is the best course of action from here?

Cheers



Andrew
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Ovirt Hosted-Engine VM iptables

[Andrew Dent]

I can add a rule into ipdates such as this
iptables -I INPUT -s 192.168.0.10 -p tcp -m tcp --dport 5666 -j ACCEPT
I can see the addition has succeeded with this
iptables-save > /etc/sysconfig/iptables

But a reboot of the Engine VM (not the Host) doesn't keep the new rule, 
and I was expecting that during bootup CentOS would read from 
/etc/sysconfig/iptables.

Alas it isn't.

Found a solution.
After reading this
https://stackoverflow.com/questions/24756240/how-can-i-use-iptables-on-centos-7
I installed iptables-services
But once installed I found that iptables -L showed no rules.
thankfully I still had the default hosted-engine rules in 
/etc/sysconfig/iptables

iptables-restore < /etc/sysconfig/iptables
Then.
service iptables save
restored the default hosted-engine rules including my rule for 5666.

Rebooting the hosted-engine VM and my rule 5666 for NRPE is still there.
Success!!

To answer your other questions

> Did you ask to configure the firewall during engine-setup?
Yes.
Looks like it setup firewalld for me.

> Alternatively, it's recommended to use firewalld.
For the moment I have disabled firewalld and are using iptablesIs 
there a reason why firewalld is preferred over iptables?


Kind regards

Andrew

-- Original Message --
From: "Yedidyah Bar David" <d...@redhat.com>
To: "Andrew Dent" <ad...@ctcroydon.com.au>
Cc: "users" <users@ovirt.org>
Sent: 29/05/2017 9:26:23 PM
Subject: Re: [ovirt-users] Ovirt Hosted-Engine VM iptables

On Mon, May 29, 2017 at 1:14 PM, Andrew Dent <ad...@ctcroydon.com.au> 
wrote:

 Hi

 I would like to add rules into the iptables of the Hosted Engine VM 
in

 Ovirt.
 I am wanting to monitor the Ovirt Engine using Nagios -> NRPE and I 
would

 like to open port 5666

 the version is oVirt Engine Version: 4.1.1.8-1.el7.centos
 I have tried using the normal process for iptables (iptables-save 
etc), but

 it seems that the file
 /etc/sysconfig/iptables
 is ignored when the Ovirt Engine VM starts.


What do you mean in "ignored"?

What's the output of 'iptables-save'?

Did you ask to configure the firewall during engine-setup?



 How can I add permanent iptables rules into the Engine VM?


On the engine vm (unlike hosts), the only thing that touches iptables
is engine-setup. Before doing that it asks you if you want to configure
the firewall. There aren't currently means to add your custom rules -
either you manage it all by yourself or you let engine-setup do that.

Alternatively, it's recommended to use firewalld. engine-setup can
add to firewalld the stuff it wants, and you still can add your own
stuff.

If I got you wrong and you refer to the hosts (not engine), see also:

https://www.ovirt.org/blog/2016/12/extension-iptables-rules-oVirt-hosts/

Best,



 Kind regards


 Andrew



 ___
 Users mailing list
 Users@ovirt.org
 http://lists.ovirt.org/mailman/listinfo/users





--
Didi


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[ovirt-users] Ovirt Hosted-Engine VM iptables

[Andrew Dent]

Hi

I would like to add rules into the iptables of the Hosted Engine VM in 
Ovirt.
I am wanting to monitor the Ovirt Engine using Nagios -> NRPE and I 
would like to open port 5666


the version is oVirt Engine Version: 4.1.1.8-1.el7.centos
I have tried using the normal process for iptables (iptables-save etc), 
but it seems that the file

/etc/sysconfig/iptables
is ignored when the Ovirt Engine VM starts.

How can I add permanent iptables rules into the Engine VM?

Kind regards



Andrew

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users