Paul
I did some more testing today and am not sure what some of the states mean.
The first test was the easiest ie. "echo c > /proc/sysrq-trigger" which crashes
the server. In my setup the VMs on the crashed node never migrate because the
server is rebooted and it comes backup before CS tries to migrate any servers.
It takes approx 4 mins for server to recover.
The next tests were by doing a hard reset on the server and then modifying
timers -
I did 4 tests and the quickest I got the VMs to failover was approx 5 and half
minutes (see below for test details).
So I have two questions really from all this -
1) why does it go from Suspect to Degraded and back to Suspect once I started
changing timers. According to the docs Degraded means a successful activity
check but the server was down so it can't have passed. And noticeably without
modifying any timers it never goes to Degraded at all.
2) what is a sensible fail over time in your experience ie. what in your
experience is a reasonable failover time ?
Thanks for any help you can give.
Tests -
1) default timers -
0:00 Suspect
9:00 recovery/Fenced
10:15 VM migrated
2) kvm.ha.activity.check.max.attempts 3 (default = 10)
0:00 Suspect
2:00 Degraded
7:00 Suspect
9:00 Recovery/Fenced
10:20 VM migrated
3) kvm.ha.activity.check.max.attempts 3 (default = 10)
kvm.ha.degraded.max.period 120 seconds (default = 300)
0:00 Suspect
2:00 Degraded
4:00 Suspect
6:00 Checking/Fenced
7:21 VM migrated
4) kvm.ha.activity.check.max.attempts 3 (default = 10)
kvm.ha.degraded.max.period 120 seconds (default = 300)
kvm.ha.activity.check.interval 30 seconds (default = 60)
0:00 Suspect
1:10 Degraded
3:10 Suspect
4:20 Recovering/Fenced
5:30 VM migrated
________________________________
From: Jon Marshall <[email protected]>
Sent: 29 March 2018 09:40
To: [email protected]
Subject: Re: Failover for VMs
Hi Paul
I did make some progress with this and seem to remember that after it said
Recovered it then went back to Suspect and finally Fenced.
I am going to rerun a lot of the tests after changing some of the kvm_ha_
timers to try and speed things up a bit.
Will update here after I have run tests to check if that is what I should be
seeing.
Many thanks
Jon
________________________________
From: Paul Angus <[email protected]>
Sent: 28 March 2018 20:01
To: [email protected]
Subject: Re: Failover for VMs
Ah.
Did you wait after the node said recovered?
That message is spurious. I've seen it also. It should say recovering. at
that time.
________________________________
From: Jon Marshall <[email protected]>
Sent: Tuesday, 27 March 2018 10:42 am
To: [email protected]
Subject: Re: Failover for VMs
Just as an update to this before I forget what I did :) -
I used "echo c /proc/sysrq-trigger" on one of the compute nodes and there was
no VM failover. Instead HA reported suspect and then IPMI rebooted the
machine, it came back online and the VM started responding to pings again.
IPMI is out of band so that seems to be reasonable behaviour but no use in
testing HA.
Next I just pulled all 3 NIC cables from the same compute node and again HA
reported suspect. Again IPMI rebooted but then HA state changed to "Recovered"
which I don't understand as the NIC cables were still disconnected so VM was
not reachable and no failover.
I don't understand how it can think the node is recovered as apart from the
IPMI out of band connection there are no network connections to this server.
Finally pulled power lead and this time HA went from suspect to Fencing and
then stayed that way. Again no VM failover. This makes sense as no power
means IPMI cannot reboot server so it never moves to Fenced I assume. Again no
failover.
I am wondering if it is to do with out of band IPMI or the way I have the NICs
setup. The management node only has one NIC in the management network but I
assume this is okay.
I may try reloading with CS v4.9 and just try failover without the new HA KVM
to see if I see anything different.
Jon
________________________________
From: Jon Marshall <[email protected]>
Sent: 27 March 2018 10:10
To: [email protected]
Subject: Re: Failover for VMs
Thanks Paul, will pick up after Easter break.
Doing some more testing with HA KVM at the moment so any progress will update
this thread
i
________________________________
From: Paul Angus <[email protected]
Sent: 27 March 2018 10:07
To: [email protected]
Subject: RE: Failover for VMr
Jon,
I've been updating the Ansible to move our physical hosts from Centos6 to
Centos7, now that's done I'll run through an HA setup and post answers
(probably after easter break).
[email protected]
www.shapeblue.com<http://www.shapeblue.com>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a
framework developed by ShapeBlue to deliver the rapid deployment of a
standardised CloudStack powered IaaS cloud for small production deployments, or
medium scale POCs or pilots.
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com<http://www.shapeblue.com>
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a
framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com<http://www.shapeblue.com>
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a
framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com<http://www.shapeblue.com>
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a
framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
[email protected]
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
-----Original Message-----
From: Jon Marshall <[email protected]>
Sent: 27 March 2018 09:19
To: [email protected]
Subject: Failover for VMs
After 3 weeks of trying multiple different setups I still have not managed to
get a VM to failover between compute nodes and am just running out of ideas.
I have 3 compute nodes each with 3 NICS (management, VMs traffic, storage), one
management node with just a single NIC connection in the management network and
a separate NFS server.
I have tried with and without the new Host HA KVM in CS v4.11 as from what I
have read even without enabling the new Host HA KVM when you power off or
reboot a compute node your VMs should still migrate.
I have tried powering off a compute node, pulling the power lead, removing the
management and NFS network cables and the management server just seems to carry
on as if nothing has happened.
Could someone explain exactly how HA is meant to work so I can look at where it
is going wrong.
