Hi Paul
My testing does indeed end up with the failed host in maintenance mode but the
VMs are never migrated. As I posted earlier the management server seems to be
saying there is no other host that the VM can be migrated to.
Couple of questions if you have the time to respond -
1) this article seems to suggest a reboot or powering off a host will end
result in the VMs being migrated and this was on CS v 4.2.1 back in 2013 so
does Host HA do something different
2) Whenever one of my two nodes is taken down in testing the active compute
nodes HA status goes from Available to Ineligible. Should this happen ie. is it
going to Ineligible stopping the manager from migrating the VMs.
Apologies for all the questions but I just can't get this to work at the
moment. If I do eventually get it working I will do a write up for others with
same issue :)
________________________________
From: Paul Angus <paul.an...@shapeblue.com>
Sent: 14 March 2018 07:45
To: users@cloudstack.apache.org
Subject: RE: KVM HostHA
Hi Parth,
Two answer your questions, VM-HA does not restart VMs on an alternate host if
the original host goes down. The management server (without host-HA) cannot
tell what happened to the host. It cannot tell if there was a failure in the
agent, loss of connectivity to the management NIC or if the host is truly down.
In the first two scenarios, the guest VMs can still be running perfectly well,
and to restart them elsewhere would be very dangerous. Therefore, the correct
thing to do is - nothing but alert the operator. These scenarios are what
Host-HA was introduced for.
Wrt to STONITH, if no disk activity is detected on the host, host-HA will try
to restart (via IPMI) the host. If, after a configurable number of attempts,
the host agent still does not check in, then host-HA will shut down the host
(via IPMA), trigger VM-HA and mark the host as in-maintenance.
paul.an...@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a
framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
-----Original Message-----
From: Parth Patel <parthpatel2...@gmail.com>
Sent: 14 March 2018 05:05
To: users@cloudstack.apache.org
Subject: Re: KVM HostHA
Hi Paul,
Thanks for the clarification. I currently don't have an ipmi enabled hardware
(in test environment), but it will be beneficial if you can help me clear out
some basic concepts of it:
- If HA-enabled VMs are autostarted on another host when current host goes
down, what is the need or purpose of HA-host? (other than management server
able to remotely control it's power interfaces)
- I understood the "Shoot-the-other-node-in-the-head" (STONITH) approach ACS
uses to fence the host, but I couldn't find what mechanism or events trigger this?
Thanks and regards,
Parth Patel
On Wed, 14 Mar 2018 at 02:22 Paul Angus <paul.an...@shapeblue.com> wrote:
The management server doesn't ping the host through IPMI. However if
IPMI is not available, you will not be able to use Host HA, as there
is no way for CloudStack to 'fence' the host - that is shut it down to
be sure that a VM cannot start again on that host.
I can explain why that is necessary if you wish.
Kind regards,
Paul Angus
paul.an...@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a
framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
-----Original Message-----
From: Parth Patel <parthpatel2...@gmail.com>
Sent: 13 March 2018 16:57
To: users@cloudstack.apache.org
Cc: Jon Marshall <jms....@hotmail.co.uk>
Subject: Re: KVM HostHA
Hi Jon and Victor,
I think the management server pings your host using ipmi (I really don't
hope this is the case).
In my case, I did not have OOBM enabled at all (my hardware didn't support
it)
I think you could disable OOBM and/or HA-Host and give that a try :)
On Tue, 13 Mar 2018 at 20:40 victor <vic...@ihnetworks.com> wrote:
Hello Guys,
I have tried the following two cases.
1, "echo c > /proc/sysrq-trigger"
2, Pulled the network cable of one of the host
In both cases, the following happened.
=====
2018-03-13 08:22:54,978 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentTaskPool-15:ctx-c8d9f5d2) (logid:c0a3d2da) Notifying other nodes
of to disconnect
2018-03-13 08:22:54,983 INFO [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-16:ctx-d8204625) (logid:ffe4a426) Host 4 is
disconnecting with event AgentDisconnected
2018-03-13 08:22:54,985 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-16:ctx-d8204625) (logid:ffe4a426) Host 4 is already
Alert
2018-03-13 08:22:54,985 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-16:ctx-d8204625) (logid:ffe4a426) Deregistering link
for
4 with state Alert
2018-03-13 08:22:54,985 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-16:ctx-d8204625) (logid:ffe4a426) Remove Agent : 4
=====
But nothing happened for the vm's in that node. I have waited for one
hour and the VM's in that node has been migrated to the other
available hosts. I think the issue is that the management server still
thinks that the VM's in that host is running. Please check the
following logs
=======
2018-03-13 11:08:25,882 DEBUG [c.c.c.CapacityManagerImpl]
(CapacityChecker:ctx-1d8378af) (logid:ae906a50) Found 1 VMs on host 4
2018-03-13 11:08:25,888 DEBUG [c.c.c.CapacityManagerImpl]
(CapacityChecker:ctx-1d8378af) (logid:ae906a50) Found 0 VM, not
running on host 4 ========
On 03/13/2018 04:20 PM, Jon Marshall wrote:
I tried "echo c > /proc/sysrq-trigger" which stopped me getting into
the
server but it did not stop the server responding to an ipmitool
request on the manager eg -
"ipmitool -I lanplus -H 172.16.7.29 -U admin3 -P letmein chassis
status"
from the management server got an answer saying the chassis power
was on
so CS never registered the compute node as down.
I am obviously doing something wrong but cannot work it out.
The management server has one NIC - 172.16.7.4
Each compute node has 3 NICs -
cnode1
cnode2
mangement NIC 172.16.7.5 172.16.7.6
vm NIC 172.16.6.130 172.16.6.131
storage - 172.16.250.4 172.16.250.5
Dell LOM (for Idrac) 172.16.7.29 172.16.7.30
the dell LOM IPs are the ones used to configure OOBM in the UI
If I pull the storage NIC presumably nothing will happen as the
ipmitool
check is running across the management NIC so I need to pull both ?
My understanding of host HA was the management server monitored the
compute nodes using ipmitool and if it did not get a response because
the host was down it would fence off that host and move the VMs to an
active compute node.
This is obviously too simplistic so could someone explain how it is
meant to work and what it is protecting against ?
________________________________
From: Paul Angus <paul.an...@shapeblue.com>
Sent: 13 March 2018 07:01
To: users@cloudstack.apache.org
Subject: RE: KVM HostHA
Hi all,
One small note, unplugging the management NIC will only cause an HA
event if the storage is running over that NIC also.
Is the storage is over a separate NIC then, the guest VMs will
continue
to run when the mgmt. NIC is unplugged, Host HA will detect the disk
activity and conclude that there is nothing it can do, as the VMs are
still running other than mark the hosts as degraded.
Kind regards,
Paul Angus
paul.an...@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a
framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]
]<
http://www.shapeblue.com/>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a
framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
Shapeblue - The CloudStack Company
<https://maps.google.com/?q=ack+Company+%0D%0A%3E+%3E+w&entry=gmail&source=g>
<http://www.shapeblue.com/>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a
framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
www.shapeblue.com<http://www.shapeblue.com>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a
framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
Rapid deployment framework for Apache CloudStack IaaS Clouds.
CSForge is
a framework developed by ShapeBlue to deli
<https://maps.google.com/?q=is+a+framework+developed+by+ShapeBlue+to+d
eli&entry=gmail&source=g>ver the rapid deployment of a standardised
...
53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
-----Original Message-----
From: Parth Patel <parthpatel2...@gmail.com>
Sent: 12 March 2018 17:35
To: users@cloudstack.apache.org
Subject: Re: KVM HostHA
Hi Jon,
As I said, in my case, making the host HA didn't work but by just
having a HA VM running on host and executing - (WARNING) "echo c >
/proc/sysrq-trigger" to simulate a kernel crash on host, the
management server registered it as down and started the VM on another
host. I know I've suggested this before but I insist you give this a
try. Also, you don't need to completely power off the machine manually
but just plugging out the network cable works fine. The cloudstack
agent after losing connection to management server auto reboots
because of KVM heartbeat check shell script mentioned by Rohit Yadav
to one of my earlier queries in other thread.
On Mon 12 Mar, 2018, 21:23 Jon Marshall, <jms....@hotmail.co.uk>
wrote:
Hi Paul
Thanks for the response.
I think I am not understanding how it was meant to work then. My
understanding was that the manager used ipmitool to just keep querying
the compute nodes as to their status so I assumed it didn't matter how
you shut the node down, once it was down the manager would get no
response and mark it as down (which it does).
I am in testing mode so I think I will just go and pull the power and
see what happens :)
Thanks
Jon
________________________________
From: Paul Angus <paul.an...@shapeblue.com>
Sent: 12 March 2018 15:31
To: users@cloudstack.apache.org
Subject: RE: KVM HostHA
Hi Jon,
I think that what you guys are finding, is that a controlled host
shutdown, which will cause the agent to shutdown cleanly; Is not
considered an HA event. I wouldn't expect CloudStack to take any
action if you shut down a host, only if the host (agent) stops
responding.
Kind regards,
Paul Angus
paul.an...@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a
framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]
]<
http://www.shapeblue.com/>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a
framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a
framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
www.shapeblue.com<http://www.shapeblue.com>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a
framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge
is
a framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]
]
]<
http://www.shapeblue.com/>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a
framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]
]<
http://www.shapeblue.com/>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a
framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a
framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
www.shapeblue.com<http://www.shapeblue.com>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a
framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge
is
a framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a
framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]
]<
http://www.shapeblue.com/>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a
framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a
framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
www.shapeblue.com<http://www.shapeblue.com>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a
framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge
is
a framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
www.shapeblue.com<http://www.shapeblue.com>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a
framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]
]<
http://www.shapeblue.com/>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a
framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a
framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
www.shapeblue.com<http://www.shapeblue.com>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a
framework developed by ShapeBlue to deliver the rapid deployment of a
standardised ...
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge
is
a framework developed by ShapeBlue to deliver
<
https://maps.google.com/?q=framework+developed+by+ShapeBlue+to+deliver&entry=gmail&source=g
the rapid deployment of a standardised ...
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge
is a framework developed by ShapeBlue to deliver the rapid deployment
of a standardised ...
53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
-----Original Message-----
From: Jon Marshall <jms....@hotmail.co.uk>
Sent: 12 March 2018 15:15
To: users@cloudstack.apache.org
Subject: Re: KVM HostHA
I have the same issue here and am not entirely sure what the behaviour
should be.
I have one manager node and 2 compute nodes running 4.11 with ipmi
working
correctly.
From the UI under HA -
HA Enabled Yes
HA State Available
HA Provider kvmhaprovider
although interestingly from the "Details" tab it shows -
HA enabled No
which I assume is a cosmetic issue ?
On each compute node I have one HA enabled VM and one non HA enabled
VM.
I power off a compute node and the UI updates the host status and the
VMs
on that node stop responding but they never fail over to the other
node.
Couple of things I noticed -
1) as soon as i power off the compute node the HA state on the other
node
shows "Ineligible"
2) In the UI the instances all still show as green even though two of
them
are not available
Any help much appreciated
________________________________
From: victor <vic...@ihnetworks.com>
Sent: 07 March 2018 17:01
To: users@cloudstack.apache.org
Subject: KVM HostHA
Hello Guys,
I have installed cloudstack 4.11. I have enabled HA for each hosts I
have
added. I have also added ipmi successfully (using ipmi driver).
The hosts are showing like the following.
=======
HA Enabled Yes
HA State Available
HA Provider kvmhaprovider
======
Also the host is showing the following correctly
Resource state --> Enabled
State --> UP
Power state --> On
So I have shutdown one of the hosts to see how the KVM hosts Ha is
working. I have waited for half an hour. But nothing has happened.
What
will happen to the VM's in that host, if the host failed to back up.
There isn't much from logs.
Regards
Victor
