RE: 4.11 without Host-HA framework

[Paul Angus]

Hi Jon,

Don't worry, TBH I'm dubious about those claiming to have VM-HA working when a 
host crashes (but doesn't restart).
I'll check in with the guys that set values for host-ha when testing, to see 
which ones they change and what they set them to. 

paul.an...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-Original Message-
From: Jon Marshall  
Sent: 23 May 2018 21:10
To: users@cloudstack.apache.org
Subject: Re: 4.11 without Host-HA framework

Rohit / Paul


Thanks again for answering.


I am a Cisco guy with an ex Unix background but no virtualisation experience 
and I can honestly say I have never felt this stupid before 


I have Cloudstack working but failover is killing me.


When you say VM HA relies on the host telling CS the VM is down how does that 
work because if you crash the host how does it tell CS anything ? And when you 
say tell CS do you mean the CS manager  ?


I guess I am just not understanding all the moving parts. I have had HOST HA 
working (to an extent) although it takes a long time to failover even after 
tweaking the timers but the fact that I keep finding references to people 
saying even without HOST HA it should failover (and mine doesn't) makes me 
think I have configured it incorrectly somewhere along the line.


I have configured a compute offering with HA and I am crashing the host with 
the echo command as suggested but still nothing.


I understand what you are saying Paul about it not being a good idea to rely on 
VM HA so I will go back to Host HA and try to speed up failover times.


Can I ask, from your experiences, what is a realistic fail over time for CS ie. 
if a host fails for example ?


Jon





From: Paul Angus 
Sent: 23 May 2018 19:55
To: users@cloudstack.apache.org
Subject: RE: 4.11 without Host-HA framework

Jon,

As Rohit says, it is very important to understand the difference between VM HA 
and host HA.
VM HA relies on the HOST telling CloudStack that the VM is down on order for 
CloudStack start it again (wherever that ends up being).
Any sequence of events that ends up with VM HA restarting the VM when 
CloudStack can't contact the host is luck/fluke/unreliable/bad(tm)

The purpose of Host HA was to create a reliable mechanism to determine that a 
host has 'crashed' and that the VMs within it are inoperative. Then take 
appropriate action, including ultimately telling VM HA to restart the VM 
elsewhere.





paul.an...@shapeblue.com
www.shapeblue.com
Shapeblue - The CloudStack Company
www.shapeblue.com
ShapeBlue are the largest independent integrator of CloudStack technologies 
globally and are specialists in the design and implementation of IaaS cloud 
infrastructures for both private and public cloud implementations.



53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue




-Original Message-
From: Rohit Yadav 
Sent: 23 May 2018 10:45
To: users@cloudstack.apache.org
Subject: Re: 4.11 without Host-HA framework

Jon,


In the VM's compute offering, make sure that HA is ticked/enabled. Then use 
that HA-enabled VM offering while deploying a VM. Around testing - it depends 
how you're crashing. In case of KVM, you can try to cause host crash (example: 
echo c > /proc/sysrq-trigger) and see if HA-enabled VMs gets started on a 
different host.


- Rohit






From: Jon Marshall 
Sent: Tuesday, May 22, 2018 8:28:06 PM
To: users@cloudstack.apache.org
Subject: Re: 4.11 without Host-HA framework

Hi Rohit


Thanks for responding.


I have not had much luck with HA at all.  I crash a server and nothing happens  
in terms of VMs migrating to another host. Monitoring the management log file 
it seems the management server recognises the host has stopped responding to 
pings but doesn't think it has to do anything.


I am currently running v4.11 with basic network but 3 separate NICs, one for 
management, one for storage and one for VMs themselves.


Should it make it any difference ie. would it be worth trying to run management 
and storage over the same NIC ?


I am just lost as to why I see no failover at all whereas others are reporting 
it works fine.


Jon



From: Rohit Yadav 
Sent: 22 May 2018 12:12
To: users@cloudstack.apache.org
Subject: Re: 4.11 without Host-HA framework

Hi Jon,


Yes, Host-HA is different from VM-HA and without Host HA enabled a HA enabled 
VM should be recovered/run on a different host when it crashes. Historically 
the term 'HA' in CloudStack is used around high availability of a VM.


Host HA as the name tries to imply is around HA of a physical hypervisor host 
by means of out-of-band management technologies such as ipmi and currently 

Re: 4.11 without Host-HA framework

[Jon Marshall]

Rohit / Paul


Thanks again for answering.


I am a Cisco guy with an ex Unix background but no virtualisation experience 
and I can honestly say I have never felt this stupid before 


I have Cloudstack working but failover is killing me.


When you say VM HA relies on the host telling CS the VM is down how does that 
work because if you crash the host how does it tell CS anything ? And when you 
say tell CS do you mean the CS manager  ?


I guess I am just not understanding all the moving parts. I have had HOST HA 
working (to an extent) although it takes a long time to failover even after 
tweaking the timers but the fact that I keep finding references to people 
saying even without HOST HA it should failover (and mine doesn't) makes me 
think I have configured it incorrectly somewhere along the line.


I have configured a compute offering with HA and I am crashing the host with 
the echo command as suggested but still nothing.


I understand what you are saying Paul about it not being a good idea to rely on 
VM HA so I will go back to Host HA and try to speed up failover times.


Can I ask, from your experiences, what is a realistic fail over time for CS ie. 
if a host fails for example ?


Jon





From: Paul Angus 
Sent: 23 May 2018 19:55
To: users@cloudstack.apache.org
Subject: RE: 4.11 without Host-HA framework

Jon,

As Rohit says, it is very important to understand the difference between VM HA 
and host HA.
VM HA relies on the HOST telling CloudStack that the VM is down on order for 
CloudStack start it again (wherever that ends up being).
Any sequence of events that ends up with VM HA restarting the VM when 
CloudStack can't contact the host is luck/fluke/unreliable/bad(tm)

The purpose of Host HA was to create a reliable mechanism to determine that a 
host has 'crashed' and that the VMs within it are inoperative. Then take 
appropriate action, including ultimately telling VM HA to restart the VM 
elsewhere.





paul.an...@shapeblue.com
www.shapeblue.com
Shapeblue - The CloudStack Company
www.shapeblue.com
ShapeBlue are the largest independent integrator of CloudStack technologies 
globally and are specialists in the design and implementation of IaaS cloud 
infrastructures for both private and public cloud implementations.



53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue




-Original Message-
From: Rohit Yadav 
Sent: 23 May 2018 10:45
To: users@cloudstack.apache.org
Subject: Re: 4.11 without Host-HA framework

Jon,


In the VM's compute offering, make sure that HA is ticked/enabled. Then use 
that HA-enabled VM offering while deploying a VM. Around testing - it depends 
how you're crashing. In case of KVM, you can try to cause host crash (example: 
echo c > /proc/sysrq-trigger) and see if HA-enabled VMs gets started on a 
different host.


- Rohit






From: Jon Marshall 
Sent: Tuesday, May 22, 2018 8:28:06 PM
To: users@cloudstack.apache.org
Subject: Re: 4.11 without Host-HA framework

Hi Rohit


Thanks for responding.


I have not had much luck with HA at all.  I crash a server and nothing happens  
in terms of VMs migrating to another host. Monitoring the management log file 
it seems the management server recognises the host has stopped responding to 
pings but doesn't think it has to do anything.


I am currently running v4.11 with basic network but 3 separate NICs, one for 
management, one for storage and one for VMs themselves.


Should it make it any difference ie. would it be worth trying to run management 
and storage over the same NIC ?


I am just lost as to why I see no failover at all whereas others are reporting 
it works fine.


Jon



From: Rohit Yadav 
Sent: 22 May 2018 12:12
To: users@cloudstack.apache.org
Subject: Re: 4.11 without Host-HA framework

Hi Jon,


Yes, Host-HA is different from VM-HA and without Host HA enabled a HA enabled 
VM should be recovered/run on a different host when it crashes. Historically 
the term 'HA' in CloudStack is used around high availability of a VM.


Host HA as the name tries to imply is around HA of a physical hypervisor host 
by means of out-of-band management technologies such as ipmi and currently 
supporting ipmi as OOBM and KVM hosts with NFS storage.


- Rohit


[https://cloudstack.apache.org/images/monkey-144.png]

Apache CloudStack: Open Source Cloud Computing
cloudstack.apache.org
CloudStack is open source cloud computing software for creating, managing, and 
deploying infrastructure cloud services







From: Jon Marshall 
Sent: Monday, May 21, 2018 8:36:04 PM
To: 

Re: 4.11 without Host-HA framework

[Rohit Yadav]

Jon,


In the VM's compute offering, make sure that HA is ticked/enabled. Then use 
that HA-enabled VM offering while deploying a VM. Around testing - it depends 
how you're crashing. In case of KVM, you can try to cause host crash (example: 
echo c > /proc/sysrq-trigger) and see if HA-enabled VMs gets started on a 
different host.


- Rohit






From: Jon Marshall 
Sent: Tuesday, May 22, 2018 8:28:06 PM
To: users@cloudstack.apache.org
Subject: Re: 4.11 without Host-HA framework

Hi Rohit


Thanks for responding.


I have not had much luck with HA at all.  I crash a server and nothing happens  
in terms of VMs migrating to another host. Monitoring the management log file 
it seems the management server recognises the host has stopped responding to 
pings but doesn't think it has to do anything.


I am currently running v4.11 with basic network but 3 separate NICs, one for 
management, one for storage and one for VMs themselves.


Should it make it any difference ie. would it be worth trying to run management 
and storage over the same NIC ?


I am just lost as to why I see no failover at all whereas others are reporting 
it works fine.


Jon



From: Rohit Yadav 
Sent: 22 May 2018 12:12
To: users@cloudstack.apache.org
Subject: Re: 4.11 without Host-HA framework

Hi Jon,


Yes, Host-HA is different from VM-HA and without Host HA enabled a HA enabled 
VM should be recovered/run on a different host when it crashes. Historically 
the term 'HA' in CloudStack is used around high availability of a VM.


Host HA as the name tries to imply is around HA of a physical hypervisor host 
by means of out-of-band management technologies such as ipmi and currently 
supporting ipmi as OOBM and KVM hosts with NFS storage.


- Rohit


[https://cloudstack.apache.org/images/monkey-144.png]

Apache CloudStack: Open Source Cloud Computing
cloudstack.apache.org
CloudStack is open source cloud computing software for creating, managing, and 
deploying infrastructure cloud services







From: Jon Marshall 
Sent: Monday, May 21, 2018 8:36:04 PM
To: users@cloudstack.apache.org
Subject: 4.11 without Host-HA framework

I keep seeing conflicting information about this in the mailing lists and in 
blogs etc.

If I run 4.11 without enabling Host HA framework should HA still work if I 
crash a compute node because my understanding was the new framework was added 
for certain cases only.

It doesn't work for me but I can find a number of people saying you don't need 
to enable the new framework for it to work.

Thanks

Jon

rohit.ya...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue




rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

RE: 4.11 without Host-HA framework

[Paul Angus]

I'm afraid that is not a host crash.  When shutting down the guest OS, the 
CloudStack agent on the host is still able to report to the management server 
that the VM has stopped.

This is my point. VM-HA relies on the management sever communication with the 
host agent.

Kind regards,

Paul Angus

paul.an...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-Original Message-
From: Yiping Zhang  
Sent: 24 May 2018 00:44
To: users@cloudstack.apache.org
Subject: Re: 4.11 without Host-HA framework

I can say for fact that VM's using a HA enabled service offering will be 
restarted by CS on another host, assuming there are enough capacity/resources 
in the cluster, when their original host crashes, regardless that host comes 
back or not.

The simplest way to test VM HA feature with a VM instance using HA enabled 
service offering is to issue shutdown command in guest OS, and watching it gets 
restarted by CS manager.

On 5/23/18, 1:23 PM, "Paul Angus"  wrote:

Hi Jon,

Don't worry, TBH I'm dubious about those claiming to have VM-HA working 
when a host crashes (but doesn't restart).
I'll check in with the guys that set values for host-ha when testing, to 
see which ones they change and what they set them to. 

paul.an...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-Original Message-
From: Jon Marshall  
Sent: 23 May 2018 21:10
To: users@cloudstack.apache.org
Subject: Re: 4.11 without Host-HA framework

Rohit / Paul


Thanks again for answering.


I am a Cisco guy with an ex Unix background but no virtualisation 
experience and I can honestly say I have never felt this stupid before 


I have Cloudstack working but failover is killing me.


When you say VM HA relies on the host telling CS the VM is down how does 
that work because if you crash the host how does it tell CS anything ? And when 
you say tell CS do you mean the CS manager  ?


I guess I am just not understanding all the moving parts. I have had HOST 
HA working (to an extent) although it takes a long time to failover even after 
tweaking the timers but the fact that I keep finding references to people 
saying even without HOST HA it should failover (and mine doesn't) makes me 
think I have configured it incorrectly somewhere along the line.


I have configured a compute offering with HA and I am crashing the host 
with the echo command as suggested but still nothing.


I understand what you are saying Paul about it not being a good idea to 
rely on VM HA so I will go back to Host HA and try to speed up failover times.


Can I ask, from your experiences, what is a realistic fail over time for CS 
ie. if a host fails for example ?


Jon





From: Paul Angus 
Sent: 23 May 2018 19:55
To: users@cloudstack.apache.org
Subject: RE: 4.11 without Host-HA framework

Jon,

As Rohit says, it is very important to understand the difference between VM 
HA and host HA.
VM HA relies on the HOST telling CloudStack that the VM is down on order 
for CloudStack start it again (wherever that ends up being).
Any sequence of events that ends up with VM HA restarting the VM when 
CloudStack can't contact the host is luck/fluke/unreliable/bad(tm)

The purpose of Host HA was to create a reliable mechanism to determine that 
a host has 'crashed' and that the VMs within it are inoperative. Then take 
appropriate action, including ultimately telling VM HA to restart the VM 
elsewhere.





paul.an...@shapeblue.com
www.shapeblue.com
Shapeblue - The CloudStack Company
www.shapeblue.com
ShapeBlue are the largest independent integrator of CloudStack technologies 
globally and are specialists in the design and implementation of IaaS cloud 
infrastructures for both private and public cloud implementations.



53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue




-Original Message-
From: Rohit Yadav 
Sent: 23 May 2018 10:45
To: users@cloudstack.apache.org
Subject: Re: 4.11 without Host-HA framework

Jon,


In the VM's compute offering, make sure that HA is ticked/enabled. Then use 
that HA-enabled VM offering while deploying a VM. Around testing - it depends 
how you're crashing. In case of KVM, you can try to cause host crash (example: 
echo c > /proc/sysrq-trigger) and see if HA-enabled VMs gets started on a 
different host.


- Rohit

Re: 4.11 without Host-HA framework

[Parth Patel]

Hi Jon and Angus,

I did not shutdown the VMs as Yiping Zhang said, but I have confirmed this
and discussed earlier in the users list that my HA-enabled VMs got started
on another suitable available host in the cluster even when I didn't have
IPMI-enabled hardware and did no configuration for OOBM and Host-HA. I
simply pulled the ethernet cable connecting the host to entire network (I
did use just one NIC) and according to the value set in ping timeout event,
the HA-enabled VMs were restarted on another available host. I tested the
scenario using both the scenarios: the echo command as well as good old
plugging out the NIC from the host. My VMs were successfully started on
another available host after CS manager confirmed they were not reachable.

I too want to understand how the failover mechanism in CloudStack actually
works. I used ACS 4.11 packages available here:
http://cloudstack.apt-get.eu/centos/7/4.11/

Regards,
Parth Patel


On Thu, 24 May 2018 at 10:53 Paul Angus  wrote:

> I'm afraid that is not a host crash.  When shutting down the guest OS, the
> CloudStack agent on the host is still able to report to the management
> server that the VM has stopped.
>
> This is my point. VM-HA relies on the management sever communication with
> the host agent.
>
> Kind regards,
>
> Paul Angus
>
> paul.an...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>
> -Original Message-
> From: Yiping Zhang 
> Sent: 24 May 2018 00:44
> To: users@cloudstack.apache.org
> Subject: Re: 4.11 without Host-HA framework
>
> I can say for fact that VM's using a HA enabled service offering will be
> restarted by CS on another host, assuming there are enough
> capacity/resources in the cluster, when their original host crashes,
> regardless that host comes back or not.
>
> The simplest way to test VM HA feature with a VM instance using HA enabled
> service offering is to issue shutdown command in guest OS, and watching it
> gets restarted by CS manager.
>
> On 5/23/18, 1:23 PM, "Paul Angus"  wrote:
>
> Hi Jon,
>
> Don't worry, TBH I'm dubious about those claiming to have VM-HA
> working when a host crashes (but doesn't restart).
> I'll check in with the guys that set values for host-ha when testing,
> to see which ones they change and what they set them to.
>
> paul.an...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>
> -Original Message-
> From: Jon Marshall 
> Sent: 23 May 2018 21:10
> To: users@cloudstack.apache.org
> Subject: Re: 4.11 without Host-HA framework
>
> Rohit / Paul
>
>
> Thanks again for answering.
>
>
> I am a Cisco guy with an ex Unix background but no virtualisation
> experience and I can honestly say I have never felt this stupid before 
>
>
> I have Cloudstack working but failover is killing me.
>
>
> When you say VM HA relies on the host telling CS the VM is down how
> does that work because if you crash the host how does it tell CS anything ?
> And when you say tell CS do you mean the CS manager  ?
>
>
> I guess I am just not understanding all the moving parts. I have had
> HOST HA working (to an extent) although it takes a long time to failover
> even after tweaking the timers but the fact that I keep finding references
> to people saying even without HOST HA it should failover (and mine doesn't)
> makes me think I have configured it incorrectly somewhere along the line.
>
>
> I have configured a compute offering with HA and I am crashing the
> host with the echo command as suggested but still nothing.
>
>
> I understand what you are saying Paul about it not being a good idea
> to rely on VM HA so I will go back to Host HA and try to speed up failover
> times.
>
>
> Can I ask, from your experiences, what is a realistic fail over time
> for CS ie. if a host fails for example ?
>
>
> Jon
>
>
>
>
> 
> From: Paul Angus 
> Sent: 23 May 2018 19:55
> To: users@cloudstack.apache.org
> Subject: RE: 4.11 without Host-HA framework
>
> Jon,
>
> As Rohit says, it is very important to understand the difference
> between VM HA and host HA.
> VM HA relies on the HOST telling CloudStack that the VM is down on
> order for CloudStack start it again (wherever that ends up being).
> Any sequence of events that ends up with VM HA restarting the VM when
> CloudStack can't contact the host is luck/fluke/unreliable/bad(tm)
>
> The purpose of Host HA was to create a reliable mechanism to determine
> that a host has 'crashed' and that the VMs within it are inoperative. Then
> take appropriate action, including ultimately telling VM HA to restart the
> VM elsewhere.
>
>
>
>
>
> paul.an...@shapeblue.com
> 

Re: KVM Problem by deploying VPC

[Benjamin Naber]

Hi Andrija,

first of all thanks for your reply. I have now testet the setup on a Ubuntu 
Xenial. Same issu with Default VPC Offering. Redundant VPC offering works also 
without any problems. Same as on CentOS.

See delow the debug log (censored public Ips):

2018-05-23 12:58:05,161 INFO  [cloud.agent.Agent] (agentRequest-Handler-2:null) 
(logid:0806b407) Proccess agent ready command, agent id = 15
2018-05-23 12:58:05,161 INFO  [cloud.agent.Agent] (agentRequest-Handler-2:null) 
(logid:0806b407) Set agent id 15
2018-05-23 12:58:05,162 INFO  [cloud.agent.Agent] (agentRequest-Handler-2:null) 
(logid:0806b407) Ready command is processed: agent id = 15
2018-05-23 12:58:05,162 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) 
(logid:0806b407) Seq 15-5506494969390563335:  { Ans: , MgmtId: 109952567336, 
via: 15, Ver: v1, Flags: 110, 
[{"com.cloud.agent.api.ReadyAnswer":{"result":true,"wait":0}}] }
2018-05-23 12:58:05,292 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) 
(logid:0806b407) Request:Seq 15-5506494969390563336:  { Cmd , MgmtId: 
109952567336, via: 15, Ver: v1, Flags: 100111, 
[{"com.cloud.agent.api.ReadyCommand":{"dcId":1,"hostId":15,"wait":0}}] }
2018-05-23 12:58:05,292 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) 
(logid:0806b407) Processing command: com.cloud.agent.api.ReadyCommand
2018-05-23 12:58:05,292 INFO  [cloud.agent.Agent] (agentRequest-Handler-3:null) 
(logid:0806b407) Proccess agent ready command, agent id = 15
2018-05-23 12:58:05,292 INFO  [cloud.agent.Agent] (agentRequest-Handler-3:null) 
(logid:0806b407) Set agent id 15
2018-05-23 12:58:05,293 INFO  [cloud.agent.Agent] (agentRequest-Handler-3:null) 
(logid:0806b407) Ready command is processed: agent id = 15
2018-05-23 12:58:05,294 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) 
(logid:0806b407) Seq 15-5506494969390563336:  { Ans: , MgmtId: 109952567336, 
via: 15, Ver: v1, Flags: 110, 
[{"com.cloud.agent.api.ReadyAnswer":{"result":true,"wait":0}}] }
2018-05-23 12:58:43,019 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null) 
(logid:ef8b353e) Request:Seq 15-5506494969390563337:  { Cmd , MgmtId: 
109952567336, via: 15, Ver: v1, Flags: 100111, 
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"6129c76d-67e5-4454-9bc9-4e9a5a152610","origUrl":"http://packages.shapeblue.com.s3-eu-west-1.amazonaws.com/systemvmtemplate/4.10/dnsmasq/systemvm64template-4.10-kvm.qcow2.bz2","uuid":"6129c76d-67e5-4454-9bc9-4e9a5a152610","id":3,"format":"QCOW2","accountId":1,"checksum":"bc2eac46f16a2ece6c19d4b89db41de3","hvm":false,"displayText":"SystemVM
 Template 
(KVM)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-b274-961fb337e716","id":14,"poolType":"RBD","host":"ceph-mon","path":"rbd","port":6789,"url":"RBD://ceph-mon/rbd/?ROLE=Primary=2258aa76-7813-354d-b274-961fb337e716","isManaged":false}},"name":"routing-3","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-b274-961fb337e716","id":14,"poolType":"RBD","host":"ceph-mon","path":"rbd","port":6789,"url":"RBD://ceph-mon/rbd/?ROLE=Primary=2258aa76-7813-354d-b274-961fb337e716","isManaged":false}},"name":"ROOT-37","size":349945344,"volumeId":37,"vmName":"r-37-VM","accountId":2,"format":"QCOW2","provisioningType":"THIN","id":37,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":true,"options":{},"options2":{},"wait":0}}]
 }
2018-05-23 12:58:43,020 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null) 
(logid:ef8b353e) Processing command: 
org.apache.cloudstack.storage.command.CopyCommand
2018-05-23 12:58:43,021 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-4:null) (logid:ef8b353e) Trying to fetch storage pool 
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:43,021 DEBUG [kvm.resource.LibvirtConnection] 
(agentRequest-Handler-4:null) (logid:ef8b353e) Looking for libvirtd connection 
at: qemu:///system
2018-05-23 12:58:43,032 DEBUG [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-4:null) (logid:ef8b353e) Succesfully refreshed pool 
2258aa76-7813-354d-b274-961fb337e716 Capacity: 1500336095232 Used: 22039007177 
Available: 1426988417024
2018-05-23 12:58:43,032 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-4:null) (logid:ef8b353e) Trying to fetch storage pool 
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:43,032 DEBUG [kvm.resource.LibvirtConnection] 
(agentRequest-Handler-4:null) (logid:ef8b353e) Looking for libvirtd connection 
at: qemu:///system
2018-05-23 12:58:43,041 DEBUG [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-4:null) (logid:ef8b353e) Succesfully refreshed pool 
2258aa76-7813-354d-b274-961fb337e716 Capacity: 1500336095232 Used: 22039007177 

Re: KVM Problem by deploying VPC

[Andrija Panic]

Hi Ben,

interesting parts seems to be:
2018-05-23 12:59:47,213 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) getting broadcast uri for
pif enp8s0f0.233 and bridge brenp8s0f0-233
2018-05-23 12:59:47,213 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Transforming
com.cloud.agent.api.routing.IpAssocVpcCommand to ConfigItems
2018-05-23 12:59:47,581 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Processing FileConfigItem,
copying 257 characters to ip_associations.json took 340ms
2018-05-23 12:59:47,582 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Executing:
/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
update_config.py 169.254.2.247 ip_associations.json
2018-05-23 12:59:47,766 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Exit value is 1

What I believe we see here, is that IP association fails for some reason
(exit value is 1) and after that ACS will just stop the VM and do the
cleanup (all log lines after this one)...

Can you check your broadcast URI in the DB ? ACtually... I see it vlan 223
- I did have some issues in past releases, when we wanted to use untagged
vlans for Public network, but it seems not to be case here...

Not sure, if it's possible that you also SSH to the VR during this creation
process, in order to collect logs from inside the VR - before Qemu destroys
the VR?

ssh -p 3922 -i .ssh/id_rsa.cloud root@ 169.254.2.247  and then try to fetch
(SCP or something) whole /var/log/ folder to the localhost - from there,
there is cloud.log and auth.log where most of the commands outputs are
located (success or failure)

or something like rsync -av -e  "ssh -p 3992 -i .ssh.id_rsa.cloud"
root@169...
/local/dir/
and keep repeating until you fetch whatever data to be able to analyze it

Perhaps someone else will have better suggestion...

Best
Andrija


On 23 May 2018 at 13:08, Benjamin Naber 
wrote:

> Hi Andrija,
>
> first of all thanks for your reply. I have now testet the setup on a
> Ubuntu Xenial. Same issu with Default VPC Offering. Redundant VPC offering
> works also without any problems. Same as on CentOS.
>
> See delow the debug log (censored public Ips):
>
> 2018-05-23 12:58:05,161 INFO  [cloud.agent.Agent]
> (agentRequest-Handler-2:null) (logid:0806b407) Proccess agent ready
> command, agent id = 15
> 2018-05-23 12:58:05,161 INFO  [cloud.agent.Agent]
> (agentRequest-Handler-2:null) (logid:0806b407) Set agent id 15
> 2018-05-23 12:58:05,162 INFO  [cloud.agent.Agent]
> (agentRequest-Handler-2:null) (logid:0806b407) Ready command is processed:
> agent id = 15
> 2018-05-23 12:58:05,162 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-2:null) (logid:0806b407) Seq 15-5506494969390563335:
> { Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 110,
> [{"com.cloud.agent.api.ReadyAnswer":{"result":true,"wait":0}}] }
> 2018-05-23 12:58:05,292 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-3:null) (logid:0806b407) Request:Seq
> 15-5506494969390563336:  { Cmd , MgmtId: 109952567336, via: 15, Ver: v1,
> Flags: 100111, [{"com.cloud.agent.api.ReadyCo
> mmand":{"dcId":1,"hostId":15,"wait":0}}] }
> 2018-05-23 12:58:05,292 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-3:null) (logid:0806b407) Processing command:
> com.cloud.agent.api.ReadyCommand
> 2018-05-23 12:58:05,292 INFO  [cloud.agent.Agent]
> (agentRequest-Handler-3:null) (logid:0806b407) Proccess agent ready
> command, agent id = 15
> 2018-05-23 12:58:05,292 INFO  [cloud.agent.Agent]
> (agentRequest-Handler-3:null) (logid:0806b407) Set agent id 15
> 2018-05-23 12:58:05,293 INFO  [cloud.agent.Agent]
> (agentRequest-Handler-3:null) (logid:0806b407) Ready command is processed:
> agent id = 15
> 2018-05-23 12:58:05,294 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-3:null) (logid:0806b407) Seq 15-5506494969390563336:
> { Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 110,
> [{"com.cloud.agent.api.ReadyAnswer":{"result":true,"wait":0}}] }
> 2018-05-23 12:58:43,019 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-4:null) (logid:ef8b353e) Request:Seq
> 15-5506494969390563337:  { Cmd , MgmtId: 109952567336, via: 15, Ver: v1,
> Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"
> srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":
> {"path":"6129c76d-67e5-4454-9bc9-4e9a5a152610","origUrl":"
> http://packages.shapeblue.com.s3-eu-west-1.amazonaws.com/sys
> temvmtemplate/4.10/dnsmasq/systemvm64template-4.10-kvm.qcow2.bz2
> ","uuid":"6129c76d-67e5-4454-9bc9-4e9a5a152610","id":
> 3,"format":"QCOW2","accountId":1,"checksum":"bc2eac46f16a2ec
> e6c19d4b89db41de3","hvm":false,"displayText":"SystemVM Template
> (KVM)","imageDataStore":{"org.apache.cloudstack.storage.to.P
> 

Re: 4.11 without Host-HA framework

[Yiping Zhang]

I can say for fact that VM's using a HA enabled service offering will be 
restarted by CS on another host, assuming there are enough capacity/resources 
in the cluster, when their original host crashes, regardless that host comes 
back or not.

The simplest way to test VM HA feature with a VM instance using HA enabled 
service offering is to issue shutdown command in guest OS, and watching it gets 
restarted by CS manager.

On 5/23/18, 1:23 PM, "Paul Angus"  wrote:

Hi Jon,

Don't worry, TBH I'm dubious about those claiming to have VM-HA working 
when a host crashes (but doesn't restart).
I'll check in with the guys that set values for host-ha when testing, to 
see which ones they change and what they set them to. 

paul.an...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-Original Message-
From: Jon Marshall  
Sent: 23 May 2018 21:10
To: users@cloudstack.apache.org
Subject: Re: 4.11 without Host-HA framework

Rohit / Paul


Thanks again for answering.


I am a Cisco guy with an ex Unix background but no virtualisation 
experience and I can honestly say I have never felt this stupid before 


I have Cloudstack working but failover is killing me.


When you say VM HA relies on the host telling CS the VM is down how does 
that work because if you crash the host how does it tell CS anything ? And when 
you say tell CS do you mean the CS manager  ?


I guess I am just not understanding all the moving parts. I have had HOST 
HA working (to an extent) although it takes a long time to failover even after 
tweaking the timers but the fact that I keep finding references to people 
saying even without HOST HA it should failover (and mine doesn't) makes me 
think I have configured it incorrectly somewhere along the line.


I have configured a compute offering with HA and I am crashing the host 
with the echo command as suggested but still nothing.


I understand what you are saying Paul about it not being a good idea to 
rely on VM HA so I will go back to Host HA and try to speed up failover times.


Can I ask, from your experiences, what is a realistic fail over time for CS 
ie. if a host fails for example ?


Jon





From: Paul Angus 
Sent: 23 May 2018 19:55
To: users@cloudstack.apache.org
Subject: RE: 4.11 without Host-HA framework

Jon,

As Rohit says, it is very important to understand the difference between VM 
HA and host HA.
VM HA relies on the HOST telling CloudStack that the VM is down on order 
for CloudStack start it again (wherever that ends up being).
Any sequence of events that ends up with VM HA restarting the VM when 
CloudStack can't contact the host is luck/fluke/unreliable/bad(tm)

The purpose of Host HA was to create a reliable mechanism to determine that 
a host has 'crashed' and that the VMs within it are inoperative. Then take 
appropriate action, including ultimately telling VM HA to restart the VM 
elsewhere.





paul.an...@shapeblue.com
www.shapeblue.com
Shapeblue - The CloudStack Company
www.shapeblue.com
ShapeBlue are the largest independent integrator of CloudStack technologies 
globally and are specialists in the design and implementation of IaaS cloud 
infrastructures for both private and public cloud implementations.



53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue




-Original Message-
From: Rohit Yadav 
Sent: 23 May 2018 10:45
To: users@cloudstack.apache.org
Subject: Re: 4.11 without Host-HA framework

Jon,


In the VM's compute offering, make sure that HA is ticked/enabled. Then use 
that HA-enabled VM offering while deploying a VM. Around testing - it depends 
how you're crashing. In case of KVM, you can try to cause host crash (example: 
echo c > /proc/sysrq-trigger) and see if HA-enabled VMs gets started on a 
different host.


- Rohit






From: Jon Marshall 
Sent: Tuesday, May 22, 2018 8:28:06 PM
To: users@cloudstack.apache.org
Subject: Re: 4.11 without Host-HA framework

Hi Rohit


Thanks for responding.


I have not had much luck with HA at all.  I crash a server and nothing 
happens  in terms of VMs migrating to another host. Monitoring the management 
log file it seems the management server recognises the host has stopped 
responding to pings but doesn't think it has 

RE: 4.11 without Host-HA framework

[Paul Angus]

Jon,

As Rohit says, it is very important to understand the difference between VM HA 
and host HA.
VM HA relies on the HOST telling CloudStack that the VM is down on order for 
CloudStack start it again (wherever that ends up being).
Any sequence of events that ends up with VM HA restarting the VM when 
CloudStack can't contact the host is luck/fluke/unreliable/bad(tm)

The purpose of Host HA was to create a reliable mechanism to determine that a 
host has 'crashed' and that the VMs within it are inoperative. Then take 
appropriate action, including ultimately telling VM HA to restart the VM 
elsewhere.





paul.an...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-Original Message-
From: Rohit Yadav  
Sent: 23 May 2018 10:45
To: users@cloudstack.apache.org
Subject: Re: 4.11 without Host-HA framework

Jon,


In the VM's compute offering, make sure that HA is ticked/enabled. Then use 
that HA-enabled VM offering while deploying a VM. Around testing - it depends 
how you're crashing. In case of KVM, you can try to cause host crash (example: 
echo c > /proc/sysrq-trigger) and see if HA-enabled VMs gets started on a 
different host.


- Rohit






From: Jon Marshall 
Sent: Tuesday, May 22, 2018 8:28:06 PM
To: users@cloudstack.apache.org
Subject: Re: 4.11 without Host-HA framework

Hi Rohit


Thanks for responding.


I have not had much luck with HA at all.  I crash a server and nothing happens  
in terms of VMs migrating to another host. Monitoring the management log file 
it seems the management server recognises the host has stopped responding to 
pings but doesn't think it has to do anything.


I am currently running v4.11 with basic network but 3 separate NICs, one for 
management, one for storage and one for VMs themselves.


Should it make it any difference ie. would it be worth trying to run management 
and storage over the same NIC ?


I am just lost as to why I see no failover at all whereas others are reporting 
it works fine.


Jon



From: Rohit Yadav 
Sent: 22 May 2018 12:12
To: users@cloudstack.apache.org
Subject: Re: 4.11 without Host-HA framework

Hi Jon,


Yes, Host-HA is different from VM-HA and without Host HA enabled a HA enabled 
VM should be recovered/run on a different host when it crashes. Historically 
the term 'HA' in CloudStack is used around high availability of a VM.


Host HA as the name tries to imply is around HA of a physical hypervisor host 
by means of out-of-band management technologies such as ipmi and currently 
supporting ipmi as OOBM and KVM hosts with NFS storage.


- Rohit


[https://cloudstack.apache.org/images/monkey-144.png]

Apache CloudStack: Open Source Cloud Computing
cloudstack.apache.org
CloudStack is open source cloud computing software for creating, managing, and 
deploying infrastructure cloud services







From: Jon Marshall 
Sent: Monday, May 21, 2018 8:36:04 PM
To: users@cloudstack.apache.org
Subject: 4.11 without Host-HA framework

I keep seeing conflicting information about this in the mailing lists and in 
blogs etc.

If I run 4.11 without enabling Host HA framework should HA still work if I 
crash a compute node because my understanding was the new framework was added 
for certain cases only.

It doesn't work for me but I can find a number of people saying you don't need 
to enable the new framework for it to work.

Thanks

Jon

rohit.ya...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue




rohit.ya...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue