Milamber, What you mention is not a HA use case but live migration.
As mentioned by Luciano, he shutdown the KVM host "A". Mgmt server managing that host realized that and disconnected the agent for that host. I believe it is expected behavior for HA to kick in at this stage. But from the logs Luciano shared it doesn't seem to be happening. Luciano, I suggest tracking this further in the issue tracking system by raising a product bug. Regards, Somesh -----Original Message----- From: Milamber [mailto:milam...@apache.org] Sent: Saturday, July 18, 2015 12:54 PM To: users@cloudstack.apache.org Subject: Re: HA feature - KVM - CloudStack 4.5.1 On 17/07/2015 22:26, Somesh Naidu wrote: >> Perhaps, the management server don't reconize the host 3 totally down >> (ping alive? or some quorum don't ok) >> The only way to the mgt server to accept totally that the host 3 has a >> real problem that the host 3 has been reboot (around 12:44)? > The host disconnect was triggered at 12:19 on host 3. Mgmt server was pretty > sure the host is down (it was a graceful shutdown I believe) which is why it > triggered a disconnect and notified other nodes. There was no > checkhealth/checkonhost/etc. triggered; just the agent disconnected and all > listeners (ping/etc.) notified. > > At this time mgmt server should have scheduled HA on all VMs running on that > host. The HA investigators would then work their way identifying whether the > VMs are still running, if they need to be fenced, etc. But this never > happened. AFAIK, stopping the cloudstack-agent service don't allow to start the HA process for the VMs hosted by the node. Seems normal to me that the HA process don't start at this moment. If I would start the HA process on a node, I go to the Web UI (or cloudmonkey) to change the state of the Host from Up to Maintenance. (after I can stop the CS-agent service if I need for exemple reboot a node) > > Regards, > Somesh > > > -----Original Message----- > From: Milamber [mailto:milam...@apache.org] > Sent: Friday, July 17, 2015 6:01 PM > To: users@cloudstack.apache.org > Subject: Re: HA feature - KVM - CloudStack 4.5.1 > > > > On 17/07/2015 21:23, Somesh Naidu wrote: >> Ok, so here are my findings. >> >> 1. Host ID 3 was shutdown around 2015-07-16 12:19:09 at which point >> management server called a disconnect. >> 2. Based on the logs, it seems VM IDs 32, 18, 39 and 46 were running on the >> host. >> 3. No HA tasks for any of these VMs at this time. >> 5. Management server restarted at around 2015-07-16 12:30:20. >> 6. Host ID 3 connected back at around 2015-07-16 12:44:08. >> 7. Management server identified the missing VMs and triggered HA on those. >> 8. The VMs were eventually started, all 4 of them. >> >> I am not 100% sure why HA wasn't triggered until 2015-07-16 12:30 (#3), but >> I know that management server restart caused it not happen until the host >> was reconnected. > Perhaps, the management server don't reconize the host 3 totally down > (ping alive? or some quorum don't ok) > The only way to the mgt server to accept totally that the host 3 has a > real problem that the host 3 has been reboot (around 12:44)? > > What is the storage subsystem? CLVMd? > > >> Regards, >> Somesh >> >> >> -----Original Message----- >> From: Luciano Castro [mailto:luciano.cas...@gmail.com] >> Sent: Friday, July 17, 2015 12:13 PM >> To: users@cloudstack.apache.org >> Subject: Re: HA feature - KVM - CloudStack 4.5.1 >> >> No problems Somesh, thanks for your help. >> >> Link of log: >> >> https://dl.dropboxusercontent.com/u/6774061/management-server.log.2015-07-16.gz >> >> Luciano >> >> On Fri, Jul 17, 2015 at 12:00 PM, Somesh Naidu <somesh.na...@citrix.com> >> wrote: >> >>> How large is the management server logs dated 2015-07-16? I would like to >>> review the logs. All the information I need from that incident should be in >>> there so I don't need any more testing. >>> >>> Regards, >>> Somesh >>> >>> -----Original Message----- >>> From: Luciano Castro [mailto:luciano.cas...@gmail.com] >>> Sent: Friday, July 17, 2015 7:58 AM >>> To: users@cloudstack.apache.org >>> Subject: Re: HA feature - KVM - CloudStack 4.5.1 >>> >>> Hi Somesh! >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> [root@1q2 ~]# zgrep -i -E >>> >>> 'SimpleIvestigator|KVMInvestigator|PingInvestigator|ManagementIPSysVMInvestigator' >>> /var/log/cloudstack/management/management-server.log.2015-07-16.gz |tail >>> -5000 > /tmp/management.txt >>> [root@1q2 ~]# cat /tmp/management.txt >>> 2015-07-16 12:30:45,452 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:null) >>> Registering extension [KVMInvestigator] in [Ha Investigators Registry] >>> 2015-07-16 12:30:45,452 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:null) >>> Registered com.cloud.ha.KVMInvestigator@57ceec9a >>> 2015-07-16 12:30:45,927 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:null) >>> Registering extension [PingInvestigator] in [Ha Investigators Registry] >>> 2015-07-16 12:30:45,928 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:null) >>> Registering extension [ManagementIPSysVMInvestigator] in [Ha Investigators >>> Registry] >>> 2015-07-16 12:30:53,796 INFO [o.a.c.s.l.r.DumpRegistry] (main:null) >>> Registry [Ha Investigators Registry] contains [SimpleInvestigator, >>> XenServerInvestigator, KVMInv >>> >>> I searched this log before, but as I thought that had not nothing >>> special. >>> >>> If you want propose to me another scenario of test, I can do it. >>> >>> Thanks >>> >>> >>> On Thu, Jul 16, 2015 at 7:27 PM, Somesh Naidu <somesh.na...@citrix.com> >>> wrote: >>> >>>> What about other investigators, specifically " KVMInvestigator, >>>> PingInvestigator"? They report the VMs as alive=false too? >>>> >>>> Also, it is recommended that you look at the management-sever.log instead >>>> of catalina.out (for one, the latter doesn’t have timestamp). >>>> >>>> Regards, >>>> Somesh >>>> >>>> >>>> -----Original Message----- >>>> From: Luciano Castro [mailto:luciano.cas...@gmail.com] >>>> Sent: Thursday, July 16, 2015 1:14 PM >>>> To: users@cloudstack.apache.org >>>> Subject: Re: HA feature - KVM - CloudStack 4.5.1 >>>> >>>> Hi Somesh! >>>> >>>> >>>> thanks for help.. I did again ,and I collected new logs: >>>> >>>> My vm_instance name is i-2-39-VM. There was some routers in KVM host 'A' >>>> (this one that I powered off now): >>>> >>>> >>>> [root@1q2 ~]# grep -i -E 'SimpleInvestigator.*false' >>>> /var/log/cloudstack/management/catalina.out >>>> INFO [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-2:ctx-e2f91c9c >>> work-3) >>>> SimpleInvestigator found VM[DomainRouter|r-4-VM]to be alive? false >>>> INFO [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-1:ctx-729acf4f >>> work-7) >>>> SimpleInvestigator found VM[User|i-23-33-VM]to be alive? false >>>> INFO [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-4:ctx-a66a4941 >>> work-8) >>>> SimpleInvestigator found VM[DomainRouter|r-36-VM]to be alive? false >>>> INFO [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-1:ctx-5977245e >>>> work-10) SimpleInvestigator found VM[User|i-17-26-VM]to be alive? false >>>> INFO [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-1:ctx-c7f39be0 >>> work-9) >>>> SimpleInvestigator found VM[DomainRouter|r-32-VM]to be alive? false >>>> INFO [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-3:ctx-ad4f5fda >>>> work-10) SimpleInvestigator found VM[DomainRouter|r-46-VM]to be alive? >>>> false >>>> INFO [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-0:ctx-0257f5af >>>> work-11) SimpleInvestigator found VM[User|i-4-52-VM]to be alive? false >>>> INFO [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-4:ctx-7ddff382 >>>> work-12) SimpleInvestigator found VM[DomainRouter|r-32-VM]to be alive? >>>> false >>>> INFO [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-1:ctx-9f79917e >>>> work-13) SimpleInvestigator found VM[User|i-2-39-VM]to be alive? false >>>> >>>> >>>> >>>> KVM host 'B' agent log (where the machine would be migrate): >>>> >>>> 2015-07-16 16:58:56,537 INFO [kvm.resource.LibvirtComputingResource] >>>> (agentRequest-Handler-4:null) Live migration of instance i-2-39-VM >>>> initiated >>>> 2015-07-16 16:58:57,540 INFO [kvm.resource.LibvirtComputingResource] >>>> (agentRequest-Handler-4:null) Waiting for migration of i-2-39-VM to >>>> complete, waited 1000ms >>>> 2015-07-16 16:58:58,541 INFO [kvm.resource.LibvirtComputingResource] >>>> (agentRequest-Handler-4:null) Waiting for migration of i-2-39-VM to >>>> complete, waited 2000ms >>>> 2015-07-16 16:58:59,542 INFO [kvm.resource.LibvirtComputingResource] >>>> (agentRequest-Handler-4:null) Waiting for migration of i-2-39-VM to >>>> complete, waited 3000ms >>>> 2015-07-16 16:59:00,543 INFO [kvm.resource.LibvirtComputingResource] >>>> (agentRequest-Handler-4:null) Waiting for migration of i-2-39-VM to >>>> complete, waited 4000ms >>>> 2015-07-16 16:59:01,245 INFO [kvm.resource.LibvirtComputingResource] >>>> (agentRequest-Handler-4:null) Migration thread for i-2-39-VM is done >>>> >>>> It said done for my i-2-39-VM instance, but I can´t ping this host. >>>> >>>> Luciano >>>> >>> >>> -- >>> Luciano Castro >>> >>
