ah... Thanks Andrija that's good to know!
much appreciated Brian -----Original Message----- From: Andrija Panic <andrija.pa...@gmail.com<mailto:andrija%20panic%20%3candrija.pa...@gmail.com%3e>> Reply-To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org> To: users <users@cloudstack.apache.org<mailto:users%20%3cus...@cloudstack.apache.org%3e>> Subject: Re: Management server reboot appears to cause vms on other hosts to shutdown? Date: Mon, 28 Jun 2021 19:12:32 +0200 CAUTION ! This email was NOT sent using a University of Chester account, so we are unable to verify the identity of the sender. Do not click links or open attachments unless you recognise the sender and know the content is safe. ===== Each (KVM) cloudstacl agent executes a heartbeat script (every 1min or so), trying to write to a primary NFS server, ensuring it has the connectivity working. If this heartbeat fails - agent will say - "ok I'm stupid and less reliable, NFS storage must be HA, so let me reboot myself/host in order to regain access to NFS" (which is on your mgmt server) /usr/lib64/cloud/agent/scripts/vm/hypervisor/kvm/kvmheartbeat.sh Comment the line " echo b > /proc/sysrq-trigger " and KVM agent will NOT reboot itself when NFS is unavailable. Something that is also wise to do in production as well (trust me.... ... ) Cheers, On Mon, 28 Jun 2021 at 18:57, Brian Fitzpatrick < <mailto:b.fitzpatr...@chester.ac.uk> b.fitzpatr...@chester.ac.uk > wrote: Hi Andrija, Thanks for the reply There is NFS primary storage on the management server but it doesn't have any vm volumes on it, they are on other NFS primary servers. So I thought I should have been safe to reboot All vms on other hosts, vm volumes on other nfs shares living on other servers, system vms and routers also on other servers So all (I am fairly certain) that was on the server I rebooted was the one and only Management server and mysql Thanks Brian -----Original Message----- From: Andrija Panic < <mailto:andrija.pa...@gmail.com> andrija.pa...@gmail.com <mailto: <mailto:andrija%20panic%20%3candrija.pa...@gmail.com> andrija%20panic%20%3candrija.pa...@gmail.com %3e>> Reply-To: <mailto:users@cloudstack.apache.org> users@cloudstack.apache.org <mailto: <mailto:users@cloudstack.apache.org> users@cloudstack.apache.org > To: users < <mailto:users@cloudstack.apache.org> users@cloudstack.apache.org <mailto: <mailto:users%20%3cus...@cloudstack.apache.org> users%20%3cus...@cloudstack.apache.org %3e>> Subject: Re: Management server reboot appears to cause vms on other hosts to shutdown? Date: Mon, 28 Jun 2021 17:44:57 +0200 CAUTION ! This email was NOT sent using a University of Chester account, so we are unable to verify the identity of the sender. Do not click links or open attachments unless you recognise the sender and know the content is safe. ===== NFS primary storage also on your management server? On Sat, 26 Jun 2021 at 10:36, Brian Fitzpatrick < <mailto: <mailto:b.fitzpatr...@chester.ac.uk> b.fitzpatr...@chester.ac.uk > <mailto:b.fitzpatr...@chester.ac.uk> b.fitzpatr...@chester.ac.uk wrote: Hi Jordan, Thanks for your reply. Apologies I might not have been clear. The management server is aware of the vm and when I set the host that is also the same server that is running the management server (and mysql) into maintenance mode, I can see it not longer has any running vms on it. They have migrated to other hosts. Cloud stack can see them. Bu when I then to an apt update and reboot the management server, the vms on the other hosts seem to have shutdown. The reboot did take a while (15-20 mins), but I am surprised that it has affected other kvm hosts, which I thought should just carry on running. Unless I have missed something that was still on the management(and mysql) server. Thanks Brian -----Original Message----- From: Yordan Kostov < <mailto: <mailto:yord...@nsogroup.com> yord...@nsogroup.com > <mailto:yord...@nsogroup.com> yord...@nsogroup.com <mailto: <mailto: <mailto:yordan%20kostov%20%3cyord...@nsogroup.com> yordan%20kostov%20%3cyord...@nsogroup.com > <mailto:yordan%20kostov%20%3cyord...@nsogroup.com> yordan%20kostov%20%3cyord...@nsogroup.com %3e>> Reply-To: <mailto: <mailto:users@cloudstack.apache.org> users@cloudstack.apache.org > <mailto:users@cloudstack.apache.org> users@cloudstack.apache.org <mailto: <mailto: <mailto:users@cloudstack.apache.org> users@cloudstack.apache.org > <mailto:users@cloudstack.apache.org> users@cloudstack.apache.org To: <mailto: <mailto:users@cloudstack.apache.org> users@cloudstack.apache.org > <mailto:users@cloudstack.apache.org> users@cloudstack.apache.org < <mailto: <mailto:users@cloudstack.apache.org> users@cloudstack.apache.org > <mailto:users@cloudstack.apache.org> users@cloudstack.apache.org <mailto:% <mailto: <mailto:22us...@cloudstack.apache.org> 22us...@cloudstack.apache.org > <mailto:22us...@cloudstack.apache.org> 22us...@cloudstack.apache.org <mailto: <mailto:%22%20%3cus...@cloudstack.apache.org> %22%20%3cus...@cloudstack.apache.org > <mailto:%22%20%3cus...@cloudstack.apache.org> %22%20%3cus...@cloudstack.apache.org %3e>> Subject: RE: Management server reboot appears to cause vms on other hosts to shutdown? Date: Fri, 25 Jun 2021 09:10:44 +0000 CAUTION ! This email was NOT sent using a University of Chester account, so we are unable to verify the identity of the sender. Do not click links or open attachments unless you recognise the sender and know the content is safe. ===== Hello Brian, May be I did not understand very well but from what you say I get that the management server + SQL and NFS are on the same physical hosts that are being managed by cloudstack? If those VMs are not visible in Cloudstack, the system is not aware that they exist so it wont try to roll them to another host if you perform hypervisor host reboot. Best regards, Jordan -----Original Message----- From: Brian Fitzpatrick < <mailto: <mailto: <mailto:b.fitzpatr...@chester.ac.uk> b.fitzpatr...@chester.ac.uk > <mailto:b.fitzpatr...@chester.ac.uk> b.fitzpatr...@chester.ac.uk <mailto: <mailto:b.fitzpatr...@chester.ac.uk> b.fitzpatr...@chester.ac.uk > <mailto:b.fitzpatr...@chester.ac.uk> b.fitzpatr...@chester.ac.uk Sent: Friday, June 25, 2021 12:06 PM To: <mailto: <mailto: <mailto:users@cloudstack.apache.org> users@cloudstack.apache.org > <mailto:users@cloudstack.apache.org> users@cloudstack.apache.org <mailto: <mailto:users@cloudstack.apache.org> users@cloudstack.apache.org > <mailto:users@cloudstack.apache.org> users@cloudstack.apache.org Subject: Management server reboot appears to cause vms on other hosts to shutdown? [X] This message came from outside your organization Hi all, Still relatively new to CloudStack and learning, testing etc. I have created 1 management server with mysql on it and created 2 clusters with a nfs primary storage server in each and a number of hosts in each. I have been working through the servers, putting them in maintenance mode (noting the vm migrations), updating and rebooting them. All working fine I then wanted to update and reboot the server running the management and mysql. It is also a host, so I set it in maintenance mode so no vms running on it. I thought if I update it and reboot, all I would lose for a period of time was access to the management server, the vms should keep running on their various hosts The reboot, took longer than usual, it seemed to hang for 15-20mins before shutting down and rebooting. To my surprise though I lost contact to all the vms on the other hosts. They all shut down. Apologies, if I have missed something here, I thought I understood. All virtual routers and system vms appeared to be running on the other hosts. Is it because the management server took a while to reboot, the other hosts have lost contact and shutdown their vms? seems odd? Any suggestions, help welcome. As I say, still learning! Thanks Brian -- Andrija Panić -- Andrija Panić
