/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh comment the 6th from the bottom line: echo b > /proc/sysrq-trigger
That stops reboot process - been there, done that for years now :) But then investigate what is going on - since it's not "normal" to trigger this script (reboot actually) in healthy environment. FYI, there is no such heartbeat checks for CEPH, afaik. It's strange, that this also affects Secondary Storage NFS - I had idea it's only made for Primary Storage - actually I'm almost sure it's NOT checking Secondary Storage, since on PRimary NFS there is automatically created folder KVMHA, and inside it a file for each KVM node which should be updated (modification time) revery minute if KVM can access the NFS... Cheers On Tue, 27 Nov 2018 at 12:56, Adam Witwicki <awitwi...@oakfordis.com> wrote: > There seems to have been an issue with connectivity on a host that was > causing the VRs to silently fail. > > How can I stop this script rebooting when it detects an issue with NFS? > Would setting up another NFS pool work? > > Nov 27 08:23:41 OIS-MH-P1-C1-H1-C heartbeat: kvmheartbeat.sh rebooted > system because it was unable to write the heartbeat to the storage. > > > Thanks > > Adam > > -----Original Message----- > From: Adam Witwicki <awitwi...@oakfordis.com> > Sent: 27 November 2018 11:09 > To: users@cloudstack.apache.org > Subject: RE: Help > > ** This mail originated from OUTSIDE the Oakford corporate network. Treat > hyperlinks and attachments in this email with caution. ** > > Hi Swen > > I use ceph for primary storage > And NFS for secondary storage > > Using KVM hosts I got this error > Nov 27 08:23:41 OIS-MH-P1-C1-H1-C heartbeat: kvmheartbeat.sh rebooted > system because it was unable to write the heartbeat to the storage. > > I fixed the issue with NFS > And now a lot of instances are stuck in starting state, after the hosts > went into a reboot flap > > Thanks > > Adam > > -----Original Message----- > From: Swen - swen.io <m...@swen.io> > Sent: 27 November 2018 11:05 > To: users@cloudstack.apache.org > Subject: AW: Help > > ** This mail originated from OUTSIDE the Oakford corporate network. Treat > hyperlinks and attachments in this email with caution. ** > > Hi Adam, > > can you please explain a little bit more why your VR are stuck because of > your problem with secondary storage? Are you using the same storage as > primary storage for your VRs? > Is the problem already fixed with your secondary storage? Can you restart > a network? > > Cu Swen > > -----Ursprüngliche Nachricht----- > Von: Adam Witwicki <awitwi...@oakfordis.com> > Gesendet: Dienstag, 27. November 2018 11:12 > An: users@cloudstack.apache.org > Betreff: RE: Help > > > Hi Guys > > All my hosts went down due to a delay in writing to secondary nfs storage, > I have a lot of Virtual Routers in a stuck starting state. > > Any tips? > > Thanks > > Adam > > > > Disclaimer Notice: > This email has been sent by Oakford Technology Limited, while we have > checked this e-mail and any attachments for viruses, we can not guarantee > that they are virus-free. You must therefore take full responsibility for > virus checking. > This message and any attachments are confidential and should only be read > by those to whom they are addressed. If you are not the intended recipient, > please contact us, delete the message from your computer and destroy any > copies. Any distribution or copying without our prior permission is > prohibited. > Internet communications are not always secure and therefore Oakford > Technology Limited does not accept legal responsibility for this message. > The recipient is responsible for verifying its authenticity before acting > on the contents. Any views or opinions presented are solely those of the > author and do not necessarily represent those of Oakford Technology Limited. > Registered address: Oakford Technology Limited, 10 Prince Maurice Court, > Devizes, Wiltshire. SN10 2RT. > Registered in England and Wales No. 5971519 > > > > Disclaimer Notice: > This email has been sent by Oakford Technology Limited, while we have > checked this e-mail and any attachments for viruses, we can not guarantee > that they are virus-free. You must therefore take full responsibility for > virus checking. > This message and any attachments are confidential and should only be read > by those to whom they are addressed. If you are not the intended recipient, > please contact us, delete the message from your computer and destroy any > copies. Any distribution or copying without our prior permission is > prohibited. > Internet communications are not always secure and therefore Oakford > Technology Limited does not accept legal responsibility for this message. > The recipient is responsible for verifying its authenticity before acting > on the contents. Any views or opinions presented are solely those of the > author and do not necessarily represent those of Oakford Technology Limited. > Registered address: Oakford Technology Limited, 10 Prince Maurice Court, > Devizes, Wiltshire. SN10 2RT. > Registered in England and Wales No. 5971519 > > Disclaimer Notice: > This email has been sent by Oakford Technology Limited, while we have > checked this e-mail and any attachments for viruses, we can not guarantee > that they are virus-free. You must therefore take full responsibility for > virus checking. > This message and any attachments are confidential and should only be read > by those to whom they are addressed. If you are not the intended recipient, > please contact us, delete the message from your computer and destroy any > copies. Any distribution or copying without our prior permission is > prohibited. > Internet communications are not always secure and therefore Oakford > Technology Limited does not accept legal responsibility for this message. > The recipient is responsible for verifying its authenticity before acting > on the contents. Any views or opinions presented are solely those of the > author and do not necessarily represent those of Oakford Technology Limited. > Registered address: Oakford Technology Limited, 10 Prince Maurice Court, > Devizes, Wiltshire. SN10 2RT. > Registered in England and Wales No. 5971519 > > -- Andrija Panić
