Hi Vikash, Greg submit the spec [1] for intrusive instance monitoring. Your review will be highly appreciated.. [1] https://review.openstack.org/#/c/469070/ --- Regards, Sampath
On Sat, May 20, 2017 at 4:49 PM, Vikash Kumar <vikash.ku...@oneconvergence.com> wrote: > Thanks Sam > > > On Sat, 20 May 2017, 06:51 Sam P, <sam47pr...@gmail.com> wrote: >> >> Hi Vikash, >> Great... I will add you as reviewer to this spec. >> Thank you.. >> --- Regards, >> Sampath >> >> >> >> On Fri, May 19, 2017 at 1:06 PM, Vikash Kumar >> <vikash.ku...@oneconvergence.com> wrote: >> > Hi Greg, >> > >> > Please include my email in this spec also. We are also dealing with >> > HA >> > of Virtual Instances (especially for Vendors) and will participate. >> > >> > On Thu, May 18, 2017 at 11:33 PM, Waines, Greg >> > <greg.wai...@windriver.com> >> > wrote: >> >> >> >> Yes I am good with writing spec for this in masakari-spec. >> >> >> >> >> >> >> >> Do you use gerrit for this git ? >> >> >> >> Do you have a template for your specs ? >> >> >> >> >> >> >> >> Greg. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> From: Sam P <sam47pr...@gmail.com> >> >> Reply-To: "openstack-dev@lists.openstack.org" >> >> <openstack-dev@lists.openstack.org> >> >> Date: Thursday, May 18, 2017 at 1:51 PM >> >> To: "openstack-dev@lists.openstack.org" >> >> <openstack-dev@lists.openstack.org> >> >> Subject: Re: [openstack-dev] [vitrage] [nova] [HA] [masakari] VM >> >> Heartbeat >> >> / Healthcheck Monitoring >> >> >> >> >> >> >> >> Hi Greg, >> >> >> >> Thank you Adam for followup. >> >> >> >> This is new feature for masakari-monitors and think Masakari can >> >> >> >> accommodate this feature in masakari-monitors. >> >> >> >> From the implementation prospective, it is not that hard to do. >> >> >> >> However, as you can see in our Boston presentation, Masakari will >> >> >> >> replace its monitoring parts ( which is masakari-monitors) with, >> >> >> >> nova-host-alerter, **-process-alerter, and **-instance-alerter. (** >> >> >> >> part is not defined yet..:p)... >> >> >> >> Therefore, I would like to save this specifications, and make sure we >> >> >> >> will not miss anything in the transformation.. >> >> >> >> Does is make sense to write simple spec for this in masakari-spec [1]? >> >> >> >> So we can discuss about the requirements how to implement it. >> >> >> >> >> >> >> >> [1] https://github.com/openstack/masakari-specs >> >> >> >> >> >> >> >> --- Regards, >> >> >> >> Sampath >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Thu, May 18, 2017 at 2:29 AM, Adam Spiers <aspi...@suse.com> wrote: >> >> >> >> I don't see any reason why masakari couldn't handle that, but you'd >> >> >> >> have to ask Sampath and the masakari team whether they would consider >> >> >> >> that in scope for their roadmap. >> >> >> >> >> >> >> >> Waines, Greg <greg.wai...@windriver.com> wrote: >> >> >> >> >> >> >> >> Sure. I can propose a new user story. >> >> >> >> >> >> >> >> And then are you thinking of including this user story in the scope of >> >> >> >> what masakari would be looking at ? >> >> >> >> >> >> >> >> Greg. >> >> >> >> >> >> >> >> >> >> >> >> From: Adam Spiers <aspi...@suse.com> >> >> >> >> Reply-To: "openstack-dev@lists.openstack.org" >> >> >> >> <openstack-dev@lists.openstack.org> >> >> >> >> Date: Wednesday, May 17, 2017 at 10:08 AM >> >> >> >> To: "openstack-dev@lists.openstack.org" >> >> >> >> <openstack-dev@lists.openstack.org> >> >> >> >> Subject: Re: [openstack-dev] [vitrage] [nova] [HA] VM Heartbeat / >> >> >> >> Healthcheck Monitoring >> >> >> >> >> >> >> >> Thanks for the clarification Greg. This sounds like it has the >> >> >> >> potential to be a very useful capability. May I suggest that you >> >> >> >> propose a new user story for it, along similar lines to this existing >> >> >> >> one? >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> http://specs.openstack.org/openstack/openstack-user-stories/user-stories/proposed/ha_vm.html >> >> >> >> >> >> >> >> Waines, Greg >> >> <greg.wai...@windriver.com<mailto:greg.wai...@windriver.com>> >> >> >> >> wrote: >> >> >> >> Yes that’s correct. >> >> >> >> VM Heartbeating / Health-check Monitoring would introduce intrusive / >> >> >> >> white-box type monitoring of VMs / Instances. >> >> >> >> >> >> >> >> I realize this is somewhat in the gray-zone of what a cloud should be >> >> >> >> monitoring or not, >> >> >> >> but I believe it provides an alternative for Applications deployed in >> >> VMs >> >> >> >> that do not have an external monitoring/management entity like a VNF >> >> Manager >> >> >> >> in the MANO architecture. >> >> >> >> And even for VMs with VNF Managers, it provides a highly reliable >> >> >> >> alternate monitoring path that does not rely on Tenant Networking. >> >> >> >> >> >> >> >> You’re correct, that VM HB/HC Monitoring would leverage >> >> >> >> https://wiki.libvirt.org/page/Qemu_guest_agent >> >> >> >> that would require the agent to be installed in the images for talking >> >> >> >> back to the compute host. >> >> >> >> ( there are other examples of similar approaches in openstack ... the >> >> >> >> murano-agent for installation, the swift-agent for object store >> >> management >> >> ) >> >> >> >> Although here, in the case of VM HB/HC Monitoring, via the QEMU Guest >> >> >> >> Agent, the messaging path is internal thru a QEMU virtual serial >> >> device. >> >> >> >> i.e. a very simple interface with very few dependencies ... it’s up and >> >> >> >> available very early in VM lifecycle and virtually always up. >> >> >> >> >> >> >> >> Wrt failure modes / use-cases >> >> >> >> >> >> >> >> · a VM’s response to a Heartbeat Challenge Request can be as >> >> >> >> simple as just ACK-ing, >> >> >> >> this alone allows for detection of: >> >> >> >> >> >> >> >> o a failed or hung QEMU/KVM instance, or >> >> >> >> >> >> >> >> o a failed or hung VM’s OS, or >> >> >> >> >> >> >> >> o a failure of the VM’s OS to schedule the QEMU Guest Agent daemon, >> >> or >> >> >> >> >> >> >> >> o a failure of the VM to route basic IO via linux sockets. >> >> >> >> >> >> >> >> · I have had feedback that this is similar to the virtual >> >> hardware >> >> >> >> watchdog of QEMU/KVM ( >> >> >> >> https://libvirt.org/formatdomain.html#elementsWatchdog ) >> >> >> >> >> >> >> >> · However, the VM Heartbeat / Health-check Monitoring >> >> >> >> >> >> >> >> o provides a higher-level (i.e. application-level) heartbeating >> >> >> >> >> >> >> >> § i.e. if the Heartbeat requests are being answered by the Application >> >> >> >> running within the VM >> >> >> >> >> >> >> >> o provides more than just heartbeating, as the Application can use it >> >> to >> >> >> >> trigger a variety of audits, >> >> >> >> >> >> >> >> o provides a mechanism for the Application within the VM to report a >> >> >> >> Health Status / Info back to the Host / Cloud, >> >> >> >> >> >> >> >> o provides notification of the Heartbeat / Health-check status to >> >> >> >> higher-level cloud entities thru Vitrage >> >> >> >> >> >> >> >> § e.g. VM-Heartbeat-Monitor - to - Vitrage - (EventAlarm) - Aodh - >> >> ... >> >> >> >> - VNF-Manager >> >> >> >> >> >> >> >> - (StateChange) - Nova - ... - VNF Manager >> >> >> >> >> >> >> >> >> >> >> >> Greg. >> >> >> >> >> >> >> >> >> >> >> >> From: Adam Spiers <aspi...@suse.com<mailto:aspi...@suse.com>> >> >> >> >> Reply-To: >> >> >> >> >> >> >> >> "openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>" >> >> >> >> >> >> >> >> <openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>> >> >> >> >> Date: Tuesday, May 16, 2017 at 7:29 PM >> >> >> >> To: >> >> >> >> >> >> >> >> "openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>" >> >> >> >> >> >> >> >> <openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>> >> >> >> >> Subject: Re: [openstack-dev] [vitrage] [nova] [HA] VM Heartbeat / >> >> >> >> Healthcheck Monitoring >> >> >> >> >> >> >> >> Waines, Greg >> >> >> >> >> >> >> >> <greg.wai...@windriver.com<mailto:greg.wai...@windriver.com><mailto:greg.wai...@windriver.com><mailto:greg.wai...@windriver.com%3e>> >> >> >> >> wrote: >> >> >> >> thanks for the pointers Sam. >> >> >> >> >> >> >> >> I took a quick look. >> >> >> >> I agree that the VM Heartbeat / Health-check looks like a good fit into >> >> >> >> Masakari. >> >> >> >> >> >> >> >> Currently your instance monitoring looks like it is strictly black-box >> >> >> >> type monitoring thru libvirt events. >> >> >> >> Is that correct ? >> >> >> >> i.e. you do not do any intrusive type monitoring of the instance thru >> >> the >> >> >> >> QUEMU Guest Agent facility >> >> >> >> correct ? >> >> >> >> >> >> >> >> That is correct: >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> https://github.com/openstack/masakari-monitors/blob/master/masakarimonitors/instancemonitor/instance.py >> >> >> >> >> >> >> >> I think this is what VM Heartbeat / Health-check would add to Masaraki. >> >> >> >> Let me know if you agree. >> >> >> >> >> >> >> >> OK, so you are looking for something slightly different I guess, based >> >> >> >> on this QEMU guest agent? >> >> >> >> >> >> >> >> https://wiki.libvirt.org/page/Qemu_guest_agent >> >> >> >> >> >> >> >> That would require the agent to be installed in the images, which is >> >> >> >> extra work but I imagine quite easily justifiable in some scenarios. >> >> >> >> What failure modes do you have in mind for covering with this >> >> >> >> approach - things like the guest kernel freezing, for instance? >> >> >> >> >> >> >> >> >> >> >> >> >> >> __________________________________________________________________________ >> >> >> >> OpenStack Development Mailing List (not for usage questions) >> >> >> >> Unsubscribe: >> >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> >> >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> >> >> >> >> >> >> >> __________________________________________________________________________ >> >> >> >> OpenStack Development Mailing List (not for usage questions) >> >> >> >> Unsubscribe: >> >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> >> >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> >> >> >> >> >> >> >> >> >> __________________________________________________________________________ >> >> OpenStack Development Mailing List (not for usage questions) >> >> Unsubscribe: >> >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> > >> > >> > >> > -- >> > Regards, >> > Vikash >> > >> > >> > __________________________________________________________________________ >> > OpenStack Development Mailing List (not for usage questions) >> > Unsubscribe: >> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev