On 2/16/17, 5:18 AM, "Rohit Yadav" <rohit.ya...@shapeblue.com> wrote:
All, I would like to start discussion on a new feature - Host HA for CloudStack. CloudStack lacks a way to reliably fence a host, the idea of the host-ha feature is to provide a general purpose HA framework and HA provider implementation specific for hypervisor that can use additional mechanism such as OOBM (ipmi based power management) to reliably investigate, recover and fence a host. This feature can handle scenarios associated with server crash issues and reliable fencing of hosts and HA of VM. The first version will have HA provider implementation for KVM (and for simulator to test the framework implementation, and write marvin tests that can validate the feature on Travis and others). Please have a look at the FS here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Host+HA Looking forward to your comments and questions. Regards. rohit.ya...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue Rohit, First, thanks for all the work you have put into this. This is something that CS has sorely needed for a long time. A couple of items: 1.) You state the following: “Before invoking the HA provider’s fence operation, the HA resource management will place the resource in maintenance mode. The intention is to require an administrator to manually verify that a resource is ready to return service by requiring an administrator to take it out of maintenance mode.” I agree that putting a host in maintenance mode to require manual intervention in order to bring it back online is ideal and honestly how I would probably prefer to do it. However, I also like to give the end user/operator choice. Perhaps we could add an option to bring the Host out of Maintenance mode automatically if it passes all checks and comes back into an ELIGIBLE state. This way, if the operator chooses, the host could come back into full operation and start recovering VMs if needed. This could also be handy if your environment isn’t quite n+1 when it comes to host capacity and you need to have the host back up and running as soon as possible to minimize the outage duration. Again, I know it isn’t ideal, but I don’t see the harm in giving the operator the choice. 2.) You state the following: “For the initial release, only KVM with NFS storage will be supported. However, the storage check component will be implemented in a modular fashion allowing for checks using other storage platforms(e.g. Ceph) in the future. HA provider plugins can be implemented for other hypervisors.” We are using KVM with a Ceph backend and would be very interested in helping make it a part of the initial push for this feature. I have a Dev environment backed by Ceph that we could use for teseting and would be willing to help with the development of the Ceph activity checks. I’m looking forward to getting this feature added to CS. Again, great job putting this together and starting the conversation. Thanks, Mabry