@Matt Booth: I think you make a lot of good points, but I think the main gist of the "opposing" argument, so to speak, is that currently, the way we differentiate between potential compute resources (whether they be an individual hypervisor or a cluster) is by having each have its own compute node.
I think some of the reluctance here is to change that model -- the idea that a Nova compute node represents one resource which is, for all intents and purposes, atomic to OpenStack. While I get your point that this is an implementation detail, I think it's a rather large one, and a fundamental assumption in current OpenStack code (for the most part). If we change that assumption, we shouldn't really change it piecemeal. IMHO, this model (compute nodes as "atomic" resources) fits the overall design well. That being said, I personally would not be averse to something like expanding a NUMA-style API to cover the cluster, as I think this continues to fit the existing model -- a NUMA-style API breaks down an atomic resource, so for a VMWare cluster that would allow tuning to individual hypervisors, while for an individual hypervisor that would allow tuning to individual cores, etc. Best Regards, Solly Ross ----- Original Message ----- From: "Matthew Booth" <mbo...@redhat.com> To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org> Sent: Monday, April 7, 2014 10:47:35 AM Subject: Re: [openstack-dev] [OpenStack-Dev][Nova][VMWare] Enable live migration with one nova compute On 07/04/14 06:20, Jay Pipes wrote: > On Sun, 2014-04-06 at 06:59 +0000, Nandavar, Divakar Padiyar wrote: >>>> Well, it seems to me that the problem is the above blueprint and the code >>>> it introduced. This is an anti-feature IMO, and probably the best solution >>>> would be to remove the above code and go back to having a single >> >>>> nova-compute managing a single vCenter cluster, not multiple ones. >> >> Problem is not introduced by managing multiple clusters from single >> nova-compute proxy node. > > I strongly disagree. > >> Internally this proxy driver is still presenting the "compute-node" for each >> of the cluster its managing. > > In what way? > >> What we need to think about is applicability of the live migration use case >> when a "cluster" is modelled as a compute. Since the "cluster" is modelled >> as a compute, it is assumed that a typical use case of live-move is taken >> care by the underlying "cluster" itself. With this there are other use >> cases which are no-op today like host maintenance mode, live move, setting >> instance affinity etc., In order to resolve this I was thinking of >> "A way to expose operations on individual ESX Hosts like Putting host in >> maintenance mode, live move, instance affinity etc., by introducing Parent >> - Child compute node concept. Scheduling can be restricted to Parent >> compute node and Child compute node can be used for providing more drill >> down on compute and also enable additional compute operations". Any >> thoughts on this? > > The fundamental problem is that hacks were put in place in order to make > Nova defer control to vCenter, when the design of Nova and vCenter are > not compatible, and we're paying the price for that right now. > > All of the operations you describe above -- putting a host in > maintenance mode, live-migration of an instance, ensuring a new instance > is launched near or not-near another instance -- depend on a fundamental > design feature in Nova: that a nova-compute worker fully controls and > manages a host that provides a place to put server instances. We have > internal driver interfaces for the *hypervisor*, not for the *manager of > hypervisors*, because, you know, that's what Nova does. I'm going to take you to task here for use of the word 'fundamental'. What does Nova do? Apparently: 'OpenStack Nova provides a cloud computing fabric controller, supporting a wide variety of virtualization technologies, including KVM, Xen, LXC, VMware, and more. In addition to its native API, it includes compatibility with the commonly encountered Amazon EC2 and S3 APIs.' There's nothing in there about the ratio of Nova instances to hypervisors: that's an implementation detail. Now this change may or may not sit well with design decisions which have been made in the past, but the concept of managing multiple clusters from a single Nova instance is certainly not fundamentally wrong. It may not be pragmatic; it may require further changes to Nova which were not made, but there is nothing about it which is fundamentally at odds with the stated goals of the project. Why did I bother with that? I think it's in danger of being lost. Nova has been around for a while now and it has a lot of code and a lot of developers behind it. We need to remember, though, that's it's all for nothing if nobody wants to use it. VMware is different, but not wrong. Let's stay fresh. > The problem with all of the vCenter stuff is that it is trying to say to > Nova "don't worry, I got this" but unfortunately, Nova wants and needs > to manage these things, not surrender control to a different system that > handles orchestration and scheduling in its own unique way. Again, I'll flip that round. Nova *currently* manages these things, and working efficiently with a platform which also does these things would require rethinking some design above the driver level. It's not something we want to do naively, which the VMware driver is suffering from in this area. It may take time to get this right, but we shouldn't write it off as fundamentally wrong. It's useful to users and not fundamentally at odds with the project's goals. > If a shop really wants to use vCenter for scheduling and orchestration > of server instances, what exactly is the point of using OpenStack Nova > to begin with? What exactly is the point of trying to use OpenStack Nova > for scheduling and host operations when you've already shelled out US > $6,000 for vCenter Server and a boatload more money for ESX licensing? I confess I wondered this myself. However, I have now spoken to real people who are spending real money doing exactly this. The drivers seem to be: * The external API * A heterogeneous cloud vSphere isn't really designed for the former and doesn't do it well. It obviously doesn't help with the latter at all. For example, users want to be able to give non-admin customers the ability to deploy across both KVM and VMware. To my mind, a VMware cluster is an obvious deployment target. I think it's reasonable for Nova to treat it like a single hypervisor, but with non-uniform resources (which would require a change in, or at least an addition to, the current data model). Maybe something like extreme NUMA. I don't think it makes sense for Nova to concern itself with migrating VMs between hosts in a cluster. Putting a cluster into maintenance mode would involve the whole cluster, but the vSphere administrator obviously has other options. The fact that we can't migrate a VM between 2 clusters is clearly a bug. Whether a single Nova should manage multiple clusters is an open question, but it should be able to treat them as multiple targets. It's not fundamentally wrong, though. Matt -- Matthew Booth, RHCA, RHCSS Red Hat Engineering, Virtualisation Team GPG ID: D33C3490 GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490 _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev