On Tue, May 31, 2016 at 08:24:03AM -0400, Sean Dague wrote: > On 05/31/2016 05:39 AM, Daniel P. Berrange wrote: > > On Tue, May 24, 2016 at 01:59:17PM -0400, Sean Dague wrote: > >> The team working on live migration testing started with an experimental > >> job on Ubuntu 16.04 to try to be using the latest and greatest libvirt + > >> qemu under the assumption that a set of issues we were seeing are > >> solved. The short answer is, it doesn't look like this is going to work. > >> > >> We run tests on a bunch of different clouds. Those clouds expose > >> different cpu flags to us. These are not standard things that map to > >> "Haswell". It means live migration in the multinode cases can hit cpus > >> with different flags. So we found the requirement was to come up with a > >> least common denominator of cpu flags, which we call gate64, and push > >> that into the libvirt cpu_map.xml in devstack, and set whenever we are > >> in a multinode scenario. > >> (https://github.com/openstack-dev/devstack/blob/master/tools/cpu_map_update.py) > >> Not ideal, but with libvirt 1.2.2 it works fine. > >> > >> It turns out it works fine because libvirt *actually* seems to take the > >> data from cpu_map.xml and do a translation to what it believes qemu will > >> understand. On these systems apparently this turns into "-cpu > >> Opteron_G1,-pse36" > >> (http://logs.openstack.org/29/42529/24/check/gate-tempest-dsvm-multinode-full/5f504c5/logs/libvirt/qemu/instance-0000000b.txt.gz) > >> > >> At some point between libvirt 1.2.2 and 1.3.1, this changed. Now libvirt > >> seems to be passing our cpu_model directly to qemu, and assumes that as > >> a user you will be responsible for writing all the <feature/> stanzas to > >> add/remove yourself. When libvirt sends 'gate64' to qemu, this explodes, > >> as qemu has no idea what we are talking about. > >> http://logs.openstack.org/34/319934/2/experimental/gate-tempest-dsvm-multinode-live-migration/b87d689/logs/screen-n-cpu.txt.gz#_2016-05-24_15_59_12_531 > >> > >> Unlike libvirt, which has a text file (xml) that configures the cpus > >> that could exist in the world, qemu builds this in statically at compile > >> time: > >> http://git.qemu.org/?p=qemu.git;a=blob;f=target-i386/cpu.c;h=895a386d3b7a94e363ca1bb98821d3251e70c0e0;hb=HEAD#l694 > >> > >> > >> So, the existing cpu_map.xml workaround for our testing situation will > >> no longer work. > >> > >> So, we have a number of open questions: > >> > >> * Have our cloud providers standardized enough that we might get away > >> without this custom cpu model? (Have some of them done it and only use > >> those for multinode?) > >> * Is there any way to get this feature back in libvirt to do the cpu > >> computation? > >> * Would we have to build a whole nova feature around setting libvirt xml > >> <feature/> to be able to test live migration in our clouds? > >> * Other options? > >> * Do we give up and go herd goats? > > > > Rather than try to define our own custom CPU models, we can probably > > just use one of the standard CPU models and then explicitly tell > > libvirt which flags to turn off in order to get compatibility with > > our cloud environments. > > > > This is not currently possible with Nova, since our nova.conf option > > only allow us to specify a bare CPU model. We would have to extend > > nova.conf to allow us to specify a list of CPU features to add or > > remove. Libvirt should then correctly pass these changes through > > to QEMU. > > Yes, that's an option. Given that the libvirt team seemed to acknowledge > this as a regression, I'd rather not build a user exposed feature for > all of that just as a workaround for a libvirt regression.
I think that fact that we're hitting this problem in the gate though is a sign that our users will likely hit it in their own deployments if using virtualized hosts. I think it is more friendly for users to be able to customize the CPU features via nova.conf, then to repeat the hacks done for devstack with editing the libvirt cpu_map.xml file. IOW, extending nova.conf to support this officially would be a generally useful feature for nova, beyond your short term CI needs. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev