On Wed, Jul 16, 2014 at 10:15 AM, Sean Dague <s...@dague.net> wrote: > Recently the main gate updated from Ubuntu 12.04 to 14.04, and in doing > so we started executing the livesnapshot code in the nova libvirt > driver. Which fails about 20% of the time in the gate, as we're bringing > computes up and down while doing a snapshot. Dan Berange did a bunch of > debug on that and thinks it might be a qemu bug. We disabled these code > paths, so live snapshot has now been ripped out. > > In January we also triggered a libvirt bug, and had to carry a private > build of libvirt for 6 weeks in order to let people merge code in > OpenStack. > > We never were able to switch to libvirt 1.1.1 in the gate using the > Ubuntu Cloud Archive during Icehouse development, because it has a > different set of failures that would have prevented people from merging > code. > > Based on these experiences, libvirt version differences seem to be as > substantial as major hypervisor differences. There is a proposal here - > https://review.openstack.org/#/c/103923/ to hold newer versions of > libvirt to the same standard we hold xen, vmware, hyperv, docker, > ironic, etc. > > I'm somewhat concerned that the -2 pile on in this review is a double > standard of libvirt features, and features exploiting really new > upstream features. I feel like a lot of the language being used here > about the burden of doing this testing is exactly the same as was > presented by the docker team before their driver was removed, which was > ignored by the Nova team at the time. It was the concern by the freebsd > team, which was also ignored and they were told to go land libvirt > patches instead. >
For running our own CI, the burden was largely a matter of resource and time constraints for individual contributors and/or startups to setup and maintain 3rd-party CI, especially in light of a parallel requirement to pass the CI itself. I received community responses that equated to, "if you were serious, you'd dedicate several full-time developers and/or infrastructure engineers available for OpenStack development, plus several thousand a month in infrastructure itself". For Docker, these were simply not options. Back in January, putting 2-3 engineers fulltime toward OpenStack would have been a contribution of 10-20% of our engineering force. OpenStack is not more important to us than Docker itself. This thread highlights more deeply the problems for the FreeBSD folks. First, I still disagree with the recommendation that they contribute to libvirt. It's a classic example of creating two or more problems from one. Once they have support in libvirt, how long before their code is in a version of libvirt acceptable to Nova? When they hit edge-cases or bugs, requiring changes in libvirt, how long before those fixes are accepted by Nova? I concur with thoughts in the Gerrit review which suggest there should be a non-voting gate for testing against the latest libvirt. I think the ideal situation would be to functionally test against multiple versions of libvirt. We'd have at least two versions: "trunk, latest-stable". We might want "trunk, trunk-snapshot-XYZ, latest-stable, version-in-ubuntu, version-in-rhel", or any number of back-versions included in the gate. The version-in-rhel and version-in-ubuntu might be good candidates for 3rd-party CI. Regards, Eric Windisch
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev