On 04/04/2017 12:14 PM, Daniel P. Berrange wrote:
On Tue, Apr 04, 2017 at 09:21:27AM -0700, Clark Boylan wrote:
On Tue, Apr 4, 2017, at 06:36 AM, Daniel P. Berrange wrote:
On Mon, Apr 03, 2017 at 04:06:53PM -0700, Clark Boylan wrote:
I have pushed a change to devstack [3] to enable using UCA which pulls
in new Libvirt and mostly seems to work. I think we should consider
switching to UCA as this may fix our Libvirt problems and if it doesn't,
we will be closer to a version of Libvirt that upstream should be
willing to fix.
This isn't the most straightfoward switch as UCA has a different repo
for each OpenStack release. libvirt-python is sensitve to the underlying
library changing; it is backward compatible but libvirt-python built
against older libvirt won't work against new libvirt. The result is a
libvirt-python wheel built on our wheel mirror does not work with UCA.
I'm surprised about that - could you elaborate on what's broken for you.
The libvirt.so provides a stable public API, and the standalone python
binding only uses public APIs from libvirt.so. IOW you should be able
to build libvirt-python against 1.3.0 and then use it against 2.5.0 with
no problems.
NB, *before* libvirt-python lived on pypi, it used some non-public
libvirt.so symbols, so was tied to the exact libvirt.so it was built
against. Assuming you're using the pypi version this should no longer
apply.
The specific issue was "AttributeError: 'module' object has no attribute
'VIR_MIGRATE_POSTCOPY'" where module here is libvirt (full log and
traceback at [0]). The libvirt-python module here was built against
Libvirt 1.3.1 turned into a wheel and copied into our wheel mirror. Then
when running against Libvirt 2.5.0 Nova seems to have detected that
newer features should be present that are not reflected in the compiled
libvirt-python resulting in the error. This crashed nova compute.
Problem was easily corrected by preventing devstack from using our wheel
mirror for libvirt-python which resulted in a new installation built
against Libvirt 2.5.0.
It seems like the API is stable enough for backward compatibility but
not forward compatibility. Its also possible that Nova is doing version
checking in a buggy way and should be checking what the libvirt-python
version is and what it supports?
Ok, so yeah your last sentance is the correct interpretation. You've built
libvirt-python against libvirt v1.3.1, so it only includes support for
constants & methods that exist in that version. The VIR_MIGRATE_POSTCOPY
constant was introduced in v1.3.3, so it will not be included in the
libvirt-python you built.
When checking features Nova calls a libvirt API that returns the version
of the libvirtd daemon, which is v2.5.0, and then just blindly assumes
libvirt-python has the same version.
Unfortunately there is no way for Nova to determine what libvirt version
the python binding was built against, so it can't improve its version
check in this respect. To deal with this, Nova would two options:
- Provide a nova.conf parameter to force it to assume an older libvirt
version, thus disabling the features regardless of what libvirtd
supports
- Make nova check for existance of the python constants / APIs it is
trying to use, in addition to checking the libvirt version
The first option is pretty trivial to do if needed. The second option would
be the more correct approach, but a much bigger maint burden, so I'm not
convinced it is worth it.
I'm not convinced either are worth it - I think if we just stop building
libvirt wheels in our wheel mirror it'll resolve itself. Most places
that are actually running OpenStack for real aren't going to be building
their install packages blind to what version of the OS they're going to
install on - so it's really only a problem with one of our gate
optimizations.
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
