Re: [openstack-dev] [nova][libvirt] Non-readonly connection to libvirt in unit tests

2014-08-22 Thread Solly Ross
(response inline)

- Original Message -
 From: Daniel P. Berrange berra...@redhat.com
 To: Solly Ross sr...@redhat.com
 Cc: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org
 Sent: Thursday, August 21, 2014 11:23:17 AM
 Subject: Re: [openstack-dev] [nova][libvirt] Non-readonly connection to 
 libvirt in unit tests
 
 On Thu, Aug 21, 2014 at 11:14:33AM -0400, Solly Ross wrote:
  (reply inline)
  
  - Original Message -
   From: Daniel P. Berrange berra...@redhat.com
   To: OpenStack Development Mailing List (not for usage questions)
   openstack-dev@lists.openstack.org
   Sent: Thursday, August 21, 2014 11:05:18 AM
   Subject: Re: [openstack-dev] [nova][libvirt] Non-readonly connection to
   libvirt in unit tests
   
   On Thu, Aug 21, 2014 at 10:52:42AM -0400, Solly Ross wrote:
FYI, the context of this is that I would like to be able to test some
of the libvirt storage pool code against a live file system, as we
currently test the storage pool code.  To do this, we need at least to
be able to get a proper connection to a session daemon.  IMHO, since
these calls aren't expensive, so to speak, it should be fine to have
them run against a real libvirt.
   
   No it really isn't OK to run against the real libvirt host system when
   in the unit tests. Unit tests must *not* rely on external system state
   in this way because it will lead to greater instability and unreliability
   of our unit tests. If you want to test stuff against the real libvirt
   storage pools then that becomes a functional / integration test suite
   which is pretty much what tempest is targetting.
  
  That's all well and good, but we *currently* manipulates the actual file
  system manually in tests.  Should we then say that we should never
  manipulate
  the actual file system either?  In that case, there are some tests which
  need to be refactored.
 
 Places where the tests manipulate the filesystem though should be doing
 so in an isolated playpen directory, not in the live location where
 a deployed nova runs, so that's not the same thing.

Ah, but in the case I mentioned before, we're dealing with storage pools,
which can just be created in the playpen directory.  In that case, libvirt
is simply acting as a library for filesystem access.  To further ensure 
isolation,
you could also connect to a session daemon instead of a system daemon.

I'm of the opinion that requiring some form of libvirt to be installed to run 
the *libvirt*
unit tests isn't actually that big of a deal, since you can build libvirt 
without extra stuff
and get a libvirt that has just enough for you to test against.  Generally it's 
the developers
that will be running the unit tests (and the CI), and if a developer is running 
the libvirt
unit tests, he or she is probably developing for the libvirt driver, and thus 
should probably
have libvirt installed in some form. 

 
 So If we require libvirt-python for tests and that requires
 libvirt-bin, what's stopping us from just removing fakelibvirt since
 it's kind of useless now anyway, right?

The thing about fakelibvirt is that it allows us to operate against
against a libvirt API without actually doing libvirt-y things like
launching VMs.  Now, libvirt does have a test:///default URI that
IIRC has similar functionality, so we could start to phase out fake
libvirt in favor of that.  However, there are probably still some
spots where we'll want to use fakelibvirt.
   
   I'm actually increasingly of the opinion that we should not in fact
   be trying to use the real libvirt library in the unit tests at all
   as it is not really adding any value. We typically nmock out all the
   actual API calls we exercise so despite using libvirt-python we
   are not in fact exercising its code or even validating that we're
   passing the correct numbers of parameters to API calls. Pretty much
   all we really relying on is the existance of the various global
   constants that are defined, and that has been nothing but trouble
   because the constants may or may not be defined depending on the
   version.
  
  Isn't that what 'test:///default' is supposed to be?  A version of libvirt
  with libvirt not actually touching the rest of the system?
 
 Yes, that is what it allows for, however, even if we used that URI we
 still wouldn't be actually exercising any of the libvirt code in any
 meaningful way because our unit tests mock out all the API calls that
 get touched. So using libvirt-python + test:///default URI doesn't
 really seem to buy us anything, but it does still mean that developers
 need to have libvirt installed in order to run  the unit tests. I'm
 not convinced that is a beneficial tradeoff.

I think it would make writing unit tests easier, because you don't have
to worry about making sure that the fakelibvirt implementation matches
the real libvirt implementation, and you don't have

Re: [openstack-dev] [nova][libvirt] Non-readonly connection to libvirt in unit tests

2014-08-22 Thread Daniel P. Berrange
On Fri, Aug 22, 2014 at 11:32:31AM -0400, Solly Ross wrote:
 (response inline)
 
 - Original Message -
  From: Daniel P. Berrange berra...@redhat.com
  To: Solly Ross sr...@redhat.com
  Cc: OpenStack Development Mailing List (not for usage questions) 
  openstack-dev@lists.openstack.org
  Sent: Thursday, August 21, 2014 11:23:17 AM
  Subject: Re: [openstack-dev] [nova][libvirt] Non-readonly connection to 
  libvirt in unit tests
  
  On Thu, Aug 21, 2014 at 11:14:33AM -0400, Solly Ross wrote:
   (reply inline)
   
   - Original Message -
From: Daniel P. Berrange berra...@redhat.com
To: OpenStack Development Mailing List (not for usage questions)
openstack-dev@lists.openstack.org
Sent: Thursday, August 21, 2014 11:05:18 AM
Subject: Re: [openstack-dev] [nova][libvirt] Non-readonly connection to
libvirt in unit tests

On Thu, Aug 21, 2014 at 10:52:42AM -0400, Solly Ross wrote:
 FYI, the context of this is that I would like to be able to test some
 of the libvirt storage pool code against a live file system, as we
 currently test the storage pool code.  To do this, we need at least to
 be able to get a proper connection to a session daemon.  IMHO, since
 these calls aren't expensive, so to speak, it should be fine to have
 them run against a real libvirt.

No it really isn't OK to run against the real libvirt host system when
in the unit tests. Unit tests must *not* rely on external system state
in this way because it will lead to greater instability and 
unreliability
of our unit tests. If you want to test stuff against the real libvirt
storage pools then that becomes a functional / integration test suite
which is pretty much what tempest is targetting.
   
   That's all well and good, but we *currently* manipulates the actual file
   system manually in tests.  Should we then say that we should never
   manipulate
   the actual file system either?  In that case, there are some tests which
   need to be refactored.
  
  Places where the tests manipulate the filesystem though should be doing
  so in an isolated playpen directory, not in the live location where
  a deployed nova runs, so that's not the same thing.
 
 Ah, but in the case I mentioned before, we're dealing with storage pools,
 which can just be created in the playpen directory.  In that case, libvirt
 is simply acting as a library for filesystem access.  To further ensure 
 isolation,
 you could also connect to a session daemon instead of a system daemon.
 
 I'm of the opinion that requiring some form of libvirt to be installed to run 
 the *libvirt*
 unit tests isn't actually that big of a deal, since you can build libvirt 
 without extra stuff
 and get a libvirt that has just enough for you to test against.  Generally 
 it's the developers
 that will be running the unit tests (and the CI), and if a developer is 
 running the libvirt
 unit tests, he or she is probably developing for the libvirt driver, and thus 
 should probably
 have libvirt installed in some form. 

The unit tests are run regardless of whether the developer is working on
libvirt or not. The more libvirt setup we require for the tests the more
pain we're inflicting on non-libvirt developers. That is a big deal for
them.

I'm actually increasingly of the opinion that we should not in fact
be trying to use the real libvirt library in the unit tests at all
as it is not really adding any value. We typically nmock out all the
actual API calls we exercise so despite using libvirt-python we
are not in fact exercising its code or even validating that we're
passing the correct numbers of parameters to API calls. Pretty much
all we really relying on is the existance of the various global
constants that are defined, and that has been nothing but trouble
because the constants may or may not be defined depending on the
version.
   
   Isn't that what 'test:///default' is supposed to be?  A version of libvirt
   with libvirt not actually touching the rest of the system?
  
  Yes, that is what it allows for, however, even if we used that URI we
  still wouldn't be actually exercising any of the libvirt code in any
  meaningful way because our unit tests mock out all the API calls that
  get touched. So using libvirt-python + test:///default URI doesn't
  really seem to buy us anything, but it does still mean that developers
  need to have libvirt installed in order to run  the unit tests. I'm
  not convinced that is a beneficial tradeoff.
 
 I think it would make writing unit tests easier, because you don't have
 to worry about making sure that the fakelibvirt implementation matches
 the real libvirt implementation, and you don't have to go adding extra
 methods to fakelibvirt to get things to work.

Those problems are all artifacts of the way fakelibvirt is /currently/
written. You can easily solve them by auto-generating the stub classes
and methods

Re: [openstack-dev] [nova][libvirt] Non-readonly connection to libvirt in unit tests

2014-08-21 Thread Solly Ross
FYI, the context of this is that I would like to be able to test some of the 
libvirt storage pool code against a live file system, as we currently test the 
storage pool code.  To do this, we need at least to be able to get a proper 
connection to a session daemon.  IMHO, since these calls aren't expensive, so 
to speak, it should be fine to have them run against a real libvirt.

 So If we require libvirt-python for tests and that requires
 libvirt-bin, what's stopping us from just removing fakelibvirt since
 it's kind of useless now anyway, right?

The thing about fakelibvirt is that it allows us to operate against against a 
libvirt API without actually doing libvirt-y things like launching VMs.  Now, 
libvirt does have a test:///default URI that IIRC has similar functionality, 
so we could start to phase out fakelibvirt in favor of that.  However, there 
are probably still some spots where we'll want to use fakelibvirt.

Best Regards,
Solly

- Original Message -
 From: Matt Riedemann mrie...@linux.vnet.ibm.com
 To: openstack-dev@lists.openstack.org
 Sent: Wednesday, August 20, 2014 8:37:39 PM
 Subject: Re: [openstack-dev] [nova][libvirt] Non-readonly connection to 
 libvirt in unit tests
 
 
 
 On 8/11/2014 4:42 AM, Daniel P. Berrange wrote:
  On Mon, Aug 04, 2014 at 06:46:13PM -0400, Solly Ross wrote:
  Hi,
  I was wondering if there was a way to get a non-readonly connection
  to libvirt when running the unit tests
  on the CI.  If I call `LibvirtDriver._connect(LibvirtDriver.uri())`,
  it works fine locally, but the upstream
  CI barfs with libvirtError: internal error Unable to locate libvirtd
  daemon in /usr/sbin (to override, set $LIBVIRTD_PATH to the name of the
  libvirtd binary).
  If I try to connect by calling libvirt.open(None), it also barfs, saying
  I don't have permission to connect.  I could just set it to always use
  fakelibvirt,
  but it would be nice to be able to run some of the tests against a real
  target.  The tests in question are part of
  https://review.openstack.org/#/c/111459/,
  and involve manipulating directory-based libvirt storage pools.
 
  Nothing in the unit tests should rely on being able to connect to the
  libvirt daemon, as the unit tests should still be able to pass when the
  daemon is not running at all. We should be either using fakelibvirt or
  mocking any libvirt APIs that need to be tested
 
  Regards,
  Daniel
 
 
 Also, doesn't this kind of break with the test requirement on
 libvirt-python now?  Before I was on trusty and trying to install that
 it was failing because I didn't have a new enough version of libvirt-bin
 installed.  So if we require libvirt-python for tests and that requires
 libvirt-bin, what's stopping us from just removing fakelibvirt since
 it's kind of useless now anyway, right?
 
 --
 
 Thanks,
 
 Matt Riedemann
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][libvirt] Non-readonly connection to libvirt in unit tests

2014-08-21 Thread Daniel P. Berrange
On Thu, Aug 21, 2014 at 10:52:42AM -0400, Solly Ross wrote:
 FYI, the context of this is that I would like to be able to test some 
 of the libvirt storage pool code against a live file system, as we
 currently test the storage pool code.  To do this, we need at least to
 be able to get a proper connection to a session daemon.  IMHO, since
 these calls aren't expensive, so to speak, it should be fine to have
 them run against a real libvirt.

No it really isn't OK to run against the real libvirt host system when
in the unit tests. Unit tests must *not* rely on external system state
in this way because it will lead to greater instability and unreliability
of our unit tests. If you want to test stuff against the real libvirt
storage pools then that becomes a functional / integration test suite
which is pretty much what tempest is targetting.
 
  So If we require libvirt-python for tests and that requires
  libvirt-bin, what's stopping us from just removing fakelibvirt since
  it's kind of useless now anyway, right?
 
 The thing about fakelibvirt is that it allows us to operate against
 against a libvirt API without actually doing libvirt-y things like
 launching VMs.  Now, libvirt does have a test:///default URI that
 IIRC has similar functionality, so we could start to phase out fake
 libvirt in favor of that.  However, there are probably still some
 spots where we'll want to use fakelibvirt.

I'm actually increasingly of the opinion that we should not in fact
be trying to use the real libvirt library in the unit tests at all
as it is not really adding any value. We typically nmock out all the
actual API calls we exercise so despite using libvirt-python we
are not in fact exercising its code or even validating that we're
passing the correct numbers of parameters to API calls. Pretty much
all we really relying on is the existance of the various global
constants that are defined, and that has been nothing but trouble
because the constants may or may not be defined depending on the
version.

The downside of fakelibvirt is that it is a half-assed implementation
of libvirt that we evolve in an adhoc fashion. I'm exploring the idea
of using pythons introspection abilities to query the libvirt-python
API and automatically generate a better 'fakelibvirt' that we can
guarantee to match the signatures of the real libvirt library. If we
had something like that which we had more confidence in, then we could
make the unit tests use that unconditionally. This would make our unit
tests more reliable since we would not be suspectible to different API
coverage in different libvirt module versions which have tripped us up
so many times

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][libvirt] Non-readonly connection to libvirt in unit tests

2014-08-21 Thread Solly Ross
(reply inline)

- Original Message -
 From: Daniel P. Berrange berra...@redhat.com
 To: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org
 Sent: Thursday, August 21, 2014 11:05:18 AM
 Subject: Re: [openstack-dev] [nova][libvirt] Non-readonly connection to 
 libvirt in unit tests
 
 On Thu, Aug 21, 2014 at 10:52:42AM -0400, Solly Ross wrote:
  FYI, the context of this is that I would like to be able to test some
  of the libvirt storage pool code against a live file system, as we
  currently test the storage pool code.  To do this, we need at least to
  be able to get a proper connection to a session daemon.  IMHO, since
  these calls aren't expensive, so to speak, it should be fine to have
  them run against a real libvirt.
 
 No it really isn't OK to run against the real libvirt host system when
 in the unit tests. Unit tests must *not* rely on external system state
 in this way because it will lead to greater instability and unreliability
 of our unit tests. If you want to test stuff against the real libvirt
 storage pools then that becomes a functional / integration test suite
 which is pretty much what tempest is targetting.

That's all well and good, but we *currently* manipulates the actual file
system manually in tests.  Should we then say that we should never manipulate
the actual file system either?  In that case, there are some tests which
need to be refactored.

  
   So If we require libvirt-python for tests and that requires
   libvirt-bin, what's stopping us from just removing fakelibvirt since
   it's kind of useless now anyway, right?
  
  The thing about fakelibvirt is that it allows us to operate against
  against a libvirt API without actually doing libvirt-y things like
  launching VMs.  Now, libvirt does have a test:///default URI that
  IIRC has similar functionality, so we could start to phase out fake
  libvirt in favor of that.  However, there are probably still some
  spots where we'll want to use fakelibvirt.
 
 I'm actually increasingly of the opinion that we should not in fact
 be trying to use the real libvirt library in the unit tests at all
 as it is not really adding any value. We typically nmock out all the
 actual API calls we exercise so despite using libvirt-python we
 are not in fact exercising its code or even validating that we're
 passing the correct numbers of parameters to API calls. Pretty much
 all we really relying on is the existance of the various global
 constants that are defined, and that has been nothing but trouble
 because the constants may or may not be defined depending on the
 version.

Isn't that what 'test:///default' is supposed to be?  A version of libvirt
with libvirt not actually touching the rest of the system?

 
 The downside of fakelibvirt is that it is a half-assed implementation
 of libvirt that we evolve in an adhoc fashion. I'm exploring the idea
 of using pythons introspection abilities to query the libvirt-python
 API and automatically generate a better 'fakelibvirt' that we can
 guarantee to match the signatures of the real libvirt library. If we
 had something like that which we had more confidence in, then we could
 make the unit tests use that unconditionally. This would make our unit
 tests more reliable since we would not be suspectible to different API
 coverage in different libvirt module versions which have tripped us up
 so many times
 
 Regards,
 Daniel
 --
 |: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
 |: http://libvirt.org  -o- http://virt-manager.org :|
 |: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
 |: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][libvirt] Non-readonly connection to libvirt in unit tests

2014-08-21 Thread Daniel P. Berrange
On Thu, Aug 21, 2014 at 11:14:33AM -0400, Solly Ross wrote:
 (reply inline)
 
 - Original Message -
  From: Daniel P. Berrange berra...@redhat.com
  To: OpenStack Development Mailing List (not for usage questions) 
  openstack-dev@lists.openstack.org
  Sent: Thursday, August 21, 2014 11:05:18 AM
  Subject: Re: [openstack-dev] [nova][libvirt] Non-readonly connection to 
  libvirt in unit tests
  
  On Thu, Aug 21, 2014 at 10:52:42AM -0400, Solly Ross wrote:
   FYI, the context of this is that I would like to be able to test some
   of the libvirt storage pool code against a live file system, as we
   currently test the storage pool code.  To do this, we need at least to
   be able to get a proper connection to a session daemon.  IMHO, since
   these calls aren't expensive, so to speak, it should be fine to have
   them run against a real libvirt.
  
  No it really isn't OK to run against the real libvirt host system when
  in the unit tests. Unit tests must *not* rely on external system state
  in this way because it will lead to greater instability and unreliability
  of our unit tests. If you want to test stuff against the real libvirt
  storage pools then that becomes a functional / integration test suite
  which is pretty much what tempest is targetting.
 
 That's all well and good, but we *currently* manipulates the actual file
 system manually in tests.  Should we then say that we should never manipulate
 the actual file system either?  In that case, there are some tests which
 need to be refactored.

Places where the tests manipulate the filesystem though should be doing
so in an isolated playpen directory, not in the live location where
a deployed nova runs, so that's not the same thing.

So If we require libvirt-python for tests and that requires
libvirt-bin, what's stopping us from just removing fakelibvirt since
it's kind of useless now anyway, right?
   
   The thing about fakelibvirt is that it allows us to operate against
   against a libvirt API without actually doing libvirt-y things like
   launching VMs.  Now, libvirt does have a test:///default URI that
   IIRC has similar functionality, so we could start to phase out fake
   libvirt in favor of that.  However, there are probably still some
   spots where we'll want to use fakelibvirt.
  
  I'm actually increasingly of the opinion that we should not in fact
  be trying to use the real libvirt library in the unit tests at all
  as it is not really adding any value. We typically nmock out all the
  actual API calls we exercise so despite using libvirt-python we
  are not in fact exercising its code or even validating that we're
  passing the correct numbers of parameters to API calls. Pretty much
  all we really relying on is the existance of the various global
  constants that are defined, and that has been nothing but trouble
  because the constants may or may not be defined depending on the
  version.
 
 Isn't that what 'test:///default' is supposed to be?  A version of libvirt
 with libvirt not actually touching the rest of the system?

Yes, that is what it allows for, however, even if we used that URI we
still wouldn't be actually exercising any of the libvirt code in any
meaningful way because our unit tests mock out all the API calls that
get touched. So using libvirt-python + test:///default URI doesn't
really seem to buy us anything, but it does still mean that developers
need to have libvirt installed in order to run  the unit tests. I'm
not convinced that is a beneficial tradeoff.

  The downside of fakelibvirt is that it is a half-assed implementation
  of libvirt that we evolve in an adhoc fashion. I'm exploring the idea
  of using pythons introspection abilities to query the libvirt-python
  API and automatically generate a better 'fakelibvirt' that we can
  guarantee to match the signatures of the real libvirt library. If we
  had something like that which we had more confidence in, then we could
  make the unit tests use that unconditionally. This would make our unit
  tests more reliable since we would not be suspectible to different API
  coverage in different libvirt module versions which have tripped us up
  so many times

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][libvirt] Non-readonly connection to libvirt in unit tests

2014-08-21 Thread Matt Riedemann



On 8/21/2014 10:23 AM, Daniel P. Berrange wrote:

On Thu, Aug 21, 2014 at 11:14:33AM -0400, Solly Ross wrote:

(reply inline)

- Original Message -

From: Daniel P. Berrange berra...@redhat.com
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org
Sent: Thursday, August 21, 2014 11:05:18 AM
Subject: Re: [openstack-dev] [nova][libvirt] Non-readonly connection to libvirt 
in unit tests

On Thu, Aug 21, 2014 at 10:52:42AM -0400, Solly Ross wrote:

FYI, the context of this is that I would like to be able to test some
of the libvirt storage pool code against a live file system, as we
currently test the storage pool code.  To do this, we need at least to
be able to get a proper connection to a session daemon.  IMHO, since
these calls aren't expensive, so to speak, it should be fine to have
them run against a real libvirt.


No it really isn't OK to run against the real libvirt host system when
in the unit tests. Unit tests must *not* rely on external system state
in this way because it will lead to greater instability and unreliability
of our unit tests. If you want to test stuff against the real libvirt
storage pools then that becomes a functional / integration test suite
which is pretty much what tempest is targetting.


That's all well and good, but we *currently* manipulates the actual file
system manually in tests.  Should we then say that we should never manipulate
the actual file system either?  In that case, there are some tests which
need to be refactored.


Places where the tests manipulate the filesystem though should be doing
so in an isolated playpen directory, not in the live location where
a deployed nova runs, so that's not the same thing.


So If we require libvirt-python for tests and that requires
libvirt-bin, what's stopping us from just removing fakelibvirt since
it's kind of useless now anyway, right?


The thing about fakelibvirt is that it allows us to operate against
against a libvirt API without actually doing libvirt-y things like
launching VMs.  Now, libvirt does have a test:///default URI that
IIRC has similar functionality, so we could start to phase out fake
libvirt in favor of that.  However, there are probably still some
spots where we'll want to use fakelibvirt.


I'm actually increasingly of the opinion that we should not in fact
be trying to use the real libvirt library in the unit tests at all
as it is not really adding any value. We typically nmock out all the
actual API calls we exercise so despite using libvirt-python we
are not in fact exercising its code or even validating that we're
passing the correct numbers of parameters to API calls. Pretty much
all we really relying on is the existance of the various global
constants that are defined, and that has been nothing but trouble
because the constants may or may not be defined depending on the
version.


Isn't that what 'test:///default' is supposed to be?  A version of libvirt
with libvirt not actually touching the rest of the system?


Yes, that is what it allows for, however, even if we used that URI we
still wouldn't be actually exercising any of the libvirt code in any
meaningful way because our unit tests mock out all the API calls that
get touched. So using libvirt-python + test:///default URI doesn't
really seem to buy us anything, but it does still mean that developers
need to have libvirt installed in order to run  the unit tests. I'm
not convinced that is a beneficial tradeoff.


The downside of fakelibvirt is that it is a half-assed implementation
of libvirt that we evolve in an adhoc fashion. I'm exploring the idea
of using pythons introspection abilities to query the libvirt-python
API and automatically generate a better 'fakelibvirt' that we can
guarantee to match the signatures of the real libvirt library. If we
had something like that which we had more confidence in, then we could
make the unit tests use that unconditionally. This would make our unit
tests more reliable since we would not be suspectible to different API
coverage in different libvirt module versions which have tripped us up
so many times


Regards,
Daniel



+1000 to removing the need to have libvirt installed to run unit tests, 
but that's what I'm seeing today unless I'm mistaken since we require 
libvirt-python which requires libvirt as already pointed out.


If you revert the change to require libvirt-python and try to run the 
unit tests, it fails, see bug 1357437 [1].


[1] https://bugs.launchpad.net/nova/+bug/1357437

--

Thanks,

Matt Riedemann


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][libvirt] Non-readonly connection to libvirt in unit tests

2014-08-21 Thread Clark Boylan


On Thu, Aug 21, 2014, at 09:25 AM, Matt Riedemann wrote:
 
 
 On 8/21/2014 10:23 AM, Daniel P. Berrange wrote:
  On Thu, Aug 21, 2014 at 11:14:33AM -0400, Solly Ross wrote:
  (reply inline)
 
  - Original Message -
  From: Daniel P. Berrange berra...@redhat.com
  To: OpenStack Development Mailing List (not for usage questions) 
  openstack-dev@lists.openstack.org
  Sent: Thursday, August 21, 2014 11:05:18 AM
  Subject: Re: [openstack-dev] [nova][libvirt] Non-readonly connection to 
  libvirt in unit tests
 
  On Thu, Aug 21, 2014 at 10:52:42AM -0400, Solly Ross wrote:
  FYI, the context of this is that I would like to be able to test some
  of the libvirt storage pool code against a live file system, as we
  currently test the storage pool code.  To do this, we need at least to
  be able to get a proper connection to a session daemon.  IMHO, since
  these calls aren't expensive, so to speak, it should be fine to have
  them run against a real libvirt.
 
  No it really isn't OK to run against the real libvirt host system when
  in the unit tests. Unit tests must *not* rely on external system state
  in this way because it will lead to greater instability and unreliability
  of our unit tests. If you want to test stuff against the real libvirt
  storage pools then that becomes a functional / integration test suite
  which is pretty much what tempest is targetting.
 
  That's all well and good, but we *currently* manipulates the actual file
  system manually in tests.  Should we then say that we should never 
  manipulate
  the actual file system either?  In that case, there are some tests which
  need to be refactored.
 
  Places where the tests manipulate the filesystem though should be doing
  so in an isolated playpen directory, not in the live location where
  a deployed nova runs, so that's not the same thing.
 
  So If we require libvirt-python for tests and that requires
  libvirt-bin, what's stopping us from just removing fakelibvirt since
  it's kind of useless now anyway, right?
 
  The thing about fakelibvirt is that it allows us to operate against
  against a libvirt API without actually doing libvirt-y things like
  launching VMs.  Now, libvirt does have a test:///default URI that
  IIRC has similar functionality, so we could start to phase out fake
  libvirt in favor of that.  However, there are probably still some
  spots where we'll want to use fakelibvirt.
 
  I'm actually increasingly of the opinion that we should not in fact
  be trying to use the real libvirt library in the unit tests at all
  as it is not really adding any value. We typically nmock out all the
  actual API calls we exercise so despite using libvirt-python we
  are not in fact exercising its code or even validating that we're
  passing the correct numbers of parameters to API calls. Pretty much
  all we really relying on is the existance of the various global
  constants that are defined, and that has been nothing but trouble
  because the constants may or may not be defined depending on the
  version.
 
  Isn't that what 'test:///default' is supposed to be?  A version of libvirt
  with libvirt not actually touching the rest of the system?
 
  Yes, that is what it allows for, however, even if we used that URI we
  still wouldn't be actually exercising any of the libvirt code in any
  meaningful way because our unit tests mock out all the API calls that
  get touched. So using libvirt-python + test:///default URI doesn't
  really seem to buy us anything, but it does still mean that developers
  need to have libvirt installed in order to run  the unit tests. I'm
  not convinced that is a beneficial tradeoff.
 
  The downside of fakelibvirt is that it is a half-assed implementation
  of libvirt that we evolve in an adhoc fashion. I'm exploring the idea
  of using pythons introspection abilities to query the libvirt-python
  API and automatically generate a better 'fakelibvirt' that we can
  guarantee to match the signatures of the real libvirt library. If we
  had something like that which we had more confidence in, then we could
  make the unit tests use that unconditionally. This would make our unit
  tests more reliable since we would not be suspectible to different API
  coverage in different libvirt module versions which have tripped us up
  so many times
 
  Regards,
  Daniel
 
 
 +1000 to removing the need to have libvirt installed to run unit tests, 
 but that's what I'm seeing today unless I'm mistaken since we require 
 libvirt-python which requires libvirt as already pointed out.
 
 If you revert the change to require libvirt-python and try to run the 
 unit tests, it fails, see bug 1357437 [1].
 
 [1] https://bugs.launchpad.net/nova/+bug/1357437
 
Reverting the change to require libvirt-python is insufficient. That
revert will flip back to using system packages and include libvirt
python lib from your operating system. Libvirt will still be required
just via a different avenue (nova does try to fall

Re: [openstack-dev] [nova][libvirt] Non-readonly connection to libvirt in unit tests

2014-08-21 Thread Matt Riedemann



On 8/21/2014 11:37 AM, Clark Boylan wrote:



On Thu, Aug 21, 2014, at 09:25 AM, Matt Riedemann wrote:



On 8/21/2014 10:23 AM, Daniel P. Berrange wrote:

On Thu, Aug 21, 2014 at 11:14:33AM -0400, Solly Ross wrote:

(reply inline)

- Original Message -

From: Daniel P. Berrange berra...@redhat.com
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org
Sent: Thursday, August 21, 2014 11:05:18 AM
Subject: Re: [openstack-dev] [nova][libvirt] Non-readonly connection to libvirt 
in unit tests

On Thu, Aug 21, 2014 at 10:52:42AM -0400, Solly Ross wrote:

FYI, the context of this is that I would like to be able to test some
of the libvirt storage pool code against a live file system, as we
currently test the storage pool code.  To do this, we need at least to
be able to get a proper connection to a session daemon.  IMHO, since
these calls aren't expensive, so to speak, it should be fine to have
them run against a real libvirt.


No it really isn't OK to run against the real libvirt host system when
in the unit tests. Unit tests must *not* rely on external system state
in this way because it will lead to greater instability and unreliability
of our unit tests. If you want to test stuff against the real libvirt
storage pools then that becomes a functional / integration test suite
which is pretty much what tempest is targetting.


That's all well and good, but we *currently* manipulates the actual file
system manually in tests.  Should we then say that we should never manipulate
the actual file system either?  In that case, there are some tests which
need to be refactored.


Places where the tests manipulate the filesystem though should be doing
so in an isolated playpen directory, not in the live location where
a deployed nova runs, so that's not the same thing.


So If we require libvirt-python for tests and that requires
libvirt-bin, what's stopping us from just removing fakelibvirt since
it's kind of useless now anyway, right?


The thing about fakelibvirt is that it allows us to operate against
against a libvirt API without actually doing libvirt-y things like
launching VMs.  Now, libvirt does have a test:///default URI that
IIRC has similar functionality, so we could start to phase out fake
libvirt in favor of that.  However, there are probably still some
spots where we'll want to use fakelibvirt.


I'm actually increasingly of the opinion that we should not in fact
be trying to use the real libvirt library in the unit tests at all
as it is not really adding any value. We typically nmock out all the
actual API calls we exercise so despite using libvirt-python we
are not in fact exercising its code or even validating that we're
passing the correct numbers of parameters to API calls. Pretty much
all we really relying on is the existance of the various global
constants that are defined, and that has been nothing but trouble
because the constants may or may not be defined depending on the
version.


Isn't that what 'test:///default' is supposed to be?  A version of libvirt
with libvirt not actually touching the rest of the system?


Yes, that is what it allows for, however, even if we used that URI we
still wouldn't be actually exercising any of the libvirt code in any
meaningful way because our unit tests mock out all the API calls that
get touched. So using libvirt-python + test:///default URI doesn't
really seem to buy us anything, but it does still mean that developers
need to have libvirt installed in order to run  the unit tests. I'm
not convinced that is a beneficial tradeoff.


The downside of fakelibvirt is that it is a half-assed implementation
of libvirt that we evolve in an adhoc fashion. I'm exploring the idea
of using pythons introspection abilities to query the libvirt-python
API and automatically generate a better 'fakelibvirt' that we can
guarantee to match the signatures of the real libvirt library. If we
had something like that which we had more confidence in, then we could
make the unit tests use that unconditionally. This would make our unit
tests more reliable since we would not be suspectible to different API
coverage in different libvirt module versions which have tripped us up
so many times


Regards,
Daniel



+1000 to removing the need to have libvirt installed to run unit tests,
but that's what I'm seeing today unless I'm mistaken since we require
libvirt-python which requires libvirt as already pointed out.

If you revert the change to require libvirt-python and try to run the
unit tests, it fails, see bug 1357437 [1].

[1] https://bugs.launchpad.net/nova/+bug/1357437


Reverting the change to require libvirt-python is insufficient. That
revert will flip back to using system packages and include libvirt
python lib from your operating system. Libvirt will still be required
just via a different avenue (nova does try to fall back on its fake
libvirt but iirc that doesn't always work so well).

If you want to stop depending

Re: [openstack-dev] [nova][libvirt] Non-readonly connection to libvirt in unit tests

2014-08-21 Thread Matt Riedemann



On 8/21/2014 12:26 PM, Daniel P. Berrange wrote:

On Thu, Aug 21, 2014 at 12:23:12PM -0500, Matt Riedemann wrote:



On 8/21/2014 11:37 AM, Clark Boylan wrote:



On Thu, Aug 21, 2014, at 09:25 AM, Matt Riedemann wrote:



On 8/21/2014 10:23 AM, Daniel P. Berrange wrote:

On Thu, Aug 21, 2014 at 11:14:33AM -0400, Solly Ross wrote:

(reply inline)

- Original Message -

From: Daniel P. Berrange berra...@redhat.com
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org
Sent: Thursday, August 21, 2014 11:05:18 AM
Subject: Re: [openstack-dev] [nova][libvirt] Non-readonly connection to libvirt 
in unit tests

On Thu, Aug 21, 2014 at 10:52:42AM -0400, Solly Ross wrote:

FYI, the context of this is that I would like to be able to test some
of the libvirt storage pool code against a live file system, as we
currently test the storage pool code.  To do this, we need at least to
be able to get a proper connection to a session daemon.  IMHO, since
these calls aren't expensive, so to speak, it should be fine to have
them run against a real libvirt.


No it really isn't OK to run against the real libvirt host system when
in the unit tests. Unit tests must *not* rely on external system state
in this way because it will lead to greater instability and unreliability
of our unit tests. If you want to test stuff against the real libvirt
storage pools then that becomes a functional / integration test suite
which is pretty much what tempest is targetting.


That's all well and good, but we *currently* manipulates the actual file
system manually in tests.  Should we then say that we should never manipulate
the actual file system either?  In that case, there are some tests which
need to be refactored.


Places where the tests manipulate the filesystem though should be doing
so in an isolated playpen directory, not in the live location where
a deployed nova runs, so that's not the same thing.


So If we require libvirt-python for tests and that requires
libvirt-bin, what's stopping us from just removing fakelibvirt since
it's kind of useless now anyway, right?


The thing about fakelibvirt is that it allows us to operate against
against a libvirt API without actually doing libvirt-y things like
launching VMs.  Now, libvirt does have a test:///default URI that
IIRC has similar functionality, so we could start to phase out fake
libvirt in favor of that.  However, there are probably still some
spots where we'll want to use fakelibvirt.


I'm actually increasingly of the opinion that we should not in fact
be trying to use the real libvirt library in the unit tests at all
as it is not really adding any value. We typically nmock out all the
actual API calls we exercise so despite using libvirt-python we
are not in fact exercising its code or even validating that we're
passing the correct numbers of parameters to API calls. Pretty much
all we really relying on is the existance of the various global
constants that are defined, and that has been nothing but trouble
because the constants may or may not be defined depending on the
version.


Isn't that what 'test:///default' is supposed to be?  A version of libvirt
with libvirt not actually touching the rest of the system?


Yes, that is what it allows for, however, even if we used that URI we
still wouldn't be actually exercising any of the libvirt code in any
meaningful way because our unit tests mock out all the API calls that
get touched. So using libvirt-python + test:///default URI doesn't
really seem to buy us anything, but it does still mean that developers
need to have libvirt installed in order to run  the unit tests. I'm
not convinced that is a beneficial tradeoff.


The downside of fakelibvirt is that it is a half-assed implementation
of libvirt that we evolve in an adhoc fashion. I'm exploring the idea
of using pythons introspection abilities to query the libvirt-python
API and automatically generate a better 'fakelibvirt' that we can
guarantee to match the signatures of the real libvirt library. If we
had something like that which we had more confidence in, then we could
make the unit tests use that unconditionally. This would make our unit
tests more reliable since we would not be suspectible to different API
coverage in different libvirt module versions which have tripped us up
so many times


Regards,
Daniel



+1000 to removing the need to have libvirt installed to run unit tests,
but that's what I'm seeing today unless I'm mistaken since we require
libvirt-python which requires libvirt as already pointed out.

If you revert the change to require libvirt-python and try to run the
unit tests, it fails, see bug 1357437 [1].

[1] https://bugs.launchpad.net/nova/+bug/1357437


Reverting the change to require libvirt-python is insufficient. That
revert will flip back to using system packages and include libvirt
python lib from your operating system. Libvirt will still be required
just via a different avenue (nova does try

Re: [openstack-dev] [nova][libvirt] Non-readonly connection to libvirt in unit tests

2014-08-20 Thread Matt Riedemann



On 8/11/2014 4:42 AM, Daniel P. Berrange wrote:

On Mon, Aug 04, 2014 at 06:46:13PM -0400, Solly Ross wrote:

Hi,
I was wondering if there was a way to get a non-readonly connection
to libvirt when running the unit tests
on the CI.  If I call `LibvirtDriver._connect(LibvirtDriver.uri())`,
it works fine locally, but the upstream
CI barfs with libvirtError: internal error Unable to locate libvirtd
daemon in /usr/sbin (to override, set $LIBVIRTD_PATH to the name of the
libvirtd binary).
If I try to connect by calling libvirt.open(None), it also barfs, saying
I don't have permission to connect.  I could just set it to always use
fakelibvirt,
but it would be nice to be able to run some of the tests against a real
target.  The tests in question are part of 
https://review.openstack.org/#/c/111459/,
and involve manipulating directory-based libvirt storage pools.


Nothing in the unit tests should rely on being able to connect to the
libvirt daemon, as the unit tests should still be able to pass when the
daemon is not running at all. We should be either using fakelibvirt or
mocking any libvirt APIs that need to be tested

Regards,
Daniel



So this is busted then right because the new flags being used aren't 
defined in fakelibvirt:


https://github.com/openstack/nova/commit/26504d71ceaecf22f135d8321769db801290c405

--

Thanks,

Matt Riedemann


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][libvirt] Non-readonly connection to libvirt in unit tests

2014-08-20 Thread Matt Riedemann



On 8/11/2014 4:42 AM, Daniel P. Berrange wrote:

On Mon, Aug 04, 2014 at 06:46:13PM -0400, Solly Ross wrote:

Hi,
I was wondering if there was a way to get a non-readonly connection
to libvirt when running the unit tests
on the CI.  If I call `LibvirtDriver._connect(LibvirtDriver.uri())`,
it works fine locally, but the upstream
CI barfs with libvirtError: internal error Unable to locate libvirtd
daemon in /usr/sbin (to override, set $LIBVIRTD_PATH to the name of the
libvirtd binary).
If I try to connect by calling libvirt.open(None), it also barfs, saying
I don't have permission to connect.  I could just set it to always use
fakelibvirt,
but it would be nice to be able to run some of the tests against a real
target.  The tests in question are part of 
https://review.openstack.org/#/c/111459/,
and involve manipulating directory-based libvirt storage pools.


Nothing in the unit tests should rely on being able to connect to the
libvirt daemon, as the unit tests should still be able to pass when the
daemon is not running at all. We should be either using fakelibvirt or
mocking any libvirt APIs that need to be tested

Regards,
Daniel



Also, doesn't this kind of break with the test requirement on 
libvirt-python now?  Before I was on trusty and trying to install that 
it was failing because I didn't have a new enough version of libvirt-bin 
installed.  So if we require libvirt-python for tests and that requires 
libvirt-bin, what's stopping us from just removing fakelibvirt since 
it's kind of useless now anyway, right?


--

Thanks,

Matt Riedemann


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][libvirt] Non-readonly connection to libvirt in unit tests

2014-08-11 Thread Daniel P. Berrange
On Mon, Aug 04, 2014 at 06:46:13PM -0400, Solly Ross wrote:
 Hi,
 I was wondering if there was a way to get a non-readonly connection
 to libvirt when running the unit tests
 on the CI.  If I call `LibvirtDriver._connect(LibvirtDriver.uri())`,
 it works fine locally, but the upstream
 CI barfs with libvirtError: internal error Unable to locate libvirtd
 daemon in /usr/sbin (to override, set $LIBVIRTD_PATH to the name of the
 libvirtd binary).
 If I try to connect by calling libvirt.open(None), it also barfs, saying
 I don't have permission to connect.  I could just set it to always use
 fakelibvirt,
 but it would be nice to be able to run some of the tests against a real
 target.  The tests in question are part of 
 https://review.openstack.org/#/c/111459/,
 and involve manipulating directory-based libvirt storage pools.

Nothing in the unit tests should rely on being able to connect to the
libvirt daemon, as the unit tests should still be able to pass when the
daemon is not running at all. We should be either using fakelibvirt or
mocking any libvirt APIs that need to be tested

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev