We talked about this last week at the Oslo meeting [1], but I also promised to 
send an email for a broader audience.

We have recently had a couple of issues when we released a library where we 
broke unit tests running in other projects. We test the source tree of the 
libraries against the applications using the integration test suite, but we do 
not run the unit tests. This isn’t a new situation — we had similar problems in 
icehouse, and juno. We discussed setting up gate jobs to run the consuming 
project's unit tests during Juno, but eventually dropped that idea because of 
the server requirements needed to actually run all of the required jobs. That 
means we still have a small risk of breaking things with a release if we don’t 
have an API test in place for something we change, or if a test suite mocks out 
an implementation detail of a library instead of mocking the public API.

As part of releasing each library, we have tried to create test APIs and test 
fixtures that can be used to control the library’s behavior within a unit test 
suite in a well-known, testable, and supportable way. We need the liaisons to 
help identify missing fixtures from existing and not-yet-graduated libraries.

There are two main ways application test suites interact with Oslo libraries 
that we want to address: Using configuration options directly to control 
library behavior and mocking. Learning more about both will help us understand 
how the library interacts with the test, and we can then either design a 
fixture or test API for the library or modify the test (in cases where it is 
mocking implementation details).

A change of this scale is a long term project, but we need to start gathering 
data now if we want to start writing new fixtures in the next cycle. Please 
review your project’s test suite and make notes about how it uses mocks and 
configuration options, then add the information to the etherpad [2]. We will 
talk about it again at the Oslo meeting in a few weeks.


[1] http://eavesdrop.openstack.org/meetings/oslo/2014/oslo.2014-12-08-16.00.html
[2] https://etherpad.openstack.org/p/oslo-mocks-in-project-unit-tests

OpenStack-dev mailing list

Reply via email to