On 06/21/2013 07:40 AM, Roman Podolyaka wrote:
Hi, all!
In Nova we've got a DB access layer known as "DB API" and tests for it.
Currently, those tests are run only for SQLite in-memory DB, which is
great for speed, but doesn't allow us to spot backend-specific errors.
There is a blueprint
(https://blueprints.launchpad.net/nova/+spec/db-api-tests-on-all-backends)
by Boris Pavlovic, which goal is to run the DB API tests on all DB
backends (e. g. SQLite, MySQL and PosgreSQL). Recently, I've been
working on implementation of this BP
(https://review.openstack.org/#/c/33236/).
The chosen approach for implementing this is best explained by going
through a list of problems which must be solved:
1. Tests should be executed concurrently by testr.
testr creates a few worker processes each running a portion of test
cases. When SQLite in-memory DB is used for testing, each of those
processes has it's own DB in its address space, so no race conditions
are possible. If we used a shared MySQL/PostgreSQL DB, the test suite
would fail due to various race conditions. Thus, we must create a
separate DB for each of test running processes and drop those, when all
tests end.
The question is, where we should create/drop those DBs? There are a few
possible places in our code:
1) setUp()/tearDown() methods of test cases. These are executed for
each test case (there are ~370 tests in test_db_api). So it must be a
bad idea to create/apply migrations/drop DB 370 times, if MySQL or
PostgreSQL are used instead of SQLite in-memory DB
2) testr supports creation of isolated test environments
(https://testrepository.readthedocs.org/en/latest/MANUAL.html#remote-or-isolated-test-environments).
Long story short: we can specify commands to execute before tests are
run, after test have ended and how to run tests
3) module/package level setUp()/tearDown(), but these are probably
supported only in nosetest
How many Classes are we talking about? We're actually going after a
similar problem in Tempest that setUp isn't cheap, so Matt Treinish has
an experimental patch to testr which allows class level partitioning
instead. Then you can use setupClass / teardownClass for expensive
resource setup.
So:
1) before tests are run, a few test DBs are created (the number of
created DBs is equal to the used concurrency level value)
2) for each test running process an env variable, containing the
connection string to the created DB, is set;
3) after all test running processes have ended, the created DBs are
dropped.
2. Tests cleanup should be fast.
For SQLite in-memory DB we use "create DB/apply migrations/run test/drop
DB" pattern, but that would be too slow for running tests on MySQL or
PostgreSQL.
Another option would be to create DB only once for each of test running
processes, apply DB migrations and then run each test case within a DB
transaction which is rolled back after a test ends. Combining with
something like "fsync = off" option of PostgreSQL this approach works
really fast (on my PC it takes ~5 s to run DB API tests on SQLite and
~10 s on PostgreSQL).
I like the idea of creating a transaction in setup, and triggering
rollback in teardown, that's pretty clever.
3. Tests should be easy to run for developers as well as for Jenkins.
DB API tests are the only tests which should be run on different
backends. All other test cases can be run on SQLite. The convenient way
to do this is to create a separate tox env, running only DB API tests.
Developers specify the DB connection string which effectively defines
the backend that should be used for running tests.
I'd rather not run those tests 'opportunistically' in py26 and py27 as
we do for migrations, because they are going to be broken for some time
(most problems are described here
https://docs.google.com/a/mirantis.com/document/d/1H82lIxd54CRmy-22oPRUS1sBkEtiguMU8N0whtye-BE/edit).
So it would be really nice to have a separate non-voting gate test.
Seperate tox env is the right approach IMHO, that would let it run
issolated non-voting until we get to the bottom of the issues. For
simplicity I'd still use the opportunistic db user / pass, as that will
mean it could run upstream today.
-Sean
--
Sean Dague
http://dague.net
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
