One small addition I would suggest is a step to remove the unused
sqlalchemy-migrate code once this is all done. That's my main concern
with moving it to Oslo right now.
Also, is this a formal blueprint(s)? Seems like it should be.
-Ben
On 2013-07-02 12:50, Boris Pavlovic wrote:
#######################################################################
Goal
#######################################################################
We should fix work with DB, unify it in all projects and use oslo code
for all common things.
In more words:
DB API
*) Fully cover by tests.
*) Run tests against all backends (now they are runed only against
sqlite).
*) Unique constraints (instead of select + insert)
a) Provide unique constraints.
b) Add missing unique constraints.
*) DB Archiving
a) create shadow tables
b) add tests that checks that shadow and main table are synced.
c) add code that work with shadow tables.
*) DB API performance optimization
a) Remove unused joins.
b) 1 query instead of N (where it is possible).
c) Add methods that could improve performance.
d) Drop unused methods.
*) DB reconnect
a) Don't break huge task if we lost connection for a moment. just
retry DB query.
*) DB Session cleanup
a) do not use session parameter in public DB API methods.
b) fix places where we are doing N queries in N transactions instead
of 1.
c) get only data that is used (e.g. len(query.all()) =>
query.count()).
----
DB Migrations
*) Test DB Migrations against all backends and real data.
*) Fix: DB schemas after Migrations should be same in different
backends
*) Fix: hidden bugs, that are caused by wrong migrations:
a) fix indexes. e.g. 152 migration in Nova drop all Indexes that has
deleted column
b) fix wrong types
c) drop unused tables
*) Switch from sqlalchemy-migrate to something that is not death
(e.g. alembic).
----
DB Models
*) Fix: Schema that is created by Models should be the same as after
migrations.
*) Fix: Unit tests should be runed on DB that was created by Models
not migrations.
*) Add test that checks that Models are synced with migrations.
----
Oslo Code
*) Base Sqlalchemy Models.
*) Work around engine and session.
*) SqlAlchemy Utils - that helps us with migrations and tests.
*) Test migrations Base.
*) Use common test wrapper that allows us to run tests on different
backends.
#######################################################################
Implementation
#######################################################################
This is really really huge task. And we are almost done with Nova=).
In OpenStack for such work there is only one approach ("baby steps"
development deriven). So we are making tons of patches that could be
easy reviewed. But there is also minuses in such approach. It is
pretty hard to track work on high level. And sometimes there are
misunderstand.
For example with oslo code. In few words at this moment we would like
to add (for some time) in oslo monkey patching for sqlalchemy-migrate.
And I got reasonable question from Doug Hellmann. Why? I answer
because of our "baby steps". But if you don't have a list of baby
steps it is pretty hard to understand why our baby steps need this
thing. And why we don't switch to alembic firstly. So I would like to
describe our Road Map and write list of "baby steps".
-------
OSLO
*) (Merged) Base code for Models and sqlalchemy engine (session)
*) (On review) Sqlalchemy utils that are used to:
1. Fix bugs in sqlalchemy-migrate
2. Base code for migrations that provides Unique Constraints.
3. Utils for db.archiving helps us to create and check shadow tables.
*) (On review) Testtools wrapper
We should have only one testtool wrapper in all projects. And this is
the one of base steps in task of running tests against all backends.
*) (On review) Test migrations base
Base classes that provides us to test our migrations against all
backends on real data
*) (On review, not finished yet) DB Reconnect.
*) (Not finished) Test that checks that schemas and models are synced
-------
${PROJECT_NAME}
In different projects we could work absolutely simultaneously, and
first candidates are Glance and Cinder. But inside project we could
also work simultaneously. Here is the workflow:
1) (SYNC) Use base code for Models and sqlalchemy engines (from oslo)
2) (SYNC) Use test migrations base (from oslo)
3) (SYNC) Use SqlAlchemy utils (from oslo)
4) (1 patch) Switch to OSLO DB code
5) (1 patch) Remove ported test migrations
6) (1 Migration) Provide unique constraints (change type of "deleted"
column)
7) (1 Migration) Add shadow tables
a) Create shadow tables
b) Add test that checks that they are synced always
8) (N Migrations) UniqueConstraint/Session/Optimization workflow:
a) (1 patch) Add/Improve/Refactor tests for part of api (that is
connected with model)
b) (1 patch) Fix session
c) (1 patch) Optimize method
d) if required (1 Migration) Add missing Unique Constraints.
9) (M Migrations) Sync Models with DB schemas (almost independent
from 8)
a) (K pathces) Sync Models with MySQL Migrations
(because they are different in different backneds)
b) (Fix, L migrations) Fix migrations, that have bugs (e.g. 152
migraiton in Nova, drops all
Indexes that have 'deleted' column).
c) Add test for MySql
d) (Fix, 2 migrations) Sync other backends schemas with MySql schema
e) Add test for SQLite and Psql
f) Rethink our models:
*) Improve Indexes
*) Use Enum where it is required
*) Fix bugs if there are.
*) Add missing default values in schema.
10) Test should be runed on schema that is created from Models, not
migrations (depend on 9)
11) Use alembic (or something else) instead of sqlalchemy-migrate
(after 9.e. it will be safe
a) make test migrations independent from sqlalchemy-migrate
b) (If alembic) Provide work with sqlite.
c) Replace all migrations
Main part of work is in 7, 8 and 10 points.
#######################################################################
Answer for Doug's question
#######################################################################
Question:
Why we should put in oslo slqlalchemy-migrate monkey patches, when we
are planing to switch to alembic?
Answer:
If we don't put in oslo sqlalchemy-migrate monkey patches. We won't
be able to work on 7 point at all until 8 and 10 points will be
implemented in every project. Also work around 8 point is not
finished, so we are not able to implement 10 points in any of project.
So this blocks almost all work in all projects. I think that these
100-200 lines of code are not so big price for saving few cycles of
time.
#######################################################################
Current progress
#######################################################################
Nova
*) 1,4,6,7 - fully finished
*) 2,3,5 - waiting for oslo, but pretty simple to sync it
*) 8 - almost finished
a) Few patched around tests (on review)
b) 9 patches that add UniqueConstraints (on review) and few missing
c) Few patches about session (on review) and few missing
*) 9 - tons of patches that syncs models with MySql on review. Also
we are working around tests that checks that schemas that are crated
by models and migrations are same. So it will be a lot of work here.
*) 10 - pretty simple (blocked by 9)
*) 11 - tons of work, candidate for I cycle.
*) Also here is work around run test on all backends but it is not
finished yet.
-------
Cinder
*) 1,4 - finished
*) Some work around 8 (sessions and tests)
*) Other things are blocked by oslo
-------
Glance
*) 1 - finished
*) 4 - working on it
Best regards,
Boris Pavlović
(irc boris-42)
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
