Boris, what do you think? Thanks, Eugene.
On Tue, Jul 2, 2013 at 9:55 PM, Ben Nemec <openst...@nemebean.com> wrote: > One small addition I would suggest is a step to remove the unused > sqlalchemy-migrate code once this is all done. That's my main concern with > moving it to Oslo right now. > > Also, is this a formal blueprint(s)? Seems like it should be. > > -Ben > > On 2013-07-02 12:50, Boris Pavlovic wrote: > >> ##############################**##############################** >> ########### >> Goal >> >> ##############################**##############################** >> ########### >> >> We should fix work with DB, unify it in all projects and use oslo code >> for all common things. >> >> In more words: >> >> DB API >> >> *) Fully cover by tests. >> >> *) Run tests against all backends (now they are runed only against >> sqlite). >> >> *) Unique constraints (instead of select + insert) >> a) Provide unique constraints. >> b) Add missing unique constraints. >> >> *) DB Archiving >> a) create shadow tables >> b) add tests that checks that shadow and main table are synced. >> c) add code that work with shadow tables. >> >> *) DB API performance optimization >> a) Remove unused joins. >> b) 1 query instead of N (where it is possible). >> c) Add methods that could improve performance. >> d) Drop unused methods. >> >> *) DB reconnect >> a) Don't break huge task if we lost connection for a moment. just >> retry DB query. >> >> *) DB Session cleanup >> a) do not use session parameter in public DB API methods. >> b) fix places where we are doing N queries in N transactions instead >> of 1. >> c) get only data that is used (e.g. len(query.all()) => >> query.count()). >> >> ---- >> >> DB Migrations >> >> *) Test DB Migrations against all backends and real data. >> >> *) Fix: DB schemas after Migrations should be same in different >> backends >> >> *) Fix: hidden bugs, that are caused by wrong migrations: >> a) fix indexes. e.g. 152 migration in Nova drop all Indexes that has >> deleted column >> b) fix wrong types >> c) drop unused tables >> >> *) Switch from sqlalchemy-migrate to something that is not death >> (e.g. alembic). >> >> ---- >> >> DB Models >> >> *) Fix: Schema that is created by Models should be the same as after >> migrations. >> >> *) Fix: Unit tests should be runed on DB that was created by Models >> not migrations. >> >> *) Add test that checks that Models are synced with migrations. >> >> ---- >> >> Oslo Code >> >> *) Base Sqlalchemy Models. >> >> *) Work around engine and session. >> >> *) SqlAlchemy Utils - that helps us with migrations and tests. >> >> *) Test migrations Base. >> >> *) Use common test wrapper that allows us to run tests on different >> backends. >> >> ##############################**##############################** >> ########### >> Implementation >> >> ##############################**##############################** >> ########### >> >> This is really really huge task. And we are almost done with Nova=). >> >> In OpenStack for such work there is only one approach ("baby steps" >> development deriven). So we are making tons of patches that could be >> easy reviewed. But there is also minuses in such approach. It is >> pretty hard to track work on high level. And sometimes there are >> misunderstand. >> >> For example with oslo code. In few words at this moment we would like >> to add (for some time) in oslo monkey patching for sqlalchemy-migrate. >> And I got reasonable question from Doug Hellmann. Why? I answer >> because of our "baby steps". But if you don't have a list of baby >> steps it is pretty hard to understand why our baby steps need this >> thing. And why we don't switch to alembic firstly. So I would like to >> describe our Road Map and write list of "baby steps". >> >> ------- >> >> OSLO >> >> *) (Merged) Base code for Models and sqlalchemy engine (session) >> >> *) (On review) Sqlalchemy utils that are used to: >> 1. Fix bugs in sqlalchemy-migrate >> 2. Base code for migrations that provides Unique Constraints. >> 3. Utils for db.archiving helps us to create and check shadow tables. >> >> *) (On review) Testtools wrapper >> We should have only one testtool wrapper in all projects. And this is >> the one of base steps in task of running tests against all backends. >> >> *) (On review) Test migrations base >> Base classes that provides us to test our migrations against all >> backends on real data >> >> *) (On review, not finished yet) DB Reconnect. >> >> *) (Not finished) Test that checks that schemas and models are synced >> >> ------- >> >> ${PROJECT_NAME} >> >> In different projects we could work absolutely simultaneously, and >> first candidates are Glance and Cinder. But inside project we could >> also work simultaneously. Here is the workflow: >> >> 1) (SYNC) Use base code for Models and sqlalchemy engines (from oslo) >> >> 2) (SYNC) Use test migrations base (from oslo) >> >> 3) (SYNC) Use SqlAlchemy utils (from oslo) >> >> 4) (1 patch) Switch to OSLO DB code >> >> 5) (1 patch) Remove ported test migrations >> >> 6) (1 Migration) Provide unique constraints (change type of "deleted" >> column) >> >> 7) (1 Migration) Add shadow tables >> a) Create shadow tables >> b) Add test that checks that they are synced always >> >> 8) (N Migrations) UniqueConstraint/Session/**Optimization workflow: >> a) (1 patch) Add/Improve/Refactor tests for part of api (that is >> connected with model) >> b) (1 patch) Fix session >> c) (1 patch) Optimize method >> d) if required (1 Migration) Add missing Unique Constraints. >> >> 9) (M Migrations) Sync Models with DB schemas (almost independent >> from 8) >> a) (K pathces) Sync Models with MySQL Migrations >> (because they are different in different backneds) >> b) (Fix, L migrations) Fix migrations, that have bugs (e.g. 152 >> migraiton in Nova, drops all >> Indexes that have 'deleted' column). >> c) Add test for MySql >> d) (Fix, 2 migrations) Sync other backends schemas with MySql schema >> e) Add test for SQLite and Psql >> f) Rethink our models: >> *) Improve Indexes >> *) Use Enum where it is required >> *) Fix bugs if there are. >> *) Add missing default values in schema. >> >> 10) Test should be runed on schema that is created from Models, not >> migrations (depend on 9) >> >> 11) Use alembic (or something else) instead of sqlalchemy-migrate >> (after 9.e. it will be safe >> a) make test migrations independent from sqlalchemy-migrate >> b) (If alembic) Provide work with sqlite. >> c) Replace all migrations >> >> Main part of work is in 7, 8 and 10 points. >> >> ##############################**##############################** >> ########### >> Answer for Doug's question >> ##############################**##############################** >> ########### >> >> Question: >> Why we should put in oslo slqlalchemy-migrate monkey patches, when we >> are planing to switch to alembic? >> >> Answer: >> If we don't put in oslo sqlalchemy-migrate monkey patches. We won't >> be able to work on 7 point at all until 8 and 10 points will be >> implemented in every project. Also work around 8 point is not >> finished, so we are not able to implement 10 points in any of project. >> So this blocks almost all work in all projects. I think that these >> 100-200 lines of code are not so big price for saving few cycles of >> time. >> >> ##############################**##############################** >> ########### >> Current progress >> >> ##############################**##############################** >> ########### >> >> Nova >> >> *) 1,4,6,7 - fully finished >> >> *) 2,3,5 - waiting for oslo, but pretty simple to sync it >> >> *) 8 - almost finished >> a) Few patched around tests (on review) >> b) 9 patches that add UniqueConstraints (on review) and few missing >> c) Few patches about session (on review) and few missing >> >> *) 9 - tons of patches that syncs models with MySql on review. Also >> we are working around tests that checks that schemas that are crated >> by models and migrations are same. So it will be a lot of work here. >> >> *) 10 - pretty simple (blocked by 9) >> >> *) 11 - tons of work, candidate for I cycle. >> >> *) Also here is work around run test on all backends but it is not >> finished yet. >> >> ------- >> >> Cinder >> >> *) 1,4 - finished >> >> *) Some work around 8 (sessions and tests) >> >> *) Other things are blocked by oslo >> >> ------- >> >> Glance >> >> *) 1 - finished >> >> *) 4 - working on it >> >> Best regards, >> >> Boris Pavlović >> >> (irc boris-42) >> >> >> ______________________________**_________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.**org <OpenStack-dev@lists.openstack.org> >> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev> >> > > ______________________________**_________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.**org <OpenStack-dev@lists.openstack.org> > http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev> >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev