Keeping this thread to discuss details of the implementation as explain by Denis and move 5.0 related decision to http://markmail.org/message/xyr5ujxcz4rfdaug.
On Wed, Apr 17, 2013 at 9:06 AM, Thomas Mortagne <[email protected]> wrote: > +1 > > On Tue, Apr 16, 2013 at 8:36 PM, Denis Gervalle <[email protected]> wrote: >> Hi devs, >> >> I have filed today a blocker issue: http://jira.xwiki.org/browse/XWIKI-9041 >> This serious issue is the result of our current schema migration process. >> >> Currently, managing schema changes is done at two level: >> >> 1) A automatic script is computed by hibernate, comparing the current DB >> schema, and the targeted schema extracted from mapping files. >> 2) DataMigration may change schema using liquibase (git like solution for >> managing schema changes) >> >> According to the Hibernate team, 1) should only be used for development >> purpose. The Hibernate team clearly discourage usage of 1) in production, >> since it does not cover every use cases. For example, it does not: >> - drop column, table, etc... >> - change datatype >> - add constrainst >> ... >> However, this was the first, and until 4.x, the only way we had to manage >> our database schema. This is why we have introduced 2) in 4.0, to be able >> to change more information in our schema. Since 4.3, 2) was even be >> splitted in two steps, one before and one after step 1) for partially >> solving similar issue to http://jira.xwiki.org/browse/XWIKI-9041. >> >> While 1) is an uncontrollable monolithic step, 2) is like the underlying >> data migration, a chronologic process, that apply change in order. These >> two methods does not really work well join together, since parts of step 1) >> may need some part of step 2) to be done, and the reverse. >> >> For example, migration of IDs from 32bits to 64bits bases its schema >> changes on the hibernate mapping, and need the schema to be updated by >> hibernate first. A new table like xwikistringlists introduced in 5.0M2 >> implies the creation of new constraints against existing IDs, and therefore >> need the change of datatype to be processed first. There is no way out of >> that situation, and this is the cause of >> http://jira.xwiki.org/browse/XWIKI-9041. >> >> Since we do not do a lot of schema updates during a release cycle, the >> current situation has been manageable during the 4.x cycle. But, to avoid >> XWIKI-9041, it implies that user must not skip a major release during >> migration. This is a possible option, but it may implies some refactoring >> of existing migration inside a given release cycle, and probably some >> limitations as well. >> >> After brainstorming about the situation with Thomas, we have concluded that >> it would be better to get rid of step 1), to control precisely the >> evolution of the schema. However, we do not want to cause a breakage for >> existing users. So here what we proposed: >> >> A) For any existing database having a version lower than 50000, apply the >> automatic update of schema using hibernate, but based on the mapping file >> of the latest 4.5.x release, and followed by B) >> B) Any changes in mapping file done in 5.x or later (implying schema >> changes) should be backed by a corresponding DataMigration that provide >> Liquibase changelogs to reflect those changes on the database. >> C) Newly created database are setup using the latest mapping file by >> hibernate (since we manage versioning of our database, old liquibase change >> logs will never be applied to these new db) >> D) Advanced user will be allowed to also provide liquibase file, to manage >> their schema updated properly (existing hidden feature), but... >> E) The automatic hibernate schema update script is computed after the above >> steps to check that it is empty, and that the DB schema is in sync with the >> mapping file. Optionally we may allow this step to apply more schema >> updates for advanced users (at their own risk) that do not provide the >> liquibase change log required. (keeping the existing internal custom >> mapping feature, as well as the logic of our dynamic mapping). >> >> The pros: >> - New database will no more be updated blindly, and liquibase will provide >> a complete history of changes (except advanced user changes and dynamic >> mapping) >> - Existing database will start to have a complete history of schema >> changes starting from 5.x (and a somewhat partial history for 4.x versions) >> - Migration test will only be needed from pre-5.x to 5.x version, future >> migration will be more isolated, and can be tested more locally between the >> release introducing the migration and the previous one >> - We optionally keep existing functionalities (step E) >> >> The cons: >> - More work to be done while changing mapping file >> - Advanced user may need also to be more careful about their change in >> .hbm file >> - We keep using our versioning solution (we do not really use the >> liquibase way), maybe a advantage as well >> - We need some time to put that in place and we have already a slipping RC1 >> >> However, the only alternative I see to fix >> http://jira.xwiki.org/browse/XWIKI-9041 would be to revert the schema >> changes until the above, or a similar solution is available. >> >> The bad news now. Once again, I have not planned to work on this, and my >> planning is already really busy. It will also require to >> be thoroughly tested, and Sorin will not say the opposite, migration test >> is time consuming. So, this will not improve our schedule to release 5.0 >> and I will need serious support from you guys to put that in place. >> >> WDTH ? >> >> -- >> Denis Gervalle >> SOFTEC sa - CEO >> eGuilde sarl - CTO >> _______________________________________________ >> devs mailing list >> [email protected] >> http://lists.xwiki.org/mailman/listinfo/devs > > > > -- > Thomas Mortagne -- Thomas Mortagne _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
