On Wed, Apr 17, 2013 at 9:07 AM, Vincent Massol <[email protected]> wrote:
> Hi Denis, > > On Apr 16, 2013, at 8:36 PM, Denis Gervalle <[email protected]> wrote: > > > Hi devs, > > > > I have filed today a blocker issue: > http://jira.xwiki.org/browse/XWIKI-9041 > > This serious issue is the result of our current schema migration process. > > > > Currently, managing schema changes is done at two level: > > > > 1) A automatic script is computed by hibernate, comparing the current DB > > schema, and the targeted schema extracted from mapping files. > > 2) DataMigration may change schema using liquibase (git like solution for > > managing schema changes) > > > > According to the Hibernate team, 1) should only be used for development > > purpose. The Hibernate team clearly discourage usage of 1) in production, > > since it does not cover every use cases. For example, it does not: > > - drop column, table, etc... > > - change datatype > > - add constrainst > > ... > > However, this was the first, and until 4.x, the only way we had to manage > > our database schema. This is why we have introduced 2) in 4.0, to be able > > to change more information in our schema. Since 4.3, 2) was even be > > splitted in two steps, one before and one after step 1) for partially > > solving similar issue to http://jira.xwiki.org/browse/XWIKI-9041. > > > > While 1) is an uncontrollable monolithic step, 2) is like the underlying > > data migration, a chronologic process, that apply change in order. These > > two methods does not really work well join together, since parts of step > 1) > > may need some part of step 2) to be done, and the reverse. > > > > For example, migration of IDs from 32bits to 64bits bases its schema > > changes on the hibernate mapping, and need the schema to be updated by > > hibernate first. A new table like xwikistringlists introduced in 5.0M2 > > implies the creation of new constraints against existing IDs, and > therefore > > need the change of datatype to be processed first. There is no way out of > > that situation, and this is the cause of > > http://jira.xwiki.org/browse/XWIKI-9041. > > > > Since we do not do a lot of schema updates during a release cycle, the > > current situation has been manageable during the 4.x cycle. But, to avoid > > XWIKI-9041, it implies that user must not skip a major release during > > migration. This is a possible option, but it may implies some refactoring > > of existing migration inside a given release cycle, and probably some > > limitations as well. > > > > After brainstorming about the situation with Thomas, we have concluded > that > > it would be better to get rid of step 1), to control precisely the > > evolution of the schema. However, we do not want to cause a breakage for > > existing users. So here what we proposed: > > > > A) For any existing database having a version lower than 50000, apply > the > > automatic update of schema using hibernate, but based on the mapping file > > of the latest 4.5.x release, and followed by B) > > B) Any changes in mapping file done in 5.x or later (implying schema > > changes) should be backed by a corresponding DataMigration that provide > > Liquibase changelogs to reflect those changes on the database. > > Actually the nice point of 1) was that Hibernate was taking care of > differences between databases whereas if we have to write a migration we > have to do that ourselves manually. > Not so nice, since hibernate do very little things. For example, your change is not applied. You will need a migration for it anyway. > > If we are to create migrations for any schema change it would be nice to > make it as simple as possible to write a migration. I personally haven't > written one and wouldn't know how to write a good one that will work on all > DBs. At the very minimum we would need a migration writing guide published > on xwiki.org. > Creating new migration is as simple as writing a component, and in regards to schema, using Liquibase xml. I agree that some documentation would be nice to have. I will try to write some asap. > > > C) Newly created database are setup using the latest mapping file by > > hibernate (since we manage versioning of our database, old liquibase > change > > logs will never be applied to these new db) > > D) Advanced user will be allowed to also provide liquibase file, to > manage > > their schema updated properly (existing hidden feature), but... > > E) The automatic hibernate schema update script is computed after the > above > > steps to check that it is empty, and that the DB schema is in sync with > the > > mapping file. Optionally we may allow this step to apply more schema > > updates for advanced users (at their own risk) that do not provide the > > liquibase change log required. (keeping the existing internal custom > > mapping feature, as well as the logic of our dynamic mapping). > > > > The pros: > > - New database will no more be updated blindly, and liquibase will > provide > > a complete history of changes (except advanced user changes and dynamic > > mapping) > > - Existing database will start to have a complete history of schema > > changes starting from 5.x (and a somewhat partial history for 4.x > versions) > > - Migration test will only be needed from pre-5.x to 5.x version, future > > migration will be more isolated, and can be tested more locally between > the > > release introducing the migration and the previous one > > - We optionally keep existing functionalities (step E) > > > > The cons: > > - More work to be done while changing mapping file > > - Advanced user may need also to be more careful about their change in > > .hbm file > > - We keep using our versioning solution (we do not really use the > > liquibase way), maybe a advantage as well > > - We need some time to put that in place and we have already a slipping > RC1 > > > > However, the only alternative I see to fix > > http://jira.xwiki.org/browse/XWIKI-9041 would be to revert the schema > > changes until the above, or a similar solution is available. > > > > The bad news now. Once again, I have not planned to work on this, and my > > planning is already really busy. It will also require to > > be thoroughly tested, and Sorin will not say the opposite, migration test > > is time consuming. So, this will not improve our schedule to release 5.0 > > and I will need serious support from you guys to put that in place. > > > > WDTH ? > > I'll trust Thomas and you on the details because I'm not familiar with > migrations. > > My only comment is that we need to release 5.0 without this big change > since we're already very late. > I agree. > > AFAIU we could put in the Release notes of 5.0 that users upgrading from > before 4.x will need first to upgrade to 4.5.x and then only upgrade to 5.0 > and that this will be fixed in 5.1. Correct? > No, if we do so, we need to postpone our proposal. This means using hbm from 5.0 and starting the new solution with 5.1. > > That said, after more thinking, when you talk about "reverting schema > changes" what does it entail? Is it just reverting Sergiu's change that > adds the new table or does it also include reverting my change for the > oracle hbm file where I've changed from not-null="true" to not-null="false" > in 2 places? If only Sergiu's change is impacted I'd say we should revert > it till 5.1 since it's not an important change for 5.0 IMO. > Only Sergiu's change is impacted, since your changes is done in 4.5.x and not in 5.0. This is also this change that we cannot get working properly with the current solution. > > Thanks > -Vincent > > _______________________________________________ > devs mailing list > [email protected] > http://lists.xwiki.org/mailman/listinfo/devs > -- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
