€0,02: I think it is important to support per commit upgrades, so I not with Wido on this at all. Looking at your naming scheme that might be a problem. Other then this your initial proposal seems fine. The reason is not to allow for x.y.z schema changes but to be able to decide a certain commit goes in or not and have their schema changed be merged in seperately. As an RM most conflict resolution, I had to do in 4.4, where in the schema files in parts of the system I would otherwise not touch.
Downgrade might not be an issue. I dread to think it would be, so let's drop it indeed, at least for now. regards On Mon, Dec 28, 2015 at 9:34 PM, Rafael Weingärtner < rafaelweingart...@gmail.com> wrote: > Hi Wido, Rohit, > I have just read the feature suggestion. > > Wido, I am not trying to complicate things, quite the opposite, I just > illustrate a simple thing that can happen and is happening; I just pointed > how it can be easily solved. > > About the release of .Z, releases more constant and others, I do not want > to mix topics. Let’s keep this thread strict to discuss database upgrades. > > Now, about the FS. I agree with Rohit that we should have only one way of > managing database upgrades and creation. I just do not like the idea of > creating a tool that work as a wrapper on frameworks/tools such as > flywaydb. I think that those frameworks already work pretty good as they > are; and, I would rather maintain configurations than some wrapper code. > > I personally like the way ACS works during upgrades (I just do not like the > code itself and how things are structured), as a system administrator I > like to change the version in the “/etc/apt/sources.list.d/cloudstack.list” > and use the "apt-get" "update" and "install" from the command line. I do > not see the need to add another tool that is just a wrapper to the mix. If > I update ACS code to 4.7.0, why would I let the database schema in an older > version? And if we want version DB schemas and application code separately > maintaining somehow compatibility between them, which would bring a whole > other level of complexity to the code; I think we should avoid that. > > The flywaydb can be easily integrated with everything we have now; we could > have a maven profile for developers and integrate it in ACS bootstrap using > its API as a Spring bean. Therefore, we could remove the current > “DatabaseUpgradeChecker “, “DbUpgrade” and other classes that aim to do > that. We could even add the creation of the schema into the first time it > boots using flywaydb and retire the “cloudstack-setup-database” script, or > at least make it less complicated, using it just to configure the database > URL and users. > > The point is that to use Flywaydb we would have to agree upon a convention > on creating routines (java and SQL) to execute upgrades. Moreover, using a > tool such as Flywaydb we do not need to worry about upgrade paths. As I > wrote in the email I used to start this thread, the upgrade has to be > straightforward, to go to a version we have to run all of the upgrade > routines between the current version until the desired one. Our job is to > create upgrade routines that work and name them properly, the job of the > tool is to check the current version, the desired one, the upgrades that it > needs to run and execute everything properly. > > Additionally, I do not see the need to break compatibility as Rohit > suggested in the FS; in my opinion, everything we have up today can be > migrated to the new structure I proposed. If we use a tool such as > Flywaydb, I even volunteered for that. The only thing we have to discuss > and agree upon is the naming conventions for upgrades routines, where to > put them and the configurations for flywaydb. > > Thanks for your contribution and time. > > > On Mon, Dec 28, 2015 at 2:10 PM, Rohit Yadav <rohit.ya...@shapeblue.com> > wrote: > > > Hi Rafael and Wido, > > > > Thanks for starting a conversation in this regard, I could not pursue the > > Chimp tool due to other $dayjob work though it’s good to see some > > discussion has started again. Hope we’ll solve this in 2016. > > > > In my opinion, we will need to first separate the database init/migration > > tooling away from mgmt server (right now the mgmt server does db > migrations > > when it starts and there is a code/db version mismatch) and secondly make > > sure that we’re using the same code/tool to deploy database (right now, > > users use the cloudstack-setup-database python tool while developer use > the > > maven/java DatabaseCreator activated by the -Ddeploydb flag). > > > > After we’ve addressed these two issues we can look into how we can > support > > minor releases workflow (or decide to do something else, like not support > > .Z releases like Wido mentioned), and see if we can or want to use any > > existing migration tool or write a wrapper tool “chimp” that uses > existing > > tools (some of those are mentioned in the Chimp FS like flywaydb etc). > For > > allowing users to go back and forth from a db schema/version, we’ll also > > need some new DB migration conventions/versioning/rules/static-checking, > > and how developer need to write such paths (forward and reverse) etc. > > > > The best approach I figured at the time was to decide that we’ll use the > > previous db upgrade path mechanism till a certain CloudStack version (say > > 4.8.0) and after that we’ll use the new approach or tooling to > > upgrade/downgrade DB schemas (thereby retiring away from the old DB > upgrade > > path mess). > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com> Rohit Yadav Software > > Architect , ShapeBlue d: * | s: +44 203 603 0540* > > <%7C%20s:%20+44%20203%20603%200540> | m: *+91 8826230892* > > <+91%208826230892> e: *rohit.ya...@shapeblue.com | t: * > > <rohit.ya...@shapeblue.com%20%7C%20t:> | w: *www.shapeblue.com* > > <http://www.shapeblue.com> a: > > 53 Chandos Place, Covent Garden London WC2N 4HS UK Shape Blue Ltd is a > > company incorporated in England & Wales. ShapeBlue Services India LLP is > a > > company incorporated in India and is operated under license from Shape > Blue > > Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in > Brasil > > and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd > is > > a company registered by The Republic of South Africa and is traded under > > license from Shape Blue Ltd. ShapeBlue is a registered trademark. > > This email and any attachments to it may be confidential and are intended > > solely for the use of the individual to whom it is addressed. Any views > or > > opinions expressed are solely those of the author and do not necessarily > > represent those of Shape Blue Ltd or related companies. If you are not > the > > intended recipient of this email, you must neither take any action based > > upon its contents, nor copy or show it to anyone. Please contact the > sender > > if you believe you have received this email in error. > > > > > > On 28-Dec-2015, at 9:10 PM, Wido den Hollander <w...@widodh.nl> wrote: > > > > > > > > > > > > On 28-12-15 16:21, Rafael Weingärtner wrote: > > >> Thanks for your contribution Wido, > > >> I have not seen Rohit’s email; I will take a look at it. > > >> > > > > > > Ok, he has a FS here: > > > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+Chimp > > > > > >> About database schema changes happening only in X.Y, I also agree with > > you > > >> (that is a convention we all could agree on, and such as conding and > > >> release procedures we could have a wiki page for that). However, I > > think we > > >> still might have scripts in versions X.Y.Z to add data to a table such > > as > > >> “guest_os_hypervisor”. > > >> > > > > > > Yes, that is true. A bugfix could be a addition into the database, but > > > we have to prevent it as much as possible. > > > > > >> The point to manage such scripts is that, if we are in version such as > > >> 4.7.0 and a new script emerges in version 4.5.3, we would have to > > decide to > > >> run or not to run it. I would rather not run them, since if they add > > >> something to the code base; those changes should also be applied into > > >> master and as a consequence it will be available in a future update. > > >> > > > > > > I understand, but this is where our release cycle becomes the problem. > > > It is because we release a X.Y.Z release we run into these kind of > > problems. > > > > > > If we as a project simple do not release the .Z releases we would be > > > fine as well ;) > > > > > > You can try to complicate things with technical things, or if we > release > > > every two / three weeks we don't run into these kind of situations :) > > > > > > We might even cut the database version loose from the code version. > > > > > > Database version is simple 100, 101, 102, 103, 104, 105. And a code > > > version requires a certain version of the database. > > > > > > Wido > > > > > >> On Mon, Dec 28, 2015 at 12:50 PM, Wido den Hollander <w...@widodh.nl> > > wrote: > > >> > > >>> > > >>> > > >>> On 28-12-15 14:16, Rafael Weingärtner wrote: > > >>>> Hi all devs, > > >>>> First of all, sorry the long text, but I hope we can start a > > discussion > > >>>> here and improve that part of ACS. > > >>>> > > >>>> A while ago I have faced the code that Apache CloudStack (ACS) uses > to > > >>>> upgrade from a version to newer one and that did not seem to be a > good > > >>> way > > >>>> to execute our upgrades. Therefore, I decided to use some time to > > search > > >>>> for alternatives. > > >>>> > > >>> > > >>> I think we all saw that happen once or more :) > > >>> > > >>>> I have read some material about versioning of scripts used to > upgrade > > a > > >>>> database (DB) of a system and went through some frameworks that > could > > >>> help > > >>>> us. > > >>>> > > >>>> In the literature of software engineering, it is firmly stated that > we > > >>> have > > >>>> to version DB scripts as we do with the source code of the > > application, > > >>>> using the baseline approach. Gladly, we were not that bad at this > > point, > > >>> we > > >>>> already versioned our routines for DB upgrade (.sql and .java). > > >>> Therefore, > > >>>> it seemed that we just did not have used a practical approach to > help > > us > > >>>> during DB upgrades. > > >>>> > > >>>> From my readings and looking at the ACS source code I raised the > > >>> following > > >>>> requirement: > > >>>> • We should be able to write more than one routine to upgrade to a > > >>>> version; those routines can be written in Java and SQL. We might > have > > >>> more > > >>>> than a routine to be executed for each version and we should be able > > to > > >>>> define an order of execution. Additionally, to go to an upper > > version, we > > >>>> have to run all of the routines from smaller versions first, until > we > > >>>> achieve the desired version. > > >>>> > > >>>> We could also add another requirement that is the downgrade from a > > >>> version, > > >>>> which we currently do not support. With that comes my first question > > for > > >>>> discussion: > > >>>> • Do we want/need a method to downgrade from a version to a previous > > >>> one? > > >>>> > > >>> > > >>> I personally do not care. Usually people should create a backup PRIOR > > to > > >>> a upgrade. If that fails they can restore the backup. > > >>> > > >>>> I found an explanation for not supporting downgrades, and I liked > it: > > >>>> http://flywaydb.org/documentation/faq.html#downgrade > > >>>> > > >>>> So, what I devised for us: > > >>>> First the bureaucracy part - our migrations occur basically in three > > (3) > > >>>> steps, first we have a "prepare script", then a cleanup script and > > >>> finally > > >>>> the migration per se that is written in Java, at least, that is what > > we > > >>> can > > >>>> expect when reading the interface “com.cloud.upgrade.dao.DbUpgrade”. > > >>>> > > >>>> Additionally, our scripts have the following naming convention: > > >>>> schema-<currentVersion>to<desiredVersion>, which in IMHO may cause > > some > > >>>> confusion because at first sight we may think that from the same > > version > > >>> we > > >>>> could have different paths to an upper version, which in practice is > > not > > >>>> happening. Instead of a <currentVersion>to<version> we could simply > > use > > >>>> V_<numberOfVersion>_<sequencial>.<fileExtension>, giving that, we > > have to > > >>>> execute all of the V_<version> scripts that are smaller than the > > version > > >>> we > > >>>> want to upgrade. > > >>>> > > >>>> To clarify what I am saying, I will use an example. Let’s say we > have > > >>> just > > >>>> installed ACS and ran the cloudstack-setup-database. That command > will > > >>>> create a database schema in version 4.0.0. To upgrade that schema to > > >>>> version 4.3.0 (it is just an example, it could be any other > version), > > ACS > > >>>> will use the following mapping: > > >>>> > > >>>> _upgradeMap.put("4.0.0", new DbUpgrade[] {new Upgrade40to41(), new > > >>>> Upgrade410to420(), new Upgrade420to421(), new Upgrade421to430()) > > >>>> > > >>>> After loading the mapping, ACS will execute the scripts defined in > > each > > >>> one > > >>>> of the Upgrade path classes and the migration code per se. > > >>>> > > >>>> Now, let’s say we change the “.sql” scripts name to the pattern I > > >>>> mentioned, we would have the following scripts; those are the > scripts > > >>> found > > >>>> that aim to upgrade to versions between the interval 4.0.0 – 4.3.0 > > >>>> (considering 4.3.0, since that is the goal version): > > >>>> > > >>>> > > >>>> - schema-40to410, can be named to: V_410_A.sql > > >>>> - schema-40to410-cleanup, can be named to: V_410_B.sql > > >>>> - schema-410to420, can be named to: V_420_A.sql > > >>>> - schema-410to420-cleanup , can be named to: V_420_b.sql > > >>>> - schema-420to421, can be named to: V_421_A.sql > > >>>> - schema-421to430, can be named to: V_430_A.sql > > >>>> - schema-421to430-cleanup, can be named to: V_430_B.sql > > >>>> > > >>>> > > >>>> Additionally, all of the java code would have to follow the same > > >>>> convention. For instance, we have > > “com.cloud.upgrade.dao.Upgrade40to41”, > > >>>> which has some java code to migrate from 4.0.0 to 4.1.0. The idea is > > to > > >>>> extract that migration code to a Java class named: V_410_C.java, > > giving > > >>>> that it has to execute the SQL scripts before the java code. > > >>>> > > >>>> In order to go from a smaller version (4.0.0) to an upper one > > (4.3.0), we > > >>>> have to run all of the migration routines from intermediate > versions. > > >>> That > > >>>> is what we are already doing, but we do all of that manually. > > >>>> > > >>>> Bottom line, I think we could simple use the convention > > >>>> V_<numberOfVersion>_<sequencial>.<fileExtension> to name upgrade > > >>> routines. > > >>>> That would facilitate us to use a framework to help us with that > > process. > > >>>> Additionally, I believe that we should always assume that to go > from a > > >>>> smaller version to a higher one, we should run all of the scripts > that > > >>>> exist between them. What do you guys think of that? > > >>>> > > >>> > > >>> That seems good to me. But we still have to prevent that we perform > > >>> database changes in a X.Y.Z release since that is branched off to a > > >>> different branch. > > >>> > > >>> Imho database changes should only happen in X.Y releases. > > >>> > > >>>> After the bureaucracy, we can discuss tools. If we use that > > convention to > > >>>> name migration (upgrade) routines, we can start thinking on tools to > > >>>> support our migration process. I found two (2) promising ones: > > Liquibase > > >>>> and Flywaydb (both seem to be under Apache license, but the first > one > > has > > >>>> an enterprise version?!). After reading the documentation and some > > usage > > >>>> examples I found the flywaydb easier and simpler to use. > > >>>> > > >>>> What are the options of tools that we can use to help us manage the > > >>>> database upgrade, without needing to code the upgrade path that you > > know? > > >>>> > > >>>> After that, I think we should decide if we should create another > > >>>> project/component to take care of migrations, or we can just add the > > >>>> dependency of the tool to a project such as “cloud-framework-db” and > > >>> start > > >>>> using it. > > >>>> > > >>>> The “cloud-framework-db” project seems to have a focus on other > things > > >>> such > > >>>> as managing transactions and generating SQLs from annotations (?!? > > That > > >>>> should be a topic for another discussion). Therefore, I would rather > > >>> create > > >>>> a new project that has the specific goal of managing ACS DB > upgrades. > > I > > >>>> would also move all of the routines (SQL and Java) to this new > > project. > > >>>> This project would be a module of the CloudStack project and it > would > > >>>> execute the upgrade routines at the startup of ACS. > > >>>> > > >>>> I believe that going from a homemade solution to one that is more > > >>>> consolidated and used by other communities would be the way to go. > > >>>> > > >>>> I can volunteer myself to create a PR with the aforementioned > changes > > and > > >>>> using flywaydb to manage our upgrades. However, I prefer to have a > > good > > >>>> discussion with other devs first, before starting coding. > > >>>> > > >>>> Do you have suggestions or points that should be raised before we > > start > > >>>> working on that? > > >>> > > >>> Rohit suggested Chimp earlier this year: > > >>> > > >>> > > > http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201508.mbox/%3c677bd09f-fc75-4888-8dc8-2b7af7439...@shapeblue.com%3E > > >>> > > >>> The thread is called: "[DISCUSS] Let's fix CloudStack Upgrades and DB > > >>> migrations with CloudStack Chimp" > > >>> > > >>> Maybe there is something good in there. > > >>> > > >>>> > > >>>> -- > > >>>> Rafael Weingärtner > > >>>> > > >>> > > >> > > >> > > >> > > > > Regards. > > > > Find out more about ShapeBlue and our range of CloudStack related > services: > > IaaS Cloud Design & Build > > <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid > > IaaS deployment framework <http://shapeblue.com/csforge/> > > CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/> | > CloudStack > > Software Engineering > > <http://shapeblue.com/cloudstack-software-engineering/> > > CloudStack Infrastructure Support > > <http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack > > Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/> > > > > > > -- > Rafael Weingärtner > -- Daan
