Rohit, I've actually been trying to reach you on irc regarding this. Let's see if we can hook up tonight to get this done. I should be on around 9p.m. pst.
--Alex > -----Original Message----- > From: [email protected] [mailto:[email protected]] On Behalf > Of Rohit Yadav > Sent: Wednesday, February 13, 2013 1:25 AM > To: [email protected] > Subject: Re: [DISCUSS] Fixing cloud-setup-databases > > Unassigned CLOUDSTACK-1019 to be picked up by someone else. > I'm not sure how to do it correctly, picked up other things to work on. > > TODO includes from the discussion on this thread: > - Fix cloudstack-setup-databases to use DatabaseCreator and to do that > we require configuring packaging so a classpath is expanded and used > to call DatabaseCreator > - Refactor and move out logic that does upgrades from mgmt server to > databasecreator. > > Regards. > > On Tue, Feb 12, 2013 at 5:02 PM, Abhinandan Prateek > <[email protected]> wrote: > > We can discuss this one ? > > > > On 12/02/13 3:46 PM, "Rohit Yadav" <[email protected]> wrote: > > > >>On Mon, Feb 11, 2013 at 11:45 PM, Alex Huang <[email protected]> > >>wrote: > >>> Rohit, > >>> > >>> The two should be one. Having the two separate is a bad idea because > >>>it means developers is running something very different from the actual > >>>deployment version which means actual deployment version is not > tested > >>>until QA hits it. The setup database script should give options and the > >>>target in maven should just have all of the options standardized for > >>>developer. > >>> > >>> The other thing is you should announce this to the list with big > >>>headlines. Starting 4.1, we no longer modify the create-schema.sql. > >>>Only modify the upgrade schemas. The process should look like this and > >>>should be documented. > >> > >>+1 > >> > >>> - For every major release (first dot release), we need to roll-up all > >>>of the changes in the schema and produce a create-schema.sql. The > >>>version recorded inside the create-schema.sql is the version of the > >>>major release. The steps to do this should be recorded in a wiki for > >>>the release manager to use. > >> > >>Okay will start a wiki on this. I just want to say that I don't know > >>how to fix it, I don't understand the requirements clearly but will > >>try my best to fix it. > >> > >>> - Every release from this major release, we always create the database > >>>from create-schema.sql and then run the upgrades. The upgrades are > >>>exactly the same as what the management server would do to upgrade > from > >>>existing version to current. > >>> > >>> We should discuss the following things: > >>> - Management Server no longer auto-upgrades. It brings more > problems > >>>than it solves. > >> > >>Alright, so there are a lot of Upgrade versionx-versiony classes which > >>are used by DatabaseUpgradeChecker. I did not write DatabaseCreator in > >>python because I wanted to skip rewriting logic for these upgrade > >>paths again. Do we want to keep it that way or should I start porting > >>them to Python? > >> > >>> It should just detect that the version in the database is different > >>>than its software version and exit. We can provide a flag to override > >>>that function. I just didn't think this through when I designed it but > >>>the good thing is it's all self-contained in a plugin so it's not > >>>difficult to change it. > >> > >>Instead of management server calling DatabaseUpgradeChecker, > >>DatabaseCreator should call it. The upgrade path does not work in > >>DatabaseCreator because I'll have to clean the spring stuff and get > >>upgrading logic out of mgmt server if we don't want mgmt server to do > >>it. > >> > >>> - DatabaseCreator should take care of both upgrade situations and > >>>initial schema creation (which I believe you did already) and upgrade > >>>cleanup (not sure if you did this part) after all management servers > >>>have been upgraded. > >>> > >> > >>Yes, the initial schema creation work and no the upgrade and cleanup > >>does not work because they need some refactoring and mgmt server in > >>that case does not need to call DatabaseUpgradeChecker. > >> > >>> This means > >>> - Developers don't write schemas in two places. It's always written in > >>>the upgrade files. > >> > >>+1 > >> > >>> - Upgrade process for the admin is multi-steps. > >> > >>So, do we require sysadmin to run DatabaseCreator from version to > >>version or should n't it just automatically upgrade to latest version? > >> > >>> - Run DatabaseCreator (may be in a script) to do pre-upgrade > >>>and as part of this step should be the script to dump the database in > >>>case problems happen. > >>> - Upgrade all management server > >>> - Run DatabaseCreator (again maybe in a script) to do > >>>post-upgrade cleanup. > >>> - Release manager produces the create-schema.sql for every major > >>>release. > >>> > >>> For 4.1, we need to get rid of the 4.1-new-schema. I have no idea why > >>>that's created as it obviously doesn't fit into our upgrade strategy. > >>>Everything in that file should be moved into the DB upgrade scripts and > >>>that file removed in git and from the developer/pom.xml. Also, the > >>>version in create-schema.sql needs to stay the same and not changed > with > >>>the release version any more. > >> > >>Alright. > >> > >>Regards. > >> > >>> > >>> --Alex > >>> > >>>> -----Original Message----- > >>>> From: [email protected] [mailto:[email protected]] On > Behalf > >>>> Of Rohit Yadav > >>>> Sent: Monday, February 11, 2013 9:13 AM > >>>> To: Hugo Trippaers > >>>> Cc: [email protected] > >>>> Subject: Re: [DISCUSS] Fixing cloud-setup-databases > >>>> > >>>> Forgot to mention, one more reason why it was written in Java, as we > >>>> pass a database upgrade class which is suppose to do the upgrade, > >>>> tests etc. If we take the python route, these things can be python > >>>> modules instead of java classes to do the magic. > >>>> > >>>> Regards. > >>>> > >>>> On Mon, Feb 11, 2013 at 10:38 PM, Rohit Yadav <[email protected]> > >>>> wrote: > >>>> > Initially I thought to write it in python, which I can do it now as > >>>> > well. I just wanted to re-use some of the stuff that Transaction and > >>>> > ScriptRunner provides. Just let me know if it's a pain to package, we > >>>> > don't need the java one, besides it's really small and I can write a > >>>> > replacement in python (because it's available on most linux distros > >>>> > and osx, can write in any other language as well) in no time. > >>>> > > >>>> > Regards. > >>>> > > >>>> > On Mon, Feb 11, 2013 at 10:32 PM, Hugo Trippaers > >>>> > <[email protected]> wrote: > >>>> >> > >>>> >> > >>>> >>> -----Original Message----- > >>>> >>> From: [email protected] [mailto:[email protected]] > On > >>>> Behalf > >>>> >>> Of Rohit Yadav > >>>> >>> Sent: Monday, February 11, 2013 5:49 PM > >>>> >>> To: Hugo Trippaers > >>>> >>> Cc: [email protected] > >>>> >>> Subject: Re: [DISCUSS] Fixing cloud-setup-databases > >>>> >>> > >>>> >>> On Mon, Feb 11, 2013 at 9:57 PM, Hugo Trippaers > >>>> >>> <[email protected]> wrote: > >>>> >>> > Hey Rohit, > >>>> >>> > > >>>> >>> > As far as I know these are two disctict use cases. > >>>> >>> > > >>>> >>> I agree, but only in few sets of sql files. The reasons why > >>>> DatabaseCreator > >>>> >>> was introduced: > >>>> >>> > >>>> >>> 0. Make life of sysadmins and developers easier 1. If passed > >>>>--database, > >>>> it > >>>> >>> would drop and recreate databases 2. Utility to run sql files, > >>>>based on > >>>> passwd > >>>> >>> db.properties file (no hardcoded db.props file) 3. Run db upgrader > >>>>class, > >>>> so > >>>> >>> we won't have to run custom upgrade path scripts > >>>> >>> > >>>> >>> > The maven deploy db is just for developers or at least people > >>>>who are > >>>> just > >>>> >>> working from source directly, and should provide a convenient way > >>>>for > >>>> >>> developers to refresh their test database or deploy a new schema. > >>>> >>> > >>>> >>> So, maven's deploydb and cloud-setup-databases (or > >>>>cloudstack-setup- > >>>> >>> databases now) take different routes. Channeling both of them > >>>>through > >>>> >>> DatabaseCreator would ensure same kind of database > >>>> deploying/creating > >>>> >>> environment for both developers and sysadmins, less bugs and > >>>>issues. > >>>> >>> > >>>> >>> It's not required that you've to pass the same set of sql files or > >>>> configuration > >>>> >>> in both cases, for example the developer-prefill or devcloud.sql > >>>>should > >>>> not > >>>> >>> be part of cloudstack-setup-databases. > >>>> >>> > >>>> >>> I saw three files which were not part of one of the use cases, so > >>>>I asked > >>>> why > >>>> >>> they are there and who's using them. I don't know so I ask > >>>> >>> :) > >>>> >>> > >>>> >>> > The cloud-setup-databases is a tool that is used the sysadmins > >>>>to setup > >>>> a > >>>> >>> database after they have deployed the packages. It has several > >>>> purposes, > >>>> >>> setup db.properties, configure encryption and load a database (if > >>>>the > >>>> >>> supplied database is empty). > >>>> >>> > >>>> >>> Yes, my point is as a developer I want to test/play around them as > >>>>well. > >>>> >>> > >>>> >>> > > >>>> >>> > In my view both should remain for these purposes, but they > >>>>should be > >>>> in- > >>>> >>> sync and also the upgrade sql script should be in sync with these > >>>>changes. > >>>> >>> > >>>> >>> You got it! Keeping them in sync is an issue, channeling them > >>>>through > >>>> >>> DatabaseCreator would make things little more maintainable. I also > >>>>see > >>>> >>> DatabaseCreator as a utility that can be perhaps used by plugins > >>>>and > >>>> external > >>>> >>> tools as well (distant future) to do magical stuff, reusable and > >>>> maintainable. > >>>> >> > >>>> >> Sounds good to me :-) Anything that makes my life as an admin > >>>>easier > >>>> gets a +1. Downside it that it is another java tool that has > >>>>requirements and > >>>> dependencies and needs to be packaged to be use full. We are already > >>>> sneaking in some jasypt jars into /usr/share/java to not break the > >>>>existing > >>>> scripts. Having a separate java bases tool would probably require its > >>>>own set > >>>> of dependencies. > >>>> >> > >>>> >>> > >>>> >>> Thoughts, comments? > >>>> >>> > >>>> >>> Regards. > >>>> >>> > >>>> >>> > > >>>> >>> > Cheers, > >>>> >>> > > >>>> >>> > Hugo > >>>> >>> > > >>>> >>> >> -----Original Message----- > >>>> >>> >> From: [email protected] > [mailto:[email protected]] On > >>>> >>> >> Behalf Of Rohit Yadav > >>>> >>> >> Sent: Monday, February 11, 2013 11:15 AM > >>>> >>> >> To: [email protected] > >>>> >>> >> Subject: [DISCUSS] Fixing cloud-setup-databases > >>>> >>> >> > >>>> >>> >> DatabaseCreator was introduce for 4.1 so both maven deploydb > >>>> target > >>>> >>> >> and cloud-setup-databases would use that. > >>>> >>> >> Following does not run on maven's deploydb: > >>>> >>> >> cloudbridge_db.sql > >>>> >>> >> schema-level.sql > >>>> >>> >> server-setup.sql > >>>> >>> >> > >>>> >>> >> Following runs only from maven's deploydb: > >>>> >>> >> com.cloud.upgrade.DatabaseUpgradeChecker > >>>> >>> >> > >>>> >>> >> Following runs only from cloud-setup-databases: > >>>> >>> >> com.cloud.test.DatabaseConfig > >>>> >>> >> > >>>> >>> >> Pl. advise if we need to run those as part of maven's deploydb, > >>>>or > >>>> >>> >> cloud- setup-database or both? > >>>> >>> >> > >>>> >>> >> Regards. > >
