We can discuss this one ? On 12/02/13 3:46 PM, "Rohit Yadav" <[email protected]> wrote:
>On Mon, Feb 11, 2013 at 11:45 PM, Alex Huang <[email protected]> >wrote: >> Rohit, >> >> The two should be one. Having the two separate is a bad idea because >>it means developers is running something very different from the actual >>deployment version which means actual deployment version is not tested >>until QA hits it. The setup database script should give options and the >>target in maven should just have all of the options standardized for >>developer. >> >> The other thing is you should announce this to the list with big >>headlines. Starting 4.1, we no longer modify the create-schema.sql. >>Only modify the upgrade schemas. The process should look like this and >>should be documented. > >+1 > >> - For every major release (first dot release), we need to roll-up all >>of the changes in the schema and produce a create-schema.sql. The >>version recorded inside the create-schema.sql is the version of the >>major release. The steps to do this should be recorded in a wiki for >>the release manager to use. > >Okay will start a wiki on this. I just want to say that I don't know >how to fix it, I don't understand the requirements clearly but will >try my best to fix it. > >> - Every release from this major release, we always create the database >>from create-schema.sql and then run the upgrades. The upgrades are >>exactly the same as what the management server would do to upgrade from >>existing version to current. >> >> We should discuss the following things: >> - Management Server no longer auto-upgrades. It brings more problems >>than it solves. > >Alright, so there are a lot of Upgrade versionx-versiony classes which >are used by DatabaseUpgradeChecker. I did not write DatabaseCreator in >python because I wanted to skip rewriting logic for these upgrade >paths again. Do we want to keep it that way or should I start porting >them to Python? > >> It should just detect that the version in the database is different >>than its software version and exit. We can provide a flag to override >>that function. I just didn't think this through when I designed it but >>the good thing is it's all self-contained in a plugin so it's not >>difficult to change it. > >Instead of management server calling DatabaseUpgradeChecker, >DatabaseCreator should call it. The upgrade path does not work in >DatabaseCreator because I'll have to clean the spring stuff and get >upgrading logic out of mgmt server if we don't want mgmt server to do >it. > >> - DatabaseCreator should take care of both upgrade situations and >>initial schema creation (which I believe you did already) and upgrade >>cleanup (not sure if you did this part) after all management servers >>have been upgraded. >> > >Yes, the initial schema creation work and no the upgrade and cleanup >does not work because they need some refactoring and mgmt server in >that case does not need to call DatabaseUpgradeChecker. > >> This means >> - Developers don't write schemas in two places. It's always written in >>the upgrade files. > >+1 > >> - Upgrade process for the admin is multi-steps. > >So, do we require sysadmin to run DatabaseCreator from version to >version or should n't it just automatically upgrade to latest version? > >> - Run DatabaseCreator (may be in a script) to do pre-upgrade >>and as part of this step should be the script to dump the database in >>case problems happen. >> - Upgrade all management server >> - Run DatabaseCreator (again maybe in a script) to do >>post-upgrade cleanup. >> - Release manager produces the create-schema.sql for every major >>release. >> >> For 4.1, we need to get rid of the 4.1-new-schema. I have no idea why >>that's created as it obviously doesn't fit into our upgrade strategy. >>Everything in that file should be moved into the DB upgrade scripts and >>that file removed in git and from the developer/pom.xml. Also, the >>version in create-schema.sql needs to stay the same and not changed with >>the release version any more. > >Alright. > >Regards. > >> >> --Alex >> >>> -----Original Message----- >>> From: [email protected] [mailto:[email protected]] On Behalf >>> Of Rohit Yadav >>> Sent: Monday, February 11, 2013 9:13 AM >>> To: Hugo Trippaers >>> Cc: [email protected] >>> Subject: Re: [DISCUSS] Fixing cloud-setup-databases >>> >>> Forgot to mention, one more reason why it was written in Java, as we >>> pass a database upgrade class which is suppose to do the upgrade, >>> tests etc. If we take the python route, these things can be python >>> modules instead of java classes to do the magic. >>> >>> Regards. >>> >>> On Mon, Feb 11, 2013 at 10:38 PM, Rohit Yadav <[email protected]> >>> wrote: >>> > Initially I thought to write it in python, which I can do it now as >>> > well. I just wanted to re-use some of the stuff that Transaction and >>> > ScriptRunner provides. Just let me know if it's a pain to package, we >>> > don't need the java one, besides it's really small and I can write a >>> > replacement in python (because it's available on most linux distros >>> > and osx, can write in any other language as well) in no time. >>> > >>> > Regards. >>> > >>> > On Mon, Feb 11, 2013 at 10:32 PM, Hugo Trippaers >>> > <[email protected]> wrote: >>> >> >>> >> >>> >>> -----Original Message----- >>> >>> From: [email protected] [mailto:[email protected]] On >>> Behalf >>> >>> Of Rohit Yadav >>> >>> Sent: Monday, February 11, 2013 5:49 PM >>> >>> To: Hugo Trippaers >>> >>> Cc: [email protected] >>> >>> Subject: Re: [DISCUSS] Fixing cloud-setup-databases >>> >>> >>> >>> On Mon, Feb 11, 2013 at 9:57 PM, Hugo Trippaers >>> >>> <[email protected]> wrote: >>> >>> > Hey Rohit, >>> >>> > >>> >>> > As far as I know these are two disctict use cases. >>> >>> > >>> >>> I agree, but only in few sets of sql files. The reasons why >>> DatabaseCreator >>> >>> was introduced: >>> >>> >>> >>> 0. Make life of sysadmins and developers easier 1. If passed >>>--database, >>> it >>> >>> would drop and recreate databases 2. Utility to run sql files, >>>based on >>> passwd >>> >>> db.properties file (no hardcoded db.props file) 3. Run db upgrader >>>class, >>> so >>> >>> we won't have to run custom upgrade path scripts >>> >>> >>> >>> > The maven deploy db is just for developers or at least people >>>who are >>> just >>> >>> working from source directly, and should provide a convenient way >>>for >>> >>> developers to refresh their test database or deploy a new schema. >>> >>> >>> >>> So, maven's deploydb and cloud-setup-databases (or >>>cloudstack-setup- >>> >>> databases now) take different routes. Channeling both of them >>>through >>> >>> DatabaseCreator would ensure same kind of database >>> deploying/creating >>> >>> environment for both developers and sysadmins, less bugs and >>>issues. >>> >>> >>> >>> It's not required that you've to pass the same set of sql files or >>> configuration >>> >>> in both cases, for example the developer-prefill or devcloud.sql >>>should >>> not >>> >>> be part of cloudstack-setup-databases. >>> >>> >>> >>> I saw three files which were not part of one of the use cases, so >>>I asked >>> why >>> >>> they are there and who's using them. I don't know so I ask >>> >>> :) >>> >>> >>> >>> > The cloud-setup-databases is a tool that is used the sysadmins >>>to setup >>> a >>> >>> database after they have deployed the packages. It has several >>> purposes, >>> >>> setup db.properties, configure encryption and load a database (if >>>the >>> >>> supplied database is empty). >>> >>> >>> >>> Yes, my point is as a developer I want to test/play around them as >>>well. >>> >>> >>> >>> > >>> >>> > In my view both should remain for these purposes, but they >>>should be >>> in- >>> >>> sync and also the upgrade sql script should be in sync with these >>>changes. >>> >>> >>> >>> You got it! Keeping them in sync is an issue, channeling them >>>through >>> >>> DatabaseCreator would make things little more maintainable. I also >>>see >>> >>> DatabaseCreator as a utility that can be perhaps used by plugins >>>and >>> external >>> >>> tools as well (distant future) to do magical stuff, reusable and >>> maintainable. >>> >> >>> >> Sounds good to me :-) Anything that makes my life as an admin >>>easier >>> gets a +1. Downside it that it is another java tool that has >>>requirements and >>> dependencies and needs to be packaged to be use full. We are already >>> sneaking in some jasypt jars into /usr/share/java to not break the >>>existing >>> scripts. Having a separate java bases tool would probably require its >>>own set >>> of dependencies. >>> >> >>> >>> >>> >>> Thoughts, comments? >>> >>> >>> >>> Regards. >>> >>> >>> >>> > >>> >>> > Cheers, >>> >>> > >>> >>> > Hugo >>> >>> > >>> >>> >> -----Original Message----- >>> >>> >> From: [email protected] [mailto:[email protected]] On >>> >>> >> Behalf Of Rohit Yadav >>> >>> >> Sent: Monday, February 11, 2013 11:15 AM >>> >>> >> To: [email protected] >>> >>> >> Subject: [DISCUSS] Fixing cloud-setup-databases >>> >>> >> >>> >>> >> DatabaseCreator was introduce for 4.1 so both maven deploydb >>> target >>> >>> >> and cloud-setup-databases would use that. >>> >>> >> Following does not run on maven's deploydb: >>> >>> >> cloudbridge_db.sql >>> >>> >> schema-level.sql >>> >>> >> server-setup.sql >>> >>> >> >>> >>> >> Following runs only from maven's deploydb: >>> >>> >> com.cloud.upgrade.DatabaseUpgradeChecker >>> >>> >> >>> >>> >> Following runs only from cloud-setup-databases: >>> >>> >> com.cloud.test.DatabaseConfig >>> >>> >> >>> >>> >> Pl. advise if we need to run those as part of maven's deploydb, >>>or >>> >>> >> cloud- setup-database or both? >>> >>> >> >>> >>> >> Regards.
