Rohit,
The two should be one. Having the two separate is a bad idea because it means
developers is running something very different from the actual deployment
version which means actual deployment version is not tested until QA hits it.
The setup database script should give options and the target in maven should
just have all of the options standardized for developer.
The other thing is you should announce this to the list with big headlines.
Starting 4.1, we no longer modify the create-schema.sql. Only modify the
upgrade schemas. The process should look like this and should be documented.
- For every major release (first dot release), we need to roll-up all of the
changes in the schema and produce a create-schema.sql. The version recorded
inside the create-schema.sql is the version of the major release. The steps to
do this should be recorded in a wiki for the release manager to use.
- Every release from this major release, we always create the database from
create-schema.sql and then run the upgrades. The upgrades are exactly the same
as what the management server would do to upgrade from existing version to
current.
We should discuss the following things:
- Management Server no longer auto-upgrades. It brings more problems than it
solves. It should just detect that the version in the database is different
than its software version and exit. We can provide a flag to override that
function. I just didn't think this through when I designed it but the good
thing is it's all self-contained in a plugin so it's not difficult to change it.
- DatabaseCreator should take care of both upgrade situations and initial
schema creation (which I believe you did already) and upgrade cleanup (not sure
if you did this part) after all management servers have been upgraded.
This means
- Developers don't write schemas in two places. It's always written in the
upgrade files.
- Upgrade process for the admin is multi-steps.
- Run DatabaseCreator (may be in a script) to do pre-upgrade and as
part of this step should be the script to dump the database in case problems
happen.
- Upgrade all management server
- Run DatabaseCreator (again maybe in a script) to do post-upgrade
cleanup.
- Release manager produces the create-schema.sql for every major release.
For 4.1, we need to get rid of the 4.1-new-schema. I have no idea why that's
created as it obviously doesn't fit into our upgrade strategy. Everything in
that file should be moved into the DB upgrade scripts and that file removed in
git and from the developer/pom.xml. Also, the version in create-schema.sql
needs to stay the same and not changed with the release version any more.
--Alex
> -----Original Message-----
> From: [email protected] [mailto:[email protected]] On Behalf
> Of Rohit Yadav
> Sent: Monday, February 11, 2013 9:13 AM
> To: Hugo Trippaers
> Cc: [email protected]
> Subject: Re: [DISCUSS] Fixing cloud-setup-databases
>
> Forgot to mention, one more reason why it was written in Java, as we
> pass a database upgrade class which is suppose to do the upgrade,
> tests etc. If we take the python route, these things can be python
> modules instead of java classes to do the magic.
>
> Regards.
>
> On Mon, Feb 11, 2013 at 10:38 PM, Rohit Yadav <[email protected]>
> wrote:
> > Initially I thought to write it in python, which I can do it now as
> > well. I just wanted to re-use some of the stuff that Transaction and
> > ScriptRunner provides. Just let me know if it's a pain to package, we
> > don't need the java one, besides it's really small and I can write a
> > replacement in python (because it's available on most linux distros
> > and osx, can write in any other language as well) in no time.
> >
> > Regards.
> >
> > On Mon, Feb 11, 2013 at 10:32 PM, Hugo Trippaers
> > <[email protected]> wrote:
> >>
> >>
> >>> -----Original Message-----
> >>> From: [email protected] [mailto:[email protected]] On
> Behalf
> >>> Of Rohit Yadav
> >>> Sent: Monday, February 11, 2013 5:49 PM
> >>> To: Hugo Trippaers
> >>> Cc: [email protected]
> >>> Subject: Re: [DISCUSS] Fixing cloud-setup-databases
> >>>
> >>> On Mon, Feb 11, 2013 at 9:57 PM, Hugo Trippaers
> >>> <[email protected]> wrote:
> >>> > Hey Rohit,
> >>> >
> >>> > As far as I know these are two disctict use cases.
> >>> >
> >>> I agree, but only in few sets of sql files. The reasons why
> DatabaseCreator
> >>> was introduced:
> >>>
> >>> 0. Make life of sysadmins and developers easier 1. If passed --database,
> it
> >>> would drop and recreate databases 2. Utility to run sql files, based on
> passwd
> >>> db.properties file (no hardcoded db.props file) 3. Run db upgrader class,
> so
> >>> we won't have to run custom upgrade path scripts
> >>>
> >>> > The maven deploy db is just for developers or at least people who are
> just
> >>> working from source directly, and should provide a convenient way for
> >>> developers to refresh their test database or deploy a new schema.
> >>>
> >>> So, maven's deploydb and cloud-setup-databases (or cloudstack-setup-
> >>> databases now) take different routes. Channeling both of them through
> >>> DatabaseCreator would ensure same kind of database
> deploying/creating
> >>> environment for both developers and sysadmins, less bugs and issues.
> >>>
> >>> It's not required that you've to pass the same set of sql files or
> configuration
> >>> in both cases, for example the developer-prefill or devcloud.sql should
> not
> >>> be part of cloudstack-setup-databases.
> >>>
> >>> I saw three files which were not part of one of the use cases, so I asked
> why
> >>> they are there and who's using them. I don't know so I ask
> >>> :)
> >>>
> >>> > The cloud-setup-databases is a tool that is used the sysadmins to setup
> a
> >>> database after they have deployed the packages. It has several
> purposes,
> >>> setup db.properties, configure encryption and load a database (if the
> >>> supplied database is empty).
> >>>
> >>> Yes, my point is as a developer I want to test/play around them as well.
> >>>
> >>> >
> >>> > In my view both should remain for these purposes, but they should be
> in-
> >>> sync and also the upgrade sql script should be in sync with these changes.
> >>>
> >>> You got it! Keeping them in sync is an issue, channeling them through
> >>> DatabaseCreator would make things little more maintainable. I also see
> >>> DatabaseCreator as a utility that can be perhaps used by plugins and
> external
> >>> tools as well (distant future) to do magical stuff, reusable and
> maintainable.
> >>
> >> Sounds good to me :-) Anything that makes my life as an admin easier
> gets a +1. Downside it that it is another java tool that has requirements and
> dependencies and needs to be packaged to be use full. We are already
> sneaking in some jasypt jars into /usr/share/java to not break the existing
> scripts. Having a separate java bases tool would probably require its own set
> of dependencies.
> >>
> >>>
> >>> Thoughts, comments?
> >>>
> >>> Regards.
> >>>
> >>> >
> >>> > Cheers,
> >>> >
> >>> > Hugo
> >>> >
> >>> >> -----Original Message-----
> >>> >> From: [email protected] [mailto:[email protected]] On
> >>> >> Behalf Of Rohit Yadav
> >>> >> Sent: Monday, February 11, 2013 11:15 AM
> >>> >> To: [email protected]
> >>> >> Subject: [DISCUSS] Fixing cloud-setup-databases
> >>> >>
> >>> >> DatabaseCreator was introduce for 4.1 so both maven deploydb
> target
> >>> >> and cloud-setup-databases would use that.
> >>> >> Following does not run on maven's deploydb:
> >>> >> cloudbridge_db.sql
> >>> >> schema-level.sql
> >>> >> server-setup.sql
> >>> >>
> >>> >> Following runs only from maven's deploydb:
> >>> >> com.cloud.upgrade.DatabaseUpgradeChecker
> >>> >>
> >>> >> Following runs only from cloud-setup-databases:
> >>> >> com.cloud.test.DatabaseConfig
> >>> >>
> >>> >> Pl. advise if we need to run those as part of maven's deploydb, or
> >>> >> cloud- setup-database or both?
> >>> >>
> >>> >> Regards.
