I'm online, in another meeting but we can discuss. It's 10:43AM here, must be 9:13PST. Regards.
On Thu, Feb 14, 2013 at 6:46 AM, Alex Huang <[email protected]> wrote: > Rohit, > > I've actually been trying to reach you on irc regarding this. Let's see if > we can hook up tonight to get this done. I should be on around 9p.m. pst. > > --Alex > >> -----Original Message----- >> From: [email protected] [mailto:[email protected]] On Behalf >> Of Rohit Yadav >> Sent: Wednesday, February 13, 2013 1:25 AM >> To: [email protected] >> Subject: Re: [DISCUSS] Fixing cloud-setup-databases >> >> Unassigned CLOUDSTACK-1019 to be picked up by someone else. >> I'm not sure how to do it correctly, picked up other things to work on. >> >> TODO includes from the discussion on this thread: >> - Fix cloudstack-setup-databases to use DatabaseCreator and to do that >> we require configuring packaging so a classpath is expanded and used >> to call DatabaseCreator >> - Refactor and move out logic that does upgrades from mgmt server to >> databasecreator. >> >> Regards. >> >> On Tue, Feb 12, 2013 at 5:02 PM, Abhinandan Prateek >> <[email protected]> wrote: >> > We can discuss this one ? >> > >> > On 12/02/13 3:46 PM, "Rohit Yadav" <[email protected]> wrote: >> > >> >>On Mon, Feb 11, 2013 at 11:45 PM, Alex Huang <[email protected]> >> >>wrote: >> >>> Rohit, >> >>> >> >>> The two should be one. Having the two separate is a bad idea because >> >>>it means developers is running something very different from the actual >> >>>deployment version which means actual deployment version is not >> tested >> >>>until QA hits it. The setup database script should give options and the >> >>>target in maven should just have all of the options standardized for >> >>>developer. >> >>> >> >>> The other thing is you should announce this to the list with big >> >>>headlines. Starting 4.1, we no longer modify the create-schema.sql. >> >>>Only modify the upgrade schemas. The process should look like this and >> >>>should be documented. >> >> >> >>+1 >> >> >> >>> - For every major release (first dot release), we need to roll-up all >> >>>of the changes in the schema and produce a create-schema.sql. The >> >>>version recorded inside the create-schema.sql is the version of the >> >>>major release. The steps to do this should be recorded in a wiki for >> >>>the release manager to use. >> >> >> >>Okay will start a wiki on this. I just want to say that I don't know >> >>how to fix it, I don't understand the requirements clearly but will >> >>try my best to fix it. >> >> >> >>> - Every release from this major release, we always create the database >> >>>from create-schema.sql and then run the upgrades. The upgrades are >> >>>exactly the same as what the management server would do to upgrade >> from >> >>>existing version to current. >> >>> >> >>> We should discuss the following things: >> >>> - Management Server no longer auto-upgrades. It brings more >> problems >> >>>than it solves. >> >> >> >>Alright, so there are a lot of Upgrade versionx-versiony classes which >> >>are used by DatabaseUpgradeChecker. I did not write DatabaseCreator in >> >>python because I wanted to skip rewriting logic for these upgrade >> >>paths again. Do we want to keep it that way or should I start porting >> >>them to Python? >> >> >> >>> It should just detect that the version in the database is different >> >>>than its software version and exit. We can provide a flag to override >> >>>that function. I just didn't think this through when I designed it but >> >>>the good thing is it's all self-contained in a plugin so it's not >> >>>difficult to change it. >> >> >> >>Instead of management server calling DatabaseUpgradeChecker, >> >>DatabaseCreator should call it. The upgrade path does not work in >> >>DatabaseCreator because I'll have to clean the spring stuff and get >> >>upgrading logic out of mgmt server if we don't want mgmt server to do >> >>it. >> >> >> >>> - DatabaseCreator should take care of both upgrade situations and >> >>>initial schema creation (which I believe you did already) and upgrade >> >>>cleanup (not sure if you did this part) after all management servers >> >>>have been upgraded. >> >>> >> >> >> >>Yes, the initial schema creation work and no the upgrade and cleanup >> >>does not work because they need some refactoring and mgmt server in >> >>that case does not need to call DatabaseUpgradeChecker. >> >> >> >>> This means >> >>> - Developers don't write schemas in two places. It's always written in >> >>>the upgrade files. >> >> >> >>+1 >> >> >> >>> - Upgrade process for the admin is multi-steps. >> >> >> >>So, do we require sysadmin to run DatabaseCreator from version to >> >>version or should n't it just automatically upgrade to latest version? >> >> >> >>> - Run DatabaseCreator (may be in a script) to do pre-upgrade >> >>>and as part of this step should be the script to dump the database in >> >>>case problems happen. >> >>> - Upgrade all management server >> >>> - Run DatabaseCreator (again maybe in a script) to do >> >>>post-upgrade cleanup. >> >>> - Release manager produces the create-schema.sql for every major >> >>>release. >> >>> >> >>> For 4.1, we need to get rid of the 4.1-new-schema. I have no idea why >> >>>that's created as it obviously doesn't fit into our upgrade strategy. >> >>>Everything in that file should be moved into the DB upgrade scripts and >> >>>that file removed in git and from the developer/pom.xml. Also, the >> >>>version in create-schema.sql needs to stay the same and not changed >> with >> >>>the release version any more. >> >> >> >>Alright. >> >> >> >>Regards. >> >> >> >>> >> >>> --Alex >> >>> >> >>>> -----Original Message----- >> >>>> From: [email protected] [mailto:[email protected]] On >> Behalf >> >>>> Of Rohit Yadav >> >>>> Sent: Monday, February 11, 2013 9:13 AM >> >>>> To: Hugo Trippaers >> >>>> Cc: [email protected] >> >>>> Subject: Re: [DISCUSS] Fixing cloud-setup-databases >> >>>> >> >>>> Forgot to mention, one more reason why it was written in Java, as we >> >>>> pass a database upgrade class which is suppose to do the upgrade, >> >>>> tests etc. If we take the python route, these things can be python >> >>>> modules instead of java classes to do the magic. >> >>>> >> >>>> Regards. >> >>>> >> >>>> On Mon, Feb 11, 2013 at 10:38 PM, Rohit Yadav <[email protected]> >> >>>> wrote: >> >>>> > Initially I thought to write it in python, which I can do it now as >> >>>> > well. I just wanted to re-use some of the stuff that Transaction and >> >>>> > ScriptRunner provides. Just let me know if it's a pain to package, we >> >>>> > don't need the java one, besides it's really small and I can write a >> >>>> > replacement in python (because it's available on most linux distros >> >>>> > and osx, can write in any other language as well) in no time. >> >>>> > >> >>>> > Regards. >> >>>> > >> >>>> > On Mon, Feb 11, 2013 at 10:32 PM, Hugo Trippaers >> >>>> > <[email protected]> wrote: >> >>>> >> >> >>>> >> >> >>>> >>> -----Original Message----- >> >>>> >>> From: [email protected] [mailto:[email protected]] >> On >> >>>> Behalf >> >>>> >>> Of Rohit Yadav >> >>>> >>> Sent: Monday, February 11, 2013 5:49 PM >> >>>> >>> To: Hugo Trippaers >> >>>> >>> Cc: [email protected] >> >>>> >>> Subject: Re: [DISCUSS] Fixing cloud-setup-databases >> >>>> >>> >> >>>> >>> On Mon, Feb 11, 2013 at 9:57 PM, Hugo Trippaers >> >>>> >>> <[email protected]> wrote: >> >>>> >>> > Hey Rohit, >> >>>> >>> > >> >>>> >>> > As far as I know these are two disctict use cases. >> >>>> >>> > >> >>>> >>> I agree, but only in few sets of sql files. The reasons why >> >>>> DatabaseCreator >> >>>> >>> was introduced: >> >>>> >>> >> >>>> >>> 0. Make life of sysadmins and developers easier 1. If passed >> >>>>--database, >> >>>> it >> >>>> >>> would drop and recreate databases 2. Utility to run sql files, >> >>>>based on >> >>>> passwd >> >>>> >>> db.properties file (no hardcoded db.props file) 3. Run db upgrader >> >>>>class, >> >>>> so >> >>>> >>> we won't have to run custom upgrade path scripts >> >>>> >>> >> >>>> >>> > The maven deploy db is just for developers or at least people >> >>>>who are >> >>>> just >> >>>> >>> working from source directly, and should provide a convenient way >> >>>>for >> >>>> >>> developers to refresh their test database or deploy a new schema. >> >>>> >>> >> >>>> >>> So, maven's deploydb and cloud-setup-databases (or >> >>>>cloudstack-setup- >> >>>> >>> databases now) take different routes. Channeling both of them >> >>>>through >> >>>> >>> DatabaseCreator would ensure same kind of database >> >>>> deploying/creating >> >>>> >>> environment for both developers and sysadmins, less bugs and >> >>>>issues. >> >>>> >>> >> >>>> >>> It's not required that you've to pass the same set of sql files or >> >>>> configuration >> >>>> >>> in both cases, for example the developer-prefill or devcloud.sql >> >>>>should >> >>>> not >> >>>> >>> be part of cloudstack-setup-databases. >> >>>> >>> >> >>>> >>> I saw three files which were not part of one of the use cases, so >> >>>>I asked >> >>>> why >> >>>> >>> they are there and who's using them. I don't know so I ask >> >>>> >>> :) >> >>>> >>> >> >>>> >>> > The cloud-setup-databases is a tool that is used the sysadmins >> >>>>to setup >> >>>> a >> >>>> >>> database after they have deployed the packages. It has several >> >>>> purposes, >> >>>> >>> setup db.properties, configure encryption and load a database (if >> >>>>the >> >>>> >>> supplied database is empty). >> >>>> >>> >> >>>> >>> Yes, my point is as a developer I want to test/play around them as >> >>>>well. >> >>>> >>> >> >>>> >>> > >> >>>> >>> > In my view both should remain for these purposes, but they >> >>>>should be >> >>>> in- >> >>>> >>> sync and also the upgrade sql script should be in sync with these >> >>>>changes. >> >>>> >>> >> >>>> >>> You got it! Keeping them in sync is an issue, channeling them >> >>>>through >> >>>> >>> DatabaseCreator would make things little more maintainable. I also >> >>>>see >> >>>> >>> DatabaseCreator as a utility that can be perhaps used by plugins >> >>>>and >> >>>> external >> >>>> >>> tools as well (distant future) to do magical stuff, reusable and >> >>>> maintainable. >> >>>> >> >> >>>> >> Sounds good to me :-) Anything that makes my life as an admin >> >>>>easier >> >>>> gets a +1. Downside it that it is another java tool that has >> >>>>requirements and >> >>>> dependencies and needs to be packaged to be use full. We are already >> >>>> sneaking in some jasypt jars into /usr/share/java to not break the >> >>>>existing >> >>>> scripts. Having a separate java bases tool would probably require its >> >>>>own set >> >>>> of dependencies. >> >>>> >> >> >>>> >>> >> >>>> >>> Thoughts, comments? >> >>>> >>> >> >>>> >>> Regards. >> >>>> >>> >> >>>> >>> > >> >>>> >>> > Cheers, >> >>>> >>> > >> >>>> >>> > Hugo >> >>>> >>> > >> >>>> >>> >> -----Original Message----- >> >>>> >>> >> From: [email protected] >> [mailto:[email protected]] On >> >>>> >>> >> Behalf Of Rohit Yadav >> >>>> >>> >> Sent: Monday, February 11, 2013 11:15 AM >> >>>> >>> >> To: [email protected] >> >>>> >>> >> Subject: [DISCUSS] Fixing cloud-setup-databases >> >>>> >>> >> >> >>>> >>> >> DatabaseCreator was introduce for 4.1 so both maven deploydb >> >>>> target >> >>>> >>> >> and cloud-setup-databases would use that. >> >>>> >>> >> Following does not run on maven's deploydb: >> >>>> >>> >> cloudbridge_db.sql >> >>>> >>> >> schema-level.sql >> >>>> >>> >> server-setup.sql >> >>>> >>> >> >> >>>> >>> >> Following runs only from maven's deploydb: >> >>>> >>> >> com.cloud.upgrade.DatabaseUpgradeChecker >> >>>> >>> >> >> >>>> >>> >> Following runs only from cloud-setup-databases: >> >>>> >>> >> com.cloud.test.DatabaseConfig >> >>>> >>> >> >> >>>> >>> >> Pl. advise if we need to run those as part of maven's deploydb, >> >>>>or >> >>>> >>> >> cloud- setup-database or both? >> >>>> >>> >> >> >>>> >>> >> Regards. >> >
