Thanks for the detailed explanation Christian . I have a clear picture now. really appreciated.
Don't think it is a good idea to ignore the rules applied to schema and just -execute all the way in every release. As eventually I will run into much bigger problem in the future. I revisited the fields mentioned in --hint and able to trace down to some customization done quite a while ago. Attempting to turn the schema back to sane without losing data. But thanks for the info and help so far. Jasper On Saturday, January 16, 2016 at 11:00:35 AM UTC-5, Christian Hammond wrote: > > Hi Jasper, > > --execute is fine, and it's what rb-site runs. It's the combination of > that and --hint that gets you into trouble. I'll explain what's going on. > > Since we need to make changes to the schema for the database every so > often, and since we support a variety of databases, we need to use a tool > to let us record the changes we need to apply at a high level and manage > applying those when it's time to upgrade. This tool is called > django-evolution, and it's something we maintain these days. > > In order to be able to sanely apply changes, every single change ever must > be recorded as part of the product, so that they can be scanned and matched > up during the upgrade process. These are stored in "evolution files" > (little bits of Python). Once an upgrade is performed, new state is > recorded in a couple tables in the database containing the entire list of > evolutions applied and the resulting signature of the database. > > In your case, at some point somebody modified the Review Board code and > added new fields to some of our tables. Maybe evolution files were added, > or maybe --hint --execute was used. Either way, not being part of the > product meant that future upgrades had no way of factoring in these fields. > Django-evolution knows it once existed but no longer does, and doesn't know > what to do about this. It has no instructions on why those fields are gone > or how to remove them. > > Every time you run --hint --execute, you're telling it "Ignore all the > carefully-specified rules that were written to upgrade and migrate data, > and just try to figure it out yourself." It'll do this by diffing the last > stored schema and the schema made up of all the database models and fields > in the product. > > This schema will be incomplete. We have updates that carefully set > defaults for new fields on migrated data, for example. You probably don't > have these applied now. That could lead to some hard-to-diagnose bug in the > future. > > That's why, ideally, you'd temporarily add those fields back. The upgrade > process would see that they're still around and wouldn't worry about them, > allowing it to focus on our carefully crafted upgrade rules. Once applied, > you could then brute-force just their removal with --hint --execute. Of > course, if you've already ran this, it's too late to do that. It might be > fine, but heads up that it might lead to future unexpected behavior. > > There's nothing available that can figure out whether you have the exact > right schema now, and that data was sanely migrated. You'd have to examine > the difference between your database schema and that of a new system, and > examine all the evolution files that were applied, hand-applying any > missing changes carefully to your database. Database repair operations like > this are something we offer under a 1 year Premium Support contract. > > Christian > > > On Friday, January 15, 2016, Jasper Chow <[email protected] <javascript:>> > wrote: > >> Thanks for the feedback Christian >> >> I actually managed to get further by even smaller increments. 1.7.1 -> >> 1.7.2 ->1.7.28 and finally 2.0 >> The process is slow and i have to resolve things like Djblet needs to be >> updated, importlib missing etc >> >> so it seems to be working so far - why is --execute a last resort? what >> does it really do? >> Unfortunately I have to do a --hint --execute EVERYTIME before a rb-site >> upgrade. as rb-site upgrade gives out different conflicts every time - i >> thought as long as i can upgrade once, then subsequent should be good. - ( >> since i am installing using egg from >> http://downloads.reviewboard.org/releases/ReviewBoard/ ) >> >> I looked into our cloned repository and i don't think there is any model >> change, It seems all the changes were cosmetic. Is there a documentation / >> certain step that i can perform to check the discrepancy systematically.? >> >> >> >> >> >> On Friday, January 15, 2016 at 5:12:04 AM UTC-5, Christian Hammond wrote >>> >>> Hi Jasper, >>> >>> This definitely looks to be due to custom model changes made by someone >>> there. You'll need to do some special stuff to disentangle these. You >>> definitely don't want to use --hint --execute is a recipe for disaster (got >>> a backup?). >>> >>> What you're going to have to do is take those old model changes from >>> your custom version of Review Board and port them back to the new version. >>> Then try to do an upgrade. If that works properly, you can then remove >>> those fields again, and *then* run --hint --execute, to get the database >>> evolution history into a sane state. >>> >>> Of course, if you have stuff in those columns that you need still, they >>> will be lost. We have an extra_data field on all these objects, which is >>> where custom state should be stored, so you may want to figure out whether >>> you want to migrate that and update whatever it is that was using these >>> fields. >>> >>> Christian >>> >>> -- >>> Christian Hammond - [email protected] >>> Review Board - https://www.reviewboard.org >>> Beanbag, Inc. - https://www.beanbaginc.com >>> >>> On Wed, Jan 13, 2016 at 1:47 PM, Jasper Chow <[email protected]> wrote: >>> >>>> Update: >>>> >>>> tried rb-site manage /path/to/site evolve -- --hint --execute >>>> but ends with Duplicate column error (password , or permission) >>>> >>>> >>>> hmm. further research it may not be due to our customization , but >>>> another case that the database somehow is in a bad state. >>>> I recall reading from some posts then manual manipulation of database >>>> schema is not recommended. >>>> >>>> >>>> >>>> >>>> On Wednesday, January 13, 2016 at 1:06:43 PM UTC-5, Jasper Chow wrote: >>>>> >>>>> >>>>> Attach evolve log ( rb-site manage /var/www/reviewboard evolve -- >>>>> --hint ) for diagnosis >>>>> >>>>> thanks >>>>> >>>>> >>>>> On Tuesday, January 12, 2016 at 12:58:01 PM UTC-5, Jasper Chow wrote: >>>>>> >>>>>> Hello >>>>>> >>>>>> I am having issue similar to previous reported, while running rb-site >>>>>> upgrade >>>>>> >>>>>> https://hellosplat.com/s/beanbag/tickets/3967/ >>>>>> >>>>>> https://www.mail-archive.com/[email protected]/msg15947.html >>>>>> >>>>>> >>>>>> information: >>>>>> on Linux (CentOS) reviewboard 1.7 -> 2.0.16 ( intended to upgrade to >>>>>> latest but ran into some problem, thought I would do it with a smaller >>>>>> increment ) >>>>>> >>>>>> >>>>>> easy_install ReviewBoard==2.0.16 (successful) >>>>>> rb-site upgrade [path] fails with following error: >>>>>> >>>>>> >>>>>> >>>>>> In model reviews.ReviewRequest: >>>>>> Field 'related_review_number' has been deleted >>>>>> In model reviews.Review >>>>>> Field 'notify_only_submitter' has been deleted >>>>>> In model reviews.ReviewRequestDraft >>>>>> Field 'related_review_number' has been deleted >>>>>> In model accounts.Profile >>>>>> Field 'eclipse_diff_view' has been deleted >>>>>> >>>>>> >>>>>> Judging by the name of the columns it seems some customization done >>>>>> by previous admin here. So question is >>>>>> What is the best practice to migrate these extension columns? >>>>>> >>>>>> >>>>>> I tried >>>>>> - drop the columns from the database ( and to restore them later ) , >>>>>> and run upgrade again >>>>>> - or used the trick provided in another post >>>>>> >>>>>> >>> from django_evolution.models import Version >>>>>> >>> v = Version.objects.all()[0] >>>>>> >>> print v >>>>>> Hinted version, updated on 2015-02-12 02:23:19+00:00 >>>>>> >>> v.delete() >>>>>> >>> >>>>>> >>>>>> but it doesn't seems to help either >>>>>> >>>>>> Or maybe it should be other way around that I need to modify the >>>>>> schema somewhere before i run the upgrade script? >>>>>> >>>>>> any help / suggestion is appreciated.. thank you >>>>>> >>>>>> >>>>>> -- >>>> Supercharge your Review Board with Power Pack: >>>> https://www.reviewboard.org/powerpack/ >>>> Want us to host Review Board for you? Check out RBCommons: >>>> https://rbcommons.com/ >>>> Happy user? Let us know! https://www.reviewboard.org/users/ >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "reviewboard" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> Supercharge your Review Board with Power Pack: >> https://www.reviewboard.org/powerpack/ >> Want us to host Review Board for you? Check out RBCommons: >> https://rbcommons.com/ >> Happy user? Let us know! https://www.reviewboard.org/users/ >> --- >> You received this message because you are subscribed to the Google Groups >> "reviewboard" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> > > > -- > -- > Christian Hammond - [email protected] <javascript:> > Review Board - https://www.reviewboard.org > Beanbag, Inc. - https://www.beanbaginc.com > > -- Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/ Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/ Happy user? Let us know! https://www.reviewboard.org/users/ --- You received this message because you are subscribed to the Google Groups "reviewboard" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
