Thanks for the detailed explanation Christian . I have a clear picture now. 
really appreciated. 

Don't think it is a good idea to ignore the rules applied to schema and 
just -execute all the way in every release.  As eventually I will run into 
much bigger problem in the future. I revisited the fields mentioned in 
--hint and able to trace down to some customization done quite a while ago. 

Attempting to turn the schema back to sane without losing data. But thanks 
for the info and help so far. 

Jasper





On Saturday, January 16, 2016 at 11:00:35 AM UTC-5, Christian Hammond wrote:
>
> Hi Jasper,
>
> --execute is fine, and it's what rb-site runs. It's the combination of 
> that and --hint that gets you into trouble. I'll explain what's going on.
>
> Since we need to make changes to the schema for the database every so 
> often, and since we support a variety of databases, we need to use a tool 
> to let us record the changes we need to apply at a high level and manage 
> applying those when it's time to upgrade. This tool is called 
> django-evolution, and it's something we maintain these days.
>
> In order to be able to sanely apply changes, every single change ever must 
> be recorded as part of the product, so that they can be scanned and matched 
> up during the upgrade process. These are stored in "evolution files" 
> (little bits of Python). Once an upgrade is performed, new state is 
> recorded in a couple tables in the database containing the entire list of 
> evolutions applied and the resulting signature of the database.
>
> In your case, at some point somebody modified the Review Board code and 
> added new fields to some of our tables. Maybe evolution files were added, 
> or maybe --hint --execute was used. Either way, not being part of the 
> product meant that future upgrades had no way of factoring in these fields. 
> Django-evolution knows it once existed but no longer does, and doesn't know 
> what to do about this. It has no instructions on why those fields are gone 
> or how to remove them.
>
> Every time you run --hint --execute, you're telling it "Ignore all the 
> carefully-specified rules that were written to upgrade and migrate data, 
> and just try to figure it out yourself." It'll do this by diffing the last 
> stored schema and the schema made up of all the database models and fields 
> in the product.
>
> This schema will be incomplete. We have updates that carefully set 
> defaults for new fields on migrated data, for example. You probably don't 
> have these applied now. That could lead to some hard-to-diagnose bug in the 
> future.
>
> That's why, ideally, you'd temporarily add those fields back. The upgrade 
> process would see that they're still around and wouldn't worry about them, 
> allowing it to focus on our carefully crafted upgrade rules. Once applied, 
> you could then brute-force just their removal with --hint --execute. Of 
> course, if you've already ran this, it's too late to do that. It might be 
> fine, but heads up that it might lead to future unexpected behavior.
>
> There's nothing available that can figure out whether you have the exact 
> right schema now, and that data was sanely migrated. You'd have to examine 
> the difference between your database schema and that of a new system, and 
> examine all the evolution files that were applied, hand-applying any 
> missing changes carefully to your database. Database repair operations like 
> this are something we offer under a 1 year Premium Support contract.
>
> Christian
>
>
> On Friday, January 15, 2016, Jasper Chow <jaspe...@gmail.com <javascript:>> 
> wrote:
>
>> Thanks for the feedback Christian 
>>
>> I actually managed to get further by even smaller increments. 1.7.1 -> 
>> 1.7.2 ->1.7.28 and finally 2.0   
>> The process is slow and i have to resolve things like Djblet needs to be 
>> updated, importlib missing etc
>>
>> so it seems to be working so far - why is --execute a last resort? what 
>> does it really do?
>> Unfortunately I have to do  a --hint --execute EVERYTIME before a rb-site 
>> upgrade.   as rb-site upgrade gives out different conflicts every time - i 
>> thought as long as i can upgrade once, then subsequent should be good. - ( 
>> since i am installing using egg from 
>> http://downloads.reviewboard.org/releases/ReviewBoard/ )  
>>
>> I looked into our cloned repository and i don't think there is any model 
>> change, It seems all the changes were cosmetic. Is there a documentation / 
>> certain step that i can perform to check the discrepancy systematically.?  
>>
>>
>>
>>
>>
>> On Friday, January 15, 2016 at 5:12:04 AM UTC-5, Christian Hammond wrote
>>>
>>> Hi Jasper,
>>>
>>> This definitely looks to be due to custom model changes made by someone 
>>> there. You'll need to do some special stuff to disentangle these. You 
>>> definitely don't want to use --hint --execute is a recipe for disaster (got 
>>> a backup?).
>>>
>>> What you're going to have to do is take those old model changes from 
>>> your custom version of Review Board and port them back to the new version. 
>>> Then try to do an upgrade. If that works properly, you can then remove 
>>> those fields again, and *then* run --hint --execute, to get the database 
>>> evolution history into a sane state.
>>>
>>> Of course, if you have stuff in those columns that you need still, they 
>>> will be lost. We have an extra_data field on all these objects, which is 
>>> where custom state should be stored, so you may want to figure out whether 
>>> you want to migrate that and update whatever it is that was using these 
>>> fields.
>>>
>>> Christian
>>>
>>> -- 
>>> Christian Hammond - chri...@beanbaginc.com
>>> Review Board - https://www.reviewboard.org
>>> Beanbag, Inc. - https://www.beanbaginc.com
>>>
>>> On Wed, Jan 13, 2016 at 1:47 PM, Jasper Chow <jaspe...@gmail.com> wrote:
>>>
>>>> Update:
>>>>
>>>> tried rb-site manage /path/to/site evolve -- --hint --execute
>>>> but ends with Duplicate column error (password , or permission)
>>>>
>>>>
>>>> hmm. further research it may not be due to our customization , but 
>>>> another case that the database somehow is in a bad state.  
>>>> I recall reading from some posts then manual manipulation of database 
>>>> schema is not recommended. 
>>>>
>>>>
>>>>
>>>>
>>>> On Wednesday, January 13, 2016 at 1:06:43 PM UTC-5, Jasper Chow wrote:
>>>>>
>>>>>
>>>>> Attach evolve log (  rb-site manage /var/www/reviewboard evolve -- 
>>>>> --hint )  for diagnosis
>>>>>
>>>>> thanks
>>>>>
>>>>>
>>>>> On Tuesday, January 12, 2016 at 12:58:01 PM UTC-5, Jasper Chow wrote:
>>>>>>
>>>>>> Hello 
>>>>>>
>>>>>> I am having issue similar to previous reported, while running rb-site 
>>>>>> upgrade
>>>>>>
>>>>>> https://hellosplat.com/s/beanbag/tickets/3967/
>>>>>>
>>>>>> https://www.mail-archive.com/reviewboard@googlegroups.com/msg15947.html
>>>>>>
>>>>>>
>>>>>> information:
>>>>>> on Linux (CentOS) reviewboard 1.7 -> 2.0.16 ( intended to upgrade to 
>>>>>> latest but ran into some problem, thought I would do it with a smaller 
>>>>>> increment )
>>>>>>
>>>>>>
>>>>>> easy_install ReviewBoard==2.0.16   (successful)
>>>>>> rb-site upgrade [path]      fails with following error:
>>>>>>
>>>>>>
>>>>>>
>>>>>> In model reviews.ReviewRequest: 
>>>>>> Field 'related_review_number' has been deleted
>>>>>> In model reviews.Review
>>>>>> Field 'notify_only_submitter' has been deleted
>>>>>> In model reviews.ReviewRequestDraft
>>>>>> Field 'related_review_number' has been deleted
>>>>>> In model accounts.Profile
>>>>>> Field 'eclipse_diff_view' has been deleted
>>>>>>
>>>>>>
>>>>>> Judging by the name of the columns it seems some customization done 
>>>>>> by previous admin here. So question is
>>>>>> What is the best practice to  migrate these extension columns?
>>>>>>
>>>>>>
>>>>>> I tried 
>>>>>> - drop the columns from the database ( and to restore them later ) , 
>>>>>> and run upgrade again 
>>>>>> - or used the trick provided in another post 
>>>>>>
>>>>>> >>> from django_evolution.models import Version
>>>>>> >>> v = Version.objects.all()[0]
>>>>>> >>> print v
>>>>>> Hinted version, updated on 2015-02-12 02:23:19+00:00
>>>>>> >>> v.delete()
>>>>>> >>> 
>>>>>>
>>>>>>  but it doesn't seems to help either 
>>>>>>
>>>>>> Or maybe it should be other way around that  I need to modify the 
>>>>>> schema somewhere before i run the upgrade script?
>>>>>>
>>>>>> any help / suggestion is appreciated.. thank you
>>>>>>
>>>>>>
>>>>>> -- 
>>>> Supercharge your Review Board with Power Pack: 
>>>> https://www.reviewboard.org/powerpack/
>>>> Want us to host Review Board for you? Check out RBCommons: 
>>>> https://rbcommons.com/
>>>> Happy user? Let us know! https://www.reviewboard.org/users/
>>>> --- 
>>>> You received this message because you are subscribed to the Google 
>>>> Groups "reviewboard" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send 
>>>> an email to reviewboard...@googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>> -- 
>> Supercharge your Review Board with Power Pack: 
>> https://www.reviewboard.org/powerpack/
>> Want us to host Review Board for you? Check out RBCommons: 
>> https://rbcommons.com/
>> Happy user? Let us know! https://www.reviewboard.org/users/
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "reviewboard" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to reviewboard+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> -- 
> -- 
> Christian Hammond - chri...@beanbaginc.com <javascript:>
> Review Board - https://www.reviewboard.org
> Beanbag, Inc. - https://www.beanbaginc.com
>
>

-- 
Supercharge your Review Board with Power Pack: 
https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: 
https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to