Re: Mercurial repository access error post rb-site upgrade "Something broke! (Error 500)"

[harish varadarajan]

Hi Christian,

Thanks for the inputs .

As updated in the other thread, copying over the SECRET_KEY from older 
instance to newer instance helped solve the problem, though the server logs 
& the error in itself were misleading.

--
Harish

On Sunday, 31 July 2016 01:50:16 UTC+5:30, Christian Hammond wrote:
>
> Hi Harish,
>
> Can you show me the entire stack trace from the reviewboard.log file?
>
> The database may have been set for UTF-8, but that doesn't mean the tables 
> or data was converted for that. If you look at the MySQL dump, you'll see 
> that each table specifies its encoding. By default for some installs, this 
> is latin1.
>
> It's not sufficient to simply change that in the database dump or set 
> character_set_database. You'll actually need to convert each table:
>
> ALTER TABLE  CONVERT TO CHARACTER SET utf8 COLLATE 
> utf8_unicode_ci;
>
> That may not address this particular issue, though. I'll need a lot more 
> information to know what's breaking and how, and how your repro steps.
>
> Christian
>
> -- 
> Christian Hammond
> President/CEO of Beanbag 
> Makers of Review Board 
>
> On Fri, Jul 29, 2016 at 7:12 PM, harish varadarajan <
> harish.va...@gmail.com > wrote:
>
>> Further, the reviewboard database was created with UTF8 character set.
>>
>> MariaDB [reviewboard]> show variables like "character_set_database";
>> ++---+
>> | Variable_name  | Value |
>> ++---+
>> | character_set_database | utf8  |
>>
>> The specific error from the server log seems to be
>>
>> DjangoUnicodeDecodeError: 'utf8' codec can't decode byte 0x8b in position 
>> 0: invalid start byte. You passed in > at 0x7f4caebd0fd0> ()
>>
>>
>>
>> On Friday, 29 July 2016 20:26:38 UTC+5:30, harish varadarajan wrote:
>>>
>>> Hi,
>>>
>>  
>>
>>>
>>> I am pretty new to reviewboard and have attempted to migrate reviewboard 
>>> instance across hosts and versions.
>>> Our older RB version was and the upgrade was to 2.5.6.1
>>>
>>> Steps followed for the upgrade were
>>> (1) Dump data on the older server (mysql)
>>> (2) Drop the previously created reviewboard database, created by rb-site 
>>> install
>>> (3) Manually create a database named reviewboard with UTF8
>>> (4) load the data from the mysql dump
>>> (5) Run rb-site upgrade.
>>>
>>> Post the upgrade which completed successfully, the dashboard, contents 
>>> and other information came up properly, including the repository listing.
>>> But, the issue is that neither the properties of repositories are 
>>> editable in newer instance nor or the older diff's from reviews accessible 
>>> . The creation of newer reviews for the repositories, which get imported 
>>> are as well not happening due to this..
>>>
>>> While trying to access the repository from Manage -> Repositories -> 
>>> Select repository to change is as below.
>>> Something broke! (Error 500)
>>>
>>> It appears something broke when you tried to go to here. This is either 
>>> a bug in Review Board or a server configuration error. Please report this 
>>> to your administrator.
>>>
>>>
>>> Please note that the creation of newer repositories and using them for 
>>> review requests.
>>>
>>>
>>> The errors in the server error log are similar to the ones below.
>>>
>>> Can someone please throw light on what the issue is and how it could be 
>>> fixed.
>>>
>>>
>>> Unexpected error fetching file from $"http path of the file": 'ascii' 
>>> codec can't decode byte 0xfe in position 3: ordinal not in range(128)
>>>
>>>
>>> - /usr/lib/python2.7/site-packages/django/forms/widgets.py:237: 
>>> UnicodeWarning: Unicode unequal comparison failed to convert both arguments 
>>> to Unicode - interpreting them as being unequal
>>>   if value != '':
>>>
>>>
>>> Thanks.
>>>
>>>
>>> Harish.
>>>
>> -- 
>> Supercharge your Review Board with Power Pack: 
>> https://www.reviewboard.org/powerpack/
>> Want us to host Review Board for you? Check out RBCommons: 
>> https://rbcommons.com/
>> Happy user? Let us know! https://www.reviewboard.org/users/
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "reviewboard" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to reviewboard...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
Supercharge your Review Board with Power Pack: 
https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: 
https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Backup and Restore of Reviewboard 2.0 failing (mysqldump issues)

[harish varadarajan]

Hi Christian,

Thanks for looking into this.
As suggested by Eric, and per update in your mail, I copied over the 
SECRET_KEY, and restarted httpd, and now the repositories are accessible 
and also, the previous review requests created in the older server show up 
their diff's correcty, which was not the case before.

Thanks for your inputs and certainly going over the contents in this group, 
help solved a lot of initial niggles.

--
Harish

On Sunday, 31 July 2016 01:43:53 UTC+5:30, Christian Hammond wrote:
>
> Ah, I see you have another thread on this. I'll respond there.
>
> Christian
>
> -- 
> Christian Hammond
> President/CEO of Beanbag 
> Makers of Review Board 
>
> On Sat, Jul 30, 2016 at 1:12 PM, Christian Hammond  > wrote:
>
>> Hi Harish,
>>
>> I'm not sure I fully understand what problems you're hitting, but a 
>> couple notes:
>>
>> 1) If you're using a database that was set up on an older instance, then 
>> yes, SECRET_KEY must be copied over. If it was not, a number of things like 
>> stored database passwords or hosting service auth tokens will be 
>> inaccessible.
>>
>> Entering a new password for each repository should take care of that. It 
>> sounds like you're hitting a problem doing this? Define "not editable?"
>>
>> 2) You should not have DEBUG = True on, for any reason, really. This is 
>> intended for development environments, and will alter behaviors that are 
>> not suitable for production environments.
>>
>>
>> Can you describe what you're trying to do and exactly what's going wrong?
>>
>> Thanks,
>>
>> Christian
>>
>> -- 
>> Christian Hammond
>> President/CEO of Beanbag 
>> Makers of Review Board 
>>
>> On Fri, Jul 29, 2016 at 7:34 PM, harish varadarajan <
>> harish.va...@gmail.com > wrote:
>>
>>> Hi Eric,
>>>
>>> Thanks for your post, that I could understand the issue that was 
>>> happening at my reviewboard instance.
>>>
>>> Did you figure out the solution for this.
>>> With your approach to turning the Debug = True, I am currently able to 
>>> access the repositories / enter the edit screen.
>>> But unfortunately, the password field for the user who access on 
>>> repositories is not editable.
>>>
>>> But, still while creating newer reviews for older repositories, it 
>>> states that the content / change is not available.
>>> Is the only way to make it work, is to copy the SECRET_KEY from the 
>>> older instance ? 
>>> any ideas on how this could be solved, if access to the older host / 
>>> instance is not available.
>>>
>>> Thanks.
>>>
>>> Harish
>>>
>>> On Thursday, 29 October 2015 12:19:12 UTC+5:30, er...@tibco.com wrote:

 I figured out what my issue was. Copying the database from one machine 
 to another isn't sufficient.

 ReviewBoard has a "SECRET_KEY" that is used when encrypting passwords 
 into the database. If you're restoring your database backup to a different 
 machine (as I was doing), and you don't copy over the SECRET_KEY, then 
 stuff will break.

 If you re-save the passwords from the web UI, it uses the SECRET_KEY 
 when re-obfuscating the password.

 Might that be the problem?

 Eric.

 On Wednesday, October 28, 2015 at 9:56:15 AM UTC-7, er...@tibco.com 
 wrote:
>
> Turns out I've been fighting this issue on and off for several days 
> now.
>
> This looks like it is one of those particularly evil bugs. I've 
> captured a lot of details about it below:
>
> Situation very similar - I'm using mysqldump to backup a database. In 
> my case, I'm trying to restore to a different machine for testing 
> purposes.
>
> When I bring up the test instance, I get an error 500 when I attempt 
> to examine the details of a configured repository.
>
> Email contains this (trimmed somewhat to avoid the long stack trace):
>   File "/usr/lib64/python2.7/site-packages/django/template/base.py", 
> line 897, in render
> return render_value_in_context(output, context)
>
>   File "/usr/lib64/python2.7/site-packages/django/template/base.py", 
> line 875, in render_value_in_context
> value = force_text(value)
>
>   File "/usr/lib64/python2.7/site-packages/django/utils/encoding.py", 
> line 116, in force_text
> raise DjangoUnicodeDecodeError(s, *e.args)
>
> DjangoUnicodeDecodeError: 'utf8' codec can't decode byte 0xad in 
> position 0: invalid start byte. You passed in 
>  ( 'django.forms.forms.BoundField'>)
>
> 
> So, seems obvious - an encoding issue. Except, it isn't.
>
> I've been very careful to ensure that MySQL uses UTF-8 everywhere (on 
> both production and testing). A mysqldump of the tables on the production 
> database, and the test instance database shows that the 
> scmtools_repository 

Re: Mercurial repository access error post rb-site upgrade "Something broke! (Error 500)"

[Christian Hammond]

Hi Harish,

Can you show me the entire stack trace from the reviewboard.log file?

The database may have been set for UTF-8, but that doesn't mean the tables
or data was converted for that. If you look at the MySQL dump, you'll see
that each table specifies its encoding. By default for some installs, this
is latin1.

It's not sufficient to simply change that in the database dump or set
character_set_database. You'll actually need to convert each table:

ALTER TABLE  CONVERT TO CHARACTER SET utf8 COLLATE
utf8_unicode_ci;

That may not address this particular issue, though. I'll need a lot more
information to know what's breaking and how, and how your repro steps.

Christian

-- 
Christian Hammond
President/CEO of Beanbag 
Makers of Review Board 

On Fri, Jul 29, 2016 at 7:12 PM, harish varadarajan <
harish.varadara...@gmail.com> wrote:

> Further, the reviewboard database was created with UTF8 character set.
>
> MariaDB [reviewboard]> show variables like "character_set_database";
> ++---+
> | Variable_name  | Value |
> ++---+
> | character_set_database | utf8  |
>
> The specific error from the server log seems to be
>
> DjangoUnicodeDecodeError: 'utf8' codec can't decode byte 0x8b in position
> 0: invalid start byte. You passed in  at 0x7f4caebd0fd0> ()
>
>
>
> On Friday, 29 July 2016 20:26:38 UTC+5:30, harish varadarajan wrote:
>>
>> Hi,
>>
>
>
>>
>> I am pretty new to reviewboard and have attempted to migrate reviewboard
>> instance across hosts and versions.
>> Our older RB version was and the upgrade was to 2.5.6.1
>>
>> Steps followed for the upgrade were
>> (1) Dump data on the older server (mysql)
>> (2) Drop the previously created reviewboard database, created by rb-site
>> install
>> (3) Manually create a database named reviewboard with UTF8
>> (4) load the data from the mysql dump
>> (5) Run rb-site upgrade.
>>
>> Post the upgrade which completed successfully, the dashboard, contents
>> and other information came up properly, including the repository listing.
>> But, the issue is that neither the properties of repositories are
>> editable in newer instance nor or the older diff's from reviews accessible
>> . The creation of newer reviews for the repositories, which get imported
>> are as well not happening due to this..
>>
>> While trying to access the repository from Manage -> Repositories ->
>> Select repository to change is as below.
>> Something broke! (Error 500)
>>
>> It appears something broke when you tried to go to here. This is either a
>> bug in Review Board or a server configuration error. Please report this to
>> your administrator.
>>
>>
>> Please note that the creation of newer repositories and using them for
>> review requests.
>>
>>
>> The errors in the server error log are similar to the ones below.
>>
>> Can someone please throw light on what the issue is and how it could be
>> fixed.
>>
>>
>> Unexpected error fetching file from $"http path of the file": 'ascii'
>> codec can't decode byte 0xfe in position 3: ordinal not in range(128)
>>
>>
>> - /usr/lib/python2.7/site-packages/django/forms/widgets.py:237: 
>> UnicodeWarning: Unicode unequal comparison failed to convert both arguments 
>> to Unicode - interpreting them as being unequal
>>   if value != '':
>>
>>
>> Thanks.
>>
>>
>> Harish.
>>
> --
> Supercharge your Review Board with Power Pack:
> https://www.reviewboard.org/powerpack/
> Want us to host Review Board for you? Check out RBCommons:
> https://rbcommons.com/
> Happy user? Let us know! https://www.reviewboard.org/users/
> ---
> You received this message because you are subscribed to the Google Groups
> "reviewboard" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to reviewboard+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
Supercharge your Review Board with Power Pack: 
https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: 
https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Backup and Restore of Reviewboard 2.0 failing (mysqldump issues)

[Christian Hammond]

Ah, I see you have another thread on this. I'll respond there.

Christian

-- 
Christian Hammond
President/CEO of Beanbag 
Makers of Review Board 

On Sat, Jul 30, 2016 at 1:12 PM, Christian Hammond  wrote:

> Hi Harish,
>
> I'm not sure I fully understand what problems you're hitting, but a couple
> notes:
>
> 1) If you're using a database that was set up on an older instance, then
> yes, SECRET_KEY must be copied over. If it was not, a number of things like
> stored database passwords or hosting service auth tokens will be
> inaccessible.
>
> Entering a new password for each repository should take care of that. It
> sounds like you're hitting a problem doing this? Define "not editable?"
>
> 2) You should not have DEBUG = True on, for any reason, really. This is
> intended for development environments, and will alter behaviors that are
> not suitable for production environments.
>
>
> Can you describe what you're trying to do and exactly what's going wrong?
>
> Thanks,
>
> Christian
>
> --
> Christian Hammond
> President/CEO of Beanbag 
> Makers of Review Board 
>
> On Fri, Jul 29, 2016 at 7:34 PM, harish varadarajan <
> harish.varadara...@gmail.com> wrote:
>
>> Hi Eric,
>>
>> Thanks for your post, that I could understand the issue that was
>> happening at my reviewboard instance.
>>
>> Did you figure out the solution for this.
>> With your approach to turning the Debug = True, I am currently able to
>> access the repositories / enter the edit screen.
>> But unfortunately, the password field for the user who access on
>> repositories is not editable.
>>
>> But, still while creating newer reviews for older repositories, it states
>> that the content / change is not available.
>> Is the only way to make it work, is to copy the SECRET_KEY from the older
>> instance ?
>> any ideas on how this could be solved, if access to the older host /
>> instance is not available.
>>
>> Thanks.
>>
>> Harish
>>
>> On Thursday, 29 October 2015 12:19:12 UTC+5:30, er...@tibco.com wrote:
>>>
>>> I figured out what my issue was. Copying the database from one machine
>>> to another isn't sufficient.
>>>
>>> ReviewBoard has a "SECRET_KEY" that is used when encrypting passwords
>>> into the database. If you're restoring your database backup to a different
>>> machine (as I was doing), and you don't copy over the SECRET_KEY, then
>>> stuff will break.
>>>
>>> If you re-save the passwords from the web UI, it uses the SECRET_KEY
>>> when re-obfuscating the password.
>>>
>>> Might that be the problem?
>>>
>>> Eric.
>>>
>>> On Wednesday, October 28, 2015 at 9:56:15 AM UTC-7, er...@tibco.com
>>> wrote:

 Turns out I've been fighting this issue on and off for several days now.

 This looks like it is one of those particularly evil bugs. I've
 captured a lot of details about it below:

 Situation very similar - I'm using mysqldump to backup a database. In
 my case, I'm trying to restore to a different machine for testing purposes.

 When I bring up the test instance, I get an error 500 when I attempt to
 examine the details of a configured repository.

 Email contains this (trimmed somewhat to avoid the long stack trace):
   File "/usr/lib64/python2.7/site-packages/django/template/base.py",
 line 897, in render
 return render_value_in_context(output, context)

   File "/usr/lib64/python2.7/site-packages/django/template/base.py",
 line 875, in render_value_in_context
 value = force_text(value)

   File "/usr/lib64/python2.7/site-packages/django/utils/encoding.py",
 line 116, in force_text
 raise DjangoUnicodeDecodeError(s, *e.args)

 DjangoUnicodeDecodeError: 'utf8' codec can't decode byte 0xad in
 position 0: invalid start byte. You passed in
  (>>> 'django.forms.forms.BoundField'>)

 
 So, seems obvious - an encoding issue. Except, it isn't.

 I've been very careful to ensure that MySQL uses UTF-8 everywhere (on
 both production and testing). A mysqldump of the tables on the production
 database, and the test instance database shows that the scmtools_repository
 tables are *identical*, including the schema, and show use of UTF-8 as
 the charset. I even did a mysql --xml -e query of the tables, and found
 that they're identical. A hexdump of the results shows no non-ASCII
 characters. In fact, the only odd character is a 09 at the beginning of the
 password field (which looks to be intentional?). Mind you, this is as
 expected, because the password field looks to be a base64 encoding of an
 actual password, so it necessarily wouldn't encode any non-ASCII 
 characters.

 I've confirmed that changing the password field using SQL, then
 resetting the password in the web UI makes the problem go away 

Re: Backup and Restore of Reviewboard 2.0 failing (mysqldump issues)

[Christian Hammond]

Hi Harish,

I'm not sure I fully understand what problems you're hitting, but a couple
notes:

1) If you're using a database that was set up on an older instance, then
yes, SECRET_KEY must be copied over. If it was not, a number of things like
stored database passwords or hosting service auth tokens will be
inaccessible.

Entering a new password for each repository should take care of that. It
sounds like you're hitting a problem doing this? Define "not editable?"

2) You should not have DEBUG = True on, for any reason, really. This is
intended for development environments, and will alter behaviors that are
not suitable for production environments.


Can you describe what you're trying to do and exactly what's going wrong?

Thanks,

Christian

-- 
Christian Hammond
President/CEO of Beanbag 
Makers of Review Board 

On Fri, Jul 29, 2016 at 7:34 PM, harish varadarajan <
harish.varadara...@gmail.com> wrote:

> Hi Eric,
>
> Thanks for your post, that I could understand the issue that was happening
> at my reviewboard instance.
>
> Did you figure out the solution for this.
> With your approach to turning the Debug = True, I am currently able to
> access the repositories / enter the edit screen.
> But unfortunately, the password field for the user who access on
> repositories is not editable.
>
> But, still while creating newer reviews for older repositories, it states
> that the content / change is not available.
> Is the only way to make it work, is to copy the SECRET_KEY from the older
> instance ?
> any ideas on how this could be solved, if access to the older host /
> instance is not available.
>
> Thanks.
>
> Harish
>
> On Thursday, 29 October 2015 12:19:12 UTC+5:30, er...@tibco.com wrote:
>>
>> I figured out what my issue was. Copying the database from one machine to
>> another isn't sufficient.
>>
>> ReviewBoard has a "SECRET_KEY" that is used when encrypting passwords
>> into the database. If you're restoring your database backup to a different
>> machine (as I was doing), and you don't copy over the SECRET_KEY, then
>> stuff will break.
>>
>> If you re-save the passwords from the web UI, it uses the SECRET_KEY when
>> re-obfuscating the password.
>>
>> Might that be the problem?
>>
>> Eric.
>>
>> On Wednesday, October 28, 2015 at 9:56:15 AM UTC-7, er...@tibco.com
>> wrote:
>>>
>>> Turns out I've been fighting this issue on and off for several days now.
>>>
>>> This looks like it is one of those particularly evil bugs. I've captured
>>> a lot of details about it below:
>>>
>>> Situation very similar - I'm using mysqldump to backup a database. In my
>>> case, I'm trying to restore to a different machine for testing purposes.
>>>
>>> When I bring up the test instance, I get an error 500 when I attempt to
>>> examine the details of a configured repository.
>>>
>>> Email contains this (trimmed somewhat to avoid the long stack trace):
>>>   File "/usr/lib64/python2.7/site-packages/django/template/base.py",
>>> line 897, in render
>>> return render_value_in_context(output, context)
>>>
>>>   File "/usr/lib64/python2.7/site-packages/django/template/base.py",
>>> line 875, in render_value_in_context
>>> value = force_text(value)
>>>
>>>   File "/usr/lib64/python2.7/site-packages/django/utils/encoding.py",
>>> line 116, in force_text
>>> raise DjangoUnicodeDecodeError(s, *e.args)
>>>
>>> DjangoUnicodeDecodeError: 'utf8' codec can't decode byte 0xad in
>>> position 0: invalid start byte. You passed in
>>>  (>> 'django.forms.forms.BoundField'>)
>>>
>>> 
>>> So, seems obvious - an encoding issue. Except, it isn't.
>>>
>>> I've been very careful to ensure that MySQL uses UTF-8 everywhere (on
>>> both production and testing). A mysqldump of the tables on the production
>>> database, and the test instance database shows that the scmtools_repository
>>> tables are *identical*, including the schema, and show use of UTF-8 as
>>> the charset. I even did a mysql --xml -e query of the tables, and found
>>> that they're identical. A hexdump of the results shows no non-ASCII
>>> characters. In fact, the only odd character is a 09 at the beginning of the
>>> password field (which looks to be intentional?). Mind you, this is as
>>> expected, because the password field looks to be a base64 encoding of an
>>> actual password, so it necessarily wouldn't encode any non-ASCII characters.
>>>
>>> I've confirmed that changing the password field using SQL, then
>>> resetting the password in the web UI makes the problem go away (for how
>>> long?).
>>>
>>> So now for the *really* messed up part. To try to get more / better
>>> diagnostics, I went to the "conf" folder for my ReviewBoard instance. I
>>> changed "Debug" flag to "True". Restart Apache (hosted via wsgi, FYI).
>>>
>>> Ready to look at details problem reports. Except ... problem gone
>>>
>>> I've now repeated this experiment several times. Set "Debug" to "False",
>>> problem occurs. "Debug = True",