I have finally re-run my entire automated migration process from start to 
finish.

After running condensediffs *before* the upgrade, I was then able to run 
condensediffs after the upgrade as well, all without any warnings or errors.

Perhaps you should add a step to the upgrade guide, advising Administrators 
to rebuild their search index, and also condense diffs?

So I seem to have found a work-around to the problem - in case anyone else 
runs into it.

Eric.

On Saturday, May 14, 2016 at 1:40:06 PM UTC-7, er...@tibco.com wrote:
>
> I re-ran the migration, and before doing anything, set the DEBUG flag to 
> True before fetching any diffs.
>
> I can confirm that it fails on *every* diff, not just random ones here and 
> there.
>
> Running with DEBUG = False, the warning doesn't throw an exception, but I 
> do see an entry in the log:
> WARNING:py.warnings:/usr/lib64/python2.7/site-packages/django/db/backends/mysql/base.py:124:
>  
> Warning: Invalid utf8 character string: '81FE48'
>
> After I've viewed the diffs of a review request once, the warnings stop 
> appearing (consistent with the data being migrated to compressed form.). 
> Switching back to DEBUG = True, and I no longer see failures for the review 
> request diffs that I looked at while DEBUG = False
>
> Trying a different approach, I ran "rb-site manage ______ -- 
> condensediffs", and that also generated the same warnings. Here's a sample:
>
> WARNING:py.warnings:/usr/lib64/python2.7/site-packages/django/db/backends/mysql/base.py:124:
>  
> Warning: Invalid utf8 character string: 'D76700'
>
>   return self.cursor.execute(query, args)
>
>
> WARNING:py.warnings:/usr/lib64/python2.7/site-packages/django/db/backends/mysql/base.py:124:
>  
> Warning: Invalid utf8 character string: 'A9C813'
>
>   return self.cursor.execute(query, args)
>
>
> ...
>
>
> This generated 169000+ lines of output, corresponding to 56435 individual 
> warning messages out of a total of 76032 diff files condensed.
>
>
> This gave me an idea.
>
>
> I ran "rb-site ... condensediffs" before the upgrade, then once again 
> after the upgrade.
>
>
> Problem went away.
>
>
> Weird. Do you want me to try to find out more, and if so, what?
>
>
> Eric.
>
>
>
> On Friday, May 13, 2016 at 1:49:53 PM UTC-7, Christian Hammond wrote:
>>
>> Hi Eric,
>>
>> Hmm, we'll need to look into that. Is there a way you'd be able to send 
>> us the diff for that? (I can help you find it.) We will need a copy in 
>> order to diagnose this. We can sign an NDA for it.
>>
>> Christian
>>
>>
>> On Friday, May 13, 2016, eric via reviewboard <revie...@googlegroups.com> 
>> wrote:
>>
>>> After I migrated my server to 2.5.4, I'm seeing a weird error. I 
>>> restarted both memcached and apache2, and then browse to a specific review 
>>> request.
>>>
>>> Then I click on the "Diff" tab. (After I turned on DEBUG = True in the 
>>> settings_local.py file) I see this instead of diffs.
>>>
>>> Traceback (most recent call last):
>>>   File 
>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/views.py",
>>>  
>>> line 275, in get
>>>     response = renderer.render_to_response(request)
>>>   File 
>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/renderers.py",
>>>  
>>> line 56, in render_to_response
>>>     return HttpResponse(self.render_to_string(request))
>>>   File 
>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/renderers.py",
>>>  
>>> line 74, in render_to_string
>>>     large_data=True)
>>>   File 
>>> "/usr/lib64/python2.7/site-packages/Djblets-0.9.3-py2.7.egg/djblets/cache/backend.py",
>>>  
>>> line 295, in cache_memoize
>>>     compress_large_data))
>>>   File 
>>> "/usr/lib64/python2.7/site-packages/Djblets-0.9.3-py2.7.egg/djblets/cache/backend.py",
>>>  
>>> line 249, in cache_memoize_iter
>>>     items = items_or_callable()
>>>   File 
>>> "/usr/lib64/python2.7/site-packages/Djblets-0.9.3-py2.7.egg/djblets/cache/backend.py",
>>>  
>>> line 292, in <lambda>
>>>     lambda: [lookup_callable()],
>>>   File 
>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/renderers.py",
>>>  
>>> line 73, in <lambda>
>>>     lambda: self.render_to_string_uncached(request),
>>>   File 
>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/renderers.py",
>>>  
>>> line 87, in render_to_string_uncached
>>>     request=request)
>>>   File 
>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/diffutils.py",
>>>  
>>> line 429, in populate_diff_chunks
>>>     chunks = list(generator.get_chunks())
>>>   File 
>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/chunk_generator.py",
>>>  
>>> line 756, in get_chunks
>>>     for chunk in super(DiffChunkGenerator, self).get_chunks(cache_key):
>>>   File 
>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/chunk_generator.py",
>>>  
>>> line 107, in get_chunks
>>>     large_data=True)
>>>   File 
>>> "/usr/lib64/python2.7/site-packages/Djblets-0.9.3-py2.7.egg/djblets/cache/backend.py",
>>>  
>>> line 295, in cache_memoize
>>>     compress_large_data))
>>>   File 
>>> "/usr/lib64/python2.7/site-packages/Djblets-0.9.3-py2.7.egg/djblets/cache/backend.py",
>>>  
>>> line 249, in cache_memoize_iter
>>>     items = items_or_callable()
>>>   File 
>>> "/usr/lib64/python2.7/site-packages/Djblets-0.9.3-py2.7.egg/djblets/cache/backend.py",
>>>  
>>> line 292, in <lambda>
>>>     lambda: [lookup_callable()],
>>>   File 
>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/chunk_generator.py",
>>>  
>>> line 106, in <lambda>
>>>     lambda: list(self.get_chunks_uncached()),
>>>   File 
>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/chunk_generator.py",
>>>  
>>> line 763, in get_chunks_uncached
>>>     new = get_patched_file(old, self.filediff, self.request)
>>>   File 
>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/diffutils.py",
>>>  
>>> line 230, in get_patched_file
>>>     diff = tool.normalize_patch(filediff.diff, filediff.source_file,
>>>   File 
>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/models.py",
>>>  
>>> line 218, in _get_diff
>>>     self._migrate_diff_data()
>>>   File 
>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/models.py",
>>>  
>>> line 421, in _migrate_diff_data
>>>     diff_hash_is_new = self._set_diff(self.legacy_diff_hash.binary)
>>>   File 
>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/models.py",
>>>  
>>> line 225, in _set_diff
>>>     RawFileDiffData.objects.get_or_create_from_data(diff)
>>>   File 
>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/managers.py",
>>>  
>>> line 345, in get_or_create_from_data
>>>     'compression': compression,
>>>   File "/usr/lib64/python2.7/site-packages/django/db/models/manager.py", 
>>> line 154, in get_or_create
>>>     return self.get_queryset().get_or_create(**kwargs)
>>>   File "/usr/lib64/python2.7/site-packages/django/db/models/query.py", 
>>> line 383, in get_or_create
>>>     obj.save(force_insert=True, using=self.db)
>>>   File "/usr/lib64/python2.7/site-packages/django/db/models/base.py", 
>>> line 545, in save
>>>     force_update=force_update, update_fields=update_fields)
>>>   File "/usr/lib64/python2.7/site-packages/django/db/models/base.py", 
>>> line 573, in save_base
>>>     updated = self._save_table(raw, cls, force_insert, force_update, 
>>> using, update_fields)
>>>   File "/usr/lib64/python2.7/site-packages/django/db/models/base.py", 
>>> line 654, in _save_table
>>>     result = self._do_insert(cls._base_manager, using, fields, 
>>> update_pk, raw)
>>>   File "/usr/lib64/python2.7/site-packages/django/db/models/base.py", 
>>> line 687, in _do_insert
>>>     using=using, raw=raw)
>>>   File "/usr/lib64/python2.7/site-packages/django/db/models/manager.py", 
>>> line 232, in _insert
>>>     return insert_query(self.model, objs, fields, **kwargs)
>>>   File "/usr/lib64/python2.7/site-packages/django/db/models/query.py", 
>>> line 1514, in insert_query
>>>     return query.get_compiler(using=using).execute_sql(return_id)
>>>   File 
>>> "/usr/lib64/python2.7/site-packages/django/db/models/sql/compiler.py", line 
>>> 903, in execute_sql
>>>     cursor.execute(sql, params)
>>>   File 
>>> "/usr/lib64/python2.7/site-packages/Djblets-0.9.3-py2.7.egg/djblets/log/middleware.py",
>>>  
>>> line 32, in execute
>>>     return self.cursor.execute(sql, params)
>>>   File 
>>> "/usr/lib64/python2.7/site-packages/django/db/backends/mysql/base.py", line 
>>> 124, in execute
>>>     return self.cursor.execute(query, args)
>>>   File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 
>>> 207, in execute
>>>     if not self._defer_warnings: self._warning_check()
>>>   File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 
>>> 117, in _warning_check
>>>     warn(w[-1], self.Warning, 3)
>>> Warning: Invalid utf8 character string: '890600'
>>>
>>>    
>>> Note that if I set DEBUG = False, then I see the diffs in the browser, 
>>> but I still see warnings in the log file. I'm nervous that migrating my 
>>> production system forward to 2.5.4 will corrupt data. So now I'm holding 
>>> off.
>>>
>>> Looking back through the stack trace, I see there's some logic in there 
>>> about "_migrate_diff_data" - code which didn't exist in 2.0.X.
>>>
>>> I suspect this is a manifestation of this bug:
>>> https://hellosplat.com/s/beanbag/tickets/4007/
>>>
>>> What can I do?
>>>
>>> Eric.
>>>     
>>>
>>> -- 
>>> Supercharge your Review Board with Power Pack: 
>>> https://www.reviewboard.org/powerpack/
>>> Want us to host Review Board for you? Check out RBCommons: 
>>> https://rbcommons.com/
>>> Happy user? Let us know! https://www.reviewboard.org/users/
>>> --- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "reviewboard" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to reviewboard+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>> -- 
>> -- 
>> Christian Hammond
>> President/CEO of Beanbag <https://www.beanbaginc.com/>
>> Makers of Review Board <https://www.reviewboard.org/>
>>
>>

-- 
Supercharge your Review Board with Power Pack: 
https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: 
https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to