Hi Eric,

I'd still really love to get a copy of those entries from your database so
I can replicate the problem. I simply can't make it happen here, and have
no idea why you'd be seeing what you're seeing. It should not be necessary
to run condensediffs prior to upgrading.

Christian

-- 
Christian Hammond
President/CEO of Beanbag <https://www.beanbaginc.com/>
Makers of Review Board <https://www.reviewboard.org/>

On Sun, May 15, 2016 at 1:11 PM, eric via reviewboard <
reviewboard@googlegroups.com> wrote:

> I have finally re-run my entire automated migration process from start to
> finish.
>
> After running condensediffs *before* the upgrade, I was then able to run
> condensediffs after the upgrade as well, all without any warnings or errors.
>
> Perhaps you should add a step to the upgrade guide, advising
> Administrators to rebuild their search index, and also condense diffs?
>
> So I seem to have found a work-around to the problem - in case anyone else
> runs into it.
>
> Eric.
>
>
> On Saturday, May 14, 2016 at 1:40:06 PM UTC-7, er...@tibco.com wrote:
>>
>> I re-ran the migration, and before doing anything, set the DEBUG flag to
>> True before fetching any diffs.
>>
>> I can confirm that it fails on *every* diff, not just random ones here
>> and there.
>>
>> Running with DEBUG = False, the warning doesn't throw an exception, but I
>> do see an entry in the log:
>> WARNING:py.warnings:/usr/lib64/python2.7/site-packages/django/db/backends/mysql/base.py:124:
>> Warning: Invalid utf8 character string: '81FE48'
>>
>> After I've viewed the diffs of a review request once, the warnings stop
>> appearing (consistent with the data being migrated to compressed form.).
>> Switching back to DEBUG = True, and I no longer see failures for the review
>> request diffs that I looked at while DEBUG = False
>>
>> Trying a different approach, I ran "rb-site manage ______ --
>> condensediffs", and that also generated the same warnings. Here's a sample:
>>
>> WARNING:py.warnings:/usr/lib64/python2.7/site-packages/django/db/backends/mysql/base.py:124:
>> Warning: Invalid utf8 character string: 'D76700'
>>
>>   return self.cursor.execute(query, args)
>>
>>
>> WARNING:py.warnings:/usr/lib64/python2.7/site-packages/django/db/backends/mysql/base.py:124:
>> Warning: Invalid utf8 character string: 'A9C813'
>>
>>   return self.cursor.execute(query, args)
>>
>>
>> ...
>>
>>
>> This generated 169000+ lines of output, corresponding to 56435 individual
>> warning messages out of a total of 76032 diff files condensed.
>>
>>
>> This gave me an idea.
>>
>>
>> I ran "rb-site ... condensediffs" before the upgrade, then once again
>> after the upgrade.
>>
>>
>> Problem went away.
>>
>>
>> Weird. Do you want me to try to find out more, and if so, what?
>>
>>
>> Eric.
>>
>>
>>
>> On Friday, May 13, 2016 at 1:49:53 PM UTC-7, Christian Hammond wrote:
>>>
>>> Hi Eric,
>>>
>>> Hmm, we'll need to look into that. Is there a way you'd be able to send
>>> us the diff for that? (I can help you find it.) We will need a copy in
>>> order to diagnose this. We can sign an NDA for it.
>>>
>>> Christian
>>>
>>>
>>> On Friday, May 13, 2016, eric via reviewboard <revie...@googlegroups.com>
>>> wrote:
>>>
>>>> After I migrated my server to 2.5.4, I'm seeing a weird error. I
>>>> restarted both memcached and apache2, and then browse to a specific review
>>>> request.
>>>>
>>>> Then I click on the "Diff" tab. (After I turned on DEBUG = True in the
>>>> settings_local.py file) I see this instead of diffs.
>>>>
>>>> Traceback (most recent call last):
>>>>   File
>>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/views.py",
>>>> line 275, in get
>>>>     response = renderer.render_to_response(request)
>>>>   File
>>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/renderers.py",
>>>> line 56, in render_to_response
>>>>     return HttpResponse(self.render_to_string(request))
>>>>   File
>>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/renderers.py",
>>>> line 74, in render_to_string
>>>>     large_data=True)
>>>>   File
>>>> "/usr/lib64/python2.7/site-packages/Djblets-0.9.3-py2.7.egg/djblets/cache/backend.py",
>>>> line 295, in cache_memoize
>>>>     compress_large_data))
>>>>   File
>>>> "/usr/lib64/python2.7/site-packages/Djblets-0.9.3-py2.7.egg/djblets/cache/backend.py",
>>>> line 249, in cache_memoize_iter
>>>>     items = items_or_callable()
>>>>   File
>>>> "/usr/lib64/python2.7/site-packages/Djblets-0.9.3-py2.7.egg/djblets/cache/backend.py",
>>>> line 292, in <lambda>
>>>>     lambda: [lookup_callable()],
>>>>   File
>>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/renderers.py",
>>>> line 73, in <lambda>
>>>>     lambda: self.render_to_string_uncached(request),
>>>>   File
>>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/renderers.py",
>>>> line 87, in render_to_string_uncached
>>>>     request=request)
>>>>   File
>>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/diffutils.py",
>>>> line 429, in populate_diff_chunks
>>>>     chunks = list(generator.get_chunks())
>>>>   File
>>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/chunk_generator.py",
>>>> line 756, in get_chunks
>>>>     for chunk in super(DiffChunkGenerator, self).get_chunks(cache_key):
>>>>   File
>>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/chunk_generator.py",
>>>> line 107, in get_chunks
>>>>     large_data=True)
>>>>   File
>>>> "/usr/lib64/python2.7/site-packages/Djblets-0.9.3-py2.7.egg/djblets/cache/backend.py",
>>>> line 295, in cache_memoize
>>>>     compress_large_data))
>>>>   File
>>>> "/usr/lib64/python2.7/site-packages/Djblets-0.9.3-py2.7.egg/djblets/cache/backend.py",
>>>> line 249, in cache_memoize_iter
>>>>     items = items_or_callable()
>>>>   File
>>>> "/usr/lib64/python2.7/site-packages/Djblets-0.9.3-py2.7.egg/djblets/cache/backend.py",
>>>> line 292, in <lambda>
>>>>     lambda: [lookup_callable()],
>>>>   File
>>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/chunk_generator.py",
>>>> line 106, in <lambda>
>>>>     lambda: list(self.get_chunks_uncached()),
>>>>   File
>>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/chunk_generator.py",
>>>> line 763, in get_chunks_uncached
>>>>     new = get_patched_file(old, self.filediff, self.request)
>>>>   File
>>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/diffutils.py",
>>>> line 230, in get_patched_file
>>>>     diff = tool.normalize_patch(filediff.diff, filediff.source_file,
>>>>   File
>>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/models.py",
>>>> line 218, in _get_diff
>>>>     self._migrate_diff_data()
>>>>   File
>>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/models.py",
>>>> line 421, in _migrate_diff_data
>>>>     diff_hash_is_new = self._set_diff(self.legacy_diff_hash.binary)
>>>>   File
>>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/models.py",
>>>> line 225, in _set_diff
>>>>     RawFileDiffData.objects.get_or_create_from_data(diff)
>>>>   File
>>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/managers.py",
>>>> line 345, in get_or_create_from_data
>>>>     'compression': compression,
>>>>   File
>>>> "/usr/lib64/python2.7/site-packages/django/db/models/manager.py", line 154,
>>>> in get_or_create
>>>>     return self.get_queryset().get_or_create(**kwargs)
>>>>   File "/usr/lib64/python2.7/site-packages/django/db/models/query.py",
>>>> line 383, in get_or_create
>>>>     obj.save(force_insert=True, using=self.db)
>>>>   File "/usr/lib64/python2.7/site-packages/django/db/models/base.py",
>>>> line 545, in save
>>>>     force_update=force_update, update_fields=update_fields)
>>>>   File "/usr/lib64/python2.7/site-packages/django/db/models/base.py",
>>>> line 573, in save_base
>>>>     updated = self._save_table(raw, cls, force_insert, force_update,
>>>> using, update_fields)
>>>>   File "/usr/lib64/python2.7/site-packages/django/db/models/base.py",
>>>> line 654, in _save_table
>>>>     result = self._do_insert(cls._base_manager, using, fields,
>>>> update_pk, raw)
>>>>   File "/usr/lib64/python2.7/site-packages/django/db/models/base.py",
>>>> line 687, in _do_insert
>>>>     using=using, raw=raw)
>>>>   File
>>>> "/usr/lib64/python2.7/site-packages/django/db/models/manager.py", line 232,
>>>> in _insert
>>>>     return insert_query(self.model, objs, fields, **kwargs)
>>>>   File "/usr/lib64/python2.7/site-packages/django/db/models/query.py",
>>>> line 1514, in insert_query
>>>>     return query.get_compiler(using=using).execute_sql(return_id)
>>>>   File
>>>> "/usr/lib64/python2.7/site-packages/django/db/models/sql/compiler.py", line
>>>> 903, in execute_sql
>>>>     cursor.execute(sql, params)
>>>>   File
>>>> "/usr/lib64/python2.7/site-packages/Djblets-0.9.3-py2.7.egg/djblets/log/middleware.py",
>>>> line 32, in execute
>>>>     return self.cursor.execute(sql, params)
>>>>   File
>>>> "/usr/lib64/python2.7/site-packages/django/db/backends/mysql/base.py", line
>>>> 124, in execute
>>>>     return self.cursor.execute(query, args)
>>>>   File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line
>>>> 207, in execute
>>>>     if not self._defer_warnings: self._warning_check()
>>>>   File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line
>>>> 117, in _warning_check
>>>>     warn(w[-1], self.Warning, 3)
>>>> Warning: Invalid utf8 character string: '890600'
>>>>
>>>>
>>>> Note that if I set DEBUG = False, then I see the diffs in the browser,
>>>> but I still see warnings in the log file. I'm nervous that migrating my
>>>> production system forward to 2.5.4 will corrupt data. So now I'm holding
>>>> off.
>>>>
>>>> Looking back through the stack trace, I see there's some logic in there
>>>> about "_migrate_diff_data" - code which didn't exist in 2.0.X.
>>>>
>>>> I suspect this is a manifestation of this bug:
>>>> https://hellosplat.com/s/beanbag/tickets/4007/
>>>>
>>>> What can I do?
>>>>
>>>> Eric.
>>>>
>>>>
>>>> --
>>>> Supercharge your Review Board with Power Pack:
>>>> https://www.reviewboard.org/powerpack/
>>>> Want us to host Review Board for you? Check out RBCommons:
>>>> https://rbcommons.com/
>>>> Happy user? Let us know! https://www.reviewboard.org/users/
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "reviewboard" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to reviewboard+unsubscr...@googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>
>>> --
>>> --
>>> Christian Hammond
>>> President/CEO of Beanbag <https://www.beanbaginc.com/>
>>> Makers of Review Board <https://www.reviewboard.org/>
>>>
>>> --
> Supercharge your Review Board with Power Pack:
> https://www.reviewboard.org/powerpack/
> Want us to host Review Board for you? Check out RBCommons:
> https://rbcommons.com/
> Happy user? Let us know! https://www.reviewboard.org/users/
> ---
> You received this message because you are subscribed to the Google Groups
> "reviewboard" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to reviewboard+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
Supercharge your Review Board with Power Pack: 
https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: 
https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to