#5640: utils.encoding.force_unicode should never throw UnicodeDecodeError
---------------------------------------+------------------------------------
Reporter: [EMAIL PROTECTED] | Owner: nobody
Status: new | Component: Core framework
Version: SVN | Keywords: unicode, force_unicode,
UnicodeDecodeError
Stage: Unreviewed | Has_patch: 1
---------------------------------------+------------------------------------
migrating a large django project to trunk version > 5609 (i.e. after
Unicode branch was merged) can be very painful due to a bug(?!) in
utils.encoding.force_unicode
basically, following the instructions here
http://code.djangoproject.com/wiki/UnicodeBranch#PortingApplicationsTheQuickChecklist
is the right thing to do, but it is very difficult in a larger project to
catch all the strings that might need and u'' before them.
Make the slightest oversight, and you lose the ability to debug your
models/views/templates as utils.encoding.force_unicode will throw an
UnicodeDecodeError exception like this one:
{{{
MOD_PYTHON ERROR
ProcessId: 9493
Interpreter: 'mcc'
ServerName: 'legolas.svt.se'
DocumentRoot: '/Library/WebServer/Documents'
URI: '/kluster/errors/'
Location: '/kluster'
Directory: None
Filename: '/Library/WebServer/Documents/kluster'
PathInfo: '/errors/'
Phase: 'PythonHandler'
Handler: 'django.core.handlers.modpython'
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/mod_python/importer.py", line 1537, in HandlerDispatch
default=default_handler, arg=req, silent=hlist.silent)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/mod_python/importer.py", line 1229, in _process_target
result = _execute_target(config, req, object, arg)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/mod_python/importer.py", line 1128, in _execute_target
result = object(arg)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/core/handlers/modpython.py", line 178, in handler
return ModPythonHandler()(req)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/core/handlers/modpython.py", line 151, in __call__
response = self.get_response(request)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/core/handlers/base.py", line 53, in get_response
response = self._real_get_response(request)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/core/handlers/base.py", line 115, in
_real_get_response
return debug.technical_500_response(request, *sys.exc_info())
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/views/debug.py", line 151, in technical_500_response
return HttpResponseServerError(t.render(c), mimetype='text/html')
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/template/__init__.py", line 176, in render
return self.nodelist.render(context)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/template/__init__.py", line 732, in render
bits.append(self.render_node(node, context))
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/template/__init__.py", line 745, in render_node
return node.render(context)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/template/defaulttags.py", line 135, in render
nodelist.append(node.render(context))
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/template/defaulttags.py", line 229, in render
return self.nodelist_true.render(context)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/template/__init__.py", line 732, in render
bits.append(self.render_node(node, context))
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/template/__init__.py", line 745, in render_node
return node.render(context)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/template/defaulttags.py", line 135, in render
nodelist.append(node.render(context))
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/template/__init__.py", line 781, in render
return self.filter_expression.resolve(context)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/template/__init__.py", line 599, in resolve
obj = func(obj, *arg_vals)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/template/defaultfilters.py", line 25, in _dec
args[0] = force_unicode(args[0])
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/utils/encoding.py", line 41, in force_unicode
s = unicode(s, encoding, errors)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 3679-3681:
invalid data
}}}
from this traceback, it is very difficult for laymen (like me) to figure
out what is wrong and trace the cause of the error (in my case: a non-
ascii string in settings.py that didn't have a u'' in front of it). This
severly hampers the migration of larger projects to django trunk post
r5609.
My patch just catches the exception in force_unicode and enforces unicode
conversion with the 'ignore' flag. This is certainly not the optimal
solution, but at least it allows for debugging models/views/templates!
--
Ticket URL: <http://code.djangoproject.com/ticket/5640>
Django Code <http://code.djangoproject.com/>
The web framework for perfectionists with deadlines
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-updates?hl=en
-~----------~----~----~----~------~----~------~--~---
