#5640: Request to improve error handling in utils.encoding.force_unicode.
----------------------------------------------------------+-----------------
Reporter: [EMAIL PROTECTED] | Owner:
mtredinnick
Status: new |
Component: Core framework
Version: SVN |
Resolution:
Keywords: unicode, force_unicode, UnicodeDecodeError |
Stage: Design decision needed
Has_patch: 0 |
Needs_docs: 0
Needs_tests: 0 |
Needs_better_patch: 0
----------------------------------------------------------+-----------------
Changes (by mtredinnick):
* needs_better_patch: => 0
* needs_tests: => 0
* summary: utils.encoding.force_unicode should never throw
UnicodeDecodeError => Request to improve error
handling in utils.encoding.force_unicode.
* owner: nobody => mtredinnick
* needs_docs: => 0
* has_patch: 1 => 0
* stage: Unreviewed => Design decision needed
Old description:
> migrating a large django project to trunk version > 5609 (i.e. after
> Unicode branch was merged) can be very painful due to a bug(?!) in
> utils.encoding.force_unicode
>
> basically, following the instructions here
> http://code.djangoproject.com/wiki/UnicodeBranch#PortingApplicationsTheQuickChecklist
> is the right thing to do, but it is very difficult in a larger project to
> catch all the strings that might need and u'' before them.
>
> Make the slightest oversight, and you lose the ability to debug your
> models/views/templates as utils.encoding.force_unicode will throw an
> UnicodeDecodeError exception like this one:
>
> {{{
> MOD_PYTHON ERROR
>
> ProcessId: 9493
> Interpreter: 'mcc'
>
> ServerName: 'legolas.svt.se'
> DocumentRoot: '/Library/WebServer/Documents'
>
> URI: '/kluster/errors/'
> Location: '/kluster'
> Directory: None
> Filename: '/Library/WebServer/Documents/kluster'
> PathInfo: '/errors/'
>
> Phase: 'PythonHandler'
> Handler: 'django.core.handlers.modpython'
>
> Traceback (most recent call last):
>
> File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
> /site-packages/mod_python/importer.py", line 1537, in HandlerDispatch
> default=default_handler, arg=req, silent=hlist.silent)
>
> File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
> /site-packages/mod_python/importer.py", line 1229, in _process_target
> result = _execute_target(config, req, object, arg)
>
> File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
> /site-packages/mod_python/importer.py", line 1128, in _execute_target
> result = object(arg)
>
> File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
> /site-packages/django/core/handlers/modpython.py", line 178, in handler
> return ModPythonHandler()(req)
>
> File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
> /site-packages/django/core/handlers/modpython.py", line 151, in __call__
> response = self.get_response(request)
>
> File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
> /site-packages/django/core/handlers/base.py", line 53, in get_response
> response = self._real_get_response(request)
>
> File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
> /site-packages/django/core/handlers/base.py", line 115, in
> _real_get_response
> return debug.technical_500_response(request, *sys.exc_info())
>
> File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
> /site-packages/django/views/debug.py", line 151, in
> technical_500_response
> return HttpResponseServerError(t.render(c), mimetype='text/html')
>
> File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
> /site-packages/django/template/__init__.py", line 176, in render
> return self.nodelist.render(context)
>
> File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
> /site-packages/django/template/__init__.py", line 732, in render
> bits.append(self.render_node(node, context))
>
> File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
> /site-packages/django/template/__init__.py", line 745, in render_node
> return node.render(context)
>
> File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
> /site-packages/django/template/defaulttags.py", line 135, in render
> nodelist.append(node.render(context))
>
> File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
> /site-packages/django/template/defaulttags.py", line 229, in render
> return self.nodelist_true.render(context)
>
> File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
> /site-packages/django/template/__init__.py", line 732, in render
> bits.append(self.render_node(node, context))
>
> File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
> /site-packages/django/template/__init__.py", line 745, in render_node
> return node.render(context)
>
> File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
> /site-packages/django/template/defaulttags.py", line 135, in render
> nodelist.append(node.render(context))
>
> File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
> /site-packages/django/template/__init__.py", line 781, in render
> return self.filter_expression.resolve(context)
>
> File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
> /site-packages/django/template/__init__.py", line 599, in resolve
> obj = func(obj, *arg_vals)
>
> File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
> /site-packages/django/template/defaultfilters.py", line 25, in _dec
> args[0] = force_unicode(args[0])
>
> File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
> /site-packages/django/utils/encoding.py", line 41, in force_unicode
> s = unicode(s, encoding, errors)
>
> UnicodeDecodeError: 'utf8' codec can't decode bytes in position
> 3679-3681: invalid data
> }}}
>
> from this traceback, it is very difficult for laymen (like me) to figure
> out what is wrong and trace the cause of the error (in my case: a non-
> ascii string in settings.py that didn't have a u'' in front of it). This
> severly hampers the migration of larger projects to django trunk post
> r5609.
>
> My patch just catches the exception in force_unicode and enforces unicode
> conversion with the 'ignore' flag. This is certainly not the optimal
> solution, but at least it allows for debugging models/views/templates!
New description:
migrating a large django project to trunk version > 5609 (i.e. after
Unicode branch was merged) can be very painful due to a bug(?!) in
utils.encoding.force_unicode
basically, following the instructions here
http://code.djangoproject.com/wiki/UnicodeBranch#PortingApplicationsTheQuickChecklist
is the right thing to do, but it is very difficult in a larger project to
catch all the strings that might need and {{{u''}}} before them.
Make the slightest oversight, and you lose the ability to debug your
models/views/templates as utils.encoding.force_unicode will throw an
UnicodeDecodeError exception like this one:
{{{
MOD_PYTHON ERROR
ProcessId: 9493
Interpreter: 'mcc'
ServerName: 'legolas.svt.se'
DocumentRoot: '/Library/WebServer/Documents'
URI: '/kluster/errors/'
Location: '/kluster'
Directory: None
Filename: '/Library/WebServer/Documents/kluster'
PathInfo: '/errors/'
Phase: 'PythonHandler'
Handler: 'django.core.handlers.modpython'
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/mod_python/importer.py", line 1537, in HandlerDispatch
default=default_handler, arg=req, silent=hlist.silent)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/mod_python/importer.py", line 1229, in _process_target
result = _execute_target(config, req, object, arg)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/mod_python/importer.py", line 1128, in _execute_target
result = object(arg)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/core/handlers/modpython.py", line 178, in handler
return ModPythonHandler()(req)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/core/handlers/modpython.py", line 151, in __call__
response = self.get_response(request)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/core/handlers/base.py", line 53, in get_response
response = self._real_get_response(request)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/core/handlers/base.py", line 115, in
_real_get_response
return debug.technical_500_response(request, *sys.exc_info())
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/views/debug.py", line 151, in technical_500_response
return HttpResponseServerError(t.render(c), mimetype='text/html')
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/template/__init__.py", line 176, in render
return self.nodelist.render(context)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/template/__init__.py", line 732, in render
bits.append(self.render_node(node, context))
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/template/__init__.py", line 745, in render_node
return node.render(context)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/template/defaulttags.py", line 135, in render
nodelist.append(node.render(context))
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/template/defaulttags.py", line 229, in render
return self.nodelist_true.render(context)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/template/__init__.py", line 732, in render
bits.append(self.render_node(node, context))
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/template/__init__.py", line 745, in render_node
return node.render(context)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/template/defaulttags.py", line 135, in render
nodelist.append(node.render(context))
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/template/__init__.py", line 781, in render
return self.filter_expression.resolve(context)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/template/__init__.py", line 599, in resolve
obj = func(obj, *arg_vals)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/template/defaultfilters.py", line 25, in _dec
args[0] = force_unicode(args[0])
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/django/utils/encoding.py", line 41, in force_unicode
s = unicode(s, encoding, errors)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 3679-3681:
invalid data
}}}
from this traceback, it is very difficult for laymen (like me) to figure
out what is wrong and trace the cause of the error (in my case: a non-
ascii string in settings.py that didn't have a u'' in front of it). This
severly hampers the migration of larger projects to django trunk post
r5609.
My patch just catches the exception in force_unicode and enforces unicode
conversion with the 'ignore' flag. This is certainly not the optimal
solution, but at least it allows for debugging models/views/templates!
Comment:
Hiding genuine errors is not the solution here, so this patch can't go in.
Porting an application is a one-time job, so I'm reluctant to even add a
setting for this.
The solution is probably to write your own version of force_unicode and
assign that to encoding.force_unicode during your porting phase. We might
be able to make the error a bit more self-helpful, though.
Changing the title to reflect the real issue, since force_unicode
''should'' raise !UnicodeDecodeError when there's a problem. That's not a
bug; the data it's being given is invalid.
--
Ticket URL: <http://code.djangoproject.com/ticket/5640#comment:1>
Django Code <http://code.djangoproject.com/>
The web framework for perfectionists with deadlines
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-updates?hl=en
-~----------~----~----~----~------~----~------~--~---
