#3543: cant post the letter ö (among others)
-----------------------------------+----------------------------------------
Reporter: Bjorn Leví | Owner: cboos
Type: defect | Status: assigned
Priority: normal | Milestone: 0.10
Component: general | Version: devel
Severity: normal | Resolution:
Keywords: unicode compatibility |
-----------------------------------+----------------------------------------
Changes (by cboos):
* status: new => assigned
* owner: jonas => cboos
* version: 0.9.6 => devel
* milestone: => 0.10
* keywords: => unicode compatibility
Old description:
> Traceback (most recent call last):
> File "C:\Python23\Lib\site-packages\trac\web\main.py", line 315, in
> dispatch_request
> dispatcher.dispatch(req)
> File "C:\Python23\Lib\site-packages\trac\web\main.py", line 200, in
> dispatch
> resp = chosen_handler.process_request(req)
> File "build\bdist.win32\egg\tractags\web_ui.py", line 64, in
> process_request
> File "C:\Python23\lib\site-packages\trac\wiki\web_ui.py", line 133, in
> process_request
> self._render_view(req, db, page)
> File "C:\Python23\lib\site-packages\trac\wiki\web_ui.py", line 460, in
> _render_view
> req.hdf['wiki'] = {
> File "C:\Python23\Lib\site-packages\trac\wiki\formatter.py", line 994,
> in wiki_to_html
> Formatter(env, req, absurls, db).format(wikitext, out,
> escape_newlines)
> File "C:\Python23\Lib\site-packages\trac\wiki\formatter.py", line 820,
> in format
> result = re.sub(self.wiki.rules, self.replace, line)
> File "C:\Python23\lib\sre.py", line 143, in sub
> return _compile(pattern, 0).sub(repl, string, count)
> File "C:\Python23\Lib\site-packages\trac\wiki\formatter.py", line 768,
> in replace
> return to_unicode(replacement)
> File "C:\Python23\Lib\site-packages\trac\util\text.py", line 57, in
> to_unicode
> return unicode(text)
> File "C:\Python23\Lib\site-packages\trac\util\markup.py", line 313, in
> __str__
> return Markup(''.join(self.serialize()))
> File "C:\Python23\Lib\site-packages\trac\util\markup.py", line 452, in
> serialize
> for part in Fragment.serialize(self):
> File "C:\Python23\Lib\site-packages\trac\util\markup.py", line 308, in
> serialize
> yield unicode(child)
> File "C:\Python23\Lib\site-packages\trac\util\markup.py", line 313, in
> __str__
> return Markup(''.join(self.serialize()))
> File "C:\Python23\Lib\site-packages\trac\util\markup.py", line 452, in
> serialize
> for part in Fragment.serialize(self):
> File "C:\Python23\Lib\site-packages\trac\util\markup.py", line 310, in
> serialize
> yield escape(child, quotes=False)
> File "C:\Python23\Lib\site-packages\trac\util\markup.py", line 107, in
> escape
> text = unicode(text)
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 5:
> ordinal not in range(128)
New description:
{{{
Traceback (most recent call last):
File "C:\Python23\Lib\site-packages\trac\web\main.py", line 315, in
dispatch_request
dispatcher.dispatch(req)
File "C:\Python23\Lib\site-packages\trac\web\main.py", line 200, in
dispatch
resp = chosen_handler.process_request(req)
File "build\bdist.win32\egg\tractags\web_ui.py", line 64, in
process_request
File "C:\Python23\lib\site-packages\trac\wiki\web_ui.py", line 133, in
process_request
self._render_view(req, db, page)
File "C:\Python23\lib\site-packages\trac\wiki\web_ui.py", line 460, in
_render_view
req.hdf['wiki'] = {
File "C:\Python23\Lib\site-packages\trac\wiki\formatter.py", line 994,
in wiki_to_html
Formatter(env, req, absurls, db).format(wikitext, out,
escape_newlines)
File "C:\Python23\Lib\site-packages\trac\wiki\formatter.py", line 820,
in format
result = re.sub(self.wiki.rules, self.replace, line)
File "C:\Python23\lib\sre.py", line 143, in sub
return _compile(pattern, 0).sub(repl, string, count)
File "C:\Python23\Lib\site-packages\trac\wiki\formatter.py", line 768,
in replace
return to_unicode(replacement)
File "C:\Python23\Lib\site-packages\trac\util\text.py", line 57, in
to_unicode
return unicode(text)
File "C:\Python23\Lib\site-packages\trac\util\markup.py", line 313, in
__str__
return Markup(''.join(self.serialize()))
File "C:\Python23\Lib\site-packages\trac\util\markup.py", line 452, in
serialize
for part in Fragment.serialize(self):
File "C:\Python23\Lib\site-packages\trac\util\markup.py", line 308, in
serialize
yield unicode(child)
File "C:\Python23\Lib\site-packages\trac\util\markup.py", line 313, in
__str__
return Markup(''.join(self.serialize()))
File "C:\Python23\Lib\site-packages\trac\util\markup.py", line 452, in
serialize
for part in Fragment.serialize(self):
File "C:\Python23\Lib\site-packages\trac\util\markup.py", line 310, in
serialize
yield escape(child, quotes=False)
File "C:\Python23\Lib\site-packages\trac\util\markup.py", line 107, in
escape
text = unicode(text)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 5:
ordinal not in range(128)
}}}
Comment:
It appears you're running trunk, not 0.9.6.
Also, it appears that the error was triggered by the TracHacks:TagsPlugin.
From what I understand looking at the stack trace, that plugin is wrapping
a `str` which contains utf-8 encoded data in a Fragment or Element object
(here the `str` is: `'\xc3\xb6'`, which corresponds to
`u'ö'.decode('utf-8')`).
I always hesitated to use `to_unicode(text)` instead of `unicode(text)` in
`Markup.escape`, because it helped to catch a lot of bad uses of non-
unicode strings in the core, but I think that increased compatibility with
0.9.3+ plugins is a good reason to do so, now that the unicode issues in
the core are gone.
Can you please try the following fix:
{{{
Index: trac/util/html.py
===================================================================
--- trac/util/html.py (revision 3605)
+++ trac/util/html.py (working copy)
@@ -21,6 +21,8 @@
from StringIO import StringIO
import sys
+from trac.util.text import to_unicode
+
__all__ = ['escape', 'unescape', 'html']
_EMPTY_TAGS = frozenset(['br', 'hr', 'img', 'input'])
@@ -104,7 +106,7 @@
"""
if isinstance(text, (cls, Element)):
return text
- text = unicode(text)
+ text = to_unicode(text)
if not text:
return cls()
text = text.replace('&', '&') \
}}}
(Note that the patch is on html.py, because markup.py has been moved
lately; it should apply on markup.py however, or you could upgrade to the
latest)
--
Ticket URL: <http://trac.edgewall.org/ticket/3543#comment:1>
The Trac Project <http://trac.edgewall.com/>
_______________________________________________
Trac-Tickets mailing list
[email protected]
http://lists.edgewall.com/mailman/listinfo/trac-tickets
