Is the page html header declaring the utf8 encoding or are using in a layout that uses a different encoding?
On Jan 19, 7:21 am, Alan Etkin <[email protected]> wrote: > I found that the Unicode errors are originated because of incompatible > encodings when web2py tries to read the raw message and render the > data for browser output. I solved it encoding the RFC822 raw text > before parsing the response data as Rows. Still i am not sure if this > is the correct way for processing the response text so it can be sent > safely (without misread characters) to the user interface. Anyway, it > seems to work well, without instensive testing. > > On 18 ene, 18:40, Alan Etkin <[email protected]> wrote: > > > > > > > > > I am trying to generate sqltables with the experimental IMAPAdapter > > select output, but with some rows, an exception is raised at the > > sqlhtml module. It has to do with unicode and charsets when sqlhtml > > processes the rows object returned by the adapter's select method, but > > I cannot find a proper way of solving it. Here is the error trace: > > > Traceback (most recent call last): > > File "/home/alan/web2py-hg/gluon/restricted.py", line 204, in > > restricted > > exec ccode in environment > > File "/home/alan/web2py-hg/applications/queries/views/default/ > > index.html", line 126, in <module> > > File "/home/alan/web2py-hg/gluon/globals.py", line 181, in write > > self.body.write(xmlescape(data)) > > File "/home/alan/web2py-hg/gluon/html.py", line 114, in xmlescape > > return data.xml() > > File "/home/alan/web2py-hg/gluon/dal.py", line 7442, in xml > > return sqlhtml.SQLTABLE(self).xml() > > File "/home/alan/web2py-hg/gluon/sqlhtml.py", line 2197, in __init__ > > ur = unicode(r, 'utf8') > > File "/usr/lib/python2.6/encodings/utf_8.py", line 16, in decode > > return codecs.utf_8_decode(input, errors, True) > > UnicodeDecodeError: 'utf8' codec can't decode bytes in position > > 1227-1229: invalid data > > > I found a workaround to avoid the exception but I doubt it's the > > correct fix, because it just prevents web2py to create the unicode > > object and use the raw input instead. > > > This is the workaround: (gluon/sqlhtml.py Line 2196) > > > try: > > ur = unicode(r, 'utf-8') > > except UnicodeDecodeError, e: > > ur = r > > > Replacing this line: > > ur = unicode(r, 'utf8') > > > When creating the Row objects at the adapter, I have to handle > > different encodings depending on the message. What would be an > > appropiate way of encoding data before creating the Row objects, so > > unicode errors can be avoided? > > > This is the adapter method i am using to store the parse input for > > each text field > > > def encode_text(self, text, charset, errors="replace"): > > """ convert text for mail to unicode""" > > if text is None: > > text = "" > > else: > > if charset is not None: > > text = unicode(text, charset, errors) > > else: > > text = unicode(text, "utf-8", errors) > > return text > > > Thanks > > > I am using the last source hg version (1.99.4) with Python 2.6.5 on a > > Mandriva GNU/Linux machine.
