[TurboGears] Re: : Catwalk and (unicode, encoding) problem (more)

Sun, 04 Dec 2005 17:34:37 -0800 


Hi,
I'm aware of this problem, and yes CatWalk is the one to blame =(.
While working this week on a new tool for the ToolBox (I18n administration), I stumble upon the culprit. The problem is that all interaction in CatWalk goes through xmlHttpRequests (ajax). Before any data is send, it is escaped to avoid problems with characters as '?' or '&' But the javascript escape function does some weird stuff to unicode characters, aarggh... For the I18n admin tool, I circumvent the problem by using encodeURIComponent instead of escape. 
The same should work for CW.
I have added a ticket for it: http://trac.turbogears.org/turbogears/ ticket/178 
Cheers
Ronald

On Dec 4, 2005, at 10:40 PM, Laurent Rahuel wrote:
No, I did not found any workaround even using Unicode in my model declaration (Unicode field of SQLObject I mean). I guess the problem is coming from 
CatWalk because command line manipulations works great.

Le Dimanche 4 Décembre 2005 18:37, vous avez écrit :

Laurent RAHUEL a écrit :

When adding these lines into dev.cfg :

decodingFilter.on = True
decodingFilter.encoding = "utf8"
Nothing appends on the catwalk interface when trying to add amitié to a 
text filed. And I get these lines in the debug window :


(snip)


line 53, in beforeMain
    newValue = value.decode(enc)
  File
"/home/lrl/dev/turbogears/lang/python242/lib/python2.4/encodings/ utf_8.py 
", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 5: 
unexpected end of data


Any ideas ???


Same problem here using SQLite. When I create a record with "éé" as
title, I get :

Setup 1 :
  * StringCol
  * no decodingFilter specified in dev.cfg

  => title get saved as "'\xe9\xe9'"

Setup 2
  * StringCol
  * decodingFilter on, utf8

  => trace:
   File
"/usr/lib/python2.4/site-packages/CherryPy-2.1.0-py2.4.egg/ cherrypy/lib/fil 
ter/decodingfilter.py", line 53, in beforeMain
     newValue = value.decode(enc)
   File "/usr/lib/python2.4/encodings/utf_8.py", line 16, in decode
     return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-1:
unexpected end of data

Setup 3
  * replacing StringCol with UnicodeCol
  * no decodingFilter specified in dev.cfg

  => Now it's SQLObject that complains :(
     trace :
2005/12/04 18:32:36  INFO Traceback (most recent call last):
   File
"/usr/lib/python2.4/site-packages/CherryPy-2.1.0-py2.4.egg/ cherrypy/_cphttp 
tools.py", line 271, in run
     main()
   File
"/usr/lib/python2.4/site-packages/CherryPy-2.1.0-py2.4.egg/ cherrypy/_cphttp 
tools.py", line 502, in main
     body = page_handler(*args, **cherrypy.request.paramMap)
   File
"/usr/lib/python2.4/site-packages/TurboGears-0.9a0dev_r277- py2.4.egg/turbog 
ears/controllers.py", line 198, in newfunc
     errors, errorhandler, func, *args, **kw)
   File
"/usr/lib/python2.4/site-packages/TurboGears-0.9a0dev_r277- py2.4.egg/turbog 
ears/database.py", line 174, in run_with_transaction
     retval = func(*args, **kw)
   File
"/usr/lib/python2.4/site-packages/TurboGears-0.9a0dev_r277- py2.4.egg/turbog 
ears/controllers.py", line 225, in _execute_func
     output = func(self, *args, **kw)
   File
"/usr/lib/python2.4/site-packages/TurboGears-0.9a0dev_r277- py2.4.egg/turbog 
ears/catwalk/catwalk.py", line 696, in add
     new_object = eval(u'obj(%s)'% param_string)
   File "<string>", line 0, in ?
   File
"/usr/lib/python2.4/site-packages/SQLObject-0.7.0-py2.4.egg/ sqlobject/main. 
py", line 1183, in __init__
     self._create(id, **kw)
   File
"/usr/lib/python2.4/site-packages/SQLObject-0.7.0-py2.4.egg/ sqlobject/main. 
py", line 1207, in _create
     self.set(**kw)
   File
"/usr/lib/python2.4/site-packages/SQLObject-0.7.0-py2.4.egg/ sqlobject/main. 
py", line 1073, in set
     value = to_python(dbValue, self._SO_validatorState)
   File
"/usr/lib/python2.4/site-packages/SQLObject-0.7.0-py2.4.egg/ sqlobject/col.p 
y", line 522, in to_python
     raise UnicodeDecodeError(*args)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-1:
unexpected end of data - (self.db_encoding is : UTF-8)
(NB : I hacked sqlobject to add the db_encoding to the exception's infos) 


Laurent, did you manage to get this stuff working ?


________________________________
Ronald Jaramillo
mail: ronald AT checkandshare DOT com
blog: http://www.checkandshare.com/blog

Reply via email to