Re: [Zope] UnicodeDecodeError

2012-04-03 Thread Stefan Loidl 

Hi Yuri,

does this error occur with all browsers? I once had a similar issue
that was caused only by some browsers because they do not send a
HTTP_ACCEPT_CHARSET header.

Maybe the following bug reports helps you in solving the issue:
https://bugs.launchpad.net/zope2/+bug/160968

I'm using a workaround similar to that in the bug report in my
environment (Zope 2.13.13 and a mixture of utf-8 and iso8859).
Without the workaround the issue does still occur for me.

Greetings,
Stefan

Am 03.04.2012 09:40, schrieb Yuri:

Hi!

I've an Zope application, basically postgres + zpt.

The postgres db has been upgraded to 8.5, the data inside is utf-8
(dumped and checkd it), the pg adapeter is the latest, and the "*Z
Psycopg 2 Database Connection" is configured to use utf-8.* Zope is
2.12.20 + latest security fix.

When I try to do:

 (result came from an ZSQL method)
I get an UnicodeDecodeError:

Traceback (innermost last):
Module ZPublisher.Publish, line 127, in publish
Module ZPublisher.mapply, line 77, in mapply
Module ZPublisher.Publish, line 47, in call_object
Module Shared.DC.Scripts.Bindings, line 324, in __call__
Module Shared.DC.Scripts.Bindings, line 361, in _bindAndExec
Module Products.PageTemplates.ZopePageTemplate, line 335, in _exec
Module Products.PageTemplates.ZopePageTemplate, line 432, in pt_render
Module Products.PageTemplates.PageTemplate, line 80, in pt_render
Module zope.pagetemplate.pagetemplate, line 113, in pt_render
Module zope.tal.talinterpreter, line 271, in __call__
Module zope.tal.talinterpreter, line 343, in interpret
Module zope.tal.talinterpreter, line 888, in do_useMacro
Module zope.tal.talinterpreter, line 343, in interpret
Module zope.tal.talinterpreter, line 946, in do_defineSlot
Module zope.tal.talinterpreter, line 343, in interpret
Module zope.tal.talinterpreter, line 533, in do_optTag_tal
Module zope.tal.talinterpreter, line 518, in do_optTag
Module zope.tal.talinterpreter, line 513, in no_tag
Module zope.tal.talinterpreter, line 343, in interpret
Module zope.tal.talinterpreter, line 821, in do_loop_tal
Module zope.tal.talinterpreter, line 343, in interpret
Module zope.tal.talinterpreter, line 821, in do_loop_tal
Module zope.tal.talinterpreter, line 343, in interpret
Module zope.tal.talinterpreter, line 852, in do_condition
Module zope.tal.talinterpreter, line 343, in interpret
Module zope.tal.talinterpreter, line 376, in do_startEndTag
Module zope.tal.talinterpreter, line 405, in do_startTag
Module zope.tal.talinterpreter, line 502, in attrAction_tal
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 38:
ordinal not in range(128)

if I do:

 it
works. (I use str because I've also dates in the record, otherwise I
could not decode being it not a string)

I've several zpt to fix, and I would like it to work as it should. My
analisys is that result['column'] is seen as a plain string with \xx
characters inside, and decode just transform it in something that Zope
recognize as utf-8 when it prints it.

Anyway, any idea?
___
Zope maillist - Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope-dev )


--
Stefan Loidl  Phone:  +49 89 35831 8882
Leibniz-Rechenzentrum Fax:+49 89 35831 8682
Boltzmannstr. 1   mailto:lo...@lrz.de
85748 Garchinghttp://www.lrz.de
___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope-dev )

[Zope] UnicodeDecodeError

2012-04-03 Thread Yuri 

Hi!

 I've an Zope application, basically postgres + zpt.

 The postgres db has been upgraded to 8.5, the data inside is utf-8 
(dumped and checkd it), the pg adapeter is the latest, and the "*Z 
Psycopg 2 Database Connection" is configured to use utf-8.* Zope is  
2.12.20 + latest security fix.


 When I try to do:

 (result came from an ZSQL method) 
I get an UnicodeDecodeError:


Traceback (innermost last):
  Module ZPublisher.Publish, line 127, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 47, in call_object
  Module Shared.DC.Scripts.Bindings, line 324, in __call__
  Module Shared.DC.Scripts.Bindings, line 361, in _bindAndExec
  Module Products.PageTemplates.ZopePageTemplate, line 335, in _exec
  Module Products.PageTemplates.ZopePageTemplate, line 432, in pt_render
  Module Products.PageTemplates.PageTemplate, line 80, in pt_render
  Module zope.pagetemplate.pagetemplate, line 113, in pt_render
  Module zope.tal.talinterpreter, line 271, in __call__
  Module zope.tal.talinterpreter, line 343, in interpret
  Module zope.tal.talinterpreter, line 888, in do_useMacro
  Module zope.tal.talinterpreter, line 343, in interpret
  Module zope.tal.talinterpreter, line 946, in do_defineSlot
  Module zope.tal.talinterpreter, line 343, in interpret
  Module zope.tal.talinterpreter, line 533, in do_optTag_tal
  Module zope.tal.talinterpreter, line 518, in do_optTag
  Module zope.tal.talinterpreter, line 513, in no_tag
  Module zope.tal.talinterpreter, line 343, in interpret
  Module zope.tal.talinterpreter, line 821, in do_loop_tal
  Module zope.tal.talinterpreter, line 343, in interpret
  Module zope.tal.talinterpreter, line 821, in do_loop_tal
  Module zope.tal.talinterpreter, line 343, in interpret
  Module zope.tal.talinterpreter, line 852, in do_condition
  Module zope.tal.talinterpreter, line 343, in interpret
  Module zope.tal.talinterpreter, line 376, in do_startEndTag
  Module zope.tal.talinterpreter, line 405, in do_startTag
  Module zope.tal.talinterpreter, line 502, in attrAction_tal
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 38: 
ordinal not in range(128)


if I do:

 it 
works. (I use str because I've also dates in the record, otherwise I 
could not decode being it not a string)


I've several zpt to fix, and I would like it to work as it should. My 
analisys is that result['column'] is seen as a plain string with \xx 
characters inside, and decode just transform it in something that Zope 
recognize as utf-8 when it prints it.


Anyway, any idea?
___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] UnicodeDecodeError in Zope 2.10.4 (upgrade from 2.8.4)

2007-07-17 Thread Maciej Wisniowski 

> Both
> template id.
> and
> template id
> is like playing with fire. Don't do it.
>
> What you've got there is unicode characters written down without any
> encoding information.
This is legacy code from zope 2.8.4 that I have to
deal with during migration. All new code is supposed to use
u'' strings.
In fact, above is only an example but in real case we
have properties set for folders, eg. 'title' that
contains national characters.

You can easily check it yourself if you want.
Just create a Folder and set it's 'title' property
to one that contains some unicode characters.
You'll not be even able to add ZPT object into
that folder I think (at last it is not possible for me).

>
> It will work if you set the internal ZPT encoding to be the same as it
> was entered into the template which I can see is not ASCII which is
> what your system defaults to.
What do you mean by internal ZPT encoding? Docs says:
"Starting Zope 2.10.2 the ZPT implementation uses unicode as internal
representation"
so there should be no encoding at this level, I think.

>
> This is internally what happens:
> >>> x="\xc5"
> >>> unicode(x)
> Traceback (most recent call last):
>   File "", line 1, in 
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position
> 0: ordinal not in range(128)
> >>> import sys; sys.getdefaultencoding()
> 'ascii'
I already realized same thing and used sitecustomize.py
with sys.setdefaultencoding('utf-8') as a quick fix. This works
but is not nice solution. I rather expected Zope 2.10 resolver to
deal with this.

-- 
Maciej Wisniowski
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] UnicodeDecodeError in Zope 2.10.4 (upgrade from 2.8.4)

2007-07-17 Thread Peter Bengtsson 

Both
template id.
and
template id
is like playing with fire. Don't do it.

What you've got there is unicode characters written down without any 
encoding information.


It will work if you set the internal ZPT encoding to be the same as it 
was entered into the template which I can see is not ASCII which is what 
your system defaults to.


This is internally what happens:
>>> x="\xc5"
>>> unicode(x)
Traceback (most recent call last):
  File "", line 1, in 
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 0: 
ordinal not in range(128)

>>> import sys; sys.getdefaultencoding()
'ascii'




Maciej Wisniowski wrote:

I've upgraded my installation from Zope2.8.4 to Zope 2.10.4
(by copying data.fs, Products/ etc.). I have publisher encoding
and management_page_charset set to utf-8. Also system
default encoding is utf-8.

Zope 2.10 is said to have better Unicode support with
UnicodeEncodeConflictResolver. It is but unfortunatelly in
some cases this is not enough. Seems that
code that is 'protected' by Resolver is not the only code
that may be affected by non unicode strings.
Simple example is with 'structure' keyword.

eg:
This works (resolver resolves conflict):
template id.

This doesn't work:
template id.

Also if you have Folder instance and set it's Title
to the string that contains some i18n characters
you're not able to even add page template
inside it.

Error traceback in both cases is same:

Error Type: UnicodeDecodeError
Error Value: 'ascii' codec can't decode byte 0xc5 in position 200:
ordinal not in range(128)

Traceback (innermost last):
Module ZPublisher.Publish, line 119, in publish
Module ZPublisher.mapply, line 88, in mapply
Module ZPublisher.Publish, line 42, in call_object
Module Shared.DC.Scripts.Bindings, line 313, in __call__
Module Shared.DC.Scripts.Bindings, line 350, in _bindAndExec
Module Products.PageTemplates.PageTemplateFile, line 129, in _exec
Module Products.PageTemplates.PageTemplate, line 89, in pt_render
Module zope.pagetemplate.pagetemplate, line 117, in pt_render
Module zope.tal.talinterpreter, line 271, in __call__
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 534, in do_optTag_tal
Module zope.tal.talinterpreter, line 516, in no_tag
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 754, in do_insertStructure_tal
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position
200: ordinal not in range(128)

problem is with:
  text = unicode(structure)
at
Module zope.tal.talinterpreter, line 754, in do_insertStructure_tal

sitecustomize.py with setdefaultencoding('utf-8') solves this
but it is not nice solution. I wonder whether this should be
submitted as a bug, or maybe there is different solution that
I've missed?

For the record: one more thing that was wrong
for me during migration was 'expand' attribute of ZPT
that in case of few ZPT objects was set to true in
2.10.4 while oryginally in 2.8.4 it was false.
This caused that tales expressions disappeared
from ZPTs under 2.10.4. I've written a script that
explicitly set expand=0 to all ZPT instances in
Zope 2.8.4 and then, after migration, everything is OK.



--
Peter Bengtsson,
work www.fry-it.com
home www.peterbe.com
hobby www.issuetrackerproduct.com
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] UnicodeDecodeError in Zope 2.10.4 (upgrade from 2.8.4)

2007-07-17 Thread Maciej Wisniowski 

> If the documented hints don't work, please submit a bugreport
> with a reproducable testcase in form of a unittest.
Bug submitted with tests attached at:
http://www.zope.org/Collectors/Zope/2339

-- 
Maciej Wisniowski
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] UnicodeDecodeError in Zope 2.10.4 (upgrade from 2.8.4)

2007-07-17 Thread Maciej Wisniowski 

>
> Did you read
>
> 
Yes.

> The migration code should auto-detect ISO-8859-15 and UTF-8
> encoded page templates. For other encodings you must set
> the environment variable ZPT_PREFERRED_ENCODING.
> The migration code applies to ZopePageTemplate instances only.
I have utf-8 encoded ZPTs so I didn't used ZPT_PREFERRED_ENCODING

> When you download ZPT content through FTP or WebDAV the
> content is converted using the output_encoding property of
> the corresponding ZopePageTemplate instance.
So far I didn't even try ftp or webdav with 2.10.4. The problems are with
TTW.

> The encoding of a rendered ZPT (through HTTP) is determined through
> charset= within the content-type HTTP response header
> etc/zope.conf: default-zpublisher-encoding
> default: iso-8859-15
default-zpublisher-encoding is set to utf-8

> In order to deal with UnicodeDecodeErrors in a reasonable way,
> we added a configurable Unicode conflict resolver.
As I said before resolver is not even called when using 'structure' keyword.

-- 
Maciej Wisniowski
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] UnicodeDecodeError in Zope 2.10.4 (upgrade from 2.8.4)

2007-07-17 Thread Andreas Jung 



--On 17. Juli 2007 10:52:05 +0200 Andreas Jung <[EMAIL PROTECTED]> wrote:




--On 17. Juli 2007 10:33:39 +0200 Maciej Wisniowski
<[EMAIL PROTECTED]> wrote:


I've upgraded my installation from Zope2.8.4 to Zope 2.10.4
(by copying data.fs, Products/ etc.). I have publisher encoding
and management_page_charset set to utf-8. Also system
default encoding is utf-8


Did you read






If the documented hints don't work, please submit a bugreport
with a reproducable testcase in form of a unittest.

-aj

pgpT3AJebGwqh.pgp
Description: PGP signature
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] UnicodeDecodeError in Zope 2.10.4 (upgrade from 2.8.4)

2007-07-17 Thread Andreas Jung 



--On 17. Juli 2007 10:33:39 +0200 Maciej Wisniowski 
<[EMAIL PROTECTED]> wrote:



I've upgraded my installation from Zope2.8.4 to Zope 2.10.4
(by copying data.fs, Products/ etc.). I have publisher encoding
and management_page_charset set to utf-8. Also system
default encoding is utf-8


Did you read



?

-aj

pgpSGCPsV0oEi.pgp
Description: PGP signature
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

[Zope] UnicodeDecodeError in Zope 2.10.4 (upgrade from 2.8.4)

2007-07-17 Thread Maciej Wisniowski 
I've upgraded my installation from Zope2.8.4 to Zope 2.10.4
(by copying data.fs, Products/ etc.). I have publisher encoding
and management_page_charset set to utf-8. Also system
default encoding is utf-8.

Zope 2.10 is said to have better Unicode support with
UnicodeEncodeConflictResolver. It is but unfortunatelly in
some cases this is not enough. Seems that
code that is 'protected' by Resolver is not the only code
that may be affected by non unicode strings.
Simple example is with 'structure' keyword.

eg:
This works (resolver resolves conflict):
template id.

This doesn't work:
template id.

Also if you have Folder instance and set it's Title
to the string that contains some i18n characters
you're not able to even add page template
inside it.

Error traceback in both cases is same:

Error Type: UnicodeDecodeError
Error Value: 'ascii' codec can't decode byte 0xc5 in position 200:
ordinal not in range(128)

Traceback (innermost last):
Module ZPublisher.Publish, line 119, in publish
Module ZPublisher.mapply, line 88, in mapply
Module ZPublisher.Publish, line 42, in call_object
Module Shared.DC.Scripts.Bindings, line 313, in __call__
Module Shared.DC.Scripts.Bindings, line 350, in _bindAndExec
Module Products.PageTemplates.PageTemplateFile, line 129, in _exec
Module Products.PageTemplates.PageTemplate, line 89, in pt_render
Module zope.pagetemplate.pagetemplate, line 117, in pt_render
Module zope.tal.talinterpreter, line 271, in __call__
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 534, in do_optTag_tal
Module zope.tal.talinterpreter, line 516, in no_tag
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 754, in do_insertStructure_tal
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position
200: ordinal not in range(128)

problem is with:
  text = unicode(structure)
at
Module zope.tal.talinterpreter, line 754, in do_insertStructure_tal

sitecustomize.py with setdefaultencoding('utf-8') solves this
but it is not nice solution. I wonder whether this should be
submitted as a bug, or maybe there is different solution that
I've missed?

For the record: one more thing that was wrong
for me during migration was 'expand' attribute of ZPT
that in case of few ZPT objects was set to true in
2.10.4 while oryginally in 2.8.4 it was false.
This caused that tales expressions disappeared
from ZPTs under 2.10.4. I've written a script that
explicitly set expand=0 to all ZPT instances in
Zope 2.8.4 and then, after migration, everything is OK.

-- 
Maciej Wisniowski
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] UnicodeDecodeError inside ZMI

2006-09-05 Thread Dieter Maurer 
Remy Pinsonnault wrote at 2006-9-5 11:18 -0400:
>With Zope 2.8.7 and Python 2.3.5 (Plone 2.1.3 installer), when I search
>inside the ZMI for the spanish "ñ" character (inside a Plone site object), I
>get an UnicodeDecodeError:
>
> Site Error
>
>An error was encountered while publishing this resource.
>
>*UnicodeDecodeError*
>Sorry, a site error occurred.
>
>Traceback (innermost last):
> ...
>   - Module OFS.FindSupport, line 112, in ZopeFind
>
>UnicodeDecodeError: 'ascii' codec can't decode byte 0xf1 in position 0:
>ordinal not in range(128)

This means that unicode and (non ASCII) byte strings are mixed together.

Search the mailing list for "UnicodeDecodeError" (or "setdefaultencoding")
to find out how you can approach this problem.


-- 
Dieter
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

[Zope] UnicodeDecodeError inside ZMI

2006-09-05 Thread Remy Pinsonnault 
Hello,
 
With Zope 2.8.7 and Python 2.3.5 (Plone 2.1.3 installer), when I search inside the ZMI for the spanish "
ñ" character (inside a Plone site object), I get an UnicodeDecodeError:
 

Site Error
An error was encountered while publishing this resource. 
UnicodeDecodeErrorSorry, a site error occurred. 

Traceback (innermost last): 

Module ZPublisher.Publish, line 188, in publish_module_standard 
Module Products.PlacelessTranslationService.PatchStringIO, line 34, in new_publish 
Module ZPublisher.Publish, line 145, in publish 
Module Zope2.App.startup, line 216, in zpublisher_exception_hook 
Module ZPublisher.Publish, line 114, in publish 
Module ZPublisher.mapply, line 88, in mapply 
Module ZPublisher.Publish, line 40, in call_object 
Module Shared.DC.Scripts.Bindings, line 311, in __call__ 
Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec 
Module App.special_dtml, line 175, in _exec 
Module DocumentTemplate.DT_With, line 61, in render 
Module DocumentTemplate.DT_Util, line 196, in eval__traceback_info__: _ 
Module , line 0, in ? 
Module OFS.FindSupport, line 112, in ZopeFind UnicodeDecodeError: 'ascii' codec can't decode byte 0xf1 in position 0: ordinal not in range(128) (Also, the following error occurred while attempting to render the standard error message, please see the event log for full details: 'portlet_intranet') 

___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] UnicodeDecodeError

2006-08-14 Thread Dieter Maurer 
Luiz Fernando Bernardes Ribeiro wrote at 2006-8-14 11:03 -0300:
>Due to project demands I had to convert the html entities in all TEXT fields
>in my database to the real international characters like "á", "ç" and so on.

Good!

>The problem is that all my dynamic generated XML started to fail with
>UnicodeDecodeError. The XML encoding is set to utf-8 and the file itself is
>correctly encoded.
>
>If I inject the international characters using a Python Script with
>something like u'çáé' it works, but passing the TEXT field from the database
>generates the error.

Then, this means that the value from the database is not unicode.

Convert it to unicode by "value.decode(the_database_encoding)".



-- 
Dieter
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

[Zope] UnicodeDecodeError

2006-08-14 Thread Luiz Fernando Bernardes Ribeiro 
Hi all,Due to project demands I had to convert the html entities in all TEXT fields in my database to the real international characters like "á", "ç" and so on. The problem is that all my dynamic generated XML started to fail with UnicodeDecodeError. The XML encoding is set to utf-8 and the file itself is correctly encoded.
If I inject the international characters using a Python Script with something like u'çáé' it works, but passing the TEXT field from the database generates the error.I'm using Zope 2.9.2 and MySQL. The XML is generated using ZPT.
The XML, invokes a script to filter and add links to the text retrieved from the database, ex:... xml stuff ...        
... xml stuff ...obra is the database row and Conteudo is the TEXT field.montarCDATA is as simple as (I removed the code to build the links):return '' % textoif instead of the TEXT field I put something like:
return '' % u'çáéà'it works, which leads me to believe the problem is with the database field and XML. The strange thing is that with common html everything works as expected.Traceback (innermost last):
   Module ZPublisher.Publish, line 115, in publish   Module ZPublisher.mapply, line 88, in mapply   Module ZPublisher.Publish, line 41, in call_object  Module Shared.DC.Scripts.Bindings
, line 311, in __call__  Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec  Module Products.PageTemplates.ZopePageTemplate, line 256, in _exec  Module Products.PageTemplates.PageTemplate
, line 105, in pt_render
   Module StringIO, line 271, in getvalueUnicodeDecodeError: 'ascii' codec can't decode byte 0xe1 in position 50: ordinal not in range(128)
Thanks in advance,-- Luiz Fernando B. Ribeiro
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

[Zope] UnicodeDecodeError: 'ascii' codec can't decode

2006-01-02 Thread Jon Bowlas 
Hi all,

I'm not sure if any of you are familiar with Silva (CMS) but I’ve created a
SQL code source which works fine when tested in the ZMI but when attempting
to add this code source into a Silva document via the Silva Management
Interface I get a UnicodeDecodeError. I think I know why this is but I’m not
sure how to resolve it, so any suggestions would be appreciated. 

Basically my 'Silva SQL Source' generates a MultiListField by extracting its
values from a database however there are some quite unusual characters
contained in the returned recordset for the MultiListField, see below the
tuple I generate from the recordset.

[('Developmental Neurobiology', 'ANAT2008'), ('Neuroanatomy', 'ANAT2009'),
('The Evolution and Development of Body Form - Visceral Systems',
'ANAT2022'), ('The Evolution and\r\nDevelopment of Body Form \xbf
The\r\nMusculoskeletal System', 'ANAT2023'), ("Man's Place in
Nature,\r\n1750\xbf1900: The Debate in Britain", 'HMED3001'), ('Medicine,
Disease and\r\nSociety from Antiquity to the\r\nRenaissance', 'HMED3003'),
('Madness and Society', 'HMED3004'), ('Medicine and Modern\r\nSociety',
'HMED3006'), ('Colonial and Empire\r\nMedicine', 'HMED3009'), ('Disease in
History', 'HMED3010'), ('The Rise of Modern\r\nMedicine, 1650\xbf1900',
'HMED3013'), ('Medicine in Asia', 'HMED3014'), ('Advanced Cell Biology',
'ANAT3001'), ('Mechanisms of Development', 'ANAT3003'), ('Neural Basis of
Motivation and Learning', 'ANAT3018'), ('Advanced Neuroanatomy',
'ANAT3025'), ('Neurobiology of Neurodegenerative Diseases', 'ANAT3028'),
('Cellular and Developmental Neurobiology', 'ANAT3030'), ('The Control of
Movement', 'ANAT3031'), ('Neural Computation and Models of Brain Function',
'ANAT3041'), ('The Peripheral Nervous System', 'ANAT3033'), ('Pain',
'ANAT3042'), ('Biological Timing (Chronobiology)', 'ANAT3043'),
('Neurobiology of Vision', 'ANAT3044'), ('Integrated Downstream Processing',
'BENG4001'), ('Design and Control of Biochemical Reactors', 'BENG4002'),
('Bioprocess Validation and Quality Control', 'BENG4010'), ('Integrated
Biochemical Engineering Design', 'BENG4011'), ('Bioprocess Design and
Implementation', 'BENG4012'), ('- Research Project', 'Option 1'),
('Bioprocess Research Project and', 'BENG4005'), ('Bioprocess Experimental
Study', 'BENG4006'), ('- Biological Sciences', 'Option 2'), ('Molecular
Biology for Biochemical Engineers', 'BIOC4601'), ('Structures of Biological
Polymers', 'BIOC4602'), ('Metabolic Pathways and\r\ntheir Regulation',
'BIOC4603'), ('- Process Engineering', 'Option 3'), ('Introduction to Human
Anatomy', 'ANAT1003')]

As you can see this containes some strange chars particularly  
\r\n which I believe is a carriage return, and
\xbf which I believe represents a hyphen - 
So I’m assuming that this is the reason for the failure in the SMI. And
would like to find a way to either replace these characters or alternatively
change the recordset encoding prior to generating the MultiListField.

I tried replacing these characters by converting the tuple for my
MultiListField into a string first using a python script but it didn’t seem
to work (do anything):

modules = context.module_retriever()
mytuple = modules.tuples()
mystring =str(mytuple).replace('\xbf','-')
return mystring

N.B. module_retriever is my Z SQL method that returns the resultset.

Does anyone have any ideas as to how I can resolve this?

Jon
UCL


___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] UnicodeDecodeError

2005-05-27 Thread Chris Withers 

Dieter Maurer wrote:

  Only the default is "latin-1" (as specified by the HTML
  specification).


Indeed, but that should be configurable...

Chris

--
Simplistix - Content Management, Zope & Python Consulting
   - http://www.simplistix.co.uk

___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] UnicodeDecodeError

2005-05-26 Thread Dieter Maurer 
Chris Withers wrote at 2005-5-25 20:54 +0100:
>Joern Wallstabe wrote:
>> Seem Zopes ZPT system is trying to do the encoding, depending on on 
>> setHeader() method.
>> So if this header is set you can simply return unicode strings.
>
>It's actually lower down in the stack... ZPublisher encodes the 
>response, and has a stoopid hard coded default of 'latin-1' that it 
>returns ;-)

You are wrong:

  ZPublisher uses what it finds in the "charset" parameter
  of the "Content-Type" response header.

  Only the default is "latin-1" (as specified by the HTML
  specification).

-- 
Dieter
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] UnicodeDecodeError

2005-05-26 Thread Chris Withers 

Joern Wallstabe wrote:
Seem Zopes ZPT system is trying to do the encoding, depending on on 
setHeader() method.

So if this header is set you can simply return unicode strings.


It's actually lower down in the stack... ZPublisher encodes the 
response, and has a stoopid hard coded default of 'latin-1' that it 
returns ;-)


cheers,

Chris

--
Simplistix - Content Management, Zope & Python Consulting
   - http://www.simplistix.co.uk

___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] UnicodeDecodeError

2005-05-23 Thread Dieter Maurer 
Joern Wallstabe wrote at 2005-5-23 11:18 +0200:
> ...
>the error is raised during zpt processing.
>by the way python default default encoding is set to ascii.
> 
>
>Module StringIO, line 203, in getvalue
>UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 95: 
>ordinal not in range(128)

Your page template mixes unicode and non-unicode (which cannot be
decoded with the default encoding).

-- 
Dieter
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] UnicodeDecodeError

2005-05-23 Thread Joern Wallstabe 

Thank you all for replying - now it works I got utf-8 string in my form.

Seem Zopes ZPT system is trying to do the encoding, depending on on 
setHeader() method.

So if this header is set you can simply return unicode strings.


Strange setting cmf default charset to utf-8 doesn't work :-(


Am Montag, den 23.05.2005, 10:35 +0200 schrieb Joern Wallstabe:


Hello,

i would like to feed a zpt form with data from a python dictionary via a 
python script


zpt snippet:







This is wrong anyway. Use tal:content in the textarea instead.


o.k. replace that with







python script prinDict:

keys_ = dict.keys()
keys_.sort()
lines=[]

for key in keys_:
   lines.append(u"%s:%s\n" %(key ,dict[key]))

uni_str = u"".join(lines)
return uni_str.encode('utf-8')


function now returns unicode string without utf-8 encoding


I'm always getting a  UnicodeDecodeError.

I' using Zope 2.7.5, CMF 1.5.1, Python 2.3.5
HTML meta tag is set to

CMF default_charset = utf-8




meta charset does not count btw. CMF default should, but it really 
depends on the real content-type header you send. Make sure its

the encoding you want there (setHeader() )


That was the important hint: added to cmf main_header_template:

tal:define "
dummy python: request.RESPONSE.setHeader('content-type', 'text/html;; 
charset=utf-8');




And its important to watch the encoding of the strings you work
with. Where does the dictionary come from?


python dictionary was result form parsing xml data into that dictionary 
(all unicode)


Thanx again :-)



smime.p7s
Description: S/MIME Cryptographic Signature
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] UnicodeDecodeError

2005-05-23 Thread Tino Wildenhain 
Am Montag, den 23.05.2005, 10:35 +0200 schrieb Joern Wallstabe:
> Hello,
> 
> i would like to feed a zpt form with data from a python dictionary via a 
> python script
> 
> zpt snippet:
> 
> 
> 

This is wrong anyway. Use tal:content in the textarea instead.

> 
> 
> python script prinDict:
> 
> keys_ = dict.keys()
> keys_.sort()
> lines=[]
> 
> for key in keys_:
> lines.append(u"%s:%s\n" %(key ,dict[key]))
> 
> uni_str = u"".join(lines)
> return uni_str.encode('utf-8')
> 
> I'm always getting a  UnicodeDecodeError.
> 
> I' using Zope 2.7.5, CMF 1.5.1, Python 2.3.5
> HTML meta tag is set to
>   
> CMF default_charset = utf-8
> 

meta charset does not count btw. CMF default should, but it really 
depends on the real content-type header you send. Make sure its
the encoding you want there (setHeader() )

And its important to watch the encoding of the strings you work
with. Where does the dictionary come from?



___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] UnicodeDecodeError

2005-05-23 Thread Andreas Jung 



--On Montag, 23. Mai 2005 11:18 Uhr +0200 Joern Wallstabe 
<[EMAIL PROTECTED]> wrote:





the script decodes unicode 'Spécification' -> utf8 'Sp\xc3\xa9cification'

the error is raised during zpt processing.
by the way python default default encoding is set to ascii.



hm...I can't reproduce this behaviour :-/

ANdreas


pgpNQ9wMM1V1g.pgp
Description: PGP signature
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] UnicodeDecodeError

2005-05-23 Thread Joern Wallstabe 

Andreas Jung wrote:



--On Montag, 23. Mai 2005 10:35 Uhr +0200 Joern Wallstabe 
<[EMAIL PROTECTED]> wrote:



I'm always getting a  UnicodeDecodeError.



To every error message belongs a traceback where you can see
*where* the error occurs...another try?

-aj



the script decodes unicode 'Spécification' -> utf8 'Sp\xc3\xa9cification'

the error is raised during zpt processing.
by the way python default default encoding is set to ascii.


Traceback (innermost last):
Module ZPublisher.Publish, line 101, in publish
Module ZPublisher.mapply, line 88, in mapply
Module ZPublisher.Publish, line 39, in call_object
Module Products.CMFFormController.FSControllerPageTemplate, line 98, in 
__call__
Module Products.CMFFormController.BaseControllerPageTemplate, line 42, 
in _call

Module Shared.DC.Scripts.Bindings, line 306, in __call__
Module Shared.DC.Scripts.Bindings, line 343, in _bindAndExec
Module Products.CMFCore.FSPageTemplate, line 188, in _exec
Module Products.CMFCore.FSPageTemplate, line 127, in pt_render
Module Products.PageTemplates.PageTemplate, line 97, in pt_render


Module StringIO, line 203, in getvalue
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 95: 
ordinal not in range(128)



regards joern


smime.p7s
Description: S/MIME Cryptographic Signature
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] UnicodeDecodeError

2005-05-23 Thread Andreas Jung 



--On Montag, 23. Mai 2005 10:35 Uhr +0200 Joern Wallstabe 
<[EMAIL PROTECTED]> wrote:



I'm always getting a  UnicodeDecodeError.


To every error message belongs a traceback where you can see
*where* the error occurs...another try?

-aj


pgpo0qQ4ukTs7.pgp
Description: PGP signature
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

[Zope] UnicodeDecodeError

2005-05-23 Thread Joern Wallstabe 

Hello,

i would like to feed a zpt form with data from a python dictionary via a 
python script


zpt snippet:





python script prinDict:

keys_ = dict.keys()
keys_.sort()
lines=[]

for key in keys_:
   lines.append(u"%s:%s\n" %(key ,dict[key]))

uni_str = u"".join(lines)
return uni_str.encode('utf-8')

I'm always getting a  UnicodeDecodeError.

I' using Zope 2.7.5, CMF 1.5.1, Python 2.3.5
HTML meta tag is set to

CMF default_charset = utf-8


Regards Joern




smime.p7s
Description: S/MIME Cryptographic Signature
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )