Re: [Zope-dev] UnicodeDecodeError problems with Zope 2.12.0b4
Chris Withers wrote: Annoyingly, this required even more evil, since sitecustomize.py is no longer found anywhere sensible in Python 2.5, correction: it never was, but prior to Zope 2.12, the INSTANCE_HOME ended up on PYTHONPATH and so sitecustomize.py could be put there. Using the buildout approach for Zope 2.12, it not longer is on PYTHONPATH, so it doesn't get found... Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] UnicodeDecodeError problems with Zope 2.12.0b4
There is a related bug report on LP (with a controversial discussion) about the issue (can't find the URL right now). Andreas On 21.08.09 21:40, Chris Withers wrote: Hi All, I'm suffering with two different types of UnicodeDecodeError with a Zope 2.12 project... The first type of problem occurs when a form is submitted and the form fields contain encoded strings. Somewhere down the line I get: Module zope.i18n, line 166, in interpolate Module zope.i18n, line 161, in replace UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 5: ordinal not in range(128) So, my question is how do I get strings submitted to the publisher (ideally) end up as unicode objects not encoded strings? Failing that, how do I find out what encoding the browser has provided? The second type of problem only occurs in IE and Safari, where viewing a page that contains some non-ascii-able data: Module ZPublisher.Publish, line 127, in publish Module ZPublisher.mapply, line 77, in mapply Module ZPublisher.Publish, line 47, in call_object Module Products.Five.browser.metaconfigure, line 427, in __call__ Module Products.Five.browser.pagetemplatefile, line 126, in __call__ Module Products.Five.browser.pagetemplatefile, line 60, in __call__ Module zope.pagetemplate.pagetemplate, line 116, in pt_render - Warning: Macro expansion failed - Warning: class 'zope.tal.taldefs.METALError': macro nesting limit (100) exceeded by u'root/standard_template.pt/macros/nav' Module StringIO, line 270, in getvalue UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 10: ordinal not in range(128) Two things are weird here: - Why am I getting a macro nesting limit? This seems to be a recurring problem (excuse the pun) with Zope 2.12 page templates where if there's an error in a standard template, I get this error when a template that uses it is rendered. - Why am I getting this decoding error only with these browsers? Firefox and Chrome work just fine... Is the publisher doing something special with a header these browsers send that IE and Safari are not? If so, how do I make it do the right thing for IE and Safari? cheers, Chris -- ZOPYX Ltd. Co KG \ ZOPYX Friends Charlottenstr. 37/1 \ The experts for your Python, Zope and D-72070 Tübingen \ Plone projects www.zopyx.com, i...@zopyx.com \ www.zopyx.de/friends, frie...@zopyx.de E-Publishing, Python, Zope Plone development, Consulting attachment: lists.vcf___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] UnicodeDecodeError problems with Zope 2.12.0b4
Am 30.08.2009, 12:51 Uhr, schrieb Andreas Jung li...@zopyx.com: - Why am I getting this decoding error only with these browsers? Firefox and Chrome work just fine... Is the publisher doing something special with a header these browsers send that IE and Safari are not? If so, how do I make it do the right thing for IE and Safari? I thought this had been fixed? I'm pretty sure it's related to 'HTTP_ACCEPT_CHARSET' missing in the headers from IE and Safari and the charset detector incorrectly extracting it from the REQUEST which is always True for Zope 2. zope.publisher.http.getPreferredCharsets() is where this magic happens. Charlie -- Charlie Clark Managing Director Clark Consulting Research Ltd. German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 GSM: +49-178-782-6226 Company No. 06770088 ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] UnicodeDecodeError problems with Zope 2.12.0b4
On 30.08.09 13:01, Charlie Clark wrote: Am 30.08.2009, 12:51 Uhr, schrieb Andreas Jung li...@zopyx.com: - Why am I getting this decoding error only with these browsers? Firefox and Chrome work just fine... Is the publisher doing something special with a header these browsers send that IE and Safari are not? If so, how do I make it do the right thing for IE and Safari? I thought this had been fixed? I'm pretty sure it's related to 'HTTP_ACCEPT_CHARSET' missing in the headers from IE and Safari and the charset detector incorrectly extracting it from the REQUEST which is always True for Zope 2. zope.publisher.http.getPreferredCharsets() is where this magic happens. Possibly fixed in new versions of zope.publisher. I think there was some open issue with backporting some fix - but my brain can't remember :- Andreas attachment: lists.vcf___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] UnicodeDecodeError problems with Zope 2.12.0b4
Am 30.08.2009, 13:26 Uhr, schrieb Andreas Jung li...@zopyx.com: Possibly fixed in new versions of zope.publisher. I think there was some open issue with backporting some fix - but my brain can't remember :- See https://bugs.launchpad.net/zope2/+bug/160968 and https://bugs.launchpad.net/bugs/143873 So, it should be in the current code-base and shouldn't be happening at all in Zope 2.12. So it could be something else. I only know about it because I keep on it hitting it in Windows installs as there isn't a binary of 2.11 with the fix. Charlie -- Charlie Clark Managing Director Clark Consulting Research Ltd. German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 GSM: +49-178-782-6226 Company No. 06770088 ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] UnicodeDecodeError problems with Zope 2.12.0b4
On 30.08.09 13:58, Charlie Clark wrote: Am 30.08.2009, 13:26 Uhr, schrieb Andreas Jung li...@zopyx.com: Possibly fixed in new versions of zope.publisher. I think there was some open issue with backporting some fix - but my brain can't remember :- See https://bugs.launchpad.net/zope2/+bug/160968 and https://bugs.launchpad.net/bugs/143873 No, I am referring to a controversial discussion with Dieter - however I really can not find it anymore. Anyway...my conclusion from this discussion was that a custom unicode conflict resolver should be used for resolving such custom problems (unless with a real bug in the processing of the http headers). A clean solution would be to introduce the notion of a default charset for a site manager (as the 'default_charset' property of CMF sites or Plone sites). A unicode conflict resolver could get hold of the (local) site manager and check for a (registered) local utiltiy return the charset through some API...something like that (instead of guessing/sniffing in the HTTP client headers). Andreas attachment: lists.vcf___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] UnicodeDecodeError problems with Zope 2.12.0b4
Hi Charlie, I only know about it because I keep on it hitting it in Windows installs as there isn't a binary of 2.11 with the fix. You mean the latest binary doesn't have the fix? Or you haven't seen that a binary was released recently? http://www.zope.org/Products/Zope/2.11.4/ -- Sidnei ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] UnicodeDecodeError problems with Zope 2.12.0b4
Am 30.08.2009, 19:16 Uhr, schrieb Sidnei da Silva sidnei.da.si...@gmail.com: You mean the latest binary doesn't have the fix? Or you haven't seen that a binary was released recently? http://www.zope.org/Products/Zope/2.11.4/ ah, I hadn't seen that one - my last Windows install was in July. Will give a try this week and am very much looking forward to removing my hackish override. Charlie -- Charlie Clark Managing Director Clark Consulting Research Ltd. German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 GSM: +49-178-782-6226 Company No. 06770088 ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] UnicodeDecodeError problems with Zope 2.12.0b4
Takayuki Shimizukawa wrote:
2009/8/22 Chris Withers ch...@simplistix.co.uk:
The first type of problem occurs when a form is submitted and the form
fields contain encoded strings. Somewhere down the line I get:
The first type, I don't know.
The second type of problem only occurs in IE and Safari, where viewing a
page that contains some non-ascii-able data:
The second type, (maybe) similer probrem reported to launchpad.
https://bugs.launchpad.net/zope2/+bug/160968
- Why am I getting this decoding error only with these browsers? Firefox
and Chrome work just fine... Is the publisher doing something special
with a header these browsers send that IE and Safari are not? If so, how
do I make it do the right thing for IE and Safari?
Are concerned that HTTP_ACCEPT_CHARSET http header will probably
not be sent from client browser (IE6,7, Safari).
When HTTP_ACCEPT_CHARSET not in HTTP request headers,
zope.publisher use UTF-8 encoding for publishing.
Well, I just solved the problem with a large setdefaultencoding hammer :-)
Annoyingly, this required even more evil, since sitecustomize.py is no
longer found anywhere sensible in Python 2.5, so I ended up with a
buildout that looks like the following:
[buildout]
parts = instance
extends =
http://svn.zope.org/*checkout*/Zope/tags/2.12.0b4/versions.cfg?rev=102515
versions = versions
[versions]
zope2 = 2.12.0b4
[instance]
recipe = zc.recipe.egg
eggs = ${buildout:eggs}
interpreter = py
entry-points=
runzope=Zope2.Startup.run:run
zopectl=Zope2.Startup.zopectl:main
scripts = runzope zopectl
initialization =
import sys
reload(sys)
sys.setdefaultencoding('utf-8')
sys.argv[1:1] = ['-C','${buildout:directory}/etc/instance.conf']
Evil huh? ;-)
Hope this helps someone...
Chris
--
Simplistix - Content Management, Batch Processing Python Consulting
- http://www.simplistix.co.uk
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] UnicodeDecodeError problems with Zope 2.12.0b4
Hi Chris. 2009/8/22 Chris Withers ch...@simplistix.co.uk: The first type of problem occurs when a form is submitted and the form fields contain encoded strings. Somewhere down the line I get: The first type, I don't know. The second type of problem only occurs in IE and Safari, where viewing a page that contains some non-ascii-able data: The second type, (maybe) similer probrem reported to launchpad. https://bugs.launchpad.net/zope2/+bug/160968 - Why am I getting this decoding error only with these browsers? Firefox and Chrome work just fine... Is the publisher doing something special with a header these browsers send that IE and Safari are not? If so, how do I make it do the right thing for IE and Safari? Are concerned that HTTP_ACCEPT_CHARSET http header will probably not be sent from client browser (IE6,7, Safari). When HTTP_ACCEPT_CHARSET not in HTTP request headers, zope.publisher use UTF-8 encoding for publishing. regards. -- Takayuki Shimizukawa ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] UnicodeDecodeError problems with Zope 2.12.0b4
Hi All, I'm suffering with two different types of UnicodeDecodeError with a Zope 2.12 project... The first type of problem occurs when a form is submitted and the form fields contain encoded strings. Somewhere down the line I get: Module zope.i18n, line 166, in interpolate Module zope.i18n, line 161, in replace UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 5: ordinal not in range(128) So, my question is how do I get strings submitted to the publisher (ideally) end up as unicode objects not encoded strings? Failing that, how do I find out what encoding the browser has provided? The second type of problem only occurs in IE and Safari, where viewing a page that contains some non-ascii-able data: Module ZPublisher.Publish, line 127, in publish Module ZPublisher.mapply, line 77, in mapply Module ZPublisher.Publish, line 47, in call_object Module Products.Five.browser.metaconfigure, line 427, in __call__ Module Products.Five.browser.pagetemplatefile, line 126, in __call__ Module Products.Five.browser.pagetemplatefile, line 60, in __call__ Module zope.pagetemplate.pagetemplate, line 116, in pt_render - Warning: Macro expansion failed - Warning: class 'zope.tal.taldefs.METALError': macro nesting limit (100) exceeded by u'root/standard_template.pt/macros/nav' Module StringIO, line 270, in getvalue UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 10: ordinal not in range(128) Two things are weird here: - Why am I getting a macro nesting limit? This seems to be a recurring problem (excuse the pun) with Zope 2.12 page templates where if there's an error in a standard template, I get this error when a template that uses it is rendered. - Why am I getting this decoding error only with these browsers? Firefox and Chrome work just fine... Is the publisher doing something special with a header these browsers send that IE and Safari are not? If so, how do I make it do the right thing for IE and Safari? cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
