Re: [Zope-dev] UnicodeDecodeError from PageTemplate rendering after upgrading to Zope 2.12.7

2010-09-01 Thread Marius Gedminas
On Tue, Aug 31, 2010 at 10:40:46AM +0100, Chris Withers wrote:
 After a recent upgraded to Zope 2.12, I'm now seeing errors like the 
 following when using IE (version 8) or Safari (version 4.0.3) to view 
 page templates. Chrome and Firefox work fine.
 
 Traceback (innermost last):
Module ZPublisher.Publish, line 127, in publish
Module ZPublisher.mapply, line 77, in mapply
Module ZPublisher.Publish, line 47, in call_object
 snip
Module Shared.DC.Scripts.Bindings, line 324, in __call__
Module Shared.DC.Scripts.Bindings, line 361, in _bindAndExec
Module Products.PageTemplates.ZopePageTemplate, line 335, in _exec
Module Products.PageTemplates.ZopePageTemplate, line 432, in pt_render
Module Products.PageTemplates.PageTemplate, line 80, in pt_render
Module zope.pagetemplate.pagetemplate, line 115, in pt_render
Module zope.tal.talinterpreter, line 271, in __call__
 snip
Module zope.tal.talinterpreter, line 343, in interpret
Module zope.tal.talinterpreter, line 751, in do_insertStructure_tal
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xa0 in position 
 366: ordinal not in range(128)
 
 Has anyone seen this or anything similar before?

Yes.  Basically, every Zope site I've upgraded to 2.12 has suffered from
this.  Annoyingly this only happens with MSIE, so developers/admins tend
not to notice.

Clarifying the only with MSIE comment: this happens if and only if
the HTTP request doesn't explicitly include 'UTF-8' in Accept-Charset.
IIRC.

 Any solutions spring to mind?

Magically sprinkle .encode() / .decode() in TAL expressions/Python
scripts, where appropriate.

Most of the data stored in the ZODB is UTF-8 raw strings (things like
DTML Documents and DTML Methods don't support storing Unicode data).
Page templates now suddenly (since Zope 2.11) refuse to deal with raw
strings and insist on having Unicode.  Which means you need to do
explicit conversions not at your system boundaries, but inside your Zope
site, as you pass data between Python scripts and page templates.

Marius Gedminas
-- 
http://pov.lt/ -- Zope 3/BlueBream consulting and development


signature.asc
Description: Digital signature
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] UnicodeDecodeError from PageTemplate rendering after upgrading to Zope 2.12.7

2010-09-01 Thread Takayuki Shimizukawa
Hi Chris,

2010/9/1 Chris Withers ch...@simplistix.co.uk:
 Marius Gedminas wrote:
 Has anyone seen this or anything similar before?

 Yes.  Basically, every Zope site I've upgraded to 2.12 has suffered from
 this.  Annoyingly this only happens with MSIE, so developers/admins tend
 not to notice.

 ...well, and Safari, but no-one uses that, right? ;-)

I think of this problem like a topic same as following URLs.

 * https://mail.zope.org/pipermail/zope-dev/2009-August/037565.html
 * https://bugs.launchpad.net/zope2/+bug/160968
 * https://bugs.launchpad.net/zope2/+bug/143873
 * http://dev.plone.org/plone/ticket/8185

And why would this problem appear again?
Will this be regression?

-- 
Takayuki SHIMIZUKAWA
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] UnicodeDecodeError from PageTemplate rendering after upgrading to Zope 2.12.7

2010-08-31 Thread Chris Withers
Hi All,

After a recent upgraded to Zope 2.12, I'm now seeing errors like the 
following when using IE (version 8) or Safari (version 4.0.3) to view 
page templates. Chrome and Firefox work fine.

Traceback (innermost last):
   Module ZPublisher.Publish, line 127, in publish
   Module ZPublisher.mapply, line 77, in mapply
   Module ZPublisher.Publish, line 47, in call_object
snip
   Module Shared.DC.Scripts.Bindings, line 324, in __call__
   Module Shared.DC.Scripts.Bindings, line 361, in _bindAndExec
   Module Products.PageTemplates.ZopePageTemplate, line 335, in _exec
   Module Products.PageTemplates.ZopePageTemplate, line 432, in pt_render
   Module Products.PageTemplates.PageTemplate, line 80, in pt_render
   Module zope.pagetemplate.pagetemplate, line 115, in pt_render
   Module zope.tal.talinterpreter, line 271, in __call__
snip
   Module zope.tal.talinterpreter, line 343, in interpret
   Module zope.tal.talinterpreter, line 751, in do_insertStructure_tal
UnicodeDecodeError: 'ascii' codec can't decode byte 0xa0 in position 
366: ordinal not in range(128)

Has anyone seen this or anything similar before?
Any solutions spring to mind?

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing  Python Consulting
 - http://www.simplistix.co.uk
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] UnicodeDecodeError from PageTemplate rendering after upgrading to Zope 2.12.7

2010-08-31 Thread Marco Bizzarri
On Tue, Aug 31, 2010 at 11:40 AM, Chris Withers ch...@simplistix.co.ukwrote:

 Hi All,

 After a recent upgraded to Zope 2.12, I'm now seeing errors like the
 following when using IE (version 8) or Safari (version 4.0.3) to view
 page templates. Chrome and Firefox work fine.

 Traceback (innermost last):
   Module ZPublisher.Publish, line 127, in publish
   Module ZPublisher.mapply, line 77, in mapply
   Module ZPublisher.Publish, line 47, in call_object
 snip
   Module Shared.DC.Scripts.Bindings, line 324, in __call__
   Module Shared.DC.Scripts.Bindings, line 361, in _bindAndExec
   Module Products.PageTemplates.ZopePageTemplate, line 335, in _exec
   Module Products.PageTemplates.ZopePageTemplate, line 432, in pt_render
   Module Products.PageTemplates.PageTemplate, line 80, in pt_render
   Module zope.pagetemplate.pagetemplate, line 115, in pt_render
   Module zope.tal.talinterpreter, line 271, in __call__
 snip
   Module zope.tal.talinterpreter, line 343, in interpret
   Module zope.tal.talinterpreter, line 751, in do_insertStructure_tal
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xa0 in position
 366: ordinal not in range(128)

 Has anyone seen this or anything similar before?
 Any solutions spring to mind?



Hi Chris.

No solution, actually, but I would suggest to sniff the traffic between you
and the server; I suspect different headers cause problems, here.

Regards
Marco



 cheers,

 Chris

 --
 Simplistix - Content Management, Batch Processing  Python Consulting
 - http://www.simplistix.co.uk
 ___
 Zope-Dev maillist  -  Zope-Dev@zope.org
 https://mail.zope.org/mailman/listinfo/zope-dev
 **  No cross posts or HTML encoding!  **
 (Related lists -
  https://mail.zope.org/mailman/listinfo/zope-announce
  https://mail.zope.org/mailman/listinfo/zope )




-- 
Marco Bizzarri
http://code.google.com/p/qt-asterisk/
http://notenotturne.blogspot.com/
http://iliveinpisa.blogspot.com/
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] UnicodeDecodeError from PageTemplate rendering after upgrading to Zope 2.12.7

2010-08-31 Thread Chris Withers
Marco Bizzarri wrote:
 
 No solution, actually, but I would suggest to sniff the traffic between 
 you and the server; I suspect different headers cause problems, here.

Of that I have no doubt.

What does Zope2's publisher do with requested encodings and the like now?
Why would this be causing problems?

Errors that affect all browsers I'd be inclined to attribute to my code, 
ones that randomly affect specific browsers appear more likely to be a 
bug in Zope for me...

Chris

-- 
Simplistix - Content Management, Batch Processing  Python Consulting
 - http://www.simplistix.co.uk
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] UnicodeDecodeError from PageTemplate rendering after upgrading to Zope 2.12.7

2010-08-31 Thread Charlie Clark
Am 31.08.2010, 12:05 Uhr, schrieb Chris Withers ch...@simplistix.co.uk:

 What does Zope2's publisher do with requested encodings and the like now?
 Why would this be causing problems?

I can't remember exactly where but preferred_charset is looked for. It  
isn't set by either IE or Safari and, although it should default to UTF-8  
it seems they get served Latin 1 and you get badda boom pretty quickly.

 Errors that affect all browsers I'd be inclined to attribute to my code,
 ones that randomly affect specific browsers appear more likely to be a
 bug in Zope for me...

It is and I thought it had been squashed.

Charlie
-- 
Charlie Clark
Managing Director
Clark Consulting  Research
German Office
Helmholtzstr. 20
Düsseldorf
D- 40215
Tel: +49-211-600-3657
Mobile: +49-178-782-6226
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] UnicodeDecodeError from PageTemplate rendering after upgrading to Zope 2.12.7

2010-08-31 Thread Chris Withers
Charlie Clark wrote:
 Am 31.08.2010, 12:05 Uhr, schrieb Chris Withers ch...@simplistix.co.uk:
 
 What does Zope2's publisher do with requested encodings and the like now?
 Why would this be causing problems?
 
 I can't remember exactly where but preferred_charset is looked for. It  
 isn't set by either IE or Safari and, although it should default to UTF-8  
 it seems they get served Latin 1 and you get badda boom pretty quickly.

That's weird, the content in this app should be fine to be served latin-1.

 Errors that affect all browsers I'd be inclined to attribute to my code,
 ones that randomly affect specific browsers appear more likely to be a
 bug in Zope for me...
 
 It is and I thought it had been squashed.

Any idea what I should be looking for and in what changelog?

Chris

-- 
Simplistix - Content Management, Batch Processing  Python Consulting
 - http://www.simplistix.co.uk
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] UnicodeDecodeError from PageTemplate rendering after upgrading to Zope 2.12.7

2010-08-31 Thread Charlie Clark
Am 31.08.2010, 12:51 Uhr, schrieb Chris Withers ch...@simplistix.co.uk:

 Any idea what I should be looking for and in what changelog?

This my horribly patched version of charset.py which we just use via an  
override. I know it's wrong but it was the easiest thing to do at a time.

class HTTPCharsets(object):
 This is a complete copy of zope.publisher.http.HTTPCharset
 except if there is no charset specified. Zope defaults to
 ISO-8859-1 and we use the portal default_charset
 adapts(IHTTPRequest)
 implements(IUserPreferredCharsets)

 def __init__(self, request):
 self.request = request

 def getPreferredCharsets(self):
 '''See interface IUserPreferredCharsets'''
 charsets = []
 sawstar = sawiso88591 = 0
 header_present = 'HTTP_ACCEPT_CHARSET' in self.request
 for charset in self.request.get('HTTP_ACCEPT_CHARSET',  
'').split(','):
 charset = charset.strip().lower()
 if charset:
 if ';' in charset:
 charset, quality = charset.split(';')
 if not quality.startswith('q='):
 # not a quality parameter
 quality = 1.0
 else:
 try:
 quality = float(quality[2:])
 except ValueError:
 continue
 else:
 quality = 1.0
 if quality == 0.0:
 continue
 if charset == '*':
 sawstar = 1
 if charset == 'iso-8859-1':
 sawiso88591 = 1
 charsets.append((quality, charset))
 # Quoting RFC 2616, $14.2: If no * is present in an  
Accept-Charset
 # field, then all character sets not explicitly mentioned get a
 # quality value of 0, except for ISO-8859-1, which gets a quality
 # value of 1 if not explicitly mentioned.
 # And quoting RFC 2616, $14.2: If no Accept-Charset header is
 # present, the default is that any character set is acceptable.
 if not sawstar and not sawiso88591 and header_present:
 ptool = getUtility(IPropertiesTool)
 default_charset = ptool.getProperty('default_charset', None)
 charsets.append((1.0, default_charset))
 # charsets.append((1.0, 'iso-8859-1'))
 # UTF-8 is **always** preferred over anything else.
 # Reason: UTF-8 is not specific and can encode the entire unicode
 # range , unlike many other encodings. Since Zope can easily use  
very
 # different ranges, like providing a French-Chinese dictionary, it  
is
 # always good to use UTF-8.
 # charsets.sort(sort_charsets)
 charsets = [charset for quality, charset in charsets]
 if sawstar and charsets[0] != 'utf-8':
 if 'utf-8' in charsets:
 idx = charsets.index('utf-8')
 del charsets[idx]
 charsets.insert(0, 'utf-8')
 return charsets

The related bug in zope.publisher may or may not have found its way into  
the current release.

https://bugs.launchpad.net/zope2/+bug/143873

In addition, but I don't know how relevant this is in your case, Jens  
improved FSPageTemplate

https://bugs.launchpad.net/zope-cmf/+bug/322263

Charlie
-- 
Charlie Clark
Managing Director
Clark Consulting  Research
German Office
Helmholtzstr. 20
Düsseldorf
D- 40215
Tel: +49-211-600-3657
Mobile: +49-178-782-6226
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] UnicodeDecodeError from PageTemplate rendering after upgrading to Zope 2.12.7

2010-08-31 Thread Vladislav Vorobiev
Maybe not the best solution but I use this hack:

zope.tal-3.5.2-py2.7.egg/zope/tal/talinterpreter.py

line 751
change:

text = unicode(structure)

to

try:
 text = unicode(structure)
except:
 text=unicode(structure.decode('utf-8'))

Vlad


2010/8/31 Chris Withers ch...@simplistix.co.uk:
 Hi All,

 After a recent upgraded to Zope 2.12, I'm now seeing errors like the
 following when using IE (version 8) or Safari (version 4.0.3) to view
 page templates. Chrome and Firefox work fine.

 Traceback (innermost last):
   Module ZPublisher.Publish, line 127, in publish
   Module ZPublisher.mapply, line 77, in mapply
   Module ZPublisher.Publish, line 47, in call_object
 snip
   Module Shared.DC.Scripts.Bindings, line 324, in __call__
   Module Shared.DC.Scripts.Bindings, line 361, in _bindAndExec
   Module Products.PageTemplates.ZopePageTemplate, line 335, in _exec
   Module Products.PageTemplates.ZopePageTemplate, line 432, in pt_render
   Module Products.PageTemplates.PageTemplate, line 80, in pt_render
   Module zope.pagetemplate.pagetemplate, line 115, in pt_render
   Module zope.tal.talinterpreter, line 271, in __call__
 snip
   Module zope.tal.talinterpreter, line 343, in interpret
   Module zope.tal.talinterpreter, line 751, in do_insertStructure_tal
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xa0 in position
 366: ordinal not in range(128)

 Has anyone seen this or anything similar before?
 Any solutions spring to mind?

 cheers,

 Chris

 --
 Simplistix - Content Management, Batch Processing  Python Consulting
             - http://www.simplistix.co.uk
 ___
 Zope-Dev maillist  -  zope-...@zope.org
 https://mail.zope.org/mailman/listinfo/zope-dev
 **  No cross posts or HTML encoding!  **
 (Related lists -
  https://mail.zope.org/mailman/listinfo/zope-announce
  https://mail.zope.org/mailman/listinfo/zope )




-- 
Best Regards
Vlad Vorobiev
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] UnicodeDecodeError from PageTemplate rendering after upgrading to Zope 2.12.7

2010-08-31 Thread Chris Withers
Vladislav Vorobiev wrote:
 Maybe not the best solution but I use this hack:
 
 zope.tal-3.5.2-py2.7.egg/zope/tal/talinterpreter.py

Indeed, certainly not a good solution.
This implies that things should be unicode by the time they get here.

Now, the decoding shouldn't be dependent on anything the browser sends, 
should it? I would have thought only the encoding side of things should 
matter..

...so why on earth does the browser used have an effect on the decoding 
  (or not, as seems to be the case with the bug) of the structure?
(and where is this decoding done?)

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing  Python Consulting
 - http://www.simplistix.co.uk
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] UnicodeDecodeError from PageTemplate rendering after upgrading to Zope 2.12.7

2010-08-31 Thread Robert Casties
Hi Chris,

On 08/31/2010 11:40 AM, Chris Withers wrote:
 After a recent upgraded to Zope 2.12, I'm now seeing errors like the 
 following when using IE (version 8) or Safari (version 4.0.3) to view 
 page templates. Chrome and Firefox work fine.

This may be related to IE and Safari not sending an accept-charset
header and Zope then falling back to the Python default (often ASCII)
for encoding, which then fails.

I use the following patch to http.py (tested up to 2.12.3):

--- lib/python/zope/publisher/http.py.orig  2009-05-02
00:49:35.0 -0700
+++ lib/python/zope/publisher/http.py   2009-06-12 02:12:50.0 -0700
@@ -965,8 +965,11 @@
 # value of 1 if not explicitly mentioned.
 # And quoting RFC 2616, $14.2: If no Accept-Charset header is
 # present, the default is that any character set is acceptable.
-if not sawstar and not sawiso88591 and header_present:
+# ROC: the following checked also for header_present which
seemed wrong
+if not sawstar and not sawiso88591:
 charsets.append((1.0, 'iso-8859-1'))
+# ROC: if there is no header we assume star (to get utf-8)
+sawstar = 1
 # UTF-8 is **always** preferred over anything else.
 # Reason: UTF-8 is not specific and can encode the entire unicode
 # range , unlike many other encodings. Since Zope can easily
use very


Best
Robert


___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )