Re: [Zope] Problem with non-latin character sets in Page Templates

2010-05-17 Thread Vladislav Vorobiev
On 05/17/2010 10:03 PM, Christopher N. Deckard wrote:
> Hello,
> Due to ICANN's decision to allow for non-Latin characters in domain
> names, I thought I'd give our Zope installation a test to see how it
> would handle it.  We've had problems with people copy and pasting from
> MS Word, and its use of strange characters.  I'm sure at some point the
> faculty here at Purdue will have a need to link to web sites using
> non-Latin characters.
>
> This is my example HTML:
>
> 
>   
> URL Test
>   
>   
>  href="http://EHB9.H2'1)-'D#*5'D'*.E51">http://EHB9.H2'1)-'D#*5'D'*.E51.
>   
> 
>
>
> Not only is it a different character set, but it is a right to left
> character set.  This code works fine while editing a Page Template.
> However, when viewing it, all of those characters in the href are
> converted to question marks.  If the same HTML is pasted into a DTML
> Method or a File object, Zope will convert the characters to ASCII
> characters.  When viewed it will be viewed correctly.
>
> Any reason that Page Templates may fail to render this properly?
>
It is a bug of page template. I patched them

zope.pagetemplate-3.5.0-py2.6.egg/zope/pagetemplate/pagetemplate.py

line 116 return output.getvalue()

to

116 # --- unicode/utf-8 hotfix ---
   for idx in range(0, len(output.buflist)):
   try:
   output.buflist[idx] = unicode(output.buflist[idx], 'utf-8')
   except (UnicodeDecodeError):
   output.buflist[idx] = output.buflist[idx].decode('utf-8',
'replace')
   except:
   pass

   return output.getvalue()
   # --- unicode/utf-8 hotfix ---


___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists -
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )


___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )


[Zope] Problem with non-latin character sets in Page Templates

2010-05-17 Thread Christopher N. Deckard
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hello,
Due to ICANN's decision to allow for non-Latin characters in domain
names, I thought I'd give our Zope installation a test to see how it
would handle it.  We've had problems with people copy and pasting from
MS Word, and its use of strange characters.  I'm sure at some point the
faculty here at Purdue will have a need to link to web sites using
non-Latin characters.

This is my example HTML:


  
URL Test
  
  
http://موقع.وزارة-الأتصالات.مصر";>http://موقع.وزارة-الأتصالات.مصر.
  



Not only is it a different character set, but it is a right to left
character set.  This code works fine while editing a Page Template.
However, when viewing it, all of those characters in the href are
converted to question marks.  If the same HTML is pasted into a DTML
Method or a File object, Zope will convert the characters to ASCII
characters.  When viewed it will be viewed correctly.

Any reason that Page Templates may fail to render this properly?

- -Chris


- -- 

- 
Christopher N. Deckard  | Lead Web Systems Developer
  c...@ecn.purdue.edu|Engineering Computer Network
  http://eng.purdue.edu/ECN/| Purdue University
-  zlib.decompress('x\234K\316Kq((-J)M\325KM)\005\000)"\005w') ---

-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.12 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkvxoQcACgkQ+zGkEDwn8UZcGACgufURwanURBIBcZUSLYYRkEvV
vKwAoNnGMlmOhE/GciZuTO3ldptuBr4Z
=mypE
-END PGP SIGNATURE-

___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )