[Zope-CMF] Re: [RFC] [Patch] GenericSetup and encodings

2006-06-09 Thread yuppie
Hi! Yves Bastide wrote: yuppie wrote: As you already mentioned setting default-zpublisher-encoding to 'utf-8' doesn't really work. Just found that DT_Util.join_unicode has 'latin-1' hardcoded, so properties with other encodings are not supported by manage_propertiesForm. Given that I don

[Zope-CMF] Re: [RFC] [Patch] GenericSetup and encodings

2006-06-08 Thread Yves Bastide
yuppie wrote: As you already mentioned setting default-zpublisher-encoding to 'utf-8' doesn't really work. Just found that DT_Util.join_unicode has 'latin-1' hardcoded, so properties with other encodings are not supported by manage_propertiesForm. Given that I don't think we have to support

[Zope-CMF] Re: [RFC] [Patch] GenericSetup and encodings

2006-06-08 Thread Yves Bastide
yuppie wrote: As you already mentioned setting default-zpublisher-encoding to 'utf-8' doesn't really work. Just found that DT_Util.join_unicode has 'latin-1' hardcoded, so properties with other encodings are not supported by manage_propertiesForm. I'm just about to send a mail to zope.devel

[Zope-CMF] Re: [RFC] [Patch] GenericSetup and encodings

2006-06-08 Thread yuppie
Yves Bastide wrote: yuppie wrote: Yves Bastide wrote: converter is field2string, default_encoding is ZPublisher.HTTPRequest.default_encoding = 'iso-8859-15' (not to mistake for ZPublisher.Converters.default_encoding = 'iso-8859-15'). These default_encoding's are set by Zope2.Startup.datat

[Zope-CMF] Re: [RFC] [Patch] GenericSetup and encodings

2006-06-08 Thread Yves Bastide
yuppie wrote: Yves Bastide wrote: converter is field2string, default_encoding is ZPublisher.HTTPRequest.default_encoding = 'iso-8859-15' (not to mistake for ZPublisher.Converters.default_encoding = 'iso-8859-15'). These default_encoding's are set by Zope2.Startup.datatypes.default_zpublish

[Zope-CMF] Re: [RFC] [Patch] GenericSetup and encodings

2006-06-08 Thread yuppie
Yves Bastide wrote: Well, should have looked up in the call stack. ZPublisher.HTTPRequest.processInputs, lines 527sq (Zope trunk): 527: item = unicode(item,character_encoding) 528: if hasattr(converter,'convert_unicode'): 529: item = converter.convert_unicode(item) 530: else: 531: item

[Zope-CMF] Re: [RFC] [Patch] GenericSetup and encodings

2006-06-08 Thread Yves Bastide
Replying to myself... Yves Bastide wrote: I know of at least one point, ZPublisher.Converters (field2string). However by the time a supposedly unicode string (say title:UTF-8:string) comes here, it's already iso8859. Will look deeper ... Well, should have looked up in the call stack. ZPubli

[Zope-CMF] Re: [RFC] [Patch] GenericSetup and encodings

2006-06-08 Thread Yves Bastide
yuppie wrote: Hi! [...] With this applied, Portàl (u'Port\xe0l'), which becomes 'Port\xc3\xa0l', is displayed as Portà l ... Zope does input--output properties in utf-8, but stores them in iso8859. Sigh. I was afraid this would be complex :( That's why I only use ASCII in configuration dat

[Zope-CMF] Re: [RFC] [Patch] GenericSetup and encodings

2006-06-08 Thread yuppie
Hi! Yves Bastide wrote: yuppie wrote: Yves Bastide wrote: yuppie wrote: Here's a minimal patch for GenericSetup not to raise on the previous case (Demonstration product. Not for sale.) [...] With this applied, Portàl (u'Port\xe0l'), which becomes 'Port\xc3\xa0l', is displayed as PortÃ

[Zope-CMF] Re: [RFC] [Patch] GenericSetup and encodings

2006-06-07 Thread Yves Bastide
yuppie wrote: Hi! Yves Bastide wrote: yuppie wrote: 3.) GenericSetup is not tested with non-ASCII UTF-8 site settings. AFAIK import works, but not export. I consider this a bug. [...] UnicodeDecodeError: 'ascii' codec can't decode byte 0xf4 in position 20: ordinal not in range(128) This

[Zope-CMF] Re: [RFC] [Patch] GenericSetup and encodings

2006-06-07 Thread yuppie
Hi! Yves Bastide wrote: yuppie wrote: 2.) GenericSetup explicitly doesn't support non-UTF-8 site settings. If someone provides a good patch this feature can be added. But with the problems you mention later ('default_charset', 'management_page_charset', and so on), how would you envision

[Zope-CMF] Re: [RFC] [Patch] GenericSetup and encodings

2006-06-07 Thread Yves Bastide
yuppie wrote: Hi Yves! Yves Bastide wrote: GenericSetup has problems handling non-ASCII data. 1.) GenericSetup explicitly doesn't support non-UTF-8 XML in profiles. UTF-8 is the default encoding for XML and I can't see a need to support other XML encodings. As output, right? Agreed. 2

[Zope-CMF] Re: [RFC] [Patch] GenericSetup and encodings

2006-06-07 Thread yuppie
Hi Florent! Florent Guillaume wrote: Let's not forget also that the goal in CMF 2 (I think) is to have all content be unicode strings, never encoded ones. In that case GenericSetup only has to deal with the XML file's encoding (always UTF-8 anyway) but that's all. That's a long term goal.

[Zope-CMF] Re: [RFC] [Patch] GenericSetup and encodings

2006-06-07 Thread Florent Guillaume
yuppie wrote: Hi Yves! Yves Bastide wrote: GenericSetup has problems handling non-ASCII data. 1.) GenericSetup explicitly doesn't support non-UTF-8 XML in profiles. UTF-8 is the default encoding for XML and I can't see a need to support other XML encodings. 2.) GenericSetup explicitly do

[Zope-CMF] Re: [RFC] [Patch] GenericSetup and encodings

2006-06-07 Thread yuppie
Hi Yves! Yves Bastide wrote: GenericSetup has problems handling non-ASCII data. 1.) GenericSetup explicitly doesn't support non-UTF-8 XML in profiles. UTF-8 is the default encoding for XML and I can't see a need to support other XML encodings. 2.) GenericSetup explicitly doesn't support n