Re: [Zope-dev] [ZOPE 2.6 B1] Unicode/locale problems with OFS/dtml/properties.dtml
On Thursday 26 Sep 2002 9:06 pm, Arnar Lundesgaard wrote: > Jepp, that seems to work. I'm at home though, so I haven't been able to try > with the original browser configuration. 'wget' gives me UTF-8 though, so > it should work nicely. :-) Thanks. This is now in the cvs trunk and the 2.6 branch. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] [ZOPE 2.6 B1] Unicode/locale problems with OFS/dtml/properties.dtml
On Friday 27 Sep 2002 10:38 am, Joachim Werner wrote: > After copying the relevant lines from the new > manage_page_header in Zope 2.6 everything works fine, hurrah! > except for one > remaining issue: b > One can create properties with Umlauts or other non-ASCII characters (which > probably should be forbidden in general). While in 2.5.1 it is possible to > delete such properties again, 2.6b1 will throw an error: The form was not telling ZPublisher's parameter marshalling that it wanted the property id as a string. try this (patch includes the previous fix) Index: properties.dtml === RCS file: /cvs-repository/Zope/lib/python/OFS/dtml/properties.dtml,v retrieving revision 1.8 diff -r1.8 properties.dtml 2c2 < --- > 47c47 < http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] [ZOPE 2.6 B1] Unicode/locale problems with OFS/dtml/properties.dtml
Hi! I have tracked down the problem with Collector issue 517 (Unicode problems) now: My Product used a modified manage_page_header that did not set the meta header for the encoding. After copying the relevant lines from the new manage_page_header in Zope 2.6 everything works fine, except for one remaining issue: One can create properties with Umlauts or other non-ASCII characters (which probably should be forbidden in general). While in 2.5.1 it is possible to delete such properties again, 2.6b1 will throw an error: The property [...] does not exist Probably the lookup in the property dictionary does not use the right encoding. I am not sure if that breaks old code, but the best way of getting rid of that problem might be to catch the creation of non-ASCII property names and don't let people use them. Cheers Joachim ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] [ZOPE 2.6 B1] Unicode/locale problems with OFS/dtml/properties.dtml
Hi!
Some time ago I filed this issue as a 2.6a1 bug
(http://collector.zope.org/Zope/517), only that it was German characters
with me ...
The browsers I was using where IE 6.0 (which led to errors storing property
sheets and could not display some of the propertysheets correctly) and
Mozilla, which just displayed the wrong characters.
I am quite sure that the problem is not with the Zope setup, but a real bug
in the way the forms are generated and evaluated. My Zope setup uses the
German locale settings. But I don't think that should be a problem ...
Cheers
Joachim
- Original Message -
From: "Toby Dickenson" <[EMAIL PROTECTED]>
To: "Arnar Lundesgaard" <[EMAIL PROTECTED]>; <[EMAIL PROTECTED]>
Sent: Thursday, September 26, 2002 6:04 PM
Subject: Re: [Zope-dev] [ZOPE 2.6 B1] Unicode/locale problems with
OFS/dtml/properties.dtml
On Thursday 26 Sep 2002 4:38 pm, Arnar Lundesgaard wrote:
> We can successfully store Norwegian characters
> ('æ','ø', and 'å') in property fields, but when we reload the page, we
> get results that display wrong.
Thats bad. I will definetely resolve this asap.
> When studying the generated sources, we see that the data from ZOPE
> inserted into the form is ISO-8859-1.
what do you mean by "inserted into the form"? That the response contains a
single byte where you properties contain character whose unicode value is
greater than 127 ? how have you checked this? if so, thats a bug.
> Secondly, the type attribute of all inputfields contain an extra
> ':utf8:' that we assumed is a server directive to interpret the
> contents as UTF-8. This apperently what crashes when storing the
> second time.
This is a directive to tell zope when you submit the form that your browser
will have encoded the form response using utf-8. Browsers stupidly dont put
this information anywhere more suitable.
What browser are you using? is it correctly using utf8 for this page? (for
example, Mozilla has a View/Encoding menu that can override server-supplied
encoding information)
> lib/python/OFS/dtml/properties.dtml contains the following that seems
> to us to be debug code:
>
>
> and several ':utf8:' directives.
>
> When we removed those, it worked fine. Was this dtml-method merged in
> mistakenly
Those are supposed to be there. The first inserts the
text/html;charset=utf-8
header into the management page. The second ensures that the dtml which
computes this page content returns a unicode object.
If the dtml returns a unicode object, then ZPublisher looks at the charset
header to determine how to encode it.
>, or are there problems in our setup.
could be.
please put ib/python/OFS/dtml/properties.dtml back the way is was
originally,
then send me
1. which browser you are using
2. a export file containing one object that demonstrates the problem.
3. a copy of the page obtained using wget or similar. (please dont use your
browsers 'save' feature because that sometimes performs transcoding)
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] [ZOPE 2.6 B1] Unicode/locale problems with OFS/dtml/properties.dtml
> Thanks for taking the time to help debug this. Thanks for fixing it so quickly! > This line, intended to force the dtml to be rendered as a unicode > object, is not doing its job. If the dtml is not a unicode object > then ZPublisher's encoding mechanism is not engaged. Has there been > some recent dtml optimisation to ignore empty strings perhaps? > This patch below works for me. Please let me know if this works for > you and I will apply it to 2.6. (and various sources of documentation > that all recommend including this line.) Jepp, that seems to work. I'm at home though, so I haven't been able to try with the original browser configuration. 'wget' gives me UTF-8 though, so it should work nicely. :-) Arnar Lundesgaard ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] [ZOPE 2.6 B1] Unicode/locale problems with OFS/dtml/properties.dtml
On Thursday 26 Sep 2002 5:47 pm, Arnar Lundesgaard wrote: Thanks for taking the time to help debug this. > > > lib/python/OFS/dtml/properties.dtml contains ... > > > > > This ensures that the dtml which computes this page > > content returns a unicode object. This line, intended to force the dtml to be rendered as a unicode object, is not doing its job. If the dtml is not a unicode object then ZPublisher's encoding mechanism is not engaged. Has there been some recent dtml optimisation to ignore empty strings perhaps? This patch below works for me. Please let me know if this works for you and I will apply it to 2.6. (and various sources of documentation that all recommend including this line.) Index: properties.dtml === RCS file: /cvs-repository/Zope/lib/python/OFS/dtml/properties.dtml,v retrieving revision 1.8 diff -c -2 -r1.8 properties.dtml *** properties.dtml 7 May 2002 17:54:56 - 1.8 --- properties.dtml 26 Sep 2002 17:54:48 - *** *** 1,4 ! --- 1,4 ! ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] [ZOPE 2.6 B1] Unicode/locale problems with OFS/dtml/properties.dtml
> what do you mean by "inserted into the form"? Put there by the dtml-method in a dtml-var statement. > That the response contains a single byte where you properties > contain character whose unicode value is greater than 127 ? Yes > how have you checked this? if so, thats a bug. In a Hex editor... The character 'æ' for instance is inserted as '0xE6' in the returned HTML. > > Secondly, the type attribute of all inputfields contain an extra > > ':utf8:' that we assumed is a server directive to interpret the > > contents as UTF-8. This apperently what crashes when storing the > > second time. > > This is a directive to tell zope when you submit the form > that your browser will have encoded the form response using > utf-8. Browsers stupidly dont put this information anywhere > more suitable. Yeah, we figured this would be similar to ':method'. > What browser are you using? is it correctly using utf8 for > this page? (for example, Mozilla has a View/Encoding menu > that can override server-supplied encoding information) We have so far tried Opera 6, IE 6 and Mozilla 1.1. > > lib/python/OFS/dtml/properties.dtml contains the following > > that seems to us to be debug code: > > > > > > and several ':utf8:' directives. > > > > When we removed those, it worked fine. Was this dtml-method > > merged in mistakenly > > Those are supposed to be there. The first inserts the > text/html;charset=utf-8 header into the management page. > The second ensures that the dtml which computes this page > content returns a unicode object. > > If the dtml returns a unicode object, then ZPublisher looks > at the charset header to determine how to encode it. This is DTML/ZOPE internal, right? I though the first two letters to signify endianness 'FFFE' or 'FEFF', tells you that this is UTF-8. The u'' string is not on the firs line. > please put ib/python/OFS/dtml/properties.dtml back the way is > was originally, then send me > > 1. which browser you are using I am using Opera 6.02, but we are seeing it on all browsers. > 2. a export file containing one object that demonstrates the problem. It is on every property page in the whole site, but see the attached .zexp of /standard_template.pt > 3. a copy of the page obtained using wget or similar. > (please dont use your browsers 'save' feature because > that sometimes performs transcoding) I used wget... -- Arnar Lundesgaard standard_template.pt.zexp Description: standard_template.pt.zexp manage_propertiesForm.1 Description: manage_propertiesForm.1
Re: [Zope-dev] [ZOPE 2.6 B1] Unicode/locale problems with OFS/dtml/properties.dtml
On Thursday 26 Sep 2002 4:38 pm, Arnar Lundesgaard wrote:
> We can successfully store Norwegian characters
> ('æ','ø', and 'å') in property fields, but when we reload the page, we
> get results that display wrong.
Thats bad. I will definetely resolve this asap.
> When studying the generated sources, we see that the data from ZOPE
> inserted into the form is ISO-8859-1.
what do you mean by "inserted into the form"? That the response contains a
single byte where you properties contain character whose unicode value is
greater than 127 ? how have you checked this? if so, thats a bug.
> Secondly, the type attribute of all inputfields contain an extra
> ':utf8:' that we assumed is a server directive to interpret the
> contents as UTF-8. This apperently what crashes when storing the
> second time.
This is a directive to tell zope when you submit the form that your browser
will have encoded the form response using utf-8. Browsers stupidly dont put
this information anywhere more suitable.
What browser are you using? is it correctly using utf8 for this page? (for
example, Mozilla has a View/Encoding menu that can override server-supplied
encoding information)
> lib/python/OFS/dtml/properties.dtml contains the following that seems
> to us to be debug code:
>
>
> and several ':utf8:' directives.
>
> When we removed those, it worked fine. Was this dtml-method merged in
> mistakenly
Those are supposed to be there. The first inserts the text/html;charset=utf-8
header into the management page. The second ensures that the dtml which
computes this page content returns a unicode object.
If the dtml returns a unicode object, then ZPublisher looks at the charset
header to determine how to encode it.
>, or are there problems in our setup.
could be.
please put ib/python/OFS/dtml/properties.dtml back the way is was originally,
then send me
1. which browser you are using
2. a export file containing one object that demonstrates the problem.
3. a copy of the page obtained using wget or similar. (please dont use your
browsers 'save' feature because that sometimes performs transcoding)
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] [ZOPE 2.6 B1] Unicode/locale problems with OFS/dtml/properties.dtml
We are trying out 2.6 b1. So far it seems very snappy (!!), and the
new error-log object is exceptionally usefull! Kudos to Zope Corp and
the community. :-)
While trying the beta we have found a few problems in the handling of
propertysheets. We can successfully store Norwegian characters
('æ','ø', and 'å') in property fields, but when we reload the page, we
get results that display wrong.
When studying the generated sources, we see that the data from ZOPE
inserted into the form is ISO-8859-1.
OTOH this field is inserted in the html-header:
Therefore the browser tries to display utf-8, which of course becomes
garbage when you get an ISO-8859-1 character with a larger value than
127.
All other pages (than the "properies" tab) in the have
Secondly, the type attribute of all inputfields contain an extra
':utf8:' that we assumed is a server directive to interpret the
contents as UTF-8. This apperently what crashes when storing the
second time.
lib/python/OFS/dtml/properties.dtml contains the following that seems
to us to be debug code:
and several ':utf8:' directives.
When we removed those, it worked fine. Was this dtml-method merged in
mistakenly, or are there problems in our setup.
Arnar Lundesgaard
---
phone: (+47) 982 38 036
mailto:arnar.lundesgaard(at)creuna.no
Creuna as
Bryggegata 3
NO-0250 Oslo
phone office: (+47) 23 23 88 00
fax: (+47) 23 23 88 50
http://www.creuna.no/
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
