Re: [Zope] Character set problems
Niklas Saers wrote at 2005-9-16 21:34 +0200: ... HTTP/1.1 200 OK Date: Fri, 16 Sep 2005 19:38:27 GMT Server: Zope/(Zope 2.7.4-0, python 2.3.5, freebsd5) ZServer/1.1 Content-Type: text/html;charset=utf-8 Transfer-Encoding: chunked Do you tell Apache to add a Content-Type? Do not when you let Zope set the Content-Type (which is good). -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Character set problems
Dear Dragos and everyone,I have the problem narrowed down to the Apache/Zope connection. I've more or less decided I want to stick with ISO-8859-1 because moving everything to UTF-8 didn't really help me out. All in ISO-8859-1 works fine with Zope stand-alone, but unfortunately Apache needs to bridge it. As of right now, running it through the Apache bridge gives me the following: HTTP/1.1 200 OKDate: Fri, 16 Sep 2005 19:38:27 GMTServer: Zope/(Zope 2.7.4-0, python 2.3.5, freebsd5) ZServer/1.1Content-Type: text/html;charset=utf-8Transfer-Encoding: chunkedwhereas doing it through Zope gives me the following header: HTTP/1.1 200 OKServer: Zope/(Zope 2.7.4-0, python 2.3.5, freebsd5) ZServer/1.1Date: Fri, 16 Sep 2005 19:32:33 GMTContent-Length: 3941Content-Type: text/html; charset=iso-8859-1I'm a little bit confused as to why Apache changes the character-set and encoding. My Apache setup for the virual host is this: VirtualHost *:80 ServerAdmin [EMAIL PROTECTED] ProxyPass / http://localhost:8080/ ProxyPassReverse / http://localhost:8080/ DocumentRoot /home/mysite ServerName www.mysite.dk ServerAlias mysite.dk ErrorLog /var/log/www.mysite.dk-error_log CustomLog /var/log/www.mysite.dk-access_log common ## proxy_html_module conf ProxyHTMLURLMap http://localhost:8080 http://www.mysite.dk SetOutputFilter proxy-html RequestHeader unset Accept-Enconding AddDefaultCharset ISO-8859-1 CharsetDefault ISO-8859-1/VirtualHostWhat am I missing that makes the encoding go UTF-8? Not that it matters, but the HTML also contains META HTTPD_EQUIV=Content-Type CONTENT=text/html;charset=iso-8859-1 /Thanks for all your help so farCheers NikOn 9/9/05, Dragos Chirila [EMAIL PROTECTED] wrote: HiHere are some things you can do to handle your problem. For me, thisapproach worked just fine.1. Your HTML pages must have set the encoding to utf-8. Put this in theHEAD tag of your html pages: meta http-equiv=Content-Type content=text/html; charset=utf-8 /2. For the Root folder add a property named 'management_page_charset' oftype 'string' with the value 'utf-8'. In this way the ZMI pages will be displayed with utf-8 encoding.3. Modify your HTML code like this:input name=test:utf8:ustring /If 'test' is also a property of some object, make sure it has 'ustring' type.After that the value will be displayed correctly in all pages.Note: If your Zope runs over an Apache you must check also the Apachesettings (to serve pages utf-8 encoded)Hope this will help. Regards,DragosNiklas Saers wrote: Hey guys, I've installed Zope on a FreeBSD server. My problem domain is very small: display a list of when people are unavailable, and let people update the list. This is all in Danish, and here are come my problems. Danish has three extra letters not in the ASCII alfabet, three upper-case and three lower-case. When getting the data from a input name=test / where I've written æ ø å Æ Ø Å, I get: æ ø Ã¥ Æ Ø Ã… What is the standard way of ensuring that I get the correct data? I tried writing a little converter, but the string tekst = tekst.replace(\xc3\xa6, ae); gives me the error *UnicodeDecodeError: **'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)* Do you have any suggestion on how to overcome these problems? CheersNik ___ Zope maillist-Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding!** (Related lists -http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Character set problems
Hi Dragos, Note: If your Zope runs over an Apache you must check also the Apachesettings (to serve pages utf-8 encoded) Thank you very much for all your suggestions. Just a question to this one, what Apache directive should I set here? I was under the impression that Apache serves the pages as-is with the following directives: ProxyPass / http://localhost:8080/ ProxyPassReverse / http://localhost:8080/ ProxyHTMLURLMap http://localhost:8080 http://www.mysite.com SetOutputFilter proxy-html Cheers Nik ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Character set problems
We figured this out. Apache had AddDefaultCharset UTF-8 set. Changed it to AddDefaultCharset ISO-8859-1. Although Zope renders the page, it still has to pass through Apache on its way down to the client. Thanks for the help! Allen Niklas Saers wrote: Hi Dragos, Note: If your Zope runs over an Apache you must check also the Apache settings (to serve pages utf-8 encoded) Thank you very much for all your suggestions. Just a question to this one, what Apache directive should I set here? I was under the impression that Apache serves the pages as-is with the following directives: ProxyPass / http://localhost:8080/ ProxyPassReverse / http://localhost:8080/ ProxyHTMLURLMap http://localhost:8080 http://www.mysite.com SetOutputFilter proxy-html Cheers Nik ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Character set problems
I will let you know...but it was slightly off in the specs I gave. It was a 2.7.0 on RH9 and new install is 2.7.7 on RHE4. If I take the offending text, cut and paste, and put on a different install of 2.7.5 on RHE4 then the encoding works fine and the characters show fine. Mainly bullets, apostrophes and quotes copied from a Word doc and pasted in as text to a dtml document. I changed the LANG encoding for the zope user that zope runs as. But did not change the root LANG encoding. If zope runs as zope does it START as zope or as root? (Linux novice here if you could not tell already) This did not seem to make a difference. Only thing I can see at this point is the difference in zope version. Nik - I will let you know if I do find out anything. Allen Niklas Saers wrote: Hi Allen, I'm trying out the zope.conf solution proposed. Will you let me know if you make progress here? Cheers Nik On 9/8/05, Allen Schmidt [EMAIL PROTECTED] wrote: What timing. I am going through something similar. Had a zope 2.7.0 instance on RH9..not sure of the python. Had a bunch customer sites in it using VHM. Nothing in front of the zope. (I know) Moved to a new server, RHE4, Zope 2.7.5 on Python 2.3.5 Just moved the Data.fs and moved products and restarted. Perfect. Now one customer says they are seeing question marks in places where bullets and apostrophes, etc. should be. Looking at the encoding the browser picks up and it always goes to UTF-8. If I manually switch it to Western ISO the page looks fine. Where does this get set? In Linux? In Python? In Zope? ...and if so, where exactly and how to change it? Any help appreciated. Allen Peter Bengtsson wrote: Set the locale on the Zope installation in zope.conf If you're unsure about your code, try this $ set | grep LANG On 9/8/05, Niklas Saers [EMAIL PROTECTED] wrote: Hey guys, I've installed Zope on a FreeBSD server. My problem domain is very small: display a list of when people are unavailable, and let people update the list. This is all in Danish, and here are come my problems. Danish has three extra letters not in the ASCII alfabet, three upper-case and three lower-case. When getting the data from a input name=test / where I've written æ ø å Æ Ø Å, I get: æ ø Ã¥ Æ Ø Ã… What is the standard way of ensuring that I get the correct data? I tried writing a little converter, but the string tekst = tekst.replace(\xc3\xa6, ae); gives me the error UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128) Do you have any suggestion on how to overcome these problems? Cheers Nik ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Character set problems
I assume you use Firefox. Go to the two pages; the one that works and the one that doesn't. Right-click and select Vew Page Info. Look at what it says on Encoding On 9/9/05, Allen Schmidt [EMAIL PROTECTED] wrote: I will let you know...but it was slightly off in the specs I gave. It was a 2.7.0 on RH9 and new install is 2.7.7 on RHE4. If I take the offending text, cut and paste, and put on a different install of 2.7.5 on RHE4 then the encoding works fine and the characters show fine. Mainly bullets, apostrophes and quotes copied from a Word doc and pasted in as text to a dtml document. I changed the LANG encoding for the zope user that zope runs as. But did not change the root LANG encoding. If zope runs as zope does it START as zope or as root? (Linux novice here if you could not tell already) This did not seem to make a difference. Only thing I can see at this point is the difference in zope version. Nik - I will let you know if I do find out anything. Allen Niklas Saers wrote: Hi Allen, I'm trying out the zope.conf solution proposed. Will you let me know if you make progress here? Cheers Nik On 9/8/05, Allen Schmidt [EMAIL PROTECTED] wrote: What timing. I am going through something similar. Had a zope 2.7.0 instance on RH9..not sure of the python. Had a bunch customer sites in it using VHM. Nothing in front of the zope. (I know) Moved to a new server, RHE4, Zope 2.7.5 on Python 2.3.5 Just moved the Data.fs and moved products and restarted. Perfect. Now one customer says they are seeing question marks in places where bullets and apostrophes, etc. should be. Looking at the encoding the browser picks up and it always goes to UTF-8. If I manually switch it to Western ISO the page looks fine. Where does this get set? In Linux? In Python? In Zope? ...and if so, where exactly and how to change it? Any help appreciated. Allen Peter Bengtsson wrote: Set the locale on the Zope installation in zope.conf If you're unsure about your code, try this $ set | grep LANG On 9/8/05, Niklas Saers [EMAIL PROTECTED] wrote: Hey guys, I've installed Zope on a FreeBSD server. My problem domain is very small: display a list of when people are unavailable, and let people update the list. This is all in Danish, and here are come my problems. Danish has three extra letters not in the ASCII alfabet, three upper-case and three lower-case. When getting the data from a input name=test / where I've written æ ø å Æ Ø Å, I get: æ ø Ã¥ Æ Ø Ã… What is the standard way of ensuring that I get the correct data? I tried writing a little converter, but the string tekst = tekst.replace(\xc3\xa6, ae); gives me the error UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128) Do you have any suggestion on how to overcome these problems? Cheers Nik ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) -- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Character set problems
Hmmm... so the page setting on both zope versions is utf-8 but on the new zope, the text is actually shown with Western encoding. Warning! I'm no unicode expert at all. I live and work in the UK :) On 9/9/05, Allen Schmidt [EMAIL PROTECTED] wrote: Firefox, Netscape and IE all show the same UTF-8 encoding not matter what I do. But if I choose Western manually the characters show fine. Copy the same text out of the document and paste into a page on 2.7.5 and the encoding is Western. Thanks -Allen Peter Bengtsson wrote: I assume you use Firefox. Go to the two pages; the one that works and the one that doesn't. Right-click and select Vew Page Info. Look at what it says on Encoding On 9/9/05, Allen Schmidt [EMAIL PROTECTED] wrote: I will let you know...but it was slightly off in the specs I gave. It was a 2.7.0 on RH9 and new install is 2.7.7 on RHE4. If I take the offending text, cut and paste, and put on a different install of 2.7.5 on RHE4 then the encoding works fine and the characters show fine. Mainly bullets, apostrophes and quotes copied from a Word doc and pasted in as text to a dtml document. I changed the LANG encoding for the zope user that zope runs as. But did not change the root LANG encoding. If zope runs as zope does it START as zope or as root? (Linux novice here if you could not tell already) This did not seem to make a difference. Only thing I can see at this point is the difference in zope version. Nik - I will let you know if I do find out anything. Allen On 9/8/05, Allen Schmidt [EMAIL PROTECTED] wrote: What timing. I am going through something similar. Had a zope 2.7.0 instance on RH9..not sure of the python. Had a bunch customer sites in it using VHM. Nothing in front of the zope. (I know) Moved to a new server, RHE4, Zope 2.7.5 on Python 2.3.5 Just moved the Data.fs and moved products and restarted. Perfect. Now one customer says they are seeing question marks in places where bullets and apostrophes, etc. should be. Looking at the encoding the browser picks up and it always goes to UTF-8. If I manually switch it to Western ISO the page looks fine. Where does this get set? In Linux? In Python? In Zope? ...and if so, where exactly and how to change it? Any help appreciated. Allen Peter Bengtsson wrote: Set the locale on the Zope installation in zope.conf If you're unsure about your code, try this $ set | grep LANG -- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Character set problems
Hi Here are some things you can do to handle your problem. For me, this approach worked just fine. 1. Your HTML pages must have set the encoding to utf-8. Put this in the HEAD tag of your html pages: meta http-equiv=Content-Type content=text/html; charset=utf-8 / 2. For the Root folder add a property named 'management_page_charset' of type 'string' with the value 'utf-8'. In this way the ZMI pages will be displayed with utf-8 encoding. 3. Modify your HTML code like this: input name=test:utf8:ustring / If 'test' is also a property of some object, make sure it has 'ustring' type. After that the value will be displayed correctly in all pages. Note: If your Zope runs over an Apache you must check also the Apache settings (to serve pages utf-8 encoded) Hope this will help. Regards, Dragos Niklas Saers wrote: Hey guys, I've installed Zope on a FreeBSD server. My problem domain is very small: display a list of when people are unavailable, and let people update the list. This is all in Danish, and here are come my problems. Danish has three extra letters not in the ASCII alfabet, three upper-case and three lower-case. When getting the data from a input name=test / where I've written æ ø å Æ Ø Å, I get: æ ø Ã¥ Æ Ø Ã… What is the standard way of ensuring that I get the correct data? I tried writing a little converter, but the string tekst = tekst.replace(\xc3\xa6, ae); gives me the error *UnicodeDecodeError: **'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)* Do you have any suggestion on how to overcome these problems? Cheers Nik ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Character set problems
Niklas Saers wrote at 2005-9-8 18:59 +0200: ... What is the standard way of ensuring that I get the correct data? I tried writing a little converter, but the string tekst = tekst.replace(\xc3\xa6, ae); gives me the error *UnicodeDecodeError: **'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)* Apparently, tekst contains a unicode string. Mixing unicode and non-unicode (as you do above) is tricky in Python (avoid it, if you can). If you know that all non-unicode text uses the same encoding, you can set Python's default encoding. It can be set with sys.setdefaultencoding(encoding) -- but only during startup (usually in sitecustomize.py). Whenever, unicode and non-unicode come together, Python uses its default encoding to convert the non-unicode to unicode. If you do nothing, the default encoding is ascii -- resulting in the above error. The ideal way would be to have all your data and templates unicode and let ZPublisher (more precisely ZPublisher.HTTPResponse.HTTPResponse) convert to the output encoding (defined via the charset parameter of the Content-Type response header). Unfortunately, only few parts of Zope are already fully unicode aware: actually, only XML PageTemplates, but neither HTML PageTemplates not Python Scripts. This will change with Zope 3. Up to that time, you can live with either setting Python's default encoding or converting unicode explicitly to your encoding as soon as you get it. In either case, you *MUST* declare the encoding you are using in the charset parameter of the Content-Type response header. -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Character set problems
Set the locale on the Zope installation in zope.conf If you're unsure about your code, try this $ set | grep LANG On 9/8/05, Niklas Saers [EMAIL PROTECTED] wrote: Hey guys, I've installed Zope on a FreeBSD server. My problem domain is very small: display a list of when people are unavailable, and let people update the list. This is all in Danish, and here are come my problems. Danish has three extra letters not in the ASCII alfabet, three upper-case and three lower-case. When getting the data from a input name=test / where I've written æ ø å Æ Ø Å, I get: æ ø Ã¥ Æ Ø Ã… What is the standard way of ensuring that I get the correct data? I tried writing a little converter, but the string tekst = tekst.replace(\xc3\xa6, ae); gives me the error UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128) Do you have any suggestion on how to overcome these problems? Cheers Nik ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) -- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Character set problems
What timing. I am going through something similar. Had a zope 2.7.0 instance on RH9..not sure of the python. Had a bunch customer sites in it using VHM. Nothing in front of the zope. (I know) Moved to a new server, RHE4, Zope 2.7.5 on Python 2.3.5 Just moved the Data.fs and moved products and restarted. Perfect. Now one customer says they are seeing question marks in places where bullets and apostrophes, etc. should be. Looking at the encoding the browser picks up and it always goes to UTF-8. If I manually switch it to Western ISO the page looks fine. Where does this get set? In Linux? In Python? In Zope? ...and if so, where exactly and how to change it? Any help appreciated. Allen Peter Bengtsson wrote: Set the locale on the Zope installation in zope.conf If you're unsure about your code, try this $ set | grep LANG On 9/8/05, Niklas Saers [EMAIL PROTECTED] wrote: Hey guys, I've installed Zope on a FreeBSD server. My problem domain is very small: display a list of when people are unavailable, and let people update the list. This is all in Danish, and here are come my problems. Danish has three extra letters not in the ASCII alfabet, three upper-case and three lower-case. When getting the data from a input name=test / where I've written æ ø å Æ Ø Å, I get: æ ø Ã¥ Æ Ø Ã… What is the standard way of ensuring that I get the correct data? I tried writing a little converter, but the string tekst = tekst.replace(\xc3\xa6, ae); gives me the error UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128) Do you have any suggestion on how to overcome these problems? Cheers Nik ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
