Re: [Zope-dev] fmt=structured-text doesn't work with accented chars
Got it! StructuredText is setting locale back to C, during initialization, without setting it back to whatever it was before, in 'STletters.py'. I think it is wrong not only for this unilateral choice of locale :-), but also for trying to ignore python's notion of what the locale is when trying to calculate letters. The patch below fix both my 'Script (Python)' and the parsing problem with StructuredText. I took the liberty of removing the spurious 'import string'. --- lib/python/StructuredText/STletters.py.orig Wed Sep 12 20:00:52 2001 +++ lib/python/StructuredText/STletters.py Wed Sep 12 20:23:24 2001 @@ -1,8 +1,8 @@ -import string + try: del string import locale -locale.setlocale(locale.LC_ALL,) +#locale.setlocale(locale.LC_ALL,) except: pass Is there anywhere else I should submit it? Cheers, Leo PS: BTW, I'm still trying to figure out how to correctly set the locale on Windows, if anyone knows... :-) On Wed, Sep 12, 2001 at 06:49:21PM -0300, Leonardo Rochael Almeida wrote: Hi, I tried using structured text for some documentation I wrote in portuguese, but the accented characters like é, á, ã, or ç break the parsing of markups like *emphasis*, 'code' or _underline_, and in portuguese we get a bunch of those with every sentence, you know... :-) The result is the literal apearance of some characters that should be markup and some opening and closing mismatch of markup when there are more than one occurrence of the same markup, like when you try to *emphasize* twice in the *same* paragraph. This is probably due to parsing with python's string.letters which doesn't include accented letters unless you set the locale. Now the funny thing is that even when I set -L to, say, pt_BR at Zope start (and my glibc has correctly configured locales for pt_BR), structured-text still doesn't parse accented letters correctly. Investigating that, I found out that if you put, in a 'Script (Python)' the following:: import string return string.lowercase it will print 'abcdefghijklmnopqrstuvwxyz' instead of 'abcdefghijklmnopqrstuvwxyzµßàáâãäåæçèéêëìíîïðñòóôõöøùúûüýþÿ' Which is what you get in a python interpreter if you type:: import locale locale.setlocale(locale.LC_ALL, pt_BR) import string print string.lowercase Assuming you have a correctly configured pt_BR locale in a Linux machine. This exercise also gives the same results with a correctly configured en_US locale. Has anyone else stumbled onto this? Speaking of locales, how do I set locale for Zope in a WinNT/2000 environment? It seems like posix locale strings aren't valid strings for locale.setlocale() in Windows. Cheers, Leo ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] fmt=structured-text doesn't work with accented chars
AACK! How dumb can one get? The patch below is broken, here is the correct one: --- lib/python/StructuredText/STletters.py.orig Wed Sep 12 20:00:52 2001 +++ lib/python/StructuredText/STletters.py Wed Sep 12 20:47:36 2001 @@ -1,8 +1,7 @@ -import string + try: -del string import locale -locale.setlocale(locale.LC_ALL,) +#locale.setlocale(locale.LC_ALL,) except: pass On Wed, Sep 12, 2001 at 08:36:24PM -0300, Leonardo Rochael Almeida wrote: Got it! StructuredText is setting locale back to C, during initialization, without setting it back to whatever it was before, in 'STletters.py'. I think it is wrong not only for this unilateral choice of locale :-), but also for trying to ignore python's notion of what the locale is when trying to calculate letters. The patch below fix both my 'Script (Python)' and the parsing problem with StructuredText. I took the liberty of removing the spurious 'import string'. --- lib/python/StructuredText/STletters.py.orig Wed Sep 12 20:00:52 2001 +++ lib/python/StructuredText/STletters.pyWed Sep 12 20:23:24 2001 @@ -1,8 +1,8 @@ -import string + try: del string import locale -locale.setlocale(locale.LC_ALL,) +#locale.setlocale(locale.LC_ALL,) except: pass Is there anywhere else I should submit it? Cheers, Leo PS: BTW, I'm still trying to figure out how to correctly set the locale on Windows, if anyone knows... :-) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] fmt=structured-text doesn't work with accented chars
I fixed the locale problem in the CVS (trunk and 2.4 branch). So the fix will be in the next 2.4 release. Andreas - Original Message - From: Leonardo Rochael Almeida [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, September 12, 2001 18:36 Subject: Re: [Zope-dev] fmt=structured-text doesn't work with accented chars Got it! StructuredText is setting locale back to C, during initialization, without setting it back to whatever it was before, in 'STletters.py'. I think it is wrong not only for this unilateral choice of locale :-), but also for trying to ignore python's notion of what the locale is when trying to calculate letters. The patch below fix both my 'Script (Python)' and the parsing problem with StructuredText. I took the liberty of removing the spurious 'import string'. --- lib/python/StructuredText/STletters.py.orig Wed Sep 12 20:00:52 2001 +++ lib/python/StructuredText/STletters.py Wed Sep 12 20:23:24 2001 @@ -1,8 +1,8 @@ -import string + try: del string import locale -locale.setlocale(locale.LC_ALL,) +#locale.setlocale(locale.LC_ALL,) except: pass Is there anywhere else I should submit it? Cheers, Leo PS: BTW, I'm still trying to figure out how to correctly set the locale on Windows, if anyone knows... :-) On Wed, Sep 12, 2001 at 06:49:21PM -0300, Leonardo Rochael Almeida wrote: Hi, I tried using structured text for some documentation I wrote in portuguese, but the accented characters like é, á, ã, or ç break the parsing of markups like *emphasis*, 'code' or _underline_, and in portuguese we get a bunch of those with every sentence, you know... :-) The result is the literal apearance of some characters that should be markup and some opening and closing mismatch of markup when there are more than one occurrence of the same markup, like when you try to *emphasize* twice in the *same* paragraph. This is probably due to parsing with python's string.letters which doesn't include accented letters unless you set the locale. Now the funny thing is that even when I set -L to, say, pt_BR at Zope start (and my glibc has correctly configured locales for pt_BR), structured-text still doesn't parse accented letters correctly. Investigating that, I found out that if you put, in a 'Script (Python)' the following:: import string return string.lowercase it will print 'abcdefghijklmnopqrstuvwxyz' instead of 'abcdefghijklmnopqrstuvwxyzµßàáâãäåæçèéêëìíîïðñòóôõöøùúûüýþÿ' Which is what you get in a python interpreter if you type:: import locale locale.setlocale(locale.LC_ALL, pt_BR) import string print string.lowercase Assuming you have a correctly configured pt_BR locale in a Linux machine. This exercise also gives the same results with a correctly configured en_US locale. Has anyone else stumbled onto this? Speaking of locales, how do I set locale for Zope in a WinNT/2000 environment? It seems like posix locale strings aren't valid strings for locale.setlocale() in Windows. Cheers, Leo ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] fmt=structured-text doesn't work with accented chars
On Wed, Sep 12, 2001 at 07:53:11PM -0500, Andreas Jung wrote: I fixed the locale problem in the CVS (trunk and 2.4 branch). So the fix will be in the next 2.4 release. Cool. I took a look at CVS now and noticed how I should have just taken out the whole try/except block in my patch. As your comment there states, there is no use in just importing locale in this case. 'strings.lowercase' works just as well. Cheers, Leo - Original Message - From: Leonardo Rochael Almeida [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, September 12, 2001 18:36 Subject: Re: [Zope-dev] fmt=structured-text doesn't work with accented chars [...] The patch below fix both my 'Script (Python)' and the parsing problem with StructuredText. I took the liberty of removing the spurious 'import string'. --- lib/python/StructuredText/STletters.py.orig Wed Sep 12 20:00:52 2001 +++ lib/python/StructuredText/STletters.py Wed Sep 12 20:23:24 2001 @@ -1,8 +1,8 @@ -import string + try: del string import locale -locale.setlocale(locale.LC_ALL,) +#locale.setlocale(locale.LC_ALL,) except: pass Is there anywhere else I should submit it? Cheers, Leo PS: BTW, I'm still trying to figure out how to correctly set the locale on Windows, if anyone knows... :-) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
