Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-10-08 Thread M.-A. Lemburg
Adam Olsen wrote: On Sun, Sep 20, 2009 at 10:17, Zooko O'Whielacronx zoo...@gmail.com wrote: On Sun, Sep 20, 2009 at 8:27 AM, Antoine Pitrou solip...@pitrou.net wrote: AFAIK, C extensions should fail loading when they have the wrong UCS2/4 setting. That would be an improvement!

[Python-Dev] please consider changing --enable-unicode default to ucs4

2009-10-07 Thread Zooko O'Whielacronx
that is preventing people from distributing and using Python packages with binary extension modules on Linux. Regards, Zooko -- Forwarded message -- From: Zooko O'Whielacronx zoo...@gmail.com Date: Sun, Sep 27, 2009 at 11:43 AM Subject: Re: [Python-Dev] please consider changing --enable

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-10-07 Thread M.-A. Lemburg
Zooko O'Whielacronx wrote: Dear MAL and python-dev: I failed to explain the problem that users are having. I will try again, and this time I will omit my ideas about how to improve things and just focus on describing the problem. Some users are having trouble using Python packages

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-10-07 Thread Antoine Pitrou
Zooko O'Whielacronx zookog at gmail.com writes: I accidentally sent this letter just to MAL when I intended it to python-dev. Please read it, as it explains why the issue I'm raising is not just the we should switch to ucs4 because it is better issue that was previously settled by GvR.

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-10-07 Thread Ronald Oussoren
On 7 Oct, 2009, at 20:05, M.-A. Lemburg wrote: If we do go for a change, we should use sizeof(wchar_t) as basis for the new default - on all platforms that provide a wchar_t type. I'd be -1 on that. Sizeof(wchar_t) is 4 on OSX, but all non-Unix API's that deal with Unicode text use ucs16.

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-10-07 Thread M.-A. Lemburg
Ronald Oussoren wrote: On 7 Oct, 2009, at 20:05, M.-A. Lemburg wrote: If we do go for a change, we should use sizeof(wchar_t) as basis for the new default - on all platforms that provide a wchar_t type. I'd be -1 on that. Sizeof(wchar_t) is 4 on OSX, but all non-Unix API's that deal

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-10-07 Thread Ronald Oussoren
On 7 Oct, 2009, at 22:13, M.-A. Lemburg wrote: Ronald Oussoren wrote: On 7 Oct, 2009, at 20:05, M.-A. Lemburg wrote: If we do go for a change, we should use sizeof(wchar_t) as basis for the new default - on all platforms that provide a wchar_t type. I'd be -1 on that. Sizeof(wchar_t) is

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-10-07 Thread M.-A. Lemburg
Ronald Oussoren wrote: On 7 Oct, 2009, at 22:13, M.-A. Lemburg wrote: Ronald Oussoren wrote: On 7 Oct, 2009, at 20:05, M.-A. Lemburg wrote: If we do go for a change, we should use sizeof(wchar_t) as basis for the new default - on all platforms that provide a wchar_t type. I'd be -1

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-10-07 Thread Neil Hodgson
Ronald Oussoren: Both Carbon and the modern APIs use UTF-16. If Unicode size standardization is seen as sufficiently beneficial then UTF-16 would be more widely applicable than UTF-32. Unix mostly uses 8-bit APIs which are either explicitly UTF-8 (such as GTK+) or can accept UTF-8 when the

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-10-07 Thread Adam Olsen
On Sun, Sep 20, 2009 at 10:17, Zooko O'Whielacronx zoo...@gmail.com wrote: On Sun, Sep 20, 2009 at 8:27 AM, Antoine Pitrou solip...@pitrou.net wrote: AFAIK, C extensions should fail loading when they have the wrong UCS2/4 setting. That would be an improvement!  Unfortunately we instead get

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-10-05 Thread Jan Matejek
Dne 20.9.2009 18:42, Antoine Pitrou napsal(a): Le Sun, 20 Sep 2009 10:33:23 -0600, Zooko O'Whielacronx a écrit : By the way, I was investigating this, and discovered an issue on the Mandriva tracker which suggests that they intend to switch to UCS4 in the next release in order to avoid

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-10-05 Thread Antoine Pitrou
Le lundi 05 octobre 2009 à 19:18 +0200, Jan Matejek a écrit : I don't see what is bad about improving compatibility in a place where the setting doesn't hurt one way or the other. I can't speak for Mandriva, but I'm sure they care more about not breaking user installs when they upgrade to

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-29 Thread Björn Lindqvist
2009/9/28 James Y Knight f...@fuhm.net: People building their own Python version will usually also build their own extensions, so I don't really believe that the above scenario is very common. I'd just like to note that I've run into this trap multiple times. I built a custom python, and

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-29 Thread Antoine Pitrou
Hello, I've also encountered this trap multiple times. Obviously, the problem is not rebuilding Python which is quick, but to figure out the correct configure option to use (--enable-unicode=ucs4). Others have also spent some time scratching their heads over the strange

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-29 Thread Zooko O'Whielacronx
Dear MAL and python-dev: I failed to explain the problem that users are having. I will try again, and this time I will omit my ideas about how to improve things and just focus on describing the problem. Some users are having trouble using Python packages containing binary extensions on Linux.

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-29 Thread Martin v. Löwis
I've also encountered this trap multiple times. Obviously, the problem is not rebuilding Python which is quick, but to figure out the correct configure option to use (--enable-unicode=ucs4). Others have also spent some time scratching their heads over the strange PyUnicodeUCS4_FromUnicode

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-28 Thread M.-A. Lemburg
Zooko O'Whielacronx wrote: Folks: I'm sorry, I think I didn't make my concern clear. My users, and lots of other users, are having a problem with incompatibility between Python binary extension modules. One way to improve the situation would be if the Python devs would use their bully

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-28 Thread M.-A. Lemburg
M.-A. Lemburg wrote: Also note that Python will complain loudly when you try to load a UCS2 extension in a UCS4 build and vice-versa. We've made sure that any extension using the Python Unicode C API has to be built for the same UCS version of Python. This is done by using different names for

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-28 Thread James Y Knight
On Sep 28, 2009, at 4:25 AM, M.-A. Lemburg wrote: Distributions should really not be put in charge of upstream coding design decisions. I don't think you can blame distros for this one From PEP 0261: It is also proposed that one day --enable-unicode will just default to the width

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-28 Thread M.-A. Lemburg
James Y Knight wrote: On Sep 28, 2009, at 4:25 AM, M.-A. Lemburg wrote: Distributions should really not be put in charge of upstream coding design decisions. I don't think you can blame distros for this one From PEP 0261: It is also proposed that one day --enable-unicode will just

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-28 Thread Martin v. Löwis
James Y Knight wrote: On Sep 28, 2009, at 4:25 AM, M.-A. Lemburg wrote: Distributions should really not be put in charge of upstream coding design decisions. I don't think you can blame distros for this one From PEP 0261: It is also proposed that one day --enable-unicode will

[Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-20 Thread Zooko O'Whielacronx
Dear Pythonistas: This issue causes serious problems. Users occasionally get binaries built for a compatible Linux and Python version but with a different UCS2-vs-UCS4 setting, and those users get mysterious memory corruption errors which are hard to diagnose. It is possible that these

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-20 Thread Benjamin Peterson
2009/9/20 Zooko O'Whielacronx zoo...@gmail.com: Dear Pythonistas: This issue causes serious problems.  Users occasionally get binaries built for a compatible Linux and Python version but with a different UCS2-vs-UCS4 setting, and those users get mysterious memory corruption errors which are

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-20 Thread Zooko O'Whielacronx
I'm sorry, I should have mentioned that I did read those archives before I posted my letter. That discussion was all about whether UCS2 or UCS4 is better. I consider that question to be mostly irrelevant to this issue, which is about compatibility for people who don't choose to configure that

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-20 Thread Antoine Pitrou
Zooko O'Whielacronx zookog at gmail.com writes: Users occasionally get binaries built for a compatible Linux and Python version but with a different UCS2-vs-UCS4 setting, and those users get mysterious memory corruption errors which are hard to diagnose. What binaries are you talking about?

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-20 Thread Zooko O'Whielacronx
On Sun, Sep 20, 2009 at 8:27 AM, Antoine Pitrou solip...@pitrou.net wrote: What binaries are you talking about? I mean extension modules with native code, which means .so shared library files on unix. AFAIK, C extensions should fail loading when they have the wrong UCS2/4 setting. That

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-20 Thread Zooko O'Whielacronx
On Sun, Sep 20, 2009 at 8:27 AM, Antoine Pitrou solip...@pitrou.net wrote: For information, all Mandriva versions I've used until now have had their Python's built with UCS2 (maxunicode == 65535). By the way, I was investigating this, and discovered an issue on the Mandriva tracker which

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-20 Thread Antoine Pitrou
Le Sun, 20 Sep 2009 10:17:45 -0600, Zooko O'Whielacronx a écrit : AFAIK, C extensions should fail loading when they have the wrong UCS2/4 setting. That would be an improvement! Unfortunately we instead get mysterious misbehavior of the module, e.g.:

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-20 Thread Antoine Pitrou
Le Sun, 20 Sep 2009 10:33:23 -0600, Zooko O'Whielacronx a écrit : By the way, I was investigating this, and discovered an issue on the Mandriva tracker which suggests that they intend to switch to UCS4 in the next release in order to avoid compatibility problems like these. Trying to use a

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-20 Thread M.-A. Lemburg
Zooko O'Whielacronx wrote: On Sun, Sep 20, 2009 at 8:27 AM, Antoine Pitrou solip...@pitrou.net wrote: What binaries are you talking about? I mean extension modules with native code, which means .so shared library files on unix. Those will not load unless they are for the right UCS-version