Re: [Python-Dev] Import and unicode: part two

2011-01-27 Thread Martin v. Löwis
When switching to a UTF-8 locale, they can also change the file names of their modules to be encoded in UTF-8. It would be fairly easy to write a script that identifies non-ASCII file names in a directory and offers to transcode their names from their current encoding to UTF-8. In fact,

Re: [Python-Dev] Import and unicode: part two

2011-01-26 Thread Stephen J. Turnbull
Toshio Kuratomi writes: Sure ... but with these systems, neither read-modules-as-locale or read-modules-as-utf-8 are a good solution to work, correct? Good solution, no, but I believe that read-modules-as-locale *should* work to a great extent. AFAIK Python 3 reads Python programs as str

Re: [Python-Dev] Import and unicode: part two

2011-01-26 Thread Victor Stinner
Le lundi 24 janvier 2011 à 19:26 -0800, Toshio Kuratomi a écrit : Why not locale: * Relying on locale is simply not portable. (...) * Mixing of modules from different locales won't work. (...) I don't understand what you are talking about. When you import a module, the module name becomes a

Re: [Python-Dev] Import and unicode: part two

2011-01-26 Thread Martin v. Löwis
Am 26.01.2011 10:40, schrieb Victor Stinner: Le lundi 24 janvier 2011 à 19:26 -0800, Toshio Kuratomi a écrit : Why not locale: * Relying on locale is simply not portable. (...) * Mixing of modules from different locales won't work. (...) I don't understand what you are talking about. I

Re: [Python-Dev] Import and unicode: part two

2011-01-26 Thread Oleg Broytman
On Wed, Jan 26, 2011 at 11:12:02AM +0100, Martin v. L??wis wrote: There are cases where there is no real transfer, in the sense in which you are using the word. For example, with NFS, you can access the very same file simultaneously on two systems, with no file name conversion (unless you are

Re: [Python-Dev] Import and unicode: part two

2011-01-26 Thread Victor Stinner
Le mercredi 26 janvier 2011 à 11:12 +0100, Martin v. Löwis a écrit : There are cases where there is no real transfer, in the sense in which you are using the word. For example, with NFS, you can access the very same file simultaneously on two systems, with no file name conversion (unless you

Re: [Python-Dev] Import and unicode: part two

2011-01-26 Thread James Y Knight
On Jan 26, 2011, at 4:40 AM, Victor Stinner wrote: During Python 3.2 development, we tried to be able to use a filesystem encoding different than the locale encoding (PYTHONFSENCODING environment variable): but it doesn't work simply because Python is not alone in the OS. Except Python, all

Re: [Python-Dev] Import and unicode: part two

2011-01-26 Thread Victor Stinner
Le mercredi 26 janvier 2011 à 08:24 -0500, James Y Knight a écrit : On Jan 26, 2011, at 4:40 AM, Victor Stinner wrote: During Python 3.2 development, we tried to be able to use a filesystem encoding different than the locale encoding (PYTHONFSENCODING environment variable): but it doesn't

Re: [Python-Dev] Import and unicode: part two

2011-01-26 Thread James Y Knight
On Jan 26, 2011, at 11:47 AM, Victor Stinner wrote: Not exactly. Gtk+ uses the glib library, and to encode/decode filenames, the glib library uses: - UTF-8 on Windows - G_FILENAME_ENCODING environment variable if set (comma-separated list of encodings) - UTF-8 if G_BROKEN_FILENAMES env

Re: [Python-Dev] Import and unicode: part two

2011-01-26 Thread Martin v. Löwis
If NFSv3 doesn't reencode filenames for each client and the clients don't reencode filenames, all clients have to use the same locale encoding than the server. Otherwise, I don't see how it can work. In practice, users accept that they get mojibake - their editors can still open the files, and

Re: [Python-Dev] Import and unicode: part two

2011-01-26 Thread Toshio Kuratomi
On Wed, Jan 26, 2011 at 11:12:02AM +0100, Martin v. Löwis wrote: Am 26.01.2011 10:40, schrieb Victor Stinner: Le lundi 24 janvier 2011 à 19:26 -0800, Toshio Kuratomi a écrit : Why not locale: * Relying on locale is simply not portable. (...) * Mixing of modules from different locales

Re: [Python-Dev] Import and unicode: part two

2011-01-26 Thread Neil Hodgson
Toshio Kuratomi: When they update their OS to a version that has utf-8 python module names, they will find that they have to make a choice. They can either change their locale settings to a utf-8 encoding and have the system installed modules work or they can leave their encoding on their

Re: [Python-Dev] Import and unicode: part two

2011-01-26 Thread Glenn Linderman
On 1/26/2011 4:47 PM, Toshio Kuratomi wrote: There's one further case that I am worried about that has no real transfer. Since people here seem to think that unicode module names are the future (for instance, the comments about redefining the C locale to include utf-8 and the comments about

Re: [Python-Dev] Import and unicode: part two

2011-01-25 Thread Stephen J. Turnbull
As Nick points out, nobody really seems to think this is an argument against your patch. I'm going to bow out of this thread after this post, as I'm clearly out of my technical depth. Victor Stinner writes: Le lundi 24 janvier 2011 11:35:22, Stephen J. Turnbull a écrit : ... VFAT-formatted

Re: [Python-Dev] Import and unicode: part two

2011-01-25 Thread Xavier Morel
On 2011-01-25, at 04:26 , Toshio Kuratomi wrote: * If you can pick a set of encodings that are valid (utf-8 for Linux and MacOS HFS+ uses UTF-16 in NFD (actually in an Apple-specific variant of NFD). Right here you've already broken Python modules on OSX. And as far as I know, Linux

Re: [Python-Dev] Import and unicode: part two

2011-01-25 Thread exarkun
On 09:22 am, catch-...@masklinn.net wrote: On 2011-01-25, at 04:26 , Toshio Kuratomi wrote: * If you can pick a set of encodings that are valid (utf-8 for Linux and MacOS HFS+ uses UTF-16 in NFD (actually in an Apple-specific variant of NFD). Right here you've already broken Python

Re: [Python-Dev] Import and unicode: part two

2011-01-25 Thread Toshio Kuratomi
On Tue, Jan 25, 2011 at 10:22:41AM +0100, Xavier Morel wrote: On 2011-01-25, at 04:26 , Toshio Kuratomi wrote: * If you can pick a set of encodings that are valid (utf-8 for Linux and MacOS HFS+ uses UTF-16 in NFD (actually in an Apple-specific variant of NFD). Right here you've

Re: [Python-Dev] Import and unicode: part two

2011-01-25 Thread Stephen J. Turnbull
Toshio Kuratomi writes: On Linux there's no defined encoding that will work; file names are just bytes to the Linux kernel so based on people's argument that the convention is and should be that filenames are utf-8 and anything else is a misconfigured system -- python should mandate that

Re: [Python-Dev] Import and unicode: part two

2011-01-25 Thread Toshio Kuratomi
On Wed, Jan 26, 2011 at 11:24:54AM +0900, Stephen J. Turnbull wrote: Toshio Kuratomi writes: On Linux there's no defined encoding that will work; file names are just bytes to the Linux kernel so based on people's argument that the convention is and should be that filenames are utf-8

Re: [Python-Dev] Import and unicode: part two

2011-01-24 Thread Stephen J. Turnbull
Guido van Rossum writes: Really? I would have thought that cell phones have long been the platforms most supportive of Unicode. I would think so too, except in Japan. However, my previous phones exposed file systems with names encoded in Shift JIS to USB and IR browsers, though. (My

Re: [Python-Dev] Import and unicode: part two

2011-01-24 Thread Martin v. Löwis
Really? I would have thought that cell phones have long been the platforms most supportive of Unicode. I would think so too, except in Japan. However, my previous phones exposed file systems with names encoded in Shift JIS to USB and IR browsers, though. (My current one uses

Re: [Python-Dev] Import and unicode: part two

2011-01-24 Thread Stephen J. Turnbull
Martin v. Löwis writes: It's one thing how the file systems are formatted, but another thing how they are presented to APIs. For example, the phones using Windows CE would have to convert the file names to Unicode in the OS kernel. So: for these phones - do you know how they present

Re: [Python-Dev] Import and unicode: part two

2011-01-24 Thread Nick Coghlan
On Mon, Jan 24, 2011 at 8:35 PM, Stephen J. Turnbull step...@xemacs.org wrote: First of all, these aren't just phones; these are all kinds of gadgets (the example I gave was a camera).  They're not as smart as an Android or iPhone-like device, and I don't know what OS they use. We're getting a

Re: [Python-Dev] Import and unicode: part two

2011-01-24 Thread Victor Stinner
Le lundi 24 janvier 2011 11:35:22, Stephen J. Turnbull a écrit : ... VFAT-formatted file systems and Shift JIS file names ... I missed something: VFAT stores filenames as unicode (whereas FAT only supports byte filenames). Well, VFAT stores filenames twice: as a 8+3 byte strings and as a 255

Re: [Python-Dev] Import and unicode: part two

2011-01-24 Thread Victor Stinner
Le lundi 24 janvier 2011 16:39:39, Victor Stinner a écrit : Le lundi 24 janvier 2011 11:35:22, Stephen J. Turnbull a écrit : ... VFAT-formatted file systems and Shift JIS file names ... I missed something: VFAT stores filenames as unicode (whereas FAT only supports byte filenames). Well,

Re: [Python-Dev] Import and unicode: part two

2011-01-24 Thread Oleg Broytman
On Mon, Jan 24, 2011 at 04:39:39PM +0100, Victor Stinner wrote: I missed something: VFAT stores filenames as unicode (whereas FAT only supports byte filenames). Well, VFAT stores filenames twice: as a 8+3 byte strings and as a 255 unicode (UTF-16-LE) string (UTF-16-LE). On which OS do you

Re: [Python-Dev] Import and unicode: part two

2011-01-24 Thread Martin v. Löwis
Am 24.01.2011 16:39, schrieb Victor Stinner: Le lundi 24 janvier 2011 11:35:22, Stephen J. Turnbull a écrit : ... VFAT-formatted file systems and Shift JIS file names ... I missed something: VFAT stores filenames as unicode (whereas FAT only supports byte filenames). Well, VFAT stores

Re: [Python-Dev] Import and unicode: part two

2011-01-24 Thread Toshio Kuratomi
On Thu, Jan 20, 2011 at 03:27:08PM -0500, Glyph Lefkowitz wrote: On Jan 20, 2011, at 11:46 AM, Guido van Rossum wrote: Same here. *Most* code will never be shared, or will only be shared between users in the same community. When it goes wrong it's also a learning opportunity. :-)

Re: [Python-Dev] Import and unicode: part two

2011-01-23 Thread Stephen J. Turnbull
Martin v. Löwis writes: Actually, as long people only involve Windows, or only involve Mac, it will all work just fine. It's only when they use non-Mac Unix (such as Linux), or try to move files across systems using sub-prime technology (such as your typical Windows zip utility) they will

Re: [Python-Dev] Import and unicode: part two

2011-01-23 Thread Guido van Rossum
On Sun, Jan 23, 2011 at 6:33 PM, Stephen J. Turnbull step...@xemacs.org wrote: Martin v. Löwis writes:   Actually, as long people only involve Windows, or only involve Mac,   it will all work just fine. It's only when they use non-Mac Unix   (such as Linux), or try to move files across systems

Re: [Python-Dev] Import and unicode: part two

2011-01-21 Thread Atsuo Ishimoto
On Fri, Jan 21, 2011 at 1:46 AM, Guido van Rossum gu...@python.org wrote: On Thu, Jan 20, 2011 at 5:16 AM, Nick Coghlan ncogh...@gmail.com wrote: On Thu, Jan 20, 2011 at 10:08 PM, Simon Cross hodgestar+python...@gmail.com wrote: I'm changing my vote on this to a +1 for two reasons: *

Re: [Python-Dev] Import and unicode: part two

2011-01-21 Thread Atsuo Ishimoto
On Fri, Jan 21, 2011 at 2:59 PM, Nick Coghlan ncogh...@gmail.com wrote: These all sound like good reasons to continue to *advise* against using non-ASCII module names. But aside from that, they sound exactly like a lot of the arguments we heard when Py3k started enforcing the bytes/text

Re: [Python-Dev] Import and unicode: part two

2011-01-21 Thread Stephen J. Turnbull
Nick Coghlan writes: On Fri, Jan 21, 2011 at 3:44 PM, Atsuo Ishimoto ishim...@gembook.org wrote: I don't want Python to encourage people to use non-ascii module names. I don't think anybody is *encouraging* it. The argument is for *permitting* it, partly for consistency with other

Re: [Python-Dev] Import and unicode: part two

2011-01-21 Thread Martin v. Löwis
I don't want Python to encourage people to use non-ascii module names. I don't think the feature is open for debate anymore. PEP 3131 has been accepted (after *long* debates), and I'll pronounce that supporting non-ASCII module names is a direct consequence of having it accepted. Of course,

Re: [Python-Dev] Import and unicode: part two

2011-01-21 Thread Stephen J. Turnbull
Atsuo Ishimoto writes: Java, a leading language of IT industry, have already support non-ASCII class files for years. But I've never seen such files in production in Japan, and didn't improve situation until now. So why wouldn't Python work the same way? The rest of the world can use

Re: [Python-Dev] Import and unicode: part two

2011-01-21 Thread Antoine Pitrou
On Thu, 20 Jan 2011 22:25:17 -0500 James Y Knight f...@fuhm.net wrote: On Jan 20, 2011, at 3:55 PM, Antoine Pitrou wrote: On Thu, 20 Jan 2011 15:27:08 -0500 Glyph Lefkowitz gl...@twistedmatrix.com wrote: To support the latter, could we just make sure that zipimport has a

Re: [Python-Dev] Import and unicode: part two

2011-01-21 Thread Atsuo Ishimoto
On Fri, Jan 21, 2011 at 5:45 PM, Stephen J. Turnbull step...@xemacs.org wrote: Nick Coghlan writes:   On Fri, Jan 21, 2011 at 3:44 PM, Atsuo Ishimoto ishim...@gembook.org wrote:   I don't want Python to encourage people to use non-ascii module names. I don't think anybody is *encouraging*

Re: [Python-Dev] Import and unicode: part two

2011-01-21 Thread Martin v. Löwis
I don't think anybody is *encouraging* it. The argument is for *permitting* it, partly for consistency with other identifiers, and partly because of Python's usual consenting adults standard for permitting dangerous practices. I'm sorry, I was not clear. I was afraid that saying learning

Re: [Python-Dev] Import and unicode: part two

2011-01-20 Thread James Y Knight
On Jan 19, 2011, at 11:39 PM, Toshio Kuratomi wrote: Tangent: This is not true about Linux. UTF-8 is a matter of the interpretation of the filesystem bytes that the user specifies by setting their system locale. Setting system locale to ASCII for use in system-wide scripts, is quite common

Re: [Python-Dev] Import and unicode: part two

2011-01-20 Thread Victor Stinner
Le mercredi 19 janvier 2011 à 20:39 -0800, Toshio Kuratomi a écrit : Teaching students to write non-portable code (relying on filesystem encoding where your solution is, don't upload to pypi anything that has non-ascii filenames) seems like the exact opposite of how you'd want to shape a young

Re: [Python-Dev] Import and unicode: part two

2011-01-20 Thread Simon Cross
On Wed, Jan 19, 2011 at 5:01 PM, Simon Cross hodgestar+python...@gmail.com wrote: On Wed, Jan 19, 2011 at 2:34 PM, Victor Stinner victor.stin...@haypocalc.com wrote:  (a) Python 3 doesn't support non-ASCII module names -0: I'm vaguely against this being supported because I'd rather not have

Re: [Python-Dev] Import and unicode: part two

2011-01-20 Thread Nick Coghlan
On Thu, Jan 20, 2011 at 10:08 PM, Simon Cross hodgestar+python...@gmail.com wrote: I'm changing my vote on this to a +1 for two reasons: * Initially I thought this wasn't supported by Python at all but I see that currently it is supported but that support is broken (or at least limited to

Re: [Python-Dev] Import and unicode: part two

2011-01-20 Thread Guido van Rossum
On Thu, Jan 20, 2011 at 5:16 AM, Nick Coghlan ncogh...@gmail.com wrote: On Thu, Jan 20, 2011 at 10:08 PM, Simon Cross hodgestar+python...@gmail.com wrote: I'm changing my vote on this to a +1 for two reasons: * Initially I thought this wasn't supported by Python at all but I see that

Re: [Python-Dev] Import and unicode: part two

2011-01-20 Thread Andy Teijelo
(Hi, I'm writing from an address different to the one I'm subscribed with to the list because I don't have reverse dns in my mail server and mail.python.org rejects my messages. I hope that's not much trouble) Maybe Python should always use an ASCII encodable filename for modules: a

Re: [Python-Dev] Import and unicode: part two

2011-01-20 Thread Toshio Kuratomi
On Thu, Jan 20, 2011 at 12:51:29PM +0100, Victor Stinner wrote: Le mercredi 19 janvier 2011 à 20:39 -0800, Toshio Kuratomi a écrit : Teaching students to write non-portable code (relying on filesystem encoding where your solution is, don't upload to pypi anything that has non-ascii

Re: [Python-Dev] Import and unicode: part two

2011-01-20 Thread Alexander Belopolsky
On Thu, Jan 20, 2011 at 11:45 AM, Andy Teijelo ateij...@gmail.com wrote: .. but if the code said: import café then Python would look, in any platform, for a file named: cafeacute;.py  or  caf#233;.py  or something nicer. Something along the lines of xmlcharrefreplace. Just an idea.

Re: [Python-Dev] Import and unicode: part two

2011-01-20 Thread Alexander Belopolsky
On Thu, Jan 20, 2011 at 12:44 PM, Toshio Kuratomi a.bad...@gmail.com wrote: .. My examples that you're replying to involve two properly configured OS's.  The Linux workstations are configured with a UTF-8 locale.  The Windows OS's use wide character unicode.  The problem occurs in that the

Re: [Python-Dev] Import and unicode: part two

2011-01-20 Thread Toshio Kuratomi
On Thu, Jan 20, 2011 at 01:43:03PM -0500, Alexander Belopolsky wrote: On Thu, Jan 20, 2011 at 12:44 PM, Toshio Kuratomi a.bad...@gmail.com wrote: .. My examples that you're replying to involve two properly configured OS's.  The Linux workstations are configured with a UTF-8 locale.  The

Re: [Python-Dev] Import and unicode: part two

2011-01-20 Thread Glyph Lefkowitz
On Jan 20, 2011, at 11:46 AM, Guido van Rossum wrote: On Thu, Jan 20, 2011 at 5:16 AM, Nick Coghlan ncogh...@gmail.com wrote: On Thu, Jan 20, 2011 at 10:08 PM, Simon Cross hodgestar+python...@gmail.com wrote: I'm changing my vote on this to a +1 for two reasons: * Initially I thought this

Re: [Python-Dev] Import and unicode: part two

2011-01-20 Thread Neil Hodgson
Toshio Kuratomi: My examples that you're replying to involve two properly configured OS's.  The Linux workstations are configured with a UTF-8 locale.  The Windows OS's use wide character unicode.  The problem occurs in that the code that one of the parties develops (either the students or

Re: [Python-Dev] Import and unicode: part two

2011-01-20 Thread Antoine Pitrou
On Thu, 20 Jan 2011 15:27:08 -0500 Glyph Lefkowitz gl...@twistedmatrix.com wrote: To support the latter, could we just make sure that zipimport has a consistent, non-locale-or-operating-system-dependent interpretation of encoding? It already has, but it's dependent on a flag in the zip file

Re: [Python-Dev] Import and unicode: part two

2011-01-20 Thread Glenn Linderman
On 1/20/2011 12:27 PM, Glyph Lefkowitz wrote: To support the latter, could we just make sure that zipimport has a consistent, non-locale-or-operating-system-dependent interpretation of encoding? That way a distributed egg would be importable from a zipfile regardless of how screwed up the

Re: [Python-Dev] Import and unicode: part two

2011-01-20 Thread Georg Brandl
Am 20.01.2011 12:51, schrieb Victor Stinner: You only give theorical arguments Read Anathem lately? ;) Georg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

Re: [Python-Dev] Import and unicode: part two

2011-01-20 Thread Nick Coghlan
On Fri, Jan 21, 2011 at 5:27 AM, Toshio Kuratomi a.bad...@gmail.com wrote: I think that both ideas are inferior to mandating that every python module filename is ascii.  From what I'm getting from Victor's posts is that he, at least, considers the portability problems to be ignorable because

Re: [Python-Dev] Import and unicode: part two

2011-01-20 Thread James Y Knight
On Jan 20, 2011, at 3:55 PM, Antoine Pitrou wrote: On Thu, 20 Jan 2011 15:27:08 -0500 Glyph Lefkowitz gl...@twistedmatrix.com wrote: To support the latter, could we just make sure that zipimport has a consistent, non-locale-or-operating-system-dependent interpretation of encoding? It

Re: [Python-Dev] Import and unicode: part two

2011-01-20 Thread Nick Coghlan
On Fri, Jan 21, 2011 at 3:44 PM, Atsuo Ishimoto ishim...@gembook.org wrote: I don't want Python to encourage people to use non-ascii module names. Today, seeing UnicodeEncodingError is one of popular reasons for newbies to abandon learning Python in Japan. Non-ascii module name is an another

Re: [Python-Dev] Import and unicode: part two

2011-01-20 Thread Terry Reedy
On 1/20/2011 12:44 PM, Toshio Kuratomi wrote: The problem occurs in that the code that one of the parties develops (either the students or the professors) is developed on one of those OS's and then used on the other OS. The problem that I reported and hope will be fixed is that private code

Re: [Python-Dev] Import and unicode: part two

2011-01-20 Thread Nick Coghlan
On Fri, Jan 21, 2011 at 4:44 PM, Atsuo Ishimoto ishim...@gembook.org wrote: On Fri, Jan 21, 2011 at 2:59 PM, Nick Coghlan ncogh...@gmail.com wrote: These all sound like good reasons to continue to *advise* against using non-ASCII module names. But aside from that, they sound exactly like a

[Python-Dev] Import and unicode: part two

2011-01-19 Thread Victor Stinner
Hi, I patched Python 3.2 to support modules with non-ASCII paths (*). It works well on all operating systems. But the task is not completly done: (a) Python 3 doesn't support non-ASCII module names (b) Python 3 doesn't support unencodable characters in the module path I would like to know if

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Simon Cross
On Wed, Jan 19, 2011 at 2:34 PM, Victor Stinner victor.stin...@haypocalc.com wrote:  (a) Python 3 doesn't support non-ASCII module names -0: I'm vaguely against this being supported because I'd rather not have to deal with what happens when the guess regarding the filesystem encoding is wrong.

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Brett Cannon
On Wed, Jan 19, 2011 at 07:01, Simon Cross hodgestar+python...@gmail.com wrote: On Wed, Jan 19, 2011 at 2:34 PM, Victor Stinner victor.stin...@haypocalc.com wrote:  (a) Python 3 doesn't support non-ASCII module names -0: I'm vaguely against this being supported because I'd rather not have to

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Alexander Belopolsky
On Wed, Jan 19, 2011 at 1:23 PM, Brett Cannon br...@python.org wrote: ..  (a) Python 3 doesn't support non-ASCII module names .. -0 from me (unless the Unicode variable naming PEP says otherwise). I am not sure what exactly is not supported. On my OSX system: $ ./python.exe Python 3.2b2+ ..

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Brett Cannon
On Wed, Jan 19, 2011 at 10:38, Alexander Belopolsky alexander.belopol...@gmail.com wrote: On Wed, Jan 19, 2011 at 1:23 PM, Brett Cannon br...@python.org wrote: ..  (a) Python 3 doesn't support non-ASCII module names .. -0 from me (unless the Unicode variable naming PEP says otherwise). I

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Alexander Belopolsky
On Wed, Jan 19, 2011 at 1:42 PM, Brett Cannon br...@python.org wrote: .. I am not sure what exactly is not supported.  On my OSX system: Victor said this is a Windows-specific issue. I missed that part. In this case, I change my vote to +0 to reflect my lack of knowledge or exposure to

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Victor Stinner
Le mercredi 19 janvier 2011 à 10:42 -0800, Brett Cannon a écrit : I am not sure what exactly is not supported. On my OSX system: Victor said this is a Windows-specific issue. Autoquote: (a) (...) doesn't work with a locale encoding different than UTF-8 Hum, it's not exactly the locale

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Glenn Linderman
On 1/19/2011 11:31 AM, Victor Stinner wrote: If we decide to reject non-ASCII module names, it should be done on any operating systems, not only on Windows. Since Python allows non-ASCII variable names, I think it should allow non-ASCII module names also, on any platform that supports the

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Terry Reedy
On 1/19/2011 7:34 AM, Victor Stinner wrote: Hi, I patched Python 3.2 to support modules with non-ASCII paths (*). It works well on all operating systems. But the task is not completly done: (a) Python 3 doesn't support non-ASCII module names (b) Python 3 doesn't support unencodable characters

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Victor Stinner
Le mercredi 19 janvier 2011 à 13:38 -0500, Alexander Belopolsky a écrit : PEP 3131 does not distinguish between different types of identifiers, so I think it assumes that non-ascii module names should be supported. My opinion is that we should suport non-ASCII module names and unencodable paths

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Simon Cross
On Wed, Jan 19, 2011 at 10:32 PM, Terry Reedy tjre...@udel.edu wrote: I am a little shocked at the so-far tepid response to (a), so let me defend and explain my claim that it is a bug. In the simplest case (from 6.11. The import statement and  2.3. Identifiers and keywords) import_stmt ::=

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Georg Brandl
Am 19.01.2011 21:32, schrieb Terry Reedy: On 1/19/2011 7:34 AM, Victor Stinner wrote: Hi, I patched Python 3.2 to support modules with non-ASCII paths (*). It works well on all operating systems. But the task is not completly done: (a) Python 3 doesn't support non-ASCII module names (b)

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Terry Reedy
On 1/19/2011 4:05 PM, Simon Cross wrote: I have no problem with non-ASCII module identifiers being valid syntax. It's a question of whether attempting to translate a non-ASCII If the names are the same, ie, produced with the same sequence of keystrokes in the save-as box and importing box,

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Victor Stinner
Le mercredi 19 janvier 2011 à 12:19 -0800, Glenn Linderman a écrit : Since Python allows non-ASCII variable names, I think it should allow non-ASCII module names also, on any platform that supports the appropriate characters in the filesystem. Since some platforms already accept them,

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Alexander Belopolsky
On Wed, Jan 19, 2011 at 4:40 PM, Terry Reedy tjre...@udel.edu wrote: .. For similar reasons we tend to avoid capital letters in module names. That is a stdlib style guide followed by many, but intentionally not enforced. Indeed. Last time I looked, we still had cProfile in stdlib.

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Brett Cannon
On Wed, Jan 19, 2011 at 14:23, Alexander Belopolsky alexander.belopol...@gmail.com wrote: On Wed, Jan 19, 2011 at 4:40 PM, Terry Reedy tjre...@udel.edu wrote: .. For similar reasons we tend to avoid capital letters in module names. That is a stdlib style guide followed by many, but

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Alexander Belopolsky
On Wed, Jan 19, 2011 at 5:47 PM, Brett Cannon br...@python.org wrote: .. Indeed.  Last time I looked, we still had cProfile in stdlib. Yes, but that is because no one got around to hiding cProfile behind profile before we released Python 3.0. I would still like to see it (slowly) go away from

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Toshio Kuratomi
On Wed, Jan 19, 2011 at 04:40:24PM -0500, Terry Reedy wrote: On 1/19/2011 4:05 PM, Simon Cross wrote: I have no problem with non-ASCII module identifiers being valid syntax. It's a question of whether attempting to translate a non-ASCII If the names are the same, ie, produced with the same

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Terry Reedy
On 1/19/2011 6:44 PM, Toshio Kuratomi wrote: I believe we now have the situation that a package that works on *nix could fail on Windows, whereas I believe that patch would *improve* portability. I'm not so sure about this Forget that claim if it is not true. The patch will certainly

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Terry Reedy
On 1/19/2011 6:05 PM, Alexander Belopolsky wrote: On Wed, Jan 19, 2011 at 5:47 PM, Brett Cannonbr...@python.org wrote: .. Indeed. Last time I looked, we still had cProfile in stdlib. Yes, but that is because no one got around to hiding cProfile behind profile before we released Python 3.0.

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Victor Stinner
Le mercredi 19 janvier 2011 à 15:44 -0800, Toshio Kuratomi a écrit : Additionally, many unix filesystem don't specify a filesystem encoding for filenames; they deal in legal and illegal bytes which could lead to troubles. This problem of which encoding to use is a problem that can be seen on

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread James Y Knight
On Jan 19, 2011, at 6:44 PM, Toshio Kuratomi wrote: This problem of which encoding to use is a problem that can be seen on UNIX systems even now. Try this: echo 'print(hi)' café.py convmv -f utf-8 -t latin1 café.py python3 -c 'import café' ASCII seems very sensible to me when faced

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Toshio Kuratomi
On Wed, Jan 19, 2011 at 07:11:52PM -0500, James Y Knight wrote: On Jan 19, 2011, at 6:44 PM, Toshio Kuratomi wrote: This problem of which encoding to use is a problem that can be seen on UNIX systems even now. Try this: echo 'print(hi)' café.py convmv -f utf-8 -t latin1 café.py

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Toshio Kuratomi
On Thu, Jan 20, 2011 at 01:26:01AM +0100, Victor Stinner wrote: Le mercredi 19 janvier 2011 à 15:44 -0800, Toshio Kuratomi a écrit : Additionally, many unix filesystem don't specify a filesystem encoding for filenames; they deal in legal and illegal bytes which could lead to troubles.

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Victor Stinner
Le mercredi 19 janvier 2011 à 18:07 -0800, Toshio Kuratomi a écrit : Saying that multiple encodings on a single system is a misconfiguration every time it comes up does not make it true. Yes, each filesystem can have its own encoding. For example, this is supported by Linux. Python doesn't

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Toshio Kuratomi
On Thu, Jan 20, 2011 at 03:51:05AM +0100, Victor Stinner wrote: For a lesson at school, it is nice to write examples in the mother language, instead of using raw english with ASCII identifiers and filenames. Then use this:: import cafe as café When you do things this way you do not have to

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Alexander Belopolsky
On Wed, Jan 19, 2011 at 9:07 PM, Toshio Kuratomi a.bad...@gmail.com wrote: .. Do you have a solution to the problem?  I haven't looked at your patch so perhaps you have an ingenous method of translating from the unicode representation of the module in the import statement to the bytes in

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Glenn Linderman
On 1/19/2011 8:39 PM, Toshio Kuratomi wrote: use this:: import cafe as café When you do things this way you do not have to translate between unknown encodings into unicode. Everything is within python source where you have a defined encoding. This is a great way of converting

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Glyph Lefkowitz
On Jan 20, 2011, at 12:02 AM, Glenn Linderman wrote: But for local code, having to think up an ASCII name for a module rather than use the obvious native-language name, is just brain-burden when creating the code. Is it really? You already had to type 'import', presumably if you can think

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Alexander Belopolsky
On Wed, Jan 19, 2011 at 11:39 PM, Toshio Kuratomi a.bad...@gmail.com wrote: .. Teaching students to write non-portable code (relying on filesystem encoding where your solution is, don't upload to pypi anything that has non-ascii filenames) seems like the exact opposite of how you'd want to

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Glenn Linderman
On 1/19/2011 9:11 PM, Glyph Lefkowitz wrote: On Jan 20, 2011, at 12:02 AM, Glenn Linderman wrote: But for local code, having to think up an ASCII name for a module rather than use the obvious native-language name, is just brain-burden when creating the code. Is it really? You already had

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Glyph Lefkowitz
On Jan 20, 2011, at 12:19 AM, Glenn Linderman wrote: Now if the stuff after m_ was the hex UTF-8 of café, that could get interesting :) (As it happens, it's the hex digest of the MD5 of the UTF-8 of café... ;-))___ Python-Dev mailing list

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Alexander Belopolsky
On Thu, Jan 20, 2011 at 12:11 AM, Glyph Lefkowitz gl...@twistedmatrix.com wrote: .. But for local code, having to think up an ASCII name for a module rather than use the obvious native-language name, is just brain-burden when creating the code. Is it really?  You already had to type 'import',

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Toshio Kuratomi
On Wed, Jan 19, 2011 at 09:02:17PM -0800, Glenn Linderman wrote: On 1/19/2011 8:39 PM, Toshio Kuratomi wrote: use this:: import cafe as café When you do things this way you do not have to translate between unknown encodings into unicode. Everything is within python

Re: [Python-Dev] Import and unicode: part two

2011-01-19 Thread Glenn Linderman
On 1/19/2011 11:20 PM, Toshio Kuratomi wrote: On Wed, Jan 19, 2011 at 09:02:17PM -0800, Glenn Linderman wrote: On 1/19/2011 8:39 PM, Toshio Kuratomi wrote: use this:: import cafe as café When you do things this way you do not have to translate between unknown encodings