Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Victor Stinner
2016-08-16 8:06 GMT+02:00 eryk sun : > My proposal was to use the wide-character APIs, but transcoding CP_ACP > without best-fit characters and raising a warning whenever the default > character is used (e.g. substituting Katakana middle dot when creating > a file using a bytes path that has an inv

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Paul Moore
On 15 August 2016 at 19:26, Steve Dower wrote: > Passing path_as_bytes in that location has been deprecated since 3.3, so we > are well within our rights (and probably overdue) to make it a TypeError in > 3.6. While it's obviously an invalid assumption, for the purposes of > changing the language

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread eryk sun
On Tue, Aug 16, 2016 at 10:53 AM, Paul Moore wrote: > > Having said all this, I can't find the documentation stating that > bytes paths are deprecated - the open() documentation for 3.5 says > "file is either a string or bytes object giving the pathname (absolute > or relative to the current worki

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Stephen J. Turnbull
Nick Coghlan writes: > At an ecosystem level, that means we're faced with a choice between > implicitly encouraging folks to make their code *nix only, and > finding a way to provide a more *nix like experience when running > on Windows (where UTF-8 encoded binary data just works, and either

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Steve Dower
Hmm, doesn't seem to be explicitly listed as a deprecation, though discussion form around that time makes it clear that everyone thought it was. I also found this proposal to use strict mbcs to decode bytes for use against the file system, which is basically the same as what I'm proposing now ap

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Paul Moore
On 16 August 2016 at 14:09, eryk sun wrote: > On Tue, Aug 16, 2016 at 10:53 AM, Paul Moore wrote: >> >> Having said all this, I can't find the documentation stating that >> bytes paths are deprecated - the open() documentation for 3.5 says >> "file is either a string or bytes object giving the pa

[Python-ideas] Digests (was Re: From mailing list to GitHub issues)

2016-08-16 Thread Stephen J. Turnbull
Barry Warsaw writes: > On Aug 14, 2016, at 02:01 PM, Chris Angelico wrote: > > >The biggest problem I'm seeing is with digests. Can that feature be > >flagged off as "DO NOT USE THIS UNLESS YOU KNOW WHAT YOU ARE ASKING > >FOR"? So many people seem to select digest mode, then get extremely >

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Random832
On Tue, Aug 16, 2016, at 09:59, Paul Moore wrote: > It probably should be. Although if we're changing the deprecation to a > behaviour change, then maybe there's no point. But some of the > arguments here about breaking code are hinging on the idea that people > currently using the bytes API are us

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Chris Barker - NOAA Federal
> There also seems to be an undercurrent in the discussions we're having > now that using bytes paths and not unicode paths is somehow The Right > Thing for unix-like OSes, Almost -- from my perusing of discussions from the last few years, there do seem to be some library developers and *nix affec

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Guido van Rossum
I am going to mute this thread but I am worried about the outcome. Once there is agreement please check with me first. --Guido (mobile) ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Co

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Steve Dower
I just want to clearly address two points, since I feel like multiple posts have been unclear on them. 1. The bytes API was deprecated in 3.3 and it is listed in https://docs.python.org/3/whatsnew/3.3.html. Lack of mention in the docs is an unfortunate oversight, but it was certainly announced

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Chris Barker
Just to make sure this is clear, the Pragmatic logic is thus: * There are more *nix-centric developers in the Python ecosystem than Windows-centric (or even Windows-agnostic) developers. * The bytes path approach works fine on *nix systems. * Whatever might be Right and Just -- the reality is th

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Brendan Barnwell
On 2016-08-16 08:56, Steve Dower wrote: I just want to clearly address two points, since I feel like multiple posts have been unclear on them. 1. The bytes API was deprecated in 3.3 and it is listed in https://docs.python.org/3/whatsnew/3.3.html. Lack of mention in the docs is an unfortunate ove

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Chris Barker
Thanks for the clarity, Steve, a couple questions/thoughts: The choices are: > > * don't represent them at all (remove bytes API) > Would the bytes API be removed on *nix also? > * convert and drop characters not in the (legacy) active code page > * convert and fail on characters not in the (le

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Random832
On Tue, Aug 16, 2016, at 12:12, Chris Barker wrote: > * convert and fail on invalid surrogate pairs > > where would an invalid surrogate pair come from? never from a file system > API call, yes? In principle it could, if the filesystem contains a file with an invalid surrogate pair. Nothing else,

Re: [Python-ideas] Digests (was Re: From mailing list to GitHub issues)

2016-08-16 Thread Chris Angelico
On Wed, Aug 17, 2016 at 12:11 AM, Stephen J. Turnbull wrote: > Barry Warsaw writes: > > On Aug 14, 2016, at 02:01 PM, Chris Angelico wrote: > > > > >The biggest problem I'm seeing is with digests. Can that feature be > > >flagged off as "DO NOT USE THIS UNLESS YOU KNOW WHAT YOU ARE ASKING > >

Re: [Python-ideas] PEP 525: Asynchronous Generators

2016-08-16 Thread Moritz Sichert via Python-ideas
>>> 2. It's extremely unlikely that somebody will design a system that >>> switches coroutine runners *while async/awaiting a coroutine*. >> Yes, I guess so. >> >> >>> But even in this unlikely use case, you can >>> easily stack finalizers following this pattern: >>> >>>old_finalizer = sys.get_

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Sven R. Kunze
On 16.08.2016 18:06, Chris Barker wrote: It's clear (to me at least) that (A) it the "Right Thing", but real world experience has shown that it's unlikely to happen any time soon. Practicality beats Purity and all that -- this is a judgment call. Maybe, but even when it takes a lot of time to

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Steve Dower
On 16Aug2016 1006, Sven R. Kunze wrote: Question is: are we in a hurry? Has somebody too little time to wait for the "Right Thing" to happen? Not really in a hurry. It's just that I decided to attack a few of the encoding issues on Windows and this was one of them. Ideally I'd want to get th

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Sven R. Kunze
On 16.08.2016 19:44, Steve Dower wrote: On 16Aug2016 1006, Sven R. Kunze wrote: Question is: are we in a hurry? Has somebody too little time to wait for the "Right Thing" to happen? Not really in a hurry. It's just that I decided to attack a few of the encoding issues on Windows and this was

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Paul Moore
On 16 August 2016 at 16:56, Steve Dower wrote: > I just want to clearly address two points, since I feel like multiple posts > have been unclear on them. > > 1. The bytes API was deprecated in 3.3 and it is listed in > https://docs.python.org/3/whatsnew/3.3.html. Lack of mention in the docs is > a

[Python-ideas] Allow manual creation of DirEntry objects

2016-08-16 Thread Brendan Moloney
Hi, I have been using the 'scandir' package (https://github.com/benhoyt/scandir) for a while now to speed up some directory tree processing code. Since Python 3.5 now includes 'os.scandir' in the stdlib (https://www.python.org/dev/peps/pep-0471/) I decided to try to make my code work with the b

Re: [Python-ideas] Allow manual creation of DirEntry objects

2016-08-16 Thread Guido van Rossum
It sounds fine to just submit a patch to add and document the DirEntry constructor. I don't think anyone intended to disallow your use case, it's more likely that nobody thought of it. On Tue, Aug 16, 2016 at 12:35 PM, Brendan Moloney wrote: > Hi, > > I have been using the 'scandir' package (htt

Re: [Python-ideas] Digests (was Re: From mailing list to GitHub issues)

2016-08-16 Thread cs
On 17Aug2016 02:44, Chris Angelico wrote: On Wed, Aug 17, 2016 at 12:11 AM, Stephen J. Turnbull wrote: Barry Warsaw writes: > On Aug 14, 2016, at 02:01 PM, Chris Angelico wrote: > >The biggest problem I'm seeing is with digests. Can that feature be > >flagged off as "DO NOT USE THIS UNLESS

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Victor Stinner
2016-08-16 17:56 GMT+02:00 Steve Dower : > 2. Windows file system encoding is *always* UTF-16. There's no "assuming > mbcs" or "assuming ACP" or "assuming UTF-8" or "asking the OS what encoding > it is". We know exactly what the encoding is on every supported version of > Windows. UTF-16. I think

Re: [Python-ideas] Allow manual creation of DirEntry objects

2016-08-16 Thread Victor Stinner
2016-08-16 23:13 GMT+02:00 Guido van Rossum : > It sounds fine to just submit a patch to add and document the DirEntry > constructor. I don't think anyone intended to disallow your use case, it's > more likely that nobody thought of it. Currently, the DirEntry constructor expects data which comes

Re: [Python-ideas] Allow manual creation of DirEntry objects

2016-08-16 Thread Victor Stinner
By the way, for all these reasons, I'm not really excited by Python 3.6 change exposing os.DirEntry ( https://bugs.python.org/issue27038 ). Victor 2016-08-17 1:11 GMT+02:00 Victor Stinner : > 2016-08-16 23:13 GMT+02:00 Guido van Rossum : >> It sounds fine to just submit a patch to add and documen

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Steve Dower
On 16Aug2016 1603, Victor Stinner wrote: 2016-08-16 17:56 GMT+02:00 Steve Dower : 2. Windows file system encoding is *always* UTF-16. There's no "assuming mbcs" or "assuming ACP" or "assuming UTF-8" or "asking the OS what encoding it is". We know exactly what the encoding is on every supported v

Re: [Python-ideas] Allow manual creation of DirEntry objects

2016-08-16 Thread Guido van Rossum
On Tue, Aug 16, 2016 at 4:14 PM, Victor Stinner wrote: > By the way, for all these reasons, I'm not really excited by Python > 3.6 change exposing os.DirEntry ( https://bugs.python.org/issue27038 > ). > But that's separate from the constructor. We could expose the class with a constructor that a

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Victor Stinner
2016-08-17 1:27 GMT+02:00 Steve Dower : >> filenameb = os.listdir(b'.')[0] >> # Python 3.5 encodes Unicode (UTF-16) to the ANSI code page >> # what if Python 3.7 encodes Unicode (UTF-16) to UTF-8? >> print("filename bytes: %a" % filenameb) >> >> proc = subprocess.Popen(['py', '-

Re: [Python-ideas] Allow manual creation of DirEntry objects

2016-08-16 Thread Victor Stinner
2016-08-17 1:50 GMT+02:00 Guido van Rossum : > We could expose the class with a > constructor that always fails (the C code could construct instances through > a backdoor). Oh, in fact you cannot create an instance of os.DirEntry, it has no (Python) constructor: $ ./python Python 3.6.0a4+ (defaul

Re: [Python-ideas] Allow manual creation of DirEntry objects

2016-08-16 Thread Brett Cannon
On Tue, 16 Aug 2016 at 16:15 Victor Stinner wrote: > By the way, for all these reasons, I'm not really excited by Python > 3.6 change exposing os.DirEntry ( https://bugs.python.org/issue27038 > ). > It was exposed at Guido's request for type hinting in typeshed. -Brett > > Victor > > 2016-08-

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Steve Dower
On 16Aug2016 1650, Victor Stinner wrote: 2016-08-17 1:27 GMT+02:00 Steve Dower : filenameb = os.listdir(b'.')[0] # Python 3.5 encodes Unicode (UTF-16) to the ANSI code page # what if Python 3.7 encodes Unicode (UTF-16) to UTF-8? print("filename bytes: %a" % filenameb) proc =

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Brendan Barnwell
On 2016-08-16 17:14, Steve Dower wrote: The existence of bugs in other applications is not a good reason to help people create new bugs. I haven't been following all the details in this thread, but isn't the whole purpose of this proposed change to accommodate code (apparently on Linux?) tha

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Steve Dower
On 16Aug2016 1915, Brendan Barnwell wrote: On 2016-08-16 17:14, Steve Dower wrote: The existence of bugs in other applications is not a good reason to help people create new bugs. I haven't been following all the details in this thread, but isn't the whole purpose of this proposed change

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Steve Dower
I've just created http://bugs.python.org/issue27781 with a patch removing use of the *A API from posixmodule.c and changing the default FS encoding to utf-8. Since we're still discussing whether the encoding should be utf-8 or something else, let's keep that here. But if you want to see how th

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread eryk sun
On Tue, Aug 16, 2016 at 3:56 PM, Steve Dower wrote: > > 2. Windows file system encoding is *always* UTF-16. There's no "assuming > mbcs" or "assuming ACP" or "assuming UTF-8" or "asking the OS what encoding > it is". We know exactly what the encoding is on every supported version of > Windows. UTF