Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-29 Thread M.-A. Lemburg
Victor Stinner wrote: > Le mercredi 29 juin 2011 à 10:18 +0200, M.-A. Lemburg a écrit : >> Victor Stinner wrote: >>> Le mardi 28 juin 2011 à 16:02 +0200, M.-A. Lemburg a écrit : How about a more radical change: have open() in Py3 default to opening the file in binary mode, if no encoding

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-29 Thread Victor Stinner
Le mercredi 29 juin 2011 à 09:21 +0200, Baptiste Carvello a écrit : > By the way, I just thought that for real programming, I would love to have a > -Wcrossplatform command switch, which would warn for all unportable > constructs > in one go. That way, I don't have to remember which parts of 'os

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-29 Thread Victor Stinner
Le mercredi 29 juin 2011 à 10:18 +0200, M.-A. Lemburg a écrit : > Victor Stinner wrote: > > Le mardi 28 juin 2011 à 16:02 +0200, M.-A. Lemburg a écrit : > >> How about a more radical change: have open() in Py3 default to > >> opening the file in binary mode, if no encoding is given (even > >> if th

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-29 Thread M.-A. Lemburg
Victor Stinner wrote: > Le mardi 28 juin 2011 à 16:02 +0200, M.-A. Lemburg a écrit : >> How about a more radical change: have open() in Py3 default to >> opening the file in binary mode, if no encoding is given (even >> if the mode doesn't include 'b') ? > > I tried your suggested change: Python d

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-29 Thread Barry Warsaw
On Jun 28, 2011, at 09:42 PM, Georg Brandl wrote: >We need to stop making incompatible changes to Python 3. We had the chance >and took it to break all kinds of stuff, some of it gratuitous, with 3.0 and >even 3.1. Now the users need a period of compatibility and stability (just >like the langua

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-29 Thread Baptiste Carvello
Le 28/06/2011 16:46, Paul Moore a écrit : -1. This will make things harder for simple scripts which are not intended to be cross-platform. +1 to all you said. I frequently use the python command prompt or "python -c" for various quick tasks (mostly on linux). I would hate to replace my ugly,

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Nick Coghlan
On Wed, Jun 29, 2011 at 7:42 AM, Georg Brandl wrote: > On 28.06.2011 14:24, Terry Reedy wrote: >> I think a PEP is needed. > > Absolutely.  And I hope the hypothetical PEP would be rejected in this form. > > We need to stop making incompatible changes to Python 3.  We had the chance > and took it

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Stefan Behnel
Victor Stinner, 28.06.2011 15:43: In Python 2, open() opens the file in binary mode (e.g. file.readline() returns a byte string). codecs.open() opens the file in binary mode by default, you have to specify an encoding name to open it in text mode. In Python 3, open() opens the file in text mode

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Terry Reedy
On 6/28/2011 5:42 PM, Georg Brandl wrote: At the very least, a change like this needs a transitional strategy, like it has been used during the 2.x series: * In 3.3, accept "locale" as the encoding parameter, meaning the locale encoding * In 3.4, warn if encoding isn't given and the locale enco

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Victor Stinner
Le mardi 28 juin 2011 à 09:33 -0700, Toshio Kuratomi a écrit : > Issuing a warning like "open used without explicit encoding may lead > to errors" if open() is used without an explicit encoding would help > a little (at least, people who get errors would then have an inkling > that the culprit migh

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/28/2011 12:52 PM, Bill Janssen wrote: > M.-A. Lemburg wrote: > >> How about a more radical change: have open() in Py3 default to >> opening the file in binary mode, if no encoding is given (even >> if the mode doesn't include 'b') ? > > +1. >

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Victor Stinner
> > I don't think that Windows developer even know that they are writing > > files into the ANSI code page. MSDN documentation of > > WideCharToMultiByte() warns developer that the ANSI code page is not > > portable, even accross Windows computers: > > Probably true. But for many uses they also do

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Georg Brandl
On 28.06.2011 14:24, Terry Reedy wrote: >> As discussed before on this list, I propose to set the default encoding >> of open() to UTF-8 in Python 3.3, and add a warning in Python 3.2 if >> open() is called without an explicit encoding and if the locale encoding >> is not UTF-8. Using the warning,

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Victor Stinner
Le mardi 28 juin 2011 à 16:02 +0200, M.-A. Lemburg a écrit : > How about a more radical change: have open() in Py3 default to > opening the file in binary mode, if no encoding is given (even > if the mode doesn't include 'b') ? I tried your suggested change: Python doesn't start. sysconfig uses t

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Paul Moore
On 28 June 2011 18:22, Michael Foord wrote: > On 28/06/2011 18:06, Terry Reedy wrote: >> >> On 6/28/2011 10:46 AM, Paul Moore wrote: >> >>> I use Windows, and come from the UK, so 99% of my text files are >>> ASCII. So the majority of my code will be unaffected. But in the >>> occasional situation

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Ethan Furman
Ethan Furman wrote: Michael Foord wrote: On 28/06/2011 17:34, Terry Reedy wrote: On 6/28/2011 10:48 AM, Michael Foord wrote: On 28/06/2011 15:36, Terry Reedy wrote: S = open('myfile.txt').read() now return a text string in both Py2 and Py3 and a subsequent 'abc' in S works in both. Nope,

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Ethan Furman
Michael Foord wrote: On 28/06/2011 17:34, Terry Reedy wrote: On 6/28/2011 10:48 AM, Michael Foord wrote: On 28/06/2011 15:36, Terry Reedy wrote: S = open('myfile.txt').read() now return a text string in both Py2 and Py3 and a subsequent 'abc' in S works in both. Nope, it returns a bytestri

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Georg Brandl
On 28.06.2011 19:06, Terry Reedy wrote: > On 6/28/2011 10:46 AM, Paul Moore wrote: > >> I use Windows, and come from the UK, so 99% of my text files are >> ASCII. So the majority of my code will be unaffected. But in the >> occasional situation where I use a £ sign, I'll get encoding errors, > >

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Antoine Pitrou
On Tue, 28 Jun 2011 13:06:44 -0400 Terry Reedy wrote: > > As for practicality. Notepad++ on Windows offers ANSI, utf-8 (w,w/o > BOM), utf-16 (big/little endian). Well, that's *one* application. We would need much more data than that. > I believe that ODF documents are utf-8 > encoded xml (com

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Michael Foord
On 28/06/2011 18:06, Terry Reedy wrote: On 6/28/2011 10:46 AM, Paul Moore wrote: I use Windows, and come from the UK, so 99% of my text files are ASCII. So the majority of my code will be unaffected. But in the occasional situation where I use a £ sign, I'll get encoding errors, I do not unde

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Terry Reedy
On 6/28/2011 10:46 AM, Paul Moore wrote: I use Windows, and come from the UK, so 99% of my text files are ASCII. So the majority of my code will be unaffected. But in the occasional situation where I use a £ sign, I'll get encoding errors, I do not understand this. With utf-8 you would never g

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Bill Janssen
Terry Reedy wrote: > > Making such default encodings depend on the locale has already > > failed to work when we first introduced a default encoding in > > Py2, so I don't understand why we are repeating the same > > mistake again in Py3 (only in a different area). > > I do not remember any prop

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Bill Janssen
M.-A. Lemburg wrote: > How about a more radical change: have open() in Py3 default to > opening the file in binary mode, if no encoding is given (even > if the mode doesn't include 'b') ? +1. > That'll make it compatible to the Py2 world again and avoid > all the encoding guessing. Yep. Bill

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Michael Foord
On 28/06/2011 17:34, Terry Reedy wrote: On 6/28/2011 10:48 AM, Michael Foord wrote: On 28/06/2011 15:36, Terry Reedy wrote: S = open('myfile.txt').read() now return a text string in both Py2 and Py3 and a subsequent 'abc' in S works in both. Nope, it returns a bytestring in Python 2. Whic

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Terry Reedy
On 6/28/2011 10:48 AM, Michael Foord wrote: On 28/06/2011 15:36, Terry Reedy wrote: S = open('myfile.txt').read() now return a text string in both Py2 and Py3 and a subsequent 'abc' in S works in both. Nope, it returns a bytestring in Python 2. Which, in Py2 is a str() object. In both Pyth

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Toshio Kuratomi
On Tue, Jun 28, 2011 at 03:46:12PM +0100, Paul Moore wrote: > On 28 June 2011 14:43, Victor Stinner wrote: > > As discussed before on this list, I propose to set the default encoding > > of open() to UTF-8 in Python 3.3, and add a warning in Python 3.2 if > > open() is called without an explicit e

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Paul Moore
On 28 June 2011 16:06, Steffen Daode Nurpmeso wrote: > @ Paul Moore wrote (2011-06-28 16:46+0200): >> UTF-8 without BOM displays incorrectly in vim(1) > > Stop right now (you're oh so wrong)!  :-) Sorry. Please add "using the default settings of gvim on Windows". My context throughout was Window

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Steffen Daode Nurpmeso
@ Paul Moore wrote (2011-06-28 16:46+0200): > UTF-8 without BOM displays incorrectly in vim(1) Stop right now (you're oh so wrong)! :-) (By the way: UTF-8 and BOM? Interesting things i learn on this list. And i hope in ten years we can laugh about this -> UTF-8 transition all over the place, 'c

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Antoine Pitrou
On Tue, 28 Jun 2011 10:41:38 -0400 Terry Reedy wrote: > On 6/28/2011 10:06 AM, Antoine Pitrou wrote: > > On Tue, 28 Jun 2011 15:43:05 +0200 > > Victor Stinner wrote: > >> - ISO-8859-1 os some FreeBSD systems > >> - ANSI code page on Windows, e.g. cp1252 (close to ISO-8859-1) in > >> Western E

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Michael Foord
On 28/06/2011 15:36, Terry Reedy wrote: On 6/28/2011 10:02 AM, M.-A. Lemburg wrote: How about a more radical change: have open() in Py3 default to opening the file in binary mode, if no encoding is given (even if the mode doesn't include 'b') ? That'll make it compatible to the Py2 world again

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Paul Moore
On 28 June 2011 14:43, Victor Stinner wrote: > As discussed before on this list, I propose to set the default encoding > of open() to UTF-8 in Python 3.3, and add a warning in Python 3.2 if > open() is called without an explicit encoding and if the locale encoding > is not UTF-8. Using the warning

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Terry Reedy
On 6/28/2011 10:06 AM, Antoine Pitrou wrote: On Tue, 28 Jun 2011 15:43:05 +0200 Victor Stinner wrote: - ISO-8859-1 os some FreeBSD systems - ANSI code page on Windows, e.g. cp1252 (close to ISO-8859-1) in Western Europe, cp952 in Japan, ... - ASCII if the locale is manually set to an empt

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Terry Reedy
On 6/28/2011 10:02 AM, M.-A. Lemburg wrote: How about a more radical change: have open() in Py3 default to opening the file in binary mode, if no encoding is given (even if the mode doesn't include 'b') ? That'll make it compatible to the Py2 world again I disagree. I believe S = open('myfile

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Terry Reedy
On 6/28/2011 9:43 AM, Victor Stinner wrote: In Python 2, open() opens the file in binary mode (e.g. file.readline() returns a byte string). codecs.open() opens the file in binary mode by default, you have to specify an encoding name to open it in text mode. In Python 3, open() opens the file in

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Antoine Pitrou
On Tue, 28 Jun 2011 15:43:05 +0200 Victor Stinner wrote: > - ISO-8859-1 os some FreeBSD systems > - ANSI code page on Windows, e.g. cp1252 (close to ISO-8859-1) in > Western Europe, cp952 in Japan, ... > - ASCII if the locale is manually set to an empty string or to "C", or > if the environment

Re: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread M.-A. Lemburg
Victor Stinner wrote: > In Python 2, open() opens the file in binary mode (e.g. file.readline() > returns a byte string). codecs.open() opens the file in binary mode by > default, you have to specify an encoding name to open it in text mode. > > In Python 3, open() opens the file in text mode by d

[Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

2011-06-28 Thread Victor Stinner
In Python 2, open() opens the file in binary mode (e.g. file.readline() returns a byte string). codecs.open() opens the file in binary mode by default, you have to specify an encoding name to open it in text mode. In Python 3, open() opens the file in text mode by default. (It only opens the binar