Steven D'Aprano writes:
 > On Sat, Jan 23, 2021 at 12:40:55AM -0500, Random832 wrote:
 > > On Fri, Jan 22, 2021, at 20:34, Inada Naoki wrote:
 > > > * Default encoding is "utf-8".
 > > 
 > > it might be worthwhile to be a little more sophisticated than this.
 > > 
 > > Notepad itself uses character set detection [it might not be 
 > > reasonable to do this on the whole file as notepad does, but maybe the 
 > > first 512 bytes, or the result of read1(512)?] when opening a file of 
 > > unknown encoding, and msvcrt's "ccs=UTF-8" option to fopen will at 
 > > least detect at the presence of UTF-8 and UTF-16 BOMs [and treat the 
 > > file as UTF-16 in the latter case].
 > 
 > 
 > I like Random's idea. If we add a new "open text file" builtin
 > function, we should seriously consider having it attempt to
 > auto-detect the encoding. It need not be as sophisticated as
 > `chardet`.

It definitely should not be as sophisticated as chardet.  Detection of
ISO 8859, ISO 2022, and EUC family encodings is reliable as long as
you know that only one of each family is going to be used.  But you
cannot easily tell which of the many ISO 8859 (also Windows-12xx)
family are present, and similarly for the other families.

I see very little use in detecting the BOMs.  I haven't seen a UTF-16
BOM in the wild in a decade (as usual for me, that's Japan-specific,
and may be limited to the academic community as well), and the UTF-8
BOM is a no-op if the default is UTF-8 anyway.

I'm definitely leaning to the suggestion I made elsewhere (if it's
adopted at all): force UTF-8, and name it 'open_utf8'.

Steve
_______________________________________________
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/LPUM3JPQD3RJCYFZ42GWTISCAHKF462C/
Code of Conduct: http://python.org/psf/codeofconduct/

Reply via email to