[issue41152] IDLE: revise setting of iomenu.encoding and .errors

[miss-islington]

miss-islington  added the comment:


New changeset 00fd04b9b7537c473c3f9396a861868b8ddd3bb2 by Miss Islington (bot) 
in branch '3.8':
bpo-41152: IDLE: always use UTF-8 for standard IO streams (GH-21214)
https://github.com/python/cpython/commit/00fd04b9b7537c473c3f9396a861868b8ddd3bb2


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41152] IDLE: revise setting of iomenu.encoding and .errors

[miss-islington]

miss-islington  added the comment:


New changeset 01638ce51a63afe5af3f778e7403702703bb41b9 by Miss Islington (bot) 
in branch '3.9':
bpo-41152: IDLE: always use UTF-8 for standard IO streams (GH-21214)
https://github.com/python/cpython/commit/01638ce51a63afe5af3f778e7403702703bb41b9


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41152] IDLE: revise setting of iomenu.encoding and .errors

[miss-islington]

Change by miss-islington :


--
pull_requests: +20379
pull_request: https://github.com/python/cpython/pull/21226

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41152] IDLE: revise setting of iomenu.encoding and .errors

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 4.0 -> 5.0
pull_requests: +20378
pull_request: https://github.com/python/cpython/pull/21225

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41152] IDLE: revise setting of iomenu.encoding and .errors

[Terry J. Reedy]

Terry J. Reedy  added the comment:


New changeset 2515a28230b1a011205f30263da6b01c6bd167a3 by Serhiy Storchaka in 
branch 'master':
bpo-41152: IDLE: always use UTF-8 for standard IO streams (GH-21214)
https://github.com/python/cpython/commit/2515a28230b1a011205f30263da6b01c6bd167a3


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41152] IDLE: revise setting of iomenu.encoding and .errors

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

PR 21214 sets the encoding of stdin/stdout/stderr to 'utf-8'. Error handler is 
set to 'surrogatepass' or 'surrogateescape' because these error handlers are 
used when convert strings between Python and Tcl. It guarantees that reading 
from stdin and writing back to stdout will never fail, even if you paste 
garbage from clipboard. Printing file paths will never fail too.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41152] IDLE: revise setting of iomenu.encoding and .errors

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +20367
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/21214

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41152] IDLE: revise setting of iomenu.encoding and .errors

[Terry J. Reedy]

Terry J. Reedy  added the comment:

I got the 'within iomenu' part a bit wrong.  To open a file to edit, 
iomenu.IOBinging('IO').open tells filelist to use IO.loadfile.  This reads 
bytes 'so that we can handle end-of-line convention ourselves'.  (I suspect 
that this predates 3.x and might not be needed any more.)  IO.loadfile calls 
IO._decode which looks for a utf-8 BOM, looks for a coding cookie, tries ascii 
(not needed in 3.x), tries utf-8, and asks the user for an encoding, using 
iomenu.encoding as the initial value in the query box.  This box is deprecated 
in the sense that for 3.x, a python file should either be utf-8 or have an 
encoding cookie.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41152] IDLE: revise setting of iomenu.encoding and .errors

[Terry J. Reedy]

Terry J. Reedy  added the comment:

The main use for the iomenu settings is for the socket-transport file classes, 
in run.py.  The default encoding='utf-8' and errors='strict' are not used but 
are overriden with the iomenu values, or for stderr, 'backslashreplace'.

Since user code can print any unicode, I think the defaults should used as is 
to transparently pass on and possibly display anything the user sends.  Such a 
change should have no back-compatibility issues.

Thinking more about errors.  With utf-8 encoding of proper strings, there 
should never be any, but Python does allow construction of 'improper' strings 
with, say, single surrogates.  The transport mechanism should never raise, so 
maybe surrogateescape or backslashreplace should always be used. 

What do you two think?

Another use is for writing bytes to an OutputWindow, as with find-in-files.  
But I can think of no case where IDLE sends bytes to an OutputWindow.  User 
files are all opened in an editor.

I believe these are all the uses of 'iomenu.encoding' outside of iomenu.  'from 
iomenu ...' is never used.

Within iomenu, the only use is part of reading an encoding cookie.
# The only use of 'encoding' below is in _decode as initial value
# of deprecated block asking user for encoding.
I am not sure if this use can be reached now.  Even if so, I believe this code 
duplicates code elsewhere in the stdlib that might be used.

So maybe the encoding calculation is not really needed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41152] IDLE: revise setting of iomenu.encoding and .errors

[Terry J. Reedy]

Terry J. Reedy  added the comment:

The PR is for 1.  The *nix code is a bit clearer without the Windows code in 
the middle.

Is there a good reason why when encoding is 'utf-8', errors should be 
'surrogateescape' on Windows and 'strict' elsewhere?  Surrogateescape seems 
like it is made for when using ascii or other limited encoding.

--
stage: patch review -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41152] IDLE: revise setting of iomenu.encoding and .errors

[Terry J. Reedy]

Change by Terry J. Reedy :


--
keywords: +patch
pull_requests: +20361
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/21206

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41152] IDLE: revise setting of iomenu.encoding and .errors

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

I think it makes sense if we want to use the locale encoding for IO streams.

But on other hand, it may be worth to drop support of locale-depending and 
configurable IO encoding and always use UTF-8. It is the IO encoding always 
used on Windows and the encoding of most locales on modern Linux and macOS.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41152] IDLE: revise setting of iomenu.encoding and .errors

[Terry J. Reedy]

New submission from Terry J. Reedy :

When testing and on Windows, iomenu.encoding and .errors are set to utf-8 and 
surrogateescape*.  When running otherwise, these are set with baroque code I 
don't understand.  (Currently lines 31 to 61.)

1. Combine the two conditional statements for testing and Windows.

2. Ned, on my Catalina Macbook, the 30-line 'else' sections sets encoding, 
errors to 'utf-8', 'strict'.  Should there ever be any other result on Mac we 
care about?  If not, I would like to directly set them, as on Windows.

3. Serhiy, does the 'baroque code' look right to you, for Linux (or *nix in 
general)?

--
assignee: terry.reedy
components: IDLE
messages: 372527
nosy: ned.deily, serhiy.storchaka, taleinat, terry.reedy
priority: normal
severity: normal
stage: needs patch
status: open
title: IDLE: revise setting of iomenu.encoding and .errors
type: behavior
versions: Python 3.10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com