Eryk Sun <eryk...@gmail.com> added the comment:
> Now many people want to use Python with UTF-8 mode in PowerShell
> in Windows Terminal. And they don't want to care about legacy
> encoding at all.
Windows Terminal isn't relevant to the encoding issue. Applications interact
with the console session to which they're attached, e.g. python.exe <->
condrv.sys <-> conhost.exe (openconsole.exe). It doesn't matter whether or not
the console session is headless (conpty) and connected to another process via
pipes (i.e. a console session created via CreatePseudoConsole and set for a
child process via PROC_THREAD_ATTRIBUTE_PSEUDOCONSOLE). Some settings and
behaviors do change in conpty mode (e.g. the session has virtual-terminal mode
enabled by default), but the way encoding and decoding are implemented for
ReadFile/ReadConsoleA and WriteFile/WriteConsoleA doesn't change.
There are still a lot of programs that read and write from the console as a
regular file via ReadFile and WriteFile, so the fact that ReadFile is broken
when the input code page is set to UTF-8 is relevant to most people. However,
people who run `chcp 65001` in Western locales usually only care about being
able to write non-ASCII UTF-8 via WriteFile. Reading non-ASCII UTF-8 from
console input via ReadFile doesn't come up as a common problem, but it
definitely is a problem. For example:
>>> s = os.read(0, 12)
Привет мир
>>> s
b'\x00\x00\x00\x00\x00\x00 \x00\x00\x00\r\n'
Thus I don't like 'solving' this mojibake issue by simply recommending that
users set the console input codepage to UTF-8. I previously proposed two
solutions. (1) a radical change to get full Unicode support: modify pydoc to
temporarily change the console input codepage to UTF-8 and write the temp file
as UTF-8. (2) a conservative change just to avoid mojibake: modify pydoc to
query the console input codepage and write the file using that encoding, as
always with the backslashreplace error handler.
----------
_______________________________________
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue44275>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
