I made the following two changes to the PEP 540: * open() error handler remains "strict" * remove the "Strict UTF8 mode" which doesn't make much sense anymore
+1 — ignore my previous note. -CHB I wrote the Strict UTF-8 mode when open() used surrogateescape error handler in the UTF-8 mode. I don't think that a Strict UTF-8 mode is required just to change the error handler of stdin and stdout. Well, read the "Passthough undecodable bytes: surrogateescape" section of the PEP rationale :-) https://www.python.org/dev/peps/pep-0540/ Victor PEP: 540 Title: Add a new UTF-8 mode Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stin...@gmail.com> BDFL-Delegate: INADA Naoki Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 5-January-2016 Python-Version: 3.7 Abstract ======== Add a new UTF-8 mode to ignore the locale, use the UTF-8 encoding, and change ``stdin`` and ``stdout`` error handlers to ``surrogateescape``. This mode is enabled by default in the POSIX locale, but otherwise disabled by default. The new ``-X utf8`` command line option and ``PYTHONUTF8`` environment variable are added to control the UTF-8 mode. Rationale ========= Locale encoding and UTF-8 ------------------------- Python 3.6 uses the locale encoding for filenames, environment variables, standard streams, etc. The locale encoding is inherited from the locale; the encoding and the locale are tightly coupled. Many users inherit the ASCII encoding from the POSIX locale, aka the "C" locale, but are unable change the locale for different reasons. This encoding is very limited in term of Unicode support: any non-ASCII character is likely to cause troubles. It is not easy to get the expected locale. Locales don't get the exact same name on all Linux distributions, FreeBSD, macOS, etc. Some locales, like the recent ``C.UTF-8`` locale, are only supported by a few platforms. For example, a SSH connection can use a different encoding than the filesystem or terminal encoding of the local host. On the other side, Python 3.6 is already using UTF-8 by default on macOS, Android and Windows (PEP 529) for most functions, except of ``open()``. UTF-8 is also the default encoding of Python scripts, XML and JSON file formats. The Go programming language uses UTF-8 for strings. When all data are stored as UTF-8 but the locale is often misconfigured, an obvious solution is to ignore the locale and use UTF-8. PEP 538 attempts to mitigate this problem by coercing the C locale to a UTF-8 based locale when one is available, but that isn't a universal solution. For example, CentOS 7's container images default to the POSIX locale, and don't include the C.UTF-8 locale, so PEP 538's locale coercion is ineffective. Passthough undecodable bytes: surrogateescape --------------------------------------------- When decoding bytes from UTF-8 using the ``strict`` error handler, which is the default, Python 3 raises a ``UnicodeDecodeError`` on the first undecodable byte. Unix command line tools like ``cat`` or ``grep`` and most Python 2 applications simply do not have this class of bugs: they don't decode data, but process data as a raw bytes sequence. Python 3 already has a solution to behave like Unix tools and Python 2: the ``surrogateescape`` error handler (:pep:`383`). It allows to process data "as bytes" but uses Unicode in practice (undecodable bytes are stored as surrogate characters). The UTF-8 mode uses the ``surrogateescape`` error handler for ``stdin`` and ``stdout`` since these streams as commonly associated to Unix command line tools. However, users have a different expectation on files. Files are expected to be properly encoded. Python is expected to fail early when ``open()`` is called with the wrong options, like opening a JPEG picture in text mode. The ``open()`` default error handler remains ``strict`` for these reasons. No change by default for best backward compatibility ---------------------------------------------------- While UTF-8 is perfect in most cases, sometimes the locale encoding is actually the best encoding. This PEP changes the behaviour for the POSIX locale since this locale usually gives the ASCII encoding, whereas UTF-8 is a much better choice. It does not change the behaviour for other locales to prevent any risk or regression. As users are responsible to enable explicitly the new UTF-8 mode, they are responsible for any potential mojibake issues caused by this mode. Proposal ======== Add a new UTF-8 mode to ignore the locale, use the UTF-8 encoding, and change ``stdin`` and ``stdout`` error handlers to ``surrogateescape``. This mode is enabled by default in the POSIX locale, but otherwise disabled by default. The new ``-X utf8`` command line option and ``PYTHONUTF8`` environment variable are added. The UTF-8 mode is enabled by ``-X utf8`` or ``PYTHONUTF8=1``. The POSIX locale enables the UTF-8 mode. In this case, the UTF-8 mode can be explicitly disabled by ``-X utf8=0`` or ``PYTHONUTF8=0``. For standard streams, the ``PYTHONIOENCODING`` environment variable has priority over the UTF-8 mode. On Windows, the ``PYTHONLEGACYWINDOWSFSENCODING`` environment variable (:pep:`529`) has the priority over the UTF-8 mode. Backward Compatibility ====================== The only backward incompatible change is that the UTF-8 encoding is now used for the POSIX locale. Annex: Encodings And Error Handlers =================================== The UTF-8 mode changes the default encoding and error handler used by ``open()``, ``os.fsdecode()``, ``os.fsencode()``, ``sys.stdin``, ``sys.stdout`` and ``sys.stderr``. Encoding and error handler -------------------------- ============================ ======================= ========================== Function Default UTF-8 mode or POSIX locale ============================ ======================= ========================== open() locale/strict **UTF-8**/strict os.fsdecode(), os.fsencode() locale/surrogateescape **UTF-8**/surrogateescape sys.stdin, sys.stdout locale/strict **UTF-8/surrogateescape** sys.stderr locale/backslashreplace **UTF-8**/backslashreplace ============================ ======================= ========================== By comparison, Python 3.6 uses: ============================ ======================= ========================== Function Default POSIX locale ============================ ======================= ========================== open() locale/strict locale/strict os.fsdecode(), os.fsencode() locale/surrogateescape locale/surrogateescape sys.stdin, sys.stdout locale/strict locale/**surrogateescape** sys.stderr locale/backslashreplace locale/backslashreplace ============================ ======================= ========================== Encoding and error handler on Windows ------------------------------------- On Windows, the encodings and error handlers are different: ============================ ======================= ========================== ========================== Function Default Legacy Windows FS encoding UTF-8 mode ============================ ======================= ========================== ========================== open() mbcs/strict mbcs/strict **UTF-8**/strict os.fsdecode(), os.fsencode() UTF-8/surrogatepass **mbcs/replace** UTF-8/surrogatepass sys.stdin, sys.stdout UTF-8/surrogateescape UTF-8/surrogateescape UTF-8/surrogateescape sys.stderr UTF-8/backslashreplace UTF-8/backslashreplace UTF-8/backslashreplace ============================ ======================= ========================== ========================== By comparison, Python 3.6 uses: ============================ ======================= ========================== Function Default Legacy Windows FS encoding ============================ ======================= ========================== open() mbcs/strict mbcs/strict os.fsdecode(), os.fsencode() UTF-8/surrogatepass **mbcs/replace** sys.stdin, sys.stdout UTF-8/surrogateescape UTF-8/surrogateescape sys.stderr UTF-8/backslashreplace UTF-8/backslashreplace ============================ ======================= ========================== The "Legacy Windows FS encoding" is enabled by the ``PYTHONLEGACYWINDOWSFSENCODING`` environment variable. If stdin and/or stdout is redirected to a pipe, ``sys.stdin`` and/or ``sys.output`` use ``mbcs`` encoding by default rather than UTF-8. But in the UTF-8 mode, ``sys.stdin`` and ``sys.stdout`` always use the UTF-8 encoding. .. note: There is no POSIX locale on Windows. The ANSI code page is used to the locale encoding, and this code page never uses the ASCII encoding. Annex: Differences between PEP 538 and PEP 540 ============================================== PEP 538's locale coercion is only effective if a suitable UTF-8 based locale is available as a coercion target. PEP 540's UTF-8 mode can be enabled even for operating systems that don't provide a suitable platform locale (such as CentOS 7). PEP 538 only changes the interpreter's behaviour for the C locale. While the new UTF-8 mode of this PEP is only enabled by default in the C locale, it can also be enabled manually for any other locale. PEP 538 is implemented with ``setlocale(LC_CTYPE, "<coercion target>")`` and ``setenv("LC_CTYPE", "<coercion target>")``, so any non-Python code running in the process and any subprocesses that inherit the environment is impacted by the change. PEP 540 is implemented in Python internals and ignores the locale: non-Python running in the same process is not aware of the "Python UTF-8 mode". The benefit of the PEP 538 approach is that it helps ensure that encoding handling in binary extension modules and subprocesses is consistent with CPython's encoding handling. The upside of the PEP 540 approach is that it allows an embedding application to change the interpreter's behaviour without having to change the process global locale settings. Links ===== * `bpo-29240: Implementation of the PEP 540: Add a new UTF-8 mode <http://bugs.python.org/issue29240>`_ * `PEP 538 <https://www.python.org/dev/peps/pep-0538/>`_: "Coercing the legacy C locale to C.UTF-8" * `PEP 529 <https://www.python.org/dev/peps/pep-0529/>`_: "Change Windows filesystem encoding to UTF-8" * `PEP 528 <https://www.python.org/dev/peps/pep-0528/>`_: "Change Windows console encoding to UTF-8" * `PEP 383 <https://www.python.org/dev/peps/pep-0383/>`_: "Non-decodable Bytes in System Character Interfaces" Post History ============ * 2017-12: `[Python-Dev] PEP 540: Add a new UTF-8 mode <https://mail.python.org/pipermail/python-dev/2017-December/151054.html>`_ * 2017-04: `[Python-Dev] Proposed BDFL Delegate update for PEPs 538 & 540 (assuming UTF-8 for *nix system boundaries) <https://mail.python.org/pipermail/python-dev/2017-April/147795.html>`_ * 2017-01: `[Python-ideas] PEP 540: Add a new UTF-8 mode <https://mail.python.org/pipermail/python-ideas/2017-January/044089.html>`_ * 2017-01: `bpo-28180: Implementation of the PEP 538: coerce C locale to C.utf-8 (msg284764) <https://bugs.python.org/issue28180#msg284764>`_ * 2016-08-17: `bpo-27781: Change sys.getfilesystemencoding() on Windows to UTF-8 (msg272916) <https://bugs.python.org/issue27781#msg272916>`_ -- Victor proposed ``-X utf8`` for the :pep:`529` (Change Windows filesystem encoding to UTF-8) Copyright ========= This document has been placed in the public domain. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
