https://github.com/python/cpython/commit/4a42aedb0302326ee54837db14da111163c21805 commit: 4a42aedb0302326ee54837db14da111163c21805 branch: 3.15 author: Miss Islington (bot) <[email protected]> committer: hugovk <[email protected]> date: 2026-09-15T13:58:35Z summary:
[3.15] gh-148603: Update docs now UTF-8 is default (PEP 686) (GH-157372) (#157556) Co-authored-by: Hugo van Kemenade <[email protected]> files: M Doc/builtins/functions.rst M Doc/library/csv.rst M Doc/library/io.rst M Doc/tutorial/inputoutput.rst M Doc/using/windows.rst diff --git a/Doc/builtins/functions.rst b/Doc/builtins/functions.rst index 28f59382c6fc568..f8d848791f6cdba 100644 --- a/Doc/builtins/functions.rst +++ b/Doc/builtins/functions.rst @@ -1,7 +1,7 @@ .. XXX document all delegations to __special__ methods .. _built-in-funcs: -Built-in Functions +Built-in functions ================== The Python interpreter has a number of functions and types built into it that @@ -1394,7 +1394,8 @@ are always available. They are listed here in alphabetical order. already exists), ``'x'`` for exclusive creation, and ``'a'`` for appending (which on *some* Unix systems, means that *all* writes append to the end of the file regardless of the current seek position). In text mode, if - *encoding* is not specified the encoding used is platform-dependent: + *encoding* is not specified, UTF-8 is used by default; if + :ref:`Python UTF-8 Mode <utf8-mode>` is disabled, :func:`locale.getencoding` is called to get the current locale encoding. (For reading and writing raw bytes use binary mode and leave *encoding* unspecified.) The available modes are: @@ -1425,7 +1426,7 @@ are always available. They are listed here in alphabetical order. argument) return contents as :class:`bytes` objects without any decoding. In text mode (the default, or when ``'t'`` is included in the *mode* argument), the contents of the file are returned as :class:`str`, the bytes having been - first decoded using a platform-dependent encoding or using the specified + first decoded using the default encoding or using the specified *encoding* if given. .. note:: @@ -1454,9 +1455,11 @@ are always available. They are listed here in alphabetical order. described above for binary files. *encoding* is the name of the encoding used to decode or encode the file. - This should only be used in text mode. The default encoding is platform - dependent (whatever :func:`locale.getencoding` returns), but any - :term:`text encoding` supported by Python can be used. + This should only be used in text mode. The default encoding is UTF-8; + if :ref:`Python UTF-8 Mode <utf8-mode>` is disabled, the default is + platform-dependent (whatever :func:`locale.getencoding` returns). + Any :term:`text encoding` supported by Python can be used, and + ``encoding="locale"`` specifies the current locale encoding explicitly. See the :mod:`codecs` module for the list of supported encodings. *errors* is an optional string that specifies how encoding and decoding @@ -1573,6 +1576,10 @@ are always available. They are listed here in alphabetical order. .. versionchanged:: 3.11 The ``'U'`` mode has been removed. + .. versionchanged:: 3.15 + UTF-8 is now the default encoding, instead of the + platform-dependent locale encoding (:pep:`686`). + .. function:: ord(character, /) Return the ordinal value of a character. diff --git a/Doc/library/csv.rst b/Doc/library/csv.rst index d05cec519d4de3b..134d85969d130f2 100644 --- a/Doc/library/csv.rst +++ b/Doc/library/csv.rst @@ -1,4 +1,4 @@ -:mod:`!csv` --- CSV File Reading and Writing +:mod:`!csv` --- CSV file reading and writing ============================================ .. module:: csv @@ -42,7 +42,7 @@ using the :class:`DictReader` and :class:`DictWriter` classes. .. _csv-contents: -Module Contents +Module contents --------------- The :mod:`!csv` module defines the following functions: @@ -429,7 +429,7 @@ The :mod:`!csv` module defines the following exception: .. _csv-fmt-params: -Dialects and Formatting Parameters +Dialects and formatting parameters ---------------------------------- To make it easier to specify the format of input and output records, specific @@ -528,7 +528,7 @@ Dialects support the following attributes: .. _reader-objects: -Reader Objects +Reader objects -------------- Reader objects (:class:`DictReader` instances and objects returned by the @@ -565,7 +565,7 @@ DictReader objects have the following public attribute: -Writer Objects +Writer objects -------------- :class:`writer` objects (:class:`DictWriter` instances and objects returned by @@ -644,17 +644,16 @@ The corresponding simplest possible writing example is:: writer.writerows(someiterable) Since :func:`open` is used to open a CSV file for reading, the file -will by default be decoded into unicode using the system default -encoding (see :func:`locale.getencoding`). To decode a file +will by default be decoded into Unicode using UTF-8. To decode a file using a different encoding, use the ``encoding`` argument of open:: import csv - with open('some.csv', newline='', encoding='utf-8') as f: + with open('some.csv', newline='', encoding='latin-1') as f: reader = csv.reader(f) for row in reader: print(row) -The same applies to writing in something other than the system default +The same applies to writing in something other than the default encoding: specify the encoding argument when opening the output file. Registering a new dialect:: diff --git a/Doc/library/io.rst b/Doc/library/io.rst index d47b74efe22de9d..114cb251b335572 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -115,34 +115,21 @@ The raw stream API is described in detail in the docs of :class:`RawIOBase`. .. _io-text-encoding: -Text Encoding +Text encoding ------------- -The default encoding of :class:`TextIOWrapper` and :func:`open` is -locale-specific (:func:`locale.getencoding`). - -However, many developers forget to specify the encoding when opening text files -encoded in UTF-8 (e.g. JSON, TOML, Markdown, etc...) since most Unix -platforms use UTF-8 locale by default. This causes bugs because the locale -encoding is not UTF-8 for most Windows users. For example:: - - # May not work on Windows when non-ASCII characters in the file. - with open("README.md") as f: - long_description = f.read() - -Accordingly, it is highly recommended that you specify the encoding -explicitly when opening text files. If you want to use UTF-8, pass -``encoding="utf-8"``. To use the current locale encoding, -``encoding="locale"`` is supported since Python 3.10. +The default encoding of :class:`TextIOWrapper` and :func:`open` is UTF-8. +If :ref:`Python UTF-8 Mode <utf8-mode>` is disabled, the default encoding +is locale-specific (:func:`locale.getencoding`). .. seealso:: :ref:`utf8-mode` - Python UTF-8 Mode can be used to change the default encoding to - UTF-8 from locale-specific encoding. + Python UTF-8 Mode ignores the locale encoding and forces the use + of UTF-8. :pep:`686` - Python 3.15 will make :ref:`utf8-mode` default. + Python 3.15 made :ref:`utf8-mode` the default. .. _io-encoding-warning: @@ -152,7 +139,7 @@ Opt-in EncodingWarning .. versionadded:: 3.10 See :pep:`597` for more details. -To find where the default locale encoding is used, you can enable +To find where the default encoding is used, you can enable the :option:`-X warn_default_encoding <-X>` command line option or set the :envvar:`PYTHONWARNDEFAULTENCODING` environment variable, which will emit an :exc:`EncodingWarning` when the default encoding is used. @@ -165,7 +152,7 @@ please consider using UTF-8 by default (i.e. ``encoding="utf-8"``) for new APIs. -High-level Module Interface +High-level module interface --------------------------- .. data:: DEFAULT_BUFFER_SIZE @@ -315,7 +302,7 @@ ABC Inherits Stub Methods Mixin M ========================= ================== ======================== ================================================== -I/O Base Classes +I/O base classes ^^^^^^^^^^^^^^^^ .. class:: IOBase @@ -660,7 +647,7 @@ I/O Base Classes so the implementation should only access *b* during the method call. -Raw File I/O +Raw file I/O ^^^^^^^^^^^^ .. class:: FileIO(name, mode='r', closefd=True, opener=None) @@ -728,7 +715,7 @@ Raw File I/O given in the constructor. -Buffered Streams +Buffered streams ^^^^^^^^^^^^^^^^ Buffered I/O streams provide a higher-level interface to an I/O device @@ -990,8 +977,8 @@ Text I/O :class:`TextIOBase`. *encoding* gives the name of the encoding that the stream will be decoded or - encoded with. In :ref:`UTF-8 Mode <utf8-mode>`, this defaults to UTF-8. - Otherwise, it defaults to :func:`locale.getencoding`. + encoded with. This defaults to UTF-8; if :ref:`UTF-8 Mode <utf8-mode>` is + disabled, it defaults to :func:`locale.getencoding`. ``encoding="locale"`` can be used to specify the current locale's encoding explicitly. See :ref:`io-text-encoding` for more information. @@ -1173,7 +1160,7 @@ Text I/O It inherits from :class:`codecs.IncrementalDecoder`. -Static Typing +Static typing ------------- The following protocols can be used for annotating function and method diff --git a/Doc/tutorial/inputoutput.rst b/Doc/tutorial/inputoutput.rst index a00f06cf46c41a3..4caaecfd0135552 100644 --- a/Doc/tutorial/inputoutput.rst +++ b/Doc/tutorial/inputoutput.rst @@ -1,7 +1,7 @@ .. _tut-io: **************** -Input and Output +Input and output **************** There are several ways to present the output of a program; data can be printed @@ -11,7 +11,7 @@ discuss some of the possibilities. .. _tut-formatting: -Fancier Output Formatting +Fancier output formatting ========================= So far we've encountered two ways of writing values: *expression statements* and @@ -111,7 +111,7 @@ This syntax is easy to use, although it offers much less control for formatting. .. _tut-f-strings: -Formatted String Literals +Formatted string literals ------------------------- :ref:`Formatted string literals <f-strings>` (also called f-strings for @@ -163,7 +163,7 @@ the reference guide for the :ref:`formatspec`. .. _tut-string-format: -The String format() Method +The string format() method -------------------------- Basic usage of the :meth:`str.format` method looks like this:: @@ -240,7 +240,7 @@ For a complete overview of string formatting with :meth:`str.format`, see :ref:`formatstrings`. -Manual String Formatting +Manual string formatting ------------------------ Here's the same table of squares and cubes, formatted manually:: @@ -303,7 +303,7 @@ More information can be found in the :ref:`old-string-formatting` section. .. _tut-files: -Reading and Writing Files +Reading and writing files ========================= .. index:: @@ -311,12 +311,11 @@ Reading and Writing Files pair: object; file :func:`open` returns a :term:`file object`, and is most commonly used with -two positional arguments and one keyword argument: -``open(filename, mode, encoding=None)`` +two positional arguments: ``open(filename, mode)`` :: - >>> f = open('workfile', 'w', encoding="utf-8") + >>> f = open('workfile', 'w') .. XXX str(f) is <io.TextIOWrapper object at 0x82e8dc4> @@ -334,10 +333,7 @@ omitted. Normally, files are opened in :dfn:`text mode`, that means, you read and write strings from and to the file, which are encoded in a specific *encoding*. -If *encoding* is not specified, the default is platform dependent -(see :func:`open`). -Because UTF-8 is the modern de-facto standard, ``encoding="utf-8"`` is -recommended unless you know that you need to use a different encoding. +If *encoding* is not specified, the default is UTF-8 (see :func:`open`). Appending a ``'b'`` to the mode opens the file in :dfn:`binary mode`. Binary mode data is read and written as :class:`bytes` objects. You can not specify *encoding* when opening file in binary mode. @@ -356,7 +352,7 @@ after its suite finishes, even if an exception is raised at some point. Using :keyword:`!with` is also much shorter than writing equivalent :keyword:`try`\ -\ :keyword:`finally` blocks:: - >>> with open('workfile', encoding="utf-8") as f: + >>> with open('workfile') as f: ... read_data = f.read() >>> # We can check that the file has been automatically closed. @@ -389,7 +385,7 @@ automatically fail. :: .. _tut-filemethods: -Methods of File Objects +Methods of file objects ----------------------- The rest of the examples in this section will assume that a file object called @@ -532,8 +528,8 @@ To decode the object again, if ``f`` is a :term:`binary file` or x = json.load(f) .. note:: - JSON files must be encoded in UTF-8. Use ``encoding="utf-8"`` when opening - JSON file as a :term:`text file` for both of reading and writing. + JSON files must be encoded in UTF-8, the default encoding for + :term:`text files <text file>`. This simple serialization technique can handle lists and dictionaries, but serializing arbitrary class instances in JSON requires a bit of extra effort. diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst index 3423617c2870ccc..7c68192ad6e069c 100644 --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -1332,14 +1332,15 @@ UTF-8 mode Python UTF-8 mode is now enabled by default (:pep:`686`). Windows still uses legacy encodings for the system encoding (the ANSI Code -Page). Python uses it for the default encoding of text files (e.g. -:func:`locale.getencoding`). +Page). When the :ref:`Python UTF-8 Mode <utf8-mode>` is disabled, Python +uses the ANSI Code Page as the default encoding of text files, as +returned by :func:`locale.getencoding`. This may cause issues because UTF-8 is widely used on the internet and most Unix systems, including WSL (Windows Subsystem for Linux). -The :ref:`Python UTF-8 Mode <utf8-mode>`, enabled by default, can help by -changing the default text encoding to UTF-8. +The :ref:`Python UTF-8 Mode <utf8-mode>`, enabled by default, ignores the +system encoding and uses UTF-8 as the default text encoding. When the :ref:`UTF-8 mode <utf8-mode>` is enabled, you can still use the system encoding (the ANSI Code Page) via the "mbcs" codec. _______________________________________________ Python-checkins mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3//lists/python-checkins.python.org Member address: [email protected]
