https://github.com/python/cpython/commit/24c4aecc1674414d3dc3238625802778c4ad29d2 commit: 24c4aecc1674414d3dc3238625802778c4ad29d2 branch: main author: Barry Warsaw <[email protected]> committer: warsaw <[email protected]> date: 2026-05-03T17:17:29Z summary:
gh-148641: Implement PEP 829 - startup configuration files (#149109) Implement PEP 829 - startup configuration files Also add `pkgutil.resolve_name(..., strict=True)` Co-authored-by: Brett Cannon <[email protected]> files: A Misc/NEWS.d/next/Library/2026-04-15-21-46-52.gh-issue-148641.-aoFyC.rst A Misc/NEWS.d/next/Library/2026-04-28-16-25-40.gh-issue-148641.aFgym0.rst M Doc/deprecations/pending-removal-in-3.18.rst M Doc/deprecations/pending-removal-in-3.20.rst M Doc/library/pkgutil.rst M Doc/library/site.rst M Doc/whatsnew/3.15.rst M Lib/pkgutil.py M Lib/site.py M Lib/test/test_pkgutil.py M Lib/test/test_site.py diff --git a/Doc/deprecations/pending-removal-in-3.18.rst b/Doc/deprecations/pending-removal-in-3.18.rst index eb42fe9919eaeb..19113aab981bbc 100644 --- a/Doc/deprecations/pending-removal-in-3.18.rst +++ b/Doc/deprecations/pending-removal-in-3.18.rst @@ -10,3 +10,9 @@ Pending removal in Python 3.18 specifier ``'N'``, which is only supported in the :mod:`!decimal` module's C implementation, has been deprecated since Python 3.13. (Contributed by Serhiy Storchaka in :gh:`89902`.) + +* Deprecations defined by :pep:`829`: + + * ``import`` lines in :file:`{name}.pth` files are silently ignored. + + (Contributed by Barry Warsaw in :gh:`148641`.) diff --git a/Doc/deprecations/pending-removal-in-3.20.rst b/Doc/deprecations/pending-removal-in-3.20.rst index 12d7acf5ce05b4..011565dfbb090d 100644 --- a/Doc/deprecations/pending-removal-in-3.20.rst +++ b/Doc/deprecations/pending-removal-in-3.20.rst @@ -39,6 +39,16 @@ Pending removal in Python 3.20 (Contributed by Hugo van Kemenade and Stan Ulbrych in :gh:`76007`.) +* Deprecations defined by :pep:`829`: + + * Warnings are produced for ``import`` lines found in :file:`{name}.pth` + files. + + * :file:`{name}.pth` files are no longer decoded in the locale encoding by + default. They **MUST** be encoded in ``utf-8-sig``. + + (Contributed by Barry Warsaw in :gh:`148641`.) + * :mod:`ast`: * Creating instances of abstract AST nodes (such as :class:`ast.AST` diff --git a/Doc/library/pkgutil.rst b/Doc/library/pkgutil.rst index aa7dd71c1329df..5473a367c49a3a 100644 --- a/Doc/library/pkgutil.rst +++ b/Doc/library/pkgutil.rst @@ -194,7 +194,7 @@ support. The :mod:`importlib.resources` module provides structured access to module resources. -.. function:: resolve_name(name) +.. function:: resolve_name(name, *, strict=False) Resolve a name to an object. @@ -208,6 +208,7 @@ support. * ``W(.W)*`` * ``W(.W)*:(W(.W)*)?`` + * ``W(.W)*:(W(.W)*)`` The first form is intended for backward compatibility only. It assumes that some part of the dotted name is a package, and the rest is an object @@ -222,6 +223,11 @@ support. hierarchy within that package. Only one import is needed in this form. If it ends with the colon, then a module object is returned. + The first two forms are accepted when ``strict=False`` (the default). + + The third form requires both the module name and callable, separated by + a colon. Only this form is accepted when ``strict=True``. + The function will return an object (which might be a module), or raise one of the following exceptions: @@ -233,3 +239,7 @@ support. hierarchy within the imported package to get to the desired object. .. versionadded:: 3.9 + + .. versionchanged:: 3.15 + + The optional keyword-only ``strict`` flag was added. diff --git a/Doc/library/site.rst b/Doc/library/site.rst index 04895ae4ec524b..3703d2fa60056f 100644 --- a/Doc/library/site.rst +++ b/Doc/library/site.rst @@ -17,7 +17,7 @@ import can be suppressed using the interpreter's :option:`-S` option. Importing this module normally appends site-specific paths to the module search path and adds :ref:`callables <site-consts>`, including :func:`help` to the built-in -namespace. However, Python startup option :option:`-S` blocks this and this module +namespace. However, Python startup option :option:`-S` blocks this, and this module can be safely imported with no automatic modifications to the module search path or additions to the builtins. To explicitly trigger the usual site-specific additions, call the :func:`main` function. @@ -71,40 +71,121 @@ the user site prefixes are also implicitly not searched for site-packages. single: # (hash); comment pair: statement; import -A path configuration file is a file whose name has the form :file:`{name}.pth` -and exists in one of the four directories mentioned above; its contents are -additional items (one per line) to be added to ``sys.path``. Non-existing items -are never added to ``sys.path``, and no check is made that the item refers to a -directory rather than a file. No item is added to ``sys.path`` more than -once. Blank lines and lines beginning with ``#`` are skipped. Lines starting -with ``import`` (followed by space or tab) are executed. +The :mod:`!site` module recognizes two startup configuration files of the form +:file:`{name}.pth` for path configurations, and :file:`{name}.start` for +pre-first-line code execution. Both files can exist in one of the four +directories mentioned above. Within each directory, these files are sorted +alphabetically by filename, then parsed in sorted order. -.. note:: +.. _site-pth-files: - An executable line in a :file:`.pth` file is run at every Python startup, - regardless of whether a particular module is actually going to be used. - Its impact should thus be kept to a minimum. - The primary intended purpose of executable lines is to make the - corresponding module(s) importable - (load 3rd-party import hooks, adjust :envvar:`PATH` etc). - Any other initialization is supposed to be done upon a module's - actual import, if and when it happens. - Limiting a code chunk to a single line is a deliberate measure - to discourage putting anything more complex here. +Path extensions (:file:`.pth` files) +------------------------------------ + +:file:`{name}.pth` contains additional items (one per line) to be appended to +``sys.path``. Items that name non-existing directories are never added to +``sys.path``, and no check is made that the item refers to a directory rather +than a file. No item is added to ``sys.path`` more than once. Blank lines +and lines beginning with ``#`` are skipped. + +For backward compatibility, lines starting with ``import`` (followed by space +or tab) are executed with :func:`exec`. .. versionchanged:: 3.13 + The :file:`.pth` files are now decoded by UTF-8 at first and then by the :term:`locale encoding` if it fails. +.. versionchanged:: next + + :file:`.pth` file lines starting with ``import`` are deprecated. During + the deprecation period, such lines are still executed (except in the case + below), but a diagnostic message is emitted only when the :option:`-v` flag + is given. + + ``import`` lines in :file:`{name}.pth` are silently ignored when a + :ref:`matching <site-start-files>` :file:`{name}.start` file exists. + + Errors on individual lines no longer abort processing of the rest of the + file. Each error is reported and the remaining lines continue to be + processed. + +.. deprecated-removed:: next 3.20 + + Decoding :file:`{name}.pth` files in any encoding other than ``utf-8-sig`` + is deprecated in Python 3.15, and support for decoding from the locale + encoding will be removed in Python 3.20. + + ``import`` lines in :file:`{name}.pth` files are deprecated and will be + silently ignored in Python 3.18 and 3.19. In Python 3.20 a warning will be + produced for ``import`` lines in :file:`{name}.pth` files. + + +.. _site-start-files: + +Startup entry points (:file:`.start` files) +------------------------------------------- + +.. versionadded:: next + +A startup entry point file is a file whose name has the form +:file:`{name}.start` and exists in one of the site-packages directories +described above. Each file specifies entry points to be called during +interpreter startup, using the ``pkg.mod:callable`` syntax understood by +:func:`pkgutil.resolve_name`. + +Each non-blank line that does not begin with ``#`` must contain an entry +point reference in the form ``pkg.mod:callable``. The colon and callable +portion are mandatory. Each callable is invoked with no arguments, and +any return value is discarded. + +:file:`.start` files are processed after all :file:`.pth` path extensions +have been applied to :data:`sys.path`, ensuring that paths are available +before any startup code runs. + +Unlike :data:`sys.path` extensions from :file:`.pth` files, duplicate entry +points are **not** de-duplicated --- if an entry point appears more than once, +it will be called more than once. + +If an exception occurs during resolution or invocation of an entry point, +a traceback is printed to :data:`sys.stderr` and processing continues with +the remaining entry points. + +:file:`.start` files must be encoded in UTF-8. + +:pep:`829` defined the original specification for these features. + +.. note:: + + If a :file:`{name}.start` file exists alongside a :file:`{name}.pth` file + with the same base name, any ``import`` lines in the :file:`.pth` file are + ignored in favor of the entry points in the :file:`.start` file. + +.. note:: + + Executable lines (``import`` lines in :file:`{name}.pth` files and + :file:`{name}.start` file entry points) are always run at Python startup + (unless :option:`-S` is given to disable the ``site.py`` module entirely), + regardless of whether a particular module is actually going to be used. + +.. note:: + + :file:`{name}.start` files invoke :func:`pkgutil.resolve_name` with + ``strict=True``, which requires the full ``pkg.mod:callable`` form. + .. index:: single: package triple: path; configuration; file + +Startup file examples +--------------------- + For example, suppose ``sys.prefix`` and ``sys.exec_prefix`` are set to :file:`/usr/local`. The Python X.Y library is then installed in :file:`/usr/local/lib/python{X.Y}`. Suppose this has a subdirectory :file:`/usr/local/lib/python{X.Y}/site-packages` with three -subsubdirectories, :file:`foo`, :file:`bar` and :file:`spam`, and two path +sub-subdirectories, :file:`foo`, :file:`bar` and :file:`spam`, and two path configuration files, :file:`foo.pth` and :file:`bar.pth`. Assume :file:`foo.pth` contains the following:: @@ -131,6 +212,45 @@ directory precedes the :file:`foo` directory because :file:`bar.pth` comes alphabetically before :file:`foo.pth`; and :file:`spam` is omitted because it is not mentioned in either path configuration file. +Let's say that there is also a :file:`foo.start` file containing the +following:: + + # foo package startup code + + foo.submod:initialize + +Now, after ``sys.path`` has been extended as above, and before Python turns +control over to user code, the ``foo.submod`` module is imported and the +``initialize()`` function from that module is called. + + +.. _site-migration-guide: + +Migrating from ``import`` lines in ``.pth`` files to ``.start`` files +--------------------------------------------------------------------- + +If your package currently ships a :file:`{name}.pth` file, you can keep all +``sys.path`` extension lines unchanged. Only ``import`` lines need to be +migrated. + +To migrate, create a callable (taking zero arguments) within an importable +module in your package. Reference it as a ``pkg.mod:callable`` entry point +in a matching :file:`{name}.start` file. Move everything on your ``import`` +line after the first semi-colon into the ``callable()`` function. + +If your package must straddle older Pythons that do not support :pep:`829` +and newer Pythons that do, change the ``import`` lines in your +:file:`{name}.pth` to use the following form: + +.. code-block:: python + + import pkg.mod; pkg.mod.callable() + +Older Pythons will execute these ``import`` lines, while newer Pythons will +ignore them in favor of the :file:`{name}.start` file. After the straddling +period, remove all ``import`` lines from your :file:`.pth` files. + + :mod:`!sitecustomize` --------------------- @@ -236,10 +356,27 @@ Module contents This function used to be called unconditionally. -.. function:: addsitedir(sitedir, known_paths=None) +.. function:: addsitedir(sitedir, known_paths=None, *, defer_processing_start_files=False) + + Add a directory to sys.path and parse the :file:`.pth` and :file:`.start` + files found in that directory. Typically used in :mod:`sitecustomize` or + :mod:`usercustomize` (see above). + + The *known_paths* argument is an optional set of case-normalized paths + used to prevent duplicate :data:`sys.path` entries. When ``None`` (the + default), the set is built from the current :data:`sys.path`. + + While :file:`.pth` and :file:`.start` files are always parsed, set + *defer_processing_start_files* to ``True`` to prevent processing the + startup data found in those files, so that you can process them explicitly + (this is typically used by the :func:`main` function). + + .. versionchanged:: next - Add a directory to sys.path and process its :file:`.pth` files. Typically - used in :mod:`sitecustomize` or :mod:`usercustomize` (see above). + Also processes :file:`.start` files. See :ref:`site-start-files`. + All :file:`.pth` and :file:`.start` files are now read and + accumulated before any path extensions, ``import`` line execution, + or entry point invocations take place. .. function:: getsitepackages() @@ -308,5 +445,6 @@ value greater than 2 if there is an error. .. seealso:: * :pep:`370` -- Per user site-packages directory + * :pep:`829` -- Startup entry points and the deprecation of import lines in ``.pth`` files * :ref:`sys-path-init` -- The initialization of :data:`sys.path`. diff --git a/Doc/whatsnew/3.15.rst b/Doc/whatsnew/3.15.rst index 61a440d2ad6f8d..53628b2ff46cfc 100644 --- a/Doc/whatsnew/3.15.rst +++ b/Doc/whatsnew/3.15.rst @@ -91,6 +91,7 @@ Summary -- Release highlights * :ref:`Improved error messages <whatsnew315-improved-error-messages>` * :ref:`The official Windows 64-bit binaries now use the tail-calling interpreter <whatsnew315-windows-tail-calling-interpreter>` +* :pep:`829`: Package Startup Configuration Files New features ============ diff --git a/Lib/pkgutil.py b/Lib/pkgutil.py index 8772a66791a3c9..11c2a4b0ef4635 100644 --- a/Lib/pkgutil.py +++ b/Lib/pkgutil.py @@ -9,6 +9,9 @@ import os.path import sys +lazy import re + + __all__ = [ 'get_importer', 'iter_importers', 'walk_packages', 'iter_modules', 'get_data', @@ -398,9 +401,10 @@ def get_data(package, resource): return loader.get_data(resource_name) -_NAME_PATTERN = None +_LENIENT_PATTERN = None +_STRICT_PATTERN = None -def resolve_name(name): +def resolve_name(name, *, strict=False): """ Resolve a name to an object. @@ -410,6 +414,7 @@ def resolve_name(name): W(.W)* W(.W)*:(W(.W)*)? + W(.W)*:(W(.W)*) The first form is intended for backward compatibility only. It assumes that some part of the dotted name is a package, and the rest is an object @@ -424,6 +429,11 @@ def resolve_name(name): hierarchy within that package. Only one import is needed in this form. If it ends with the colon, then a module object is returned. + The first two forms are accepted when `strict=False` (the default). + + The third form requires both the module name and callable, separated by + a colon. Only this form is accepted when `strict=True`. + The function will return an object (which might be a module), or raise one of the following exceptions: @@ -432,18 +442,26 @@ def resolve_name(name): AttributeError - if a failure occurred when traversing the object hierarchy within the imported package to get to the desired object. """ - global _NAME_PATTERN - if _NAME_PATTERN is None: - # Lazy import to speedup Python startup time - import re - dotted_words = r'(?!\d)(\w+)(\.(?!\d)(\w+))*' - _NAME_PATTERN = re.compile(f'^(?P<pkg>{dotted_words})' - f'(?P<cln>:(?P<obj>{dotted_words})?)?$', - re.UNICODE) - - m = _NAME_PATTERN.match(name) - if not m: + global _LENIENT_PATTERN, _STRICT_PATTERN + dotted_words = r'(?!\d)(\w+)(\.(?!\d)(\w+))*' + if strict: + if _STRICT_PATTERN is None: + _STRICT_PATTERN = re.compile( + f'^(?P<pkg>{dotted_words})' + f'(?P<cln>:(?P<obj>{dotted_words}))$', + re.UNICODE) + pattern = _STRICT_PATTERN + else: + if _LENIENT_PATTERN is None: + _LENIENT_PATTERN = re.compile( + f'^(?P<pkg>{dotted_words})' + f'(?P<cln>:(?P<obj>{dotted_words})?)?$', + re.UNICODE) + pattern = _LENIENT_PATTERN + + if (m := pattern.match(name)) is None: raise ValueError(f'invalid format: {name!r}') + gd = m.groupdict() if gd.get('cln'): # there is a colon - a one-step import is all that's needed diff --git a/Lib/site.py b/Lib/site.py index 30015b3f26b4b3..52dd9648734c3e 100644 --- a/Lib/site.py +++ b/Lib/site.py @@ -18,57 +18,26 @@ it is also checked for site-packages (sys.base_prefix and sys.base_exec_prefix will always be the "real" prefixes of the Python installation). If "pyvenv.cfg" (a bootstrap configuration file) contains -the key "include-system-site-packages" is set to "true" -(case-insensitive), the system-level prefixes will still also be -searched for site-packages; otherwise they won't. If the system-level -prefixes are not included then the user site prefixes are also implicitly -not searched for site-packages. - -All of the resulting site-specific directories, if they exist, are -appended to sys.path, and also inspected for path configuration -files. - -A path configuration file is a file whose name has the form -<package>.pth; its contents are additional directories (one per line) -to be added to sys.path. Non-existing directories (or -non-directories) are never added to sys.path; no directory is added to -sys.path more than once. Blank lines and lines beginning with -'#' are skipped. Lines starting with 'import' are executed. - -For example, suppose sys.prefix and sys.exec_prefix are set to -/usr/local and there is a directory /usr/local/lib/python2.5/site-packages -with three subdirectories, foo, bar and spam, and two path -configuration files, foo.pth and bar.pth. Assume foo.pth contains the -following: - - # foo package configuration - foo - bar - bletch - -and bar.pth contains: - - # bar package configuration - bar - -Then the following directories are added to sys.path, in this order: - - /usr/local/lib/python2.5/site-packages/bar - /usr/local/lib/python2.5/site-packages/foo - -Note that bletch is omitted because it doesn't exist; bar precedes foo -because bar.pth comes alphabetically before foo.pth; and spam is -omitted because it is not mentioned in either path configuration file. - -The readline module is also automatically configured to enable -completion for systems that support it. This can be overridden in -sitecustomize, usercustomize or PYTHONSTARTUP. Starting Python in -isolated mode (-I) disables automatic readline configuration. - -After these operations, an attempt is made to import a module -named sitecustomize, which can perform arbitrary additional -site-specific customizations. If this import fails with an -ImportError exception, it is silently ignored. +the key "include-system-site-packages" set to "true" (case-insensitive), +the system-level prefixes will still also be searched for site-packages; +otherwise they won't. + +Two kinds of configuration files are processed in each site-packages +directory: + +- <name>.pth files extend sys.path with additional directories (one per + line). Lines starting with "import" are deprecated (see PEP 829). + +- <name>.start files specify startup entry points using the pkg.mod:callable + syntax. These are resolved via pkgutil.resolve_name() and called with no + arguments. + +When called from main(), all .pth path extensions are applied before any +.start entry points are executed, ensuring that paths are available before +startup code runs. + +See the documentation for the site module for full details: +https://docs.python.org/3/library/site.html """ import sys @@ -79,6 +48,11 @@ import stat import errno +lazy import locale +lazy import pkgutil +lazy import traceback +lazy import warnings + # Prefixes for site-packages; add additional prefixes like /usr/local here PREFIXES = [sys.prefix, sys.exec_prefix] # Enable per user site-packages directory @@ -92,17 +66,34 @@ USER_BASE = None -def _trace(message): +def _trace(message, exc=None): if sys.flags.verbose: - print(message, file=sys.stderr) + _print_error(message, exc) -def _warn(*args, **kwargs): - import warnings +def _print_error(message, exc=None): + """Print an error message to stderr, optionally with a formatted traceback.""" + print(message, file=sys.stderr) + if exc is not None: + for record in traceback.format_exception(exc): + for line in record.splitlines(): + print(' ' + line, file=sys.stderr) + +def _warn(*args, **kwargs): warnings.warn(*args, **kwargs) +def _warn_future_us(message, remove): + # Don't call warnings._deprecated() directly because we're lazily importing warnings and don't + # want to have to trigger an eager import if it's not necessary. Startup time matters a lot + # here and warnings isn't cheap! This inlines the check from + # warnings._py_warnings._deprecated(). + _version = sys.version_info + if (_version[:2] > remove) or (_version[:2] == remove and _version[3] != "alpha"): + warnings._deprecated(message, remove=remove) + + def makepath(*paths): dir = os.path.join(*paths) try: @@ -163,75 +154,232 @@ def _init_pathinfo(): return d -def addpackage(sitedir, name, known_paths): - """Process a .pth file within the site-packages directory: - For each line in the file, either combine it with sitedir to a path - and add that to known_paths, or execute it if it starts with 'import '. +# Accumulated entry points from .start files across all site-packages +# directories. Execution is deferred until all paths in .pth files have been +# appended to sys.path. Map the .pth/.start file the data is found in to the +# data. +_pending_entrypoints = {} +_pending_syspaths = {} +_pending_importexecs = {} + + +def _read_pthstart_file(sitedir, name, suffix): + """Parse a .start or .pth file and return (lines, filename). + + On success, ``lines`` is a (possibly empty) list of the file's lines. + On failure (file missing, hidden, unreadable, or .start with bad + encoding), ``lines`` is ``None`` so callers can distinguish a + successfully-read empty file from one that could not be read. """ - if known_paths is None: - known_paths = _init_pathinfo() - reset = True - else: - reset = False - fullname = os.path.join(sitedir, name) + filename = os.path.join(sitedir, name) + _trace(f"Reading startup configuration file: {filename}") + try: - st = os.lstat(fullname) - except OSError: - return + st = os.lstat(filename) + except OSError as exc: + _trace(f"Cannot stat {filename!r}", exc) + return None, filename + if ((getattr(st, 'st_flags', 0) & stat.UF_HIDDEN) or (getattr(st, 'st_file_attributes', 0) & stat.FILE_ATTRIBUTE_HIDDEN)): - _trace(f"Skipping hidden .pth file: {fullname!r}") - return - _trace(f"Processing .pth file: {fullname!r}") + _trace(f"Skipping hidden {suffix} file: {filename!r}") + return None, filename + + _trace(f"Processing {suffix} file: {filename!r}") try: - with io.open_code(fullname) as f: - pth_content = f.read() - except OSError: - return + with io.open_code(filename) as f: + raw_content = f.read() + except OSError as exc: + _trace(f"Cannot read {filename!r}", exc) + return None, filename try: - # Accept BOM markers in .pth files as we do in source files - # (Windows PowerShell 5.1 makes it hard to emit UTF-8 files without a BOM) - pth_content = pth_content.decode("utf-8-sig") + # Accept BOM markers in .start and .pth files as we do in source files (Windows PowerShell + # 5.1 makes it hard to emit UTF-8 files without a BOM). + content = raw_content.decode("utf-8-sig") except UnicodeDecodeError: - # Fallback to locale encoding for backward compatibility. - # We will deprecate this fallback in the future. - import locale - pth_content = pth_content.decode(locale.getencoding()) - _trace(f"Cannot read {fullname!r} as UTF-8. " - f"Using fallback encoding {locale.getencoding()!r}") - - for n, line in enumerate(pth_content.splitlines(), 1): - if line.startswith("#"): + _trace(f"Cannot read {filename!r} as UTF-8.") + # For .pth files only, and then only until Python 3.20, fallback to locale encoding for + # backward compatibility. + _warn_future_us( + ".pth files decoded to locale encoding as a fallback", + remove=(3, 20) + ) + if suffix == ".pth": + content = raw_content.decode(locale.getencoding()) + _trace(f"Using fallback encoding {locale.getencoding()!r}") + else: + return None, filename + + return content.splitlines(), filename + + +def _read_pth_file(sitedir, name, known_paths): + """Parse a .pth file, accumulating sys.path extensions and import lines. + + Errors on individual lines do not abort processing of the rest of the + file (PEP 829). + """ + lines, filename = _read_pthstart_file(sitedir, name, ".pth") + if lines is None: + return + + for n, line in enumerate(lines, 1): + line = line.strip() + if not line or line.startswith("#"): continue - if line.strip() == "": + + # In Python 3.18 and 3.19, `import` lines are silently ignored. In + # Python 3.20 and beyond, issue a warning when `import` lines in .pth + # files are detected. + if line.startswith(("import ", "import\t")): + _warn_future_us( + "import lines in .pth files are silently ignored", + remove=(3, 18) + ) + _warn_future_us( + "import lines in .pth files are noisily ignored", + remove=(3, 20) + ) + _pending_importexecs.setdefault(filename, []).append(line) continue + try: - if line.startswith(("import ", "import\t")): + dir_, dircase = makepath(sitedir, line) + except Exception as exc: + _trace(f"Error in {filename!r}, line {n:d}: {line!r}", exc) + continue + + if dircase in known_paths: + _trace(f"In {filename!r}, line {n:d}: " + f"skipping duplicate sys.path entry: {dir_}") + else: + _pending_syspaths.setdefault(filename, []).append(dir_) + known_paths.add(dircase) + + +def _read_start_file(sitedir, name): + """Parse a .start file for a list of entry point strings.""" + lines, filename = _read_pthstart_file(sitedir, name, ".start") + if lines is None: + return + + # PEP 829: the *presence* of a matching .start file disables `import` + # line processing in the matched .pth file, regardless of whether the + # .start file produced any entry points. Register the filename as a + # key now so an empty (or comment-only) .start file still suppresses. + entrypoints = _pending_entrypoints.setdefault(filename, []) + + for n, line in enumerate(lines, 1): + line = line.strip() + if not line or line.startswith("#"): + continue + # Syntax validation is deferred to entry-point execution time, + # where pkgutil.resolve_name(strict=True) enforces the + # pkg.mod:callable form. + entrypoints.append(line) + + +def _extend_syspath(): + # We've already filtered out duplicates, either in the existing sys.path + # or in all the .pth files we've seen. We've also abspath/normpath'd all + # the entries, so all that's left to do is to ensure that the path exists. + for filename, dirs in _pending_syspaths.items(): + for dir_ in dirs: + if os.path.exists(dir_): + _trace(f"Extending sys.path with {dir_} from {filename}") + sys.path.append(dir_) + else: + _print_error( + f"In {filename}: {dir_} does not exist; " + f"skipping sys.path append") + + +def _exec_imports(): + # For all the `import` lines we've seen in .pth files, exec() them in + # order. However, if they come from a file with a matching .start, then + # we ignore these import lines. For the ones we do process, print a + # warning but only when -v was given. + for filename, imports in _pending_importexecs.items(): + name, dot, pth = filename.rpartition(".") + assert dot == "." and pth == "pth", f"Bad startup filename: {filename}" + + if f"{name}.start" in _pending_entrypoints: + # Skip import lines in favor of entry points. + continue + + _trace( + f"import lines in {filename} are deprecated, " + f"use entry points in a {name}.start file instead." + ) + + for line in imports: + try: + _trace(f"Exec'ing from {filename}: {line}") exec(line) + except Exception as exc: + _print_error( + f"Error in import line from {filename}: {line}", exc) + + +def _execute_start_entrypoints(): + """Execute all accumulated .start file entry points. + + Called after all site-packages directories have been processed so that + sys.path is fully populated before any entry point code runs. Uses + pkgutil.resolve_name(strict=True) which both validates the strict + pkg.mod:callable form and resolves the entry point in one step. + """ + for filename, entrypoints in _pending_entrypoints.items(): + for entrypoint in entrypoints: + try: + _trace(f"Executing entry point: {entrypoint} from {filename}") + callable_ = pkgutil.resolve_name(entrypoint, strict=True) + except ValueError as exc: + _print_error( + f"Invalid entry point syntax in {filename}: " + f"{entrypoint!r}", exc) continue - line = line.rstrip() - dir, dircase = makepath(sitedir, line) - if dircase not in known_paths and os.path.exists(dir): - sys.path.append(dir) - known_paths.add(dircase) - except Exception as exc: - print(f"Error processing line {n:d} of {fullname}:\n", - file=sys.stderr) - import traceback - for record in traceback.format_exception(exc): - for line in record.splitlines(): - print(' '+line, file=sys.stderr) - print("\nRemainder of file ignored", file=sys.stderr) - break + except Exception as exc: + _print_error( + f"Error resolving entry point {entrypoint} " + f"from {filename}", exc) + continue + try: + callable_() + except Exception as exc: + _print_error( + f"Error in entry point {entrypoint} from {filename}", + exc) + + +def process_startup_files(): + """Flush all pending sys.path and entry points.""" + _extend_syspath() + _exec_imports() + _execute_start_entrypoints() + _pending_syspaths.clear() + _pending_importexecs.clear() + _pending_entrypoints.clear() + + +def addpackage(sitedir, name, known_paths): + """Process a .pth file within the site-packages directory.""" + if known_paths is None: + known_paths = _init_pathinfo() + reset = True + else: + reset = False + _read_pth_file(sitedir, name, known_paths) + process_startup_files() if reset: known_paths = None return known_paths -def addsitedir(sitedir, known_paths=None): - """Add 'sitedir' argument to sys.path if missing and handle .pth files in - 'sitedir'""" +def addsitedir(sitedir, known_paths=None, *, defer_processing_start_files=False): + """Add 'sitedir' argument to sys.path if missing and handle startup + files.""" _trace(f"Adding directory: {sitedir!r}") if known_paths is None: known_paths = _init_pathinfo() @@ -246,12 +394,36 @@ def addsitedir(sitedir, known_paths=None): names = os.listdir(sitedir) except OSError: return - names = [name for name in names - if name.endswith(".pth") and not name.startswith(".")] - for name in sorted(names): - addpackage(sitedir, name, known_paths) + + # The following phases are defined by PEP 829. + # Phases 1-3: Read .pth files, accumulating paths and import lines. + pth_names = sorted( + name for name in names + if name.endswith(".pth") and not name.startswith(".") + ) + for name in pth_names: + _read_pth_file(sitedir, name, known_paths) + + # Phases 6-7: Discover .start files and accumulate their entry points. + # Import lines from .pth files with a matching .start file are discarded + # at flush time by _exec_imports(). + start_names = sorted( + name for name in names + if name.endswith(".start") and not name.startswith(".") + ) + for name in start_names: + _read_start_file(sitedir, name) + + # Generally, when addsitedir() is called explicitly, we'll want to process + # all the startup file data immediately. However, when called through + # main(), we'll want to batch up all the startup file processing. main() + # will set this flag to True to defer processing. + if not defer_processing_start_files: + process_startup_files() + if reset: known_paths = None + return known_paths @@ -364,7 +536,7 @@ def getusersitepackages(): return USER_SITE -def addusersitepackages(known_paths): +def addusersitepackages(known_paths, *, defer_processing_start_files=False): """Add a per user site-package to sys.path Each user has its own python directory with site-packages in the @@ -376,7 +548,7 @@ def addusersitepackages(known_paths): user_site = getusersitepackages() if ENABLE_USER_SITE and os.path.isdir(user_site): - addsitedir(user_site, known_paths) + addsitedir(user_site, known_paths, defer_processing_start_files=defer_processing_start_files) return known_paths def getsitepackages(prefixes=None): @@ -418,12 +590,12 @@ def getsitepackages(prefixes=None): sitepackages.append(os.path.join(prefix, "Lib", "site-packages")) return sitepackages -def addsitepackages(known_paths, prefixes=None): +def addsitepackages(known_paths, prefixes=None, *, defer_processing_start_files=False): """Add site-packages to sys.path""" _trace("Processing global site-packages") for sitedir in getsitepackages(prefixes): if os.path.isdir(sitedir): - addsitedir(sitedir, known_paths) + addsitedir(sitedir, known_paths, defer_processing_start_files=defer_processing_start_files) return known_paths @@ -705,8 +877,15 @@ def main(): known_paths = venv(known_paths) if ENABLE_USER_SITE is None: ENABLE_USER_SITE = check_enableusersite() - known_paths = addusersitepackages(known_paths) - known_paths = addsitepackages(known_paths) + known_paths = addusersitepackages(known_paths, defer_processing_start_files=True) + known_paths = addsitepackages(known_paths, defer_processing_start_files=True) + # PEP 829: flush accumulated data from all .pth and .start files. + # Paths are extended first, then deprecated import lines are exec'd, + # and finally .start entry points are executed — ensuring sys.path is + # fully populated before any startup code runs. process_startup_files() + # also clears the pending state so a later addsitedir() call does + # not re-apply already-processed data. + process_startup_files() setquit() setcopyright() sethelper() diff --git a/Lib/test/test_pkgutil.py b/Lib/test/test_pkgutil.py index d4faaaeca00457..4623b7eb4434b0 100644 --- a/Lib/test/test_pkgutil.py +++ b/Lib/test/test_pkgutil.py @@ -1,3 +1,5 @@ +import logging +import logging.handlers from pathlib import Path from test.support.import_helper import unload from test.support.warnings_helper import check_warnings @@ -232,9 +234,6 @@ def test_walk_packages_raises_on_string_or_bytes_input(self): list(pkgutil.walk_packages(bytes_input)) def test_name_resolution(self): - import logging - import logging.handlers - success_cases = ( ('os', os), ('os.path', os.path), @@ -322,6 +321,53 @@ def test_name_resolution(self): with self.assertRaises(exc): pkgutil.resolve_name(s) + def test_name_resolution_strict(self): + # PEP 829: strict=True accepts only the pkg.mod:callable form + # (W(.W)*:W(.W)*) -- both the colon and the callable are required. + success_cases = ( + ('os.path:pathsep', os.path.pathsep), + ('logging.handlers:SysLogHandler', + logging.handlers.SysLogHandler), + ('logging.handlers:SysLogHandler.LOG_ALERT', + logging.handlers.SysLogHandler.LOG_ALERT), + ('builtins:int', int), + ('builtins:int.from_bytes', int.from_bytes), + ('os:path', os.path), + ) + + # All of these are accepted under strict=False but must be + # rejected under strict=True. + failure_cases = ( + 'os', # no colon (non-strict form) + 'os.path', # no colon + 'logging:', # colon, empty callable + 'os.foo:', # colon, empty callable + ':int', # empty package + 'os.path:join:extra', # extra colon + 'os.path.9abc:join', # invalid identifier in package + 'os.path:9abc', # invalid identifier in callable + '', # empty + '?abc:foo', # invalid character + ) + + for s, expected in success_cases: + with self.subTest(s=s): + self.assertEqual( + pkgutil.resolve_name(s, strict=True), expected) + + for s in failure_cases: + with self.subTest(s=s): + with self.assertRaises(ValueError): + pkgutil.resolve_name(s, strict=True) + + # Cache independence: a strict=True call must not poison + # strict=False (and vice versa). Exercise both orderings. + self.assertEqual( + pkgutil.resolve_name('os:path', strict=True), os.path) + self.assertEqual(pkgutil.resolve_name('os.path'), os.path) + self.assertEqual( + pkgutil.resolve_name('os:path', strict=True), os.path) + def test_name_resolution_import_rebinding(self): # The same data is also used for testing import in test_import and # mock.patch in test_unittest. diff --git a/Lib/test/test_site.py b/Lib/test/test_site.py index e7dc5e2611c2de..ac69e2cbdbbe54 100644 --- a/Lib/test/test_site.py +++ b/Lib/test/test_site.py @@ -16,6 +16,7 @@ from test.support.script_helper import spawn_python, kill_python import ast import builtins +import contextlib import glob import io import os @@ -27,6 +28,7 @@ import sysconfig import tempfile from textwrap import dedent +from types import SimpleNamespace import urllib.error import urllib.request from unittest import mock @@ -123,14 +125,11 @@ def test_addpackage(self): # comment or import that is a valid directory name for where the .pth # file resides; invalid directories are not added pth_file = PthFile() - pth_file.cleanup(prep=True) # to make sure that nothing is - # pre-existing that shouldn't be - try: - pth_file.create() + # Ensure we have a clean slate. + pth_file.cleanup(prep=True) + with pth_file.create(): site.addpackage(pth_file.base_dir, pth_file.filename, set()) self.pth_file_tests(pth_file) - finally: - pth_file.cleanup() def make_pth(self, contents, pth_dir='.', pth_name=TESTFN): # Create a .pth file and return its (abspath, basename). @@ -150,9 +149,6 @@ def test_addpackage_import_bad_syntax(self): self.assertRegex(err_out.getvalue(), "line 1") self.assertRegex(err_out.getvalue(), re.escape(os.path.join(pth_dir, pth_fn))) - # XXX: the previous two should be independent checks so that the - # order doesn't matter. The next three could be a single check - # but my regex foo isn't good enough to write it. self.assertRegex(err_out.getvalue(), 'Traceback') self.assertRegex(err_out.getvalue(), r'import bad-syntax') self.assertRegex(err_out.getvalue(), 'SyntaxError') @@ -162,10 +158,8 @@ def test_addpackage_import_bad_exec(self): pth_dir, pth_fn = self.make_pth("randompath\nimport nosuchmodule\n") with captured_stderr() as err_out: site.addpackage(pth_dir, pth_fn, set()) - self.assertRegex(err_out.getvalue(), "line 2") self.assertRegex(err_out.getvalue(), re.escape(os.path.join(pth_dir, pth_fn))) - # XXX: ditto previous XXX comment. self.assertRegex(err_out.getvalue(), 'Traceback') self.assertRegex(err_out.getvalue(), 'ModuleNotFoundError') @@ -178,65 +172,68 @@ def test_addpackage_empty_lines(self): def test_addpackage_import_bad_pth_file(self): # Issue 5258 pth_dir, pth_fn = self.make_pth("abc\x00def\n") - with captured_stderr() as err_out: - self.assertFalse(site.addpackage(pth_dir, pth_fn, set())) - self.maxDiff = None - self.assertEqual(err_out.getvalue(), "") for path in sys.path: if isinstance(path, str): self.assertNotIn("abc\x00def", path) def test_addsitedir(self): - # Same tests for test_addpackage since addsitedir() essentially just - # calls addpackage() for every .pth file in the directory + # addsitedir() reads .pth files and, when called standalone + # (known_paths=None), flushes paths and import lines immediately. pth_file = PthFile() - pth_file.cleanup(prep=True) # Make sure that nothing is pre-existing - # that is tested for - try: - pth_file.create() - site.addsitedir(pth_file.base_dir, set()) + # Ensure we have a clean slate. + pth_file.cleanup(prep=True) + with pth_file.create(): + site.addsitedir(pth_file.base_dir) + self.pth_file_tests(pth_file) + + def test_addsitedir_explicit_flush(self): + # addsitedir() reads .pth files and, with + # defer_processing_start_files=True, accumulates pending state + # without flushing. A subsequent process_startup_files() call + # then applies the paths and runs the import lines. + pth_file = PthFile() + # Ensure we have a clean slate. + pth_file.cleanup(prep=True) + with pth_file.create(): + # Pass defer_processing_start_files=True to prevent flushing. + site.addsitedir(pth_file.base_dir, set(), + defer_processing_start_files=True) + self.assertNotIn(pth_file.imported, sys.modules) + site.process_startup_files() self.pth_file_tests(pth_file) - finally: - pth_file.cleanup() def test_addsitedir_dotfile(self): pth_file = PthFile('.dotfile') + # Ensure we have a clean slate. pth_file.cleanup(prep=True) - try: - pth_file.create() - site.addsitedir(pth_file.base_dir, set()) + with pth_file.create(): + site.addsitedir(pth_file.base_dir) self.assertNotIn(site.makepath(pth_file.good_dir_path)[0], sys.path) self.assertIn(pth_file.base_dir, sys.path) - finally: - pth_file.cleanup() @unittest.skipUnless(hasattr(os, 'chflags'), 'test needs os.chflags()') def test_addsitedir_hidden_flags(self): pth_file = PthFile() + # Ensure we have a clean slate. pth_file.cleanup(prep=True) - try: - pth_file.create() + with pth_file.create(): st = os.stat(pth_file.file_path) os.chflags(pth_file.file_path, st.st_flags | stat.UF_HIDDEN) - site.addsitedir(pth_file.base_dir, set()) + site.addsitedir(pth_file.base_dir) self.assertNotIn(site.makepath(pth_file.good_dir_path)[0], sys.path) self.assertIn(pth_file.base_dir, sys.path) - finally: - pth_file.cleanup() @unittest.skipUnless(sys.platform == 'win32', 'test needs Windows') @support.requires_subprocess() def test_addsitedir_hidden_file_attribute(self): pth_file = PthFile() + # Ensure we have a clean slate. pth_file.cleanup(prep=True) - try: - pth_file.create() + with pth_file.create(): subprocess.check_call(['attrib', '+H', pth_file.file_path]) - site.addsitedir(pth_file.base_dir, set()) + site.addsitedir(pth_file.base_dir) self.assertNotIn(site.makepath(pth_file.good_dir_path)[0], sys.path) self.assertIn(pth_file.base_dir, sys.path) - finally: - pth_file.cleanup() # This tests _getuserbase, hence the double underline # to distinguish from a test for getuserbase @@ -400,7 +397,7 @@ def test_trace(self): self.assertEqual(sys.stderr.getvalue(), out) -class PthFile(object): +class PthFile: """Helper class for handling testing of .pth files""" def __init__(self, filename_base=TESTFN, imported="time", @@ -415,6 +412,7 @@ def __init__(self, filename_base=TESTFN, imported="time", self.good_dir_path = os.path.join(self.base_dir, self.good_dirname) self.bad_dir_path = os.path.join(self.base_dir, self.bad_dirname) + @contextlib.contextmanager def create(self): """Create a .pth file with a comment, blank lines, an ``import <self.imported>``, a line with self.good_dirname, and a line with @@ -423,8 +421,7 @@ def create(self): Creation of the directory for self.good_dir_path (based off of self.good_dirname) is also performed. - Make sure to call self.cleanup() to undo anything done by this method. - + Used as a context manager: self.cleanup() is called on exit. """ FILE = open(self.file_path, 'w') try: @@ -436,6 +433,10 @@ def create(self): finally: FILE.close() os.mkdir(self.good_dir_path) + try: + yield self + finally: + self.cleanup() def cleanup(self, prep=False): """Make sure that the .pth file is deleted, self.imported is not in @@ -908,5 +909,544 @@ def test_both_args(self): self.assertEqual(output, excepted_output) +class StartFileTests(unittest.TestCase): + """Tests for .start file processing (PEP 829).""" + + def setUp(self): + self.enterContext(import_helper.DirsOnSysPath()) + self.tmpdir = self.sitedir = self.enterContext(os_helper.temp_dir()) + # Save and clear all pending dicts. + self.saved_entrypoints = site._pending_entrypoints.copy() + self.saved_syspaths = site._pending_syspaths.copy() + self.saved_importexecs = site._pending_importexecs.copy() + site._pending_entrypoints.clear() + site._pending_syspaths.clear() + site._pending_importexecs.clear() + + def tearDown(self): + site._pending_entrypoints = self.saved_entrypoints.copy() + site._pending_syspaths = self.saved_syspaths.copy() + site._pending_importexecs = self.saved_importexecs.copy() + + def _make_start(self, content, name='testpkg'): + """Write a <name>.start file and return its basename.""" + basename = f"{name}.start" + filepath = os.path.join(self.tmpdir, basename) + with open(filepath, 'w', encoding='utf-8') as f: + f.write(content) + return basename + + def _make_pth(self, content, name='testpkg'): + """Write a <name>.pth file and return its basename.""" + basename = f"{name}.pth" + filepath = os.path.join(self.tmpdir, basename) + with open(filepath, 'w', encoding='utf-8') as f: + f.write(content) + return basename + + def _all_entrypoints(self): + """Flatten _pending_entrypoints dict into a list of (filename, entry) tuples.""" + result = [] + for filename, entries in site._pending_entrypoints.items(): + for entry in entries: + result.append((filename, entry)) + return result + + def _just_entrypoints(self): + return [entry for filename, entry in self._all_entrypoints()] + + # --- _read_start_file tests --- + + def test_read_start_file_basic(self): + self._make_start("os.path:join\n", name='foo') + site._read_start_file(self.sitedir, 'foo.start') + fullname = os.path.join(self.sitedir, 'foo.start') + self.assertEqual(site._pending_entrypoints[fullname], ['os.path:join']) + + def test_read_start_file_multiple_entries(self): + self._make_start("os.path:join\nos.path:exists\n", name='foo') + site._read_start_file(self.sitedir, 'foo.start') + fullname = os.path.join(self.sitedir, 'foo.start') + self.assertEqual(site._pending_entrypoints[fullname], + ['os.path:join', 'os.path:exists']) + + def test_read_start_file_comments_and_blanks(self): + self._make_start("# a comment\n\nos.path:join\n \n", name='foo') + site._read_start_file(self.sitedir, 'foo.start') + fullname = os.path.join(self.sitedir, 'foo.start') + self.assertEqual(site._pending_entrypoints[fullname], ['os.path:join']) + + def test_read_start_file_accepts_all_non_blank_lines(self): + # Syntax validation is deferred to entry-point execution time + # (where pkgutil.resolve_name(strict=True) enforces the strict + # pkg.mod:callable form), so parsing accepts every non-blank, + # non-comment line, including syntactically invalid ones. + content = ( + "os.path\n" # no colon + "pkg.mod:\n" # empty callable + ":callable\n" # empty module + "pkg.mod:callable:extra\n" # multiple colons + "os.path:join\n" # valid + ) + self._make_start(content, name='foo') + site._read_start_file(self.sitedir, 'foo.start') + fullname = os.path.join(self.sitedir, 'foo.start') + self.assertEqual(site._pending_entrypoints[fullname], [ + 'os.path', + 'pkg.mod:', + ':callable', + 'pkg.mod:callable:extra', + 'os.path:join', + ]) + + def test_read_start_file_empty(self): + # PEP 829: an empty .start file is still registered as present + # (with an empty entry-point list) so that it suppresses `import` + # lines in any matching .pth file. + self._make_start("", name='foo') + site._read_start_file(self.sitedir, 'foo.start') + fullname = os.path.join(self.sitedir, 'foo.start') + self.assertEqual(site._pending_entrypoints, {fullname: []}) + + def test_read_start_file_comments_only(self): + # As with an empty file, a comments-only .start file is registered + # as present so it can suppress matching .pth `import` lines. + self._make_start("# just a comment\n# another\n", name='foo') + site._read_start_file(self.sitedir, 'foo.start') + fullname = os.path.join(self.sitedir, 'foo.start') + self.assertEqual(site._pending_entrypoints, {fullname: []}) + + def test_read_start_file_nonexistent(self): + with captured_stderr(): + site._read_start_file(self.tmpdir, 'nonexistent.start') + self.assertEqual(site._pending_entrypoints, {}) + + @unittest.skipUnless(hasattr(os, 'chflags'), 'test needs os.chflags()') + def test_read_start_file_hidden_flags(self): + self._make_start("os.path:join\n", name='foo') + filepath = os.path.join(self.tmpdir, 'foo.start') + st = os.stat(filepath) + os.chflags(filepath, st.st_flags | stat.UF_HIDDEN) + site._read_start_file(self.sitedir, 'foo.start') + self.assertEqual(site._pending_entrypoints, {}) + + def test_read_start_file_duplicates_not_deduplicated(self): + # PEP 829: duplicate entry points are NOT deduplicated. + self._make_start("os.path:join\nos.path:join\n", name='foo') + site._read_start_file(self.sitedir, 'foo.start') + fullname = os.path.join(self.sitedir, 'foo.start') + self.assertEqual(site._pending_entrypoints[fullname], + ['os.path:join', 'os.path:join']) + + def test_read_start_file_accepts_utf8_bom(self): + # PEP 829: .start files MUST be utf-8-sig (UTF-8 with optional BOM). + filepath = os.path.join(self.tmpdir, 'foo.start') + with open(filepath, 'wb') as f: + f.write(b'\xef\xbb\xbf' + b'os.path:join\n') + site._read_start_file(self.sitedir, 'foo.start') + fullname = os.path.join(self.sitedir, 'foo.start') + self.assertEqual( + site._pending_entrypoints[fullname], ['os.path:join']) + + def test_read_start_file_invalid_utf8_silently_skipped(self): + # PEP 829: .start files MUST be utf-8-sig. Unlike .pth, there is + # no locale-encoding fallback -- a .start file that is not valid + # UTF-8 is silently skipped, with no key registered in + # _pending_entrypoints and no output to stderr (parsing errors + # are reported only under -v). + filepath = os.path.join(self.tmpdir, 'foo.start') + with open(filepath, 'wb') as f: + # Bare continuation byte -- invalid as a UTF-8 start byte. + f.write(b'\x80\x80\x80\n') + with captured_stderr() as err: + site._read_start_file(self.sitedir, 'foo.start') + self.assertEqual(site._pending_entrypoints, {}) + self.assertEqual(err.getvalue(), "") + + def test_two_start_files_with_duplicates_not_deduplicated(self): + self._make_start("os.path:join", name="foo") + self._make_start("os.path:join", name="bar") + site._read_start_file(self.sitedir, 'foo.start') + site._read_start_file(self.sitedir, 'bar.start') + self.assertEqual(self._just_entrypoints(), + ['os.path:join', 'os.path:join']) + + # --- _read_pth_file tests --- + + def test_read_pth_file_paths(self): + subdir = os.path.join(self.sitedir, 'mylib') + os.mkdir(subdir) + self._make_pth("mylib\n", name='foo') + site._read_pth_file(self.sitedir, 'foo.pth', set()) + fullname = os.path.join(self.sitedir, 'foo.pth') + self.assertIn(subdir, site._pending_syspaths[fullname]) + + def test_read_pth_file_imports_collected(self): + self._make_pth("import sys\n", name='foo') + site._read_pth_file(self.sitedir, 'foo.pth', set()) + fullname = os.path.join(self.sitedir, 'foo.pth') + self.assertEqual(site._pending_importexecs[fullname], ['import sys']) + + def test_read_pth_file_comments_and_blanks(self): + self._make_pth("# comment\n\n \n", name='foo') + site._read_pth_file(self.sitedir, 'foo.pth', set()) + self.assertEqual(site._pending_syspaths, {}) + self.assertEqual(site._pending_importexecs, {}) + + def test_read_pth_file_deduplication(self): + subdir = os.path.join(self.sitedir, 'mylib') + os.mkdir(subdir) + known_paths = set() + self._make_pth("mylib\n", name='a') + self._make_pth("mylib\n", name='b') + site._read_pth_file(self.sitedir, 'a.pth', known_paths) + site._read_pth_file(self.sitedir, 'b.pth', known_paths) + # Only one entry across both files. + all_dirs = [] + for dirs in site._pending_syspaths.values(): + all_dirs.extend(dirs) + self.assertEqual(all_dirs, [subdir]) + + def test_read_pth_file_bad_line_continues(self): + # PEP 829: errors on individual lines don't abort the file. + subdir = os.path.join(self.sitedir, 'goodpath') + os.mkdir(subdir) + self._make_pth("abc\x00def\ngoodpath\n", name='foo') + with captured_stderr(): + site._read_pth_file(self.sitedir, 'foo.pth', set()) + fullname = os.path.join(self.sitedir, 'foo.pth') + self.assertIn(subdir, site._pending_syspaths.get(fullname, [])) + + def _flags_with_verbose(self, verbose): + # Build a sys.flags clone with verbose overridden but every + # other field preserved, so unrelated reads like + # sys.flags.optimize during io.open_code() continue to work. + attrs = {name: getattr(sys.flags, name) + for name in sys.flags.__match_args__} + attrs['verbose'] = verbose + return SimpleNamespace(**attrs) + + def test_read_pth_file_parse_error_silent_by_default(self): + # PEP 829: parse-time errors are silent unless -v is given. + # Force the error path by making makepath() raise. + self._make_pth("badline\n", name='foo') + with mock.patch('site.makepath', side_effect=ValueError("boom")), \ + mock.patch('sys.flags', self._flags_with_verbose(False)), \ + captured_stderr() as err: + site._read_pth_file(self.sitedir, 'foo.pth', set()) + self.assertEqual(err.getvalue(), "") + + def test_read_pth_file_parse_error_reported_under_verbose(self): + # PEP 829: parse-time errors are reported when -v is given. + self._make_pth("badline\n", name='foo') + with mock.patch('site.makepath', side_effect=ValueError("boom")), \ + mock.patch('sys.flags', self._flags_with_verbose(True)), \ + captured_stderr() as err: + site._read_pth_file(self.sitedir, 'foo.pth', set()) + out = err.getvalue() + self.assertIn('Error in', out) + self.assertIn('foo.pth', out) + + def test_read_pth_file_locale_fallback(self): + # PEP 829: .pth files that fail UTF-8 decoding fall back to the + # locale encoding for backward compatibility (deprecated in + # 3.15, to be removed in 3.20). Mock locale.getencoding() so + # the test does not depend on the host's actual locale. + subdir = os.path.join(self.sitedir, 'mylib') + os.mkdir(subdir) + filepath = os.path.join(self.tmpdir, 'foo.pth') + # \xe9 is invalid UTF-8 but valid in latin-1. + with open(filepath, 'wb') as f: + f.write(b'# caf\xe9 comment\nmylib\n') + with mock.patch('locale.getencoding', return_value='latin-1'), \ + captured_stderr(): + site._read_pth_file(self.sitedir, 'foo.pth', set()) + fullname = os.path.join(self.sitedir, 'foo.pth') + self.assertIn(subdir, site._pending_syspaths.get(fullname, [])) + + # --- _execute_start_entrypoints tests --- + + def test_execute_entrypoints_with_callable(self): + # Entrypoint with callable is invoked. + mod_dir = os.path.join(self.sitedir, 'epmod') + os.mkdir(mod_dir) + init_file = os.path.join(mod_dir, '__init__.py') + with open(init_file, 'w') as f: + f.write("""\ +called = False +def startup(): + global called + called = True +""") + sys.path.insert(0, self.sitedir) + self.addCleanup(sys.modules.pop, 'epmod', None) + fullname = os.path.join(self.sitedir, 'epmod.start') + site._pending_entrypoints[fullname] = ['epmod:startup'] + site._execute_start_entrypoints() + import epmod + self.assertTrue(epmod.called) + + def test_execute_entrypoints_import_error(self): + # Import error prints traceback but continues. + fullname = os.path.join(self.sitedir, 'bad.start') + site._pending_entrypoints[fullname] = [ + 'nosuchmodule_xyz:func', 'os.path:join'] + with captured_stderr() as err: + site._execute_start_entrypoints() + self.assertIn('nosuchmodule_xyz', err.getvalue()) + # os.path:join should still have been called (no exception for it) + + def test_execute_entrypoints_strict_syntax_rejection(self): + # PEP 829: only the strict pkg.mod:callable form is valid. + # At entry-point execution, pkgutil.resolve_name(strict=True) + # raises ValueError for invalid syntax; the invalid entry is + # reported and execution continues with the next one. + fullname = os.path.join(self.sitedir, 'bad.start') + site._pending_entrypoints[fullname] = [ + 'os.path', # no colon + 'pkg.mod:', # empty callable + ':callable', # empty module + 'pkg.mod:callable:extra', # multiple colons + ] + with captured_stderr() as err: + site._execute_start_entrypoints() + out = err.getvalue() + self.assertIn('Invalid entry point syntax', out) + for bad in ('os.path', 'pkg.mod:', ':callable', + 'pkg.mod:callable:extra'): + self.assertIn(bad, out) + + def test_execute_entrypoints_callable_error(self): + # Callable that raises prints traceback but continues. + mod_dir = os.path.join(self.sitedir, 'badmod') + os.mkdir(mod_dir) + init_file = os.path.join(mod_dir, '__init__.py') + with open(init_file, 'w') as f: + f.write("""\ +def fail(): + raise RuntimeError("boom") +""") + sys.path.insert(0, self.sitedir) + self.addCleanup(sys.modules.pop, 'badmod', None) + fullname = os.path.join(self.sitedir, 'badmod.start') + site._pending_entrypoints[fullname] = ['badmod:fail'] + with captured_stderr() as err: + site._execute_start_entrypoints() + self.assertIn('RuntimeError', err.getvalue()) + self.assertIn('boom', err.getvalue()) + + def test_execute_entrypoints_duplicates_called_twice(self): + # PEP 829: duplicate entry points execute multiple times. + mod_dir = os.path.join(self.sitedir, 'countmod') + os.mkdir(mod_dir) + init_file = os.path.join(mod_dir, '__init__.py') + with open(init_file, 'w') as f: + f.write("""\ +call_count = 0 +def bump(): + global call_count + call_count += 1 +""") + sys.path.insert(0, self.sitedir) + self.addCleanup(sys.modules.pop, 'countmod', None) + fullname = os.path.join(self.sitedir, 'countmod.start') + site._pending_entrypoints[fullname] = [ + 'countmod:bump', 'countmod:bump'] + site._execute_start_entrypoints() + import countmod + self.assertEqual(countmod.call_count, 2) + + # --- _exec_imports tests --- + + def test_exec_imports_suppressed_by_matching_start(self): + # Import lines from foo.pth are suppressed when foo.start exists. + pth_fullname = os.path.join(self.sitedir, 'foo.pth') + start_fullname = os.path.join(self.sitedir, 'foo.start') + site._pending_importexecs[pth_fullname] = ['import sys'] + site._pending_entrypoints[start_fullname] = ['os.path:join'] + # Should not exec the import line; no error expected. + site._exec_imports() + + def test_exec_imports_not_suppressed_by_different_start(self): + # Import lines from foo.pth are NOT suppressed by bar.start. + pth_fullname = os.path.join(self.sitedir, 'foo.pth') + start_fullname = os.path.join(self.sitedir, 'bar.start') + site._pending_importexecs[pth_fullname] = ['import sys'] + site._pending_entrypoints[start_fullname] = ['os.path:join'] + # Should execute the import line without error. + site._exec_imports() + + def test_exec_imports_suppressed_by_empty_matching_start(self): + self._make_start("", name='foo') + self._make_pth("import epmod; epmod.startup()", name='foo') + mod_dir = os.path.join(self.sitedir, 'epmod') + os.mkdir(mod_dir) + init_file = os.path.join(mod_dir, '__init__.py') + with open(init_file, 'w') as f: + f.write("""\ +called = False +def startup(): + global called + called = True +""") + sys.path.insert(0, self.sitedir) + self.addCleanup(sys.modules.pop, 'epmod', None) + site._read_pth_file(self.sitedir, 'foo.pth', set()) + site._read_start_file(self.sitedir, 'foo.start') + site._exec_imports() + import epmod + self.assertFalse(epmod.called) + + # --- _extend_syspath tests --- + + def test_extend_syspath_existing_dir(self): + subdir = os.path.join(self.sitedir, 'extlib') + os.mkdir(subdir) + site._pending_syspaths['test.pth'] = [subdir] + site._extend_syspath() + self.assertIn(subdir, sys.path) + + def test_extend_syspath_nonexistent_dir(self): + nosuch = os.path.join(self.sitedir, 'nosuchdir') + site._pending_syspaths['test.pth'] = [nosuch] + with captured_stderr() as err: + site._extend_syspath() + self.assertNotIn(nosuch, sys.path) + self.assertIn('does not exist', err.getvalue()) + + # --- addsitedir integration tests --- + + def test_addsitedir_discovers_start_files(self): + # addsitedir() should discover .start files and accumulate entries. + self._make_start("os.path:join\n", name='foo') + site.addsitedir(self.sitedir, set(), + defer_processing_start_files=True) + fullname = os.path.join(self.sitedir, 'foo.start') + self.assertIn('os.path:join', site._pending_entrypoints[fullname]) + + def test_addsitedir_start_suppresses_pth_imports(self): + # When foo.start exists, import lines in foo.pth are skipped + # at flush time by _exec_imports(). + self._make_start("os.path:join\n", name='foo') + self._make_pth("import sys\n", name='foo') + site.addsitedir(self.sitedir, set(), + defer_processing_start_files=True) + pth_fullname = os.path.join(self.sitedir, 'foo.pth') + start_fullname = os.path.join(self.sitedir, 'foo.start') + # Import line was collected... + self.assertIn('import sys', + site._pending_importexecs.get(pth_fullname, [])) + # ...but _exec_imports() will skip it because foo.start exists. + site._exec_imports() + + def test_addsitedir_pth_paths_still_work_with_start(self): + # Path lines in .pth files still work even when a .start file exists. + subdir = os.path.join(self.sitedir, 'mylib') + os.mkdir(subdir) + self._make_start("os.path:join\n", name='foo') + self._make_pth("mylib\n", name='foo') + site.addsitedir(self.sitedir, set(), + defer_processing_start_files=True) + fullname = os.path.join(self.sitedir, 'foo.pth') + self.assertIn(subdir, site._pending_syspaths.get(fullname, [])) + + def test_addsitedir_start_alphabetical_order(self): + # Multiple .start files are discovered alphabetically. + self._make_start("os.path:join\n", name='zzz') + self._make_start("os.path:exists\n", name='aaa') + site.addsitedir(self.sitedir, set(), + defer_processing_start_files=True) + all_entries = self._all_entrypoints() + entries = [entry for _, entry in all_entries] + idx_a = entries.index('os.path:exists') + idx_z = entries.index('os.path:join') + self.assertLess(idx_a, idx_z) + + def test_addsitedir_pth_before_start(self): + # PEP 829: .pth files are scanned before .start files. + # Create a .pth and .start with the same basename; verify + # the .pth data is collected before .start data. + subdir = os.path.join(self.sitedir, 'mylib') + os.mkdir(subdir) + self._make_pth("mylib\n", name='foo') + self._make_start("os.path:join\n", name='foo') + site.addsitedir(self.sitedir, set(), + defer_processing_start_files=True) + # Both should be collected. + pth_fullname = os.path.join(self.sitedir, 'foo.pth') + start_fullname = os.path.join(self.sitedir, 'foo.start') + self.assertIn(subdir, site._pending_syspaths.get(pth_fullname, [])) + self.assertIn('os.path:join', + site._pending_entrypoints.get(start_fullname, [])) + + def test_addsitedir_dotfile_start_ignored(self): + # .start files starting with '.' are skipped. Defer flushing so + # the assertion against _pending_entrypoints is meaningful; + # otherwise process_startup_files() would clear the dict + # regardless of whether the dotfile was picked up. + self._make_start("os.path:join\n", name='.hidden') + site.addsitedir(self.sitedir, set(), + defer_processing_start_files=True) + self.assertEqual(site._pending_entrypoints, {}) + + def test_addsitedir_standalone_flushes(self): + # When called with known_paths=None (standalone), addsitedir + # flushes immediately so the caller sees the effect. + subdir = os.path.join(self.sitedir, 'flushlib') + os.mkdir(subdir) + self._make_pth("flushlib\n", name='foo') + site.addsitedir(self.sitedir) # known_paths=None + self.assertIn(subdir, sys.path) + # Pending dicts should be cleared after flush. + self.assertEqual(site._pending_syspaths, {}) + + def test_addsitedir_defer_does_not_flush(self): + # With defer_processing_start_files=True, addsitedir accumulates + # pending state but does not flush; sys.path is updated only when + # process_startup_files() is called explicitly. + subdir = os.path.join(self.sitedir, 'acclib') + os.mkdir(subdir) + self._make_pth("acclib\n", name='foo') + site.addsitedir(self.sitedir, set(), + defer_processing_start_files=True) + # Path is pending, not yet on sys.path. + self.assertNotIn(subdir, sys.path) + fullname = os.path.join(self.sitedir, 'foo.pth') + self.assertIn(subdir, site._pending_syspaths.get(fullname, [])) + + def test_pth_path_is_available_to_start_entrypoint(self): + # Core PEP 829 invariant: all .pth path extensions are applied to + # sys.path *before* any .start entry point runs, so an entry + # point may live in a module reachable only via a .pth-extended + # path. If the flush phases were inverted, resolving the entry + # point would fail with ModuleNotFoundError. + extdir = os.path.join(self.sitedir, 'extdir') + os.mkdir(extdir) + modpath = os.path.join(extdir, 'mod.py') + with open(modpath, 'w') as f: + f.write("""\ +called = False +def hook(): + global called + called = True +""") + self.addCleanup(sys.modules.pop, 'mod', None) + + # extdir is not on sys.path; only the .pth file makes it so. + self.assertNotIn(extdir, sys.path) + self._make_pth("extdir\n", name='extlib') + self._make_start("mod:hook\n", name='extlib') + + # Standalone addsitedir() triggers the full flush sequence. + site.addsitedir(self.sitedir) + + self.assertIn(extdir, sys.path) + import mod + self.assertTrue( + mod.called, + "entry point did not run; .pth path was likely not applied " + "before .start entry-point execution") + + if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Library/2026-04-15-21-46-52.gh-issue-148641.-aoFyC.rst b/Misc/NEWS.d/next/Library/2026-04-15-21-46-52.gh-issue-148641.-aoFyC.rst new file mode 100644 index 00000000000000..53779676fd100c --- /dev/null +++ b/Misc/NEWS.d/next/Library/2026-04-15-21-46-52.gh-issue-148641.-aoFyC.rst @@ -0,0 +1,3 @@ +:pep:`829` (package startup configuration files) implements a new format +``<name>.start`` parallel to ``<name>.pth`` files, to replace ``import`` +lines in the latter. diff --git a/Misc/NEWS.d/next/Library/2026-04-28-16-25-40.gh-issue-148641.aFgym0.rst b/Misc/NEWS.d/next/Library/2026-04-28-16-25-40.gh-issue-148641.aFgym0.rst new file mode 100644 index 00000000000000..3820c5f9d0e697 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2026-04-28-16-25-40.gh-issue-148641.aFgym0.rst @@ -0,0 +1,2 @@ +:func:`pkgutil.resolve_name` gets a new optional, keyword-only argument +called ``strict``. The default is ``False`` for backward compatibility. _______________________________________________ Python-checkins mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3//lists/python-checkins.python.org Member address: [email protected]
