[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

STINNER Victor added the comment: New changeset 9852cb38112a4f8d11e26c3423643ea994d5a14f by Victor Stinner in branch 'master': bpo-42955: Rename module_names to sys.stdlib_module_names (GH-24332) https://github.com/python/cpython/commit/9852cb38112a4f8d11e26c3423643ea994d5a14f --

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

Change by STINNER Victor : -- pull_requests: +23151 pull_request: https://github.com/python/cpython/pull/24332 ___ Python tracker ___

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

STINNER Victor added the comment: New changeset 483359174e92489e13959977824806734f1a8cdd by Victor Stinner in branch 'master': bpo-42955: Fix sys.module_names doc (GH-24329) https://github.com/python/cpython/commit/483359174e92489e13959977824806734f1a8cdd --

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

Change by STINNER Victor : -- pull_requests: +23148 pull_request: https://github.com/python/cpython/pull/24329 ___ Python tracker ___

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

STINNER Victor added the comment: I merged my PR, thanks for the feedback and reviews. -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

STINNER Victor added the comment: New changeset db584bdad32d81e42b71871077a8008036f5c048 by Victor Stinner in branch 'master': bpo-42955: Add sys.modules_names (GH-24238) https://github.com/python/cpython/commit/db584bdad32d81e42b71871077a8008036f5c048 --

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

Change by Andre Roberge : -- nosy: +aroberge ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

STINNER Victor added the comment: Another potential use case: restrict pydoc web server to stdlib modules, see bpo-42988. -- ___ Python tracker ___

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

Change by hai shi : -- nosy: +shihai1991 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

SylvainDe added the comment: For similar reasons as friendly-traceback, I'd be interested in a list of stdlib modules to be able to provide additional information in case of exceptions. For instance: string.ascii_lowercase > Before: NameError("name 'string' is not defined",) > After:

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

STINNER Victor added the comment: Another example: the unimport project ("linter, formatter for finding and removing unused import statements") uses the following function to guess if a module object is part of the stdlib: def is_std(package: str) -> bool: """Returns True if package

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

STINNER Victor added the comment: Another use case similar to the "isort" use case: vim plugin to insert an import, it checks if the module name is a stdlib module. It checks the module path (for Python stdlib modules) and uses an hardcoded list of builtin stdlib modules.

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

STINNER Victor added the comment: Yet another use case: the friendly-traceback project detects typos on imports using the list of stdlib module names: https://aroberge.github.io/friendly-traceback-docs/docs/html/tracebacks_en_3.8.html#standard-library-module For example, it suggest to

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

STINNER Victor added the comment: The pants project also uses a list of stdlib module names, similar to isort, to infer dependencies: https://github.com/pantsbuild/pants/tree/master/src/python/pants/backend/python/dependency_inference/python_stdlib "Pants is a scalable build system for

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

STINNER Victor added the comment: sys.module_names also solves the following old StackOverflow question: "How to check if a module/library/package is part of the python standard library?"

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

STINNER Victor added the comment: New changeset cad8020cb83ec6d904f874c0e4f599e651022196 by Victor Stinner in branch 'master': bpo-42955: Add Python/module_names.h (GH-24258) https://github.com/python/cpython/commit/cad8020cb83ec6d904f874c0e4f599e651022196 --

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

Change by STINNER Victor : -- pull_requests: +23082 pull_request: https://github.com/python/cpython/pull/24258 ___ Python tracker ___

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

STINNER Victor added the comment: > One (poorly thought out) option is to add sys._stdlib_path with the > subsection of sys.path that contains the stdlib, and a function in importlib > that returns if a spec is for a stdlib module. There is already sysconfig.get_paths()['stdlib']. Maybe we

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

STINNER Victor added the comment: Ronald: > I think we agree on that point: my counter proposal won’t work in the > faulthandler scenario, and may be problematic in the Py_FatalError case as > well. The API providing a tuple of str (sys.module_names) works with the 4 use cases that I

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

STINNER Victor added the comment: Ronald: > BTW. A list of stdlib module names is not sufficient to determine if a module > is in the stdlib, thanks to .pth files it is possible to have entries on > sys.path before the stdlib. Yeah, I wrote it in my first message. I solved this issue with

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

Ronald Oussoren added the comment: BTW. A list of stdlib module names is not sufficient to determine if a module is in the stdlib, thanks to .pth files it is possible to have entries on sys.path before the stdlib. -- ___ Python tracker

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

Ronald Oussoren added the comment: > On 19 Jan 2021, at 12:30, STINNER Victor wrote: > > Ronald: >> You wouldn't necessarily have to import a module to test, this is something >> that could be added to importlib. One (poorly thought out) option is to add >> sys._stdlib_path with the

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

STINNER Victor added the comment: Myself: > Having to actually import modules to check if it's a stdlib module or not is > not convenient. Many stdlib modules have side effects on import. The "trace" usecase needs an exhaustive list of all module names. It is even less convenient to have to

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

Ronald Oussoren added the comment: >> Wouldn't it be sufficient to somehow mark the stdlib entries on sys.path? >> Although that might give misleading answers with tools like >> pyinstaller/py2exe/py2app that package an application and its dependencies >> into a single zipfile. > Having to

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

STINNER Victor added the comment: Ronald Oussoren: > A list of stdlib modules/extensions is IMHO problematic for maintenance, esp. > if you consider that not all modules/extensions are installed on all systems > (both because dependencies aren't present and because packagers have decided >

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

Ronald Oussoren added the comment: A list of stdlib modules/extensions is IMHO problematic for maintenance, esp. if you consider that not all modules/extensions are installed on all systems (both because dependencies aren't present and because packagers have decided to unbundle parts of the

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

Change by Dong-hee Na : -- nosy: +corona10 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

Change by STINNER Victor : -- pull_requests: +23075 pull_request: https://github.com/python/cpython/pull/24254 ___ Python tracker ___

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

STINNER Victor added the comment: Note for myself: regen-keyword should be included in "make regen-all". -- ___ Python tracker ___

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

STINNER Victor added the comment: Another way to gather the list of Python modules: pydoc help("modules") uses pkgutil.walk_packages(). pydoc uses ModuleScanner which uses sys.builtin_module_names and pkgutil.walk_packages(). pkgutil.walk_packages() calls pkgutil.iter_modules() which

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

STINNER Victor added the comment: (...) This API uses objects.inv from the online Python documentation. By the way, there is also this documentation page listing Python stdlib modules: https://docs.python.org/dev/py-modindex.html -- ___ Python

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

STINNER Victor added the comment: Use cases for sys.module_names: * When computing dependencies of a project, ignore modules which are part of the stdlib: https://github.com/jackmaney/pypt/issues/3 * Trace the execution of third party code, but ignore stdlib, use --ignore-module option of

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

STINNER Victor added the comment: I wrote PR 24238 which implements an hardcoded list of stdlib module names. The PR uses the CI to ensure that the list always remain up to date. If tomorrow a new module is added, a CI job fails ("file not up to date"). > What if set __author__ = 'PSF' in

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

Change by STINNER Victor : -- keywords: +patch pull_requests: +23060 stage: -> patch review pull_request: https://github.com/python/cpython/pull/24238 ___ Python tracker ___

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

Serhiy Storchaka added the comment: There is no technical difference between stdlib modules and other modules. Stdlib modules are only special in context of copyright and responsibility. What if set __author__ = 'PSF' in every stdib module? Or other attributes which would allow to group

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

STINNER Victor added the comment: On Fedora 33, the stdlib lives in two main directories: * /usr/lib64/python3.9: Python modules * /usr/lib64/python3.9/lib-dynload: C extension modules Example: >>> import os.path >>> os.path.dirname(os.__file__) # Python '/usr/lib64/python3.9' >>>

[issue42955] Add sys.module_names: list of stdlib module names (Python and extension modules)

New submission from STINNER Victor : Some use cases require to know if a module comes from the stdlib or not. For example, I would like to only dump extension modules which don't come from the stdlib in bpo-42923: "Py_FatalError(): dump the list of extension modules". Stdlib modules are