[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

Change by STINNER Victor : -- pull_requests: +21941 pull_request: https://github.com/python/cpython/pull/23024 ___ Python tracker ___

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

Change by Pablo Galindo Salgado : -- pull_requests: +21375 pull_request: https://github.com/python/cpython/pull/22331 ___ Python tracker ___

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

STINNER Victor added the comment: I also tested bpo-41261 reproducer and https://bugs.python.org/issue41194#msg372863 reproducer on 3.9 and master branches: Python doesn't crash. -- ___ Python tracker

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

STINNER Victor added the comment: I close the issue. I tested manually: the fix works as expected. I reproduced https://bugzilla.redhat.com/show_bug.cgi?id=1871992#c1 bug (I had to comment the workaround of the mercurial package in Fedora) on 3.9: make[1]: Entering directory

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

Łukasz Langa added the comment: New changeset 55e0836849c14fb474e1ba7f37851e07660eea3c by Pablo Galindo in branch '3.9': [3.9] bpo-41631: _ast module uses again a global state (GH-21961) (GH-22258) https://github.com/python/cpython/commit/55e0836849c14fb474e1ba7f37851e07660eea3c --

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

Change by Pablo Galindo Salgado : -- pull_requests: +21313 pull_request: https://github.com/python/cpython/pull/22258 ___ Python tracker ___

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

Łukasz Langa added the comment: New changeset e5fbe0cbd4be99ced5f000ad382208ad2a561c90 by Victor Stinner in branch 'master': bpo-41631: _ast module uses again a global state (#21961) https://github.com/python/cpython/commit/e5fbe0cbd4be99ced5f000ad382208ad2a561c90 --

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

STINNER Victor added the comment: I marked bpo-41766 as a duplicate of this issue. -- ___ Python tracker ___ ___ Python-bugs-list

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

STINNER Victor added the comment: > Compared to Python 3.8, PR 21961 has no regression related to subinterpreters. Oh. I forgot that static types cannot be modified (in Python, but it's possible in C). So my PR still changed the behavior compared to 3.8: --- import _testcapi import _ast res

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

STINNER Victor added the comment: About subinterpreters. In Python 3.8, _ast.AST type is a static type: static PyTypeObject AST_type = {...}; In Python 3.9, it's now a heap type: static PyType_Spec AST_type_spec = {...}; state->AST_type = PyType_FromSpec(_type_spec); In Python 3.8, the

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

Manuel Jacob added the comment: I was running "make all" and I also ran the documentation generator command without an error. However, I tried it again and now it failed the same way as reported. With a debug build, I get "Python/Python-ast.c:231: get_ast_state: Assertion `state != NULL'

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

Petr Viktorin added the comment: That is interesting. The original post here doesn't mention that the problem occurs in Mercurial's "make all", specifically when building documentation, in the command: /usr/bin/python3.9 gendoc.py "hgrc.5" > hgrc.5.txt.tmp Could you confirm `make all`

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

Manuel Jacob added the comment: I couldn’t reproduce the problem with a freshly compiled Python 3.9.0rc1 on Arch Linux. Was this ever reproduced on a non-Red Hat system? -- nosy: +mjacob ___ Python tracker

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

Petr Viktorin added the comment: We need this bug solved for 3.9.0 rc2. Łukasz, you're the one to make the call about the approach; how can we make your job easier? My view is: - Victor's PR 21961 is going in the right direction. But, as far as I know (correct me if I'm wrong), Victor

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

Petr Viktorin added the comment: I also looked into Mercurial. They have a large list of modules [0] that don't work with the lazy-loading scheme. I don't think adding one more should be a big problem for them; but we'll reach out to them and confirm that. [0]

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

STINNER Victor added the comment: At commit ac46eb4ad6662cf6d771b20d8963658b2186c48c ("bpo-38113: Update the Python-ast.c generator to PEP384 (gh-15957)"): --- static struct PyModuleDef _astmodule = { PyModuleDef_HEAD_INIT, "_ast", NULL, sizeof(astmodulestate),

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

Change by Petr Viktorin : -- pull_requests: +21082 pull_request: https://github.com/python/cpython/pull/21973 ___ Python tracker ___

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

Pablo Galindo Salgado added the comment: > Initialized: when dlopen on import. With this I mean import-> dlopen -> dlsym for init function -> call init function -- ___ Python tracker

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

Pablo Galindo Salgado added the comment: > What I'm not yet clear on: when is that shared object is initialized and > destroyed? I am assuming that you mean at the Python level and not at the linker level. Then: * Initialized: when dlopen on import. * Destroyed: never. The interpreter

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

Petr Viktorin added the comment: > Also, option 1 is virtually equivalent to the state of the _ast module prior > to the recent changes except that the symbols are in a shared object instead > of the binary or libpython. The advantage here is not moving them out of the > shared object, is

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

Pablo Galindo Salgado added the comment: Also, adding them into a module that needs access through Python had a bootstrap problem: Basically: initializing import system -> initialize codec -> compile -> ast init -> init ast module ->  -- ___

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

Pablo Galindo Salgado added the comment: Also, option 1 is virtually equivalent to the state of the _ast module prior to the recent changes except that the symbols are in a shared object instead of the binary or libpython. The advantage here is not moving them out of the shared object, is

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

Pablo Galindo Salgado added the comment: >The mod2obj/obj2mod functions, called by e.g. compile(..., PyCF_ONLY_AST), >should: * import the _ast module * call a Python-accessible function, e.g. _ast._mod2obj Would that impact performance considerably? --

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

Petr Viktorin added the comment: Regarding ac46eb4ad6662cf6d771b20d8963658b2186c48c: Module states come and go with the modules that contain them; if a "get_global_ast_state" or "astmodulestate_global" needs to be accessed from outside the module, it shouldn't be module state :/ So,

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

Change by STINNER Victor : -- keywords: +patch pull_requests: +21071 stage: -> patch review pull_request: https://github.com/python/cpython/pull/21961 ___ Python tracker ___

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

Change by Karthikeyan Singaravelan : -- nosy: +lukasz.langa ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

Change by hai shi : -- nosy: +shihai1991 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

Pablo Galindo Salgado added the comment: Given how close are we to a release for 3.9 we should try to maximize stability. -- ___ Python tracker ___

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

Change by Pablo Galindo Salgado : -- nosy: +pablogsal ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

STINNER Victor added the comment: One option is to revert all AST changes of bpo-38113 (and following changes), to move back to the state before bpo-38113, until all issues are addressed. -- ___ Python tracker

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

Change by STINNER Victor : -- nosy: +corona10 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

Change by Batuhan Taskaya : -- nosy: +BTaskaya ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

New submission from STINNER Victor : Building Mercurial with Python 3.9.0rc1 fails with the error: SystemError: returned NULL without setting an error The problem comes from the PyAST_Check() function. This function calls get_global_ast_state() which gets the state of the _ast module. If