[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

Change by STINNER Victor :


--
components: +Subinterpreters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

STINNER Victor  added the comment:

FYI this change broke librepo which calls PyLong_FromLong() without holding the 
GIL. In Python 3.8, "it works". In Python 3.9, it does crash: get_small_int() 
gets a NULL tstate and then dereference a NULL pointer.

librepo bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1788918

IMHO it's a bug in librepo: the GIL must be held to use Python C API.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 630c8df5cf126594f8c1c4579c1888ca80a29d59 by Victor Stinner in 
branch 'master':
bpo-38858: Small integer per interpreter (GH-17315)
https://github.com/python/cpython/commit/630c8df5cf126594f8c1c4579c1888ca80a29d59


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[Eric Snow]

Eric Snow  added the comment:

Thanks for working on this.  It really does have far-reaching benefits, not 
just for the subinterpreter stuff I'm interested in. :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 080ee5a88406fb68aaab741145cd5d2a7c5f2ad6 by Victor Stinner in 
branch 'master':
bpo-38858: Fix ref leak in pycore_interp_init() (GH-17512)
https://github.com/python/cpython/commit/080ee5a88406fb68aaab741145cd5d2a7c5f2ad6


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +16988
pull_request: https://github.com/python/cpython/pull/17512

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

STINNER Victor  added the comment:

Py_NewInterpreter() (new_interpreter() in practice) and Py_InitializeEx() now 
share almost all their code.

Py_EndInterpreter() shares almost all its code with Py_Finalizer(). It's not 
perfect, but it's way better than previously.

Py_NewInterpreter() now isolates more things from the main interpreter. For 
example, builtins and sys modules no longer copy the module dictionary of the 
main interpreter, but create their own dictionary from scratch.

See each commit for the details.

Py_Finalizer() could share more code with Py_EndInterpreter(), but each 
Py_Finalizer() change is really tricky and require to pay a lot attention. The 
Python finalization is really fragile. I started to take notes on this fragile 
code:
https://pythondev.readthedocs.io/finalization.html

I consider that the initial issue is fixed, so I close the issue.

--
nosy: +eric.snow
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

STINNER Victor  added the comment:


New changeset d863ade0c7fa4826e8b71aa467809c83a711f019 by Victor Stinner in 
branch 'master':
bpo-38858: Add pycore_interp_init() code to factorize code (GH-17483)
https://github.com/python/cpython/commit/d863ade0c7fa4826e8b71aa467809c83a711f019


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +16962
pull_request: https://github.com/python/cpython/pull/17483

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 81fe5bd3d78f9bb955f8255404d99df27a31c36a by Victor Stinner in 
branch 'master':
bpo-38858: new_interpreter() reuses _PySys_Create() (GH-17481)
https://github.com/python/cpython/commit/81fe5bd3d78f9bb955f8255404d99df27a31c36a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +16960
pull_request: https://github.com/python/cpython/pull/17481

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 2ec1a1b307cc893adae4662a32e1d2e94b6908e3 by Victor Stinner in 
branch 'master':
bpo-38858: new_interpreter() uses pycore_init_import_warnings() (GH-17353)
https://github.com/python/cpython/commit/2ec1a1b307cc893adae4662a32e1d2e94b6908e3


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +16838
pull_request: https://github.com/python/cpython/pull/17353

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 2582d46fbcf7bdf86b9cf4016850b8d155267ac6 by Victor Stinner in 
branch 'master':
bpo-38858: new_interpreter() reuses pycore_init_builtins() (GH-17351)
https://github.com/python/cpython/commit/2582d46fbcf7bdf86b9cf4016850b8d155267ac6


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +16835
pull_request: https://github.com/python/cpython/pull/17351

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 82c83bd907409c287a5bd0d0f4598f2c0538f34d by Victor Stinner in 
branch 'master':
bpo-38858: _PyImport_FixupExtensionObject() handles subinterpreters (GH-17350)
https://github.com/python/cpython/commit/82c83bd907409c287a5bd0d0f4598f2c0538f34d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +16834
pull_request: https://github.com/python/cpython/pull/17350

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

STINNER Victor  added the comment:


New changeset b00513636c9891deba4cae50217e25e8faf6c6ac by Victor Stinner in 
branch 'master':
bpo-38858: Add init_interp_main() subfunction (GH-17347)
https://github.com/python/cpython/commit/b00513636c9891deba4cae50217e25e8faf6c6ac


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +16831
pull_request: https://github.com/python/cpython/pull/17347

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

STINNER Victor  added the comment:


New changeset e0c9ab8e26d1648b870b80c296b2490a5e9553e5 by Victor Stinner in 
branch 'master':
bpo-38858: Add init_set_builtins_open() subfunction (GH-17346)
https://github.com/python/cpython/commit/e0c9ab8e26d1648b870b80c296b2490a5e9553e5


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +16830
pull_request: https://github.com/python/cpython/pull/17346

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

STINNER Victor  added the comment:

I introduced a workaround to a reference leak in bpo-36854 because 
_PyImport_Cleanup() doesn't clear modules dictionary in some cases:
https://bugs.python.org/issue36854#msg357160

Maybe _PyImport_Cleanup() should be enhanced to ensure that the dictionary of 
all modules is cleared when the function exit.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 3d4833488a173c16446c3f94f58f05e2d13c5dee by Victor Stinner in 
branch 'master':
bpo-38858: Call _PyUnicode_Fini() in Py_EndInterpreter() (GH-17330)
https://github.com/python/cpython/commit/3d4833488a173c16446c3f94f58f05e2d13c5dee


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +16817
pull_request: https://github.com/python/cpython/pull/17330

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +16802
pull_request: https://github.com/python/cpython/pull/17315

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

STINNER Victor  added the comment:

> bpo-38858: Allocate small integers on the heap (GH-17301)

I associated this change to this issue because I had troubles when I tried to 
call _PyLong_Fini() in subinterpreters. I'm now trying to have different small 
integers per interpreter.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 5dcc06f6e0d7b5d6589085692b86c63e35e2325e by Victor Stinner in 
branch 'master':
bpo-38858: Allocate small integers on the heap (GH-17301)
https://github.com/python/cpython/commit/5dcc06f6e0d7b5d6589085692b86c63e35e2325e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +16793
pull_request: https://github.com/python/cpython/pull/17301

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

STINNER Victor  added the comment:


New changeset b93f31fcd9f10b213c614d4944baf9ca2df2016c by Victor Stinner in 
branch 'master':
bpo-38858: Fix Py_Finalize() when called from a subinterpreter (GH-17297)
https://github.com/python/cpython/commit/b93f31fcd9f10b213c614d4944baf9ca2df2016c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +16789
pull_request: https://github.com/python/cpython/pull/17297

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

STINNER Victor  added the comment:


New changeset fff7bbfdb6b7c143b73b6b4b6b40e828c101110c by Victor Stinner in 
branch 'master':
bpo-38858: Add _Py_IsMainInterpreter(tstate) (GH-17293)
https://github.com/python/cpython/commit/fff7bbfdb6b7c143b73b6b4b6b40e828c101110c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +16786
pull_request: https://github.com/python/cpython/pull/17293

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

STINNER Victor  added the comment:


New changeset e7e699e4df73420ddccaa0057cd07ebb3b590b9b by Victor Stinner in 
branch 'master':
bpo-38858: Fix reference leak in pycore_init_types() (GH-17286)
https://github.com/python/cpython/commit/e7e699e4df73420ddccaa0057cd07ebb3b590b9b


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

STINNER Victor  added the comment:

> New changeset ef5aa9af7c7e493402ac62009e4400aed7c3d54e by Victor Stinner in 
> branch 'master':
> bpo-38858: Reorganize pycore_init_types() (GH-17265)

This change introduced a reference leak:

https://buildbot.python.org/all/#builders/80/builds/771

test_atexit leaked [792, 792, 792] references, sum=2376
test_capi leaked [528, 528, 528] references, sum=1584
test__xxsubinterpreters leaked [17954, 17952, 17954] references, sum=53860
test_threading leaked [792, 792, 792] references, sum=2376

It should be fixed by PR 17286.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +16779
pull_request: https://github.com/python/cpython/pull/17286

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 7eee5beaf87be898a679278c480e8dd0df76d351 by Victor Stinner in 
branch 'master':
bpo-38858: Factorize Py_EndInterpreter() code (GH-17273)
https://github.com/python/cpython/commit/7eee5beaf87be898a679278c480e8dd0df76d351


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +16766
pull_request: https://github.com/python/cpython/pull/17273

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

STINNER Victor  added the comment:


New changeset ef5aa9af7c7e493402ac62009e4400aed7c3d54e by Victor Stinner in 
branch 'master':
bpo-38858: Reorganize pycore_init_types() (GH-17265)
https://github.com/python/cpython/commit/ef5aa9af7c7e493402ac62009e4400aed7c3d54e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

Change by STINNER Victor :


--
keywords: +patch
pull_requests: +16757
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/17265

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38858] new_interpreter() should reuse more Py_InitializeFromConfig() code

[STINNER Victor]

New submission from STINNER Victor :

Currently, new_interpreter() is a subset of Py_InitializeFromConfig(): the code 
was duplicated. I would prefer that both functions stay in sync and so that 
new_interpreter() reuses more Py_InitializeFromConfig() code.

--
components: Interpreter Core
messages: 357001
nosy: vstinner
priority: normal
severity: normal
status: open
title: new_interpreter() should reuse more Py_InitializeFromConfig() code
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com