[issue44441] Malformed PyImport_Inittab after re-initialization

[kryheb]

kryheb  added the comment:

Thanks vstinner for your help and for taking care of this case!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44441] Malformed PyImport_Inittab after re-initialization

[STINNER Victor]

STINNER Victor  added the comment:

Thanks kryheb for your bug report, reproducer and your fix!

The bug should now be fixed in 3.9, 3.10 and main branches.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
versions: +Python 3.11

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44441] Malformed PyImport_Inittab after re-initialization

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 5ed7827b1620f5b3729bc9767158d24a33863fa9 by Victor Stinner in 
branch '3.9':
bpo-1: _PyImport_Fini2() resets PyImport_Inittab (GH-26874) (GH-26878)
https://github.com/python/cpython/commit/5ed7827b1620f5b3729bc9767158d24a33863fa9


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44441] Malformed PyImport_Inittab after re-initialization

[STINNER Victor]

STINNER Victor  added the comment:


New changeset ece3841d3ddf38875b997eb919488cbb911a66e2 by Victor Stinner in 
branch '3.10':
bpo-1: _PyImport_Fini2() resets PyImport_Inittab (GH-26874) (GH-26877)
https://github.com/python/cpython/commit/ece3841d3ddf38875b997eb919488cbb911a66e2


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44441] Malformed PyImport_Inittab after re-initialization

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +25454
pull_request: https://github.com/python/cpython/pull/26878

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44441] Malformed PyImport_Inittab after re-initialization

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +25453
pull_request: https://github.com/python/cpython/pull/26877

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44441] Malformed PyImport_Inittab after re-initialization

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 489699ca05bed5cfd10e847d8580840812b476cd by Victor Stinner in 
branch 'main':
bpo-1: _PyImport_Fini2() resets PyImport_Inittab (GH-26874)
https://github.com/python/cpython/commit/489699ca05bed5cfd10e847d8580840812b476cd


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44441] Malformed PyImport_Inittab after re-initialization

[STINNER Victor]

STINNER Victor  added the comment:

Oh, I didn't notice your PR 26767 fix! Thanks. I wrote PR 26874 which includes 
the same fix, but adds also a regression test.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44441] Malformed PyImport_Inittab after re-initialization

[STINNER Victor]

STINNER Victor  added the comment:

Thanks for inittab-bug_no-threads.c reproducer. I managed to reproduce the 
issue and I wrote PR 26767 to fix with a regression test.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44441] Malformed PyImport_Inittab after re-initialization

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +25449
pull_request: https://github.com/python/cpython/pull/26874

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44441] Malformed PyImport_Inittab after re-initialization

[kryheb]

kryheb  added the comment:

Hi vstinner,

I apologize for the inaccurate code sample.
Please find a simplified example without threads `inittab-bug_no-threads.c` in 
attachments. Please notice that __main__.py has changed, there is no infinite 
loop anymore.

Steps to reproduce:

1. Append embedded module
2. Initialize Python from config with run_filename
3. Run main
4. Finalize
5. Repeat all above

Observed behavior:
The script is executed at the first iteration, but re-initialization fails with 
a segmentation fault:

gdb) r
Starting program: /home/kheb/proj/tmp/pyc/a.out 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Run Python Script
>>> This is the python script
Python Script completed (0)

Program received signal SIGSEGV, Segmentation fault.
__strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:65
65  VPCMPEQ (%rdi), %ymm0, %ymm1
Missing separate debuginfos, use: dnf debuginfo-install 
libxcrypt-4.4.20-2.fc33.x86_64
(gdb) bt
#0  __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:65
#1  0x77d039f0 in PyUnicode_FromString (
u=0x )
at Objects/unicodeobject.c:2309
#2  0x77dbf049 in list_builtin_module_names () at 
./Python/sysmodule.c:2056
#3  0x77dc1777 in _PySys_InitCore (tstate=tstate@entry=0x422ac0, 
sysdict=sysdict@entry=0x7fffea221070)
at ./Python/sysmodule.c:2813
#4  0x77dc36ef in _PySys_Create (tstate=tstate@entry=0x422ac0, 
sysmod_p=sysmod_p@entry=0x7fffd9d8)
at ./Python/sysmodule.c:3087
#5  0x77da9b32 in pycore_interp_init (tstate=0x422ac0) at 
Python/pylifecycle.c:824
#6  0x77da9c8e in pyinit_config (runtime=runtime@entry=0x77fbe820 
<_PyRuntime>, 
tstate_p=tstate_p@entry=0x7fffdca8, config=config@entry=0x7fffdac0) 
at Python/pylifecycle.c:866
#7  0x77daba3e in pyinit_core (runtime=runtime@entry=0x77fbe820 
<_PyRuntime>, 
src_config=src_config@entry=0x7fffdd40, 
tstate_p=tstate_p@entry=0x7fffdca8) at Python/pylifecycle.c:1029
#8  0x77dabb18 in Py_InitializeFromConfig (config=0x7fffdd40) at 
Python/pylifecycle.c:1214
#9  0x004012d3 in main (argc=1, argv=0x7fffdfc8) at 
inittab-bug_no-threads.c:45

I hope this example helps.

Best regards,
Krystian

--
Added file: https://bugs.python.org/file50125/inittab-bug_no-threads.c

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44441] Malformed PyImport_Inittab after re-initialization

[STINNER Victor]

STINNER Victor  added the comment:

inittab-bug.c uses the Python C API in 3 threads:

* Thread A (init_proc) calls Py_InitializeFromConfig()
* Thread B (run_proc) calls Py_RunMain()
* The main thread (main) calls Py_FinalizeEx()

Problem: the thread B (run_proc) doesn't hold the GIL and so I get a fatal 
error with Python built in debug mode:

$ gcc inittab-bug.c -ggdb -pthread -lpython3.11d -L. -I. -I Include/
$ PYTHONPATH=$PWD/Lib LD_LIBRARY_PATH=$PWD ./a.out
-- Modules: --
#0 'posix'
#1 'errno'
#2 'pwd'
--
Could not find platform independent libraries 
Could not find platform dependent libraries 
Consider setting $PYTHONHOME to [:]
Run Python Script
Fatal Python error: _PyMem_DebugMalloc: Python memory allocator called without 
holding the GIL
Python runtime state: initialized

Thread 0x7f61ce5c1640 (most recent call first):

Abandon (core dumped)


If I remove the thread B (comment the comment), I get a similar error in the 
main thread which calls Py_FinalizeEx().

Please fix your usage of the GIL. For example, you can try to use:

* https://docs.python.org/dev/c-api/init.html#c.PyGILState_Ensure
* https://docs.python.org/dev/c-api/init.html#c.PyGILState_Release

Usually, Python initialization and Python finalization is done in the same 
thread, but another thread can use the Python C API if it acquires the GIL 
using PyGILState_Ensure().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44441] Malformed PyImport_Inittab after re-initialization

[Petr Viktorin]

Change by Petr Viktorin :


--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44441] Malformed PyImport_Inittab after re-initialization

[kryheb]

Change by kryheb :


--
keywords: +patch
pull_requests: +25353
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/26767

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44441] Malformed PyImport_Inittab after re-initialization

[kryheb]

New submission from kryheb :

Hi all,

I observed misbehavior trying to embed the Python interpreter into a C app.
It seems that after re-initialization, PyImport_Inittab is malformed and points 
to the memory freed _PyImport_Fini2.


Steps to reproduce:

1. Append embedded module
2. Initialize Python from config with run_filename
3. Run main with an infinite loop
4. Interrupt script execution with async call
5. Finalize Python
6. Repeat all above

Observed behavior:
The script is executed at first iteration, but re-initialization fails with an 
error:
"
Traceback (most recent call last):
  File "", line 1187, in 
_install_external_importers
ModuleNotFoundError: No module named 'posix'

Error: external importer setup failed
"

Head of modules list at fist run:
-- Modules: --
#0 'posix'
#1 'errno'
#2 'pwd'
--
and after re-initialization:
-- Modules: --
#0 'P ''
#1 'errno'
#2 'pwd'
--

An issue discovered on:
Fedora 33
gcc (GCC) 10.3.1 20210422 (Red Hat 10.3.1-1)
python3-devel.x86_64 3.9.5-2.fc33

Issue still exists on the latest main and on the rc 3.10
Source code to reproduce an issue in attachment.

Best regards,
Krystian Heberlein

--
components: C API, Library (Lib)
files: inittab-bug.c
messages: 395985
nosy: kryheb
priority: normal
severity: normal
status: open
title: Malformed PyImport_Inittab after re-initialization
versions: Python 3.10, Python 3.9
Added file: https://bugs.python.org/file50115/inittab-bug.c

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com