[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[Alban Browaeys]

Alban Browaeys  added the comment:

I managed to reproduce the bug.py crash with main branch up to commit 
12c0012cf97d21bc637056983ede0eaf4c0d9c33 .

--
Added file: 
https://bugs.python.org/file50700/bug.py_asyncio_cpustressed-crash.log

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[Alban Browaeys]

Alban Browaeys  added the comment:

bisect of main for bug.py with each local python builds get me to commit 
b127e70a8a682fe869c22ce04c379bd85a00db67 "bpo-46070: Fix asyncio initialization 
guard (GH-30423)" as the one that fixed bug.py most of the time.

At times I can make bug.py segfault be it on python 3.9, 3.10 or main branch. 
It is pretty hard (I can have a batch of 200 runs without an issue) but seems 
easier to reproduce with a CPU stressed, then I can have two segfaults in a 
batch of 50 runs.

Bash:
for i in {1..50}; do ./python  ../python-crash-kodi/bug.py ; done
or sh:
for i in `seq 1 50`; do ./python  ../python-crash-kodi/bug.py ; done

with:
stress -c `nproc --all` at the same time.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[Alban Browaeys]

Alban Browaeys  added the comment:

While bisecting main branch I did not only get segfaults but also exceptions, 
namely:

$ ./python  ../python-crash-kodi/sqlite3_crash.py 
Exception ignored deletion of interned string failed:
Traceback (most recent call last):
  File 
"/home/prahal/Projects/WIP/libreelec/cpython_bisect/Lib/sqlite3/dbapi2.py", 
line 81, in register_adapters_and_converters
register_adapter(datetime.datetime, adapt_datetime)
KeyError: 'isoformat'
Exception ignored deletion of interned string failed:
Traceback (most recent call last):
  File 
"/home/prahal/Projects/WIP/libreelec/cpython_bisect/Lib/sqlite3/dbapi2.py", 
line 83, in register_adapters_and_converters
register_converter("timestamp", convert_timestamp)
KeyError: 'timepart_full'
Exception ignored deletion of interned string failed:
Traceback (most recent call last):
  File 
"/home/prahal/Projects/WIP/libreelec/cpython_bisect/Lib/sqlite3/dbapi2.py", 
line 83, in register_adapters_and_converters
register_converter("timestamp", convert_timestamp)
KeyError: 'day'
Exception ignored deletion of interned string failed:
Traceback (most recent call last):
  File 
"/home/prahal/Projects/WIP/libreelec/cpython_bisect/Lib/sqlite3/dbapi2.py", 
line 83, in register_adapters_and_converters
register_converter("timestamp", convert_timestamp)
KeyError: 'month'
Exception ignored deletion of interned string failed:
Traceback (most recent call last):
  File 
"/home/prahal/Projects/WIP/libreelec/cpython_bisect/Lib/sqlite3/dbapi2.py", 
line 83, in register_adapters_and_converters
register_converter("timestamp", convert_timestamp)
KeyError: 'year'
Exception ignored deletion of interned string failed:
Traceback (most recent call last):
  File 
"/home/prahal/Projects/WIP/libreelec/cpython_bisect/Lib/sqlite3/dbapi2.py", 
line 83, in register_adapters_and_converters
register_converter("timestamp", convert_timestamp)
KeyError: 'timepart'
Exception ignored deletion of interned string failed:
Traceback (most recent call last):
  File 
"/home/prahal/Projects/WIP/libreelec/cpython_bisect/Lib/sqlite3/dbapi2.py", 
line 83, in register_adapters_and_converters
register_converter("timestamp", convert_timestamp)
KeyError: 'datepart'
Exception ignored deletion of interned string failed:
Traceback (most recent call last):
  File "", line 688, in _load_unlocked
KeyError: 'convert_timestamp'
Exception ignored deletion of interned string failed:
Traceback (most recent call last):
  File "", line 688, in _load_unlocked
KeyError: 'convert_date'
Exception ignored deletion of interned string failed:
Traceback (most recent call last):
  File "", line 688, in _load_unlocked
KeyError: 'adapt_datetime'
Exception ignored deletion of interned string failed:
Traceback (most recent call last):
  File "", line 688, in _load_unlocked
KeyError: 'adapt_date'


The 3.10 branch bisect pointed to one commit that fix the crash after 3.10.1 
which is 72c260cf0c71eb01eb13100b751e9d5007d00b70 [3.10] bpo-46006: Revert 
"bpo-40521: Per-interpreter interned strings (GH-20085)" (GH-30422) (GH-30425) 
which makes sense regarding main branch logs. It is commit 
35d6540c904ef07b8602ff014e520603f84b5886 in the main branch.

What remains to be seen is why "bpo-46070: _PyGC_Fini() untracks objects 
(GH-30577)" looks fine in the main branch (though it has no effect on the 
import crash) and not in 3.9 and 3.10 branch.
Mind in the main branch "bpo-46006: Revert "bpo-40521: Per-interpreter interned 
strings (GH-20085)" (GH-30422)" was already applied when "bpo-46070: 
_PyGC_Fini() untracks objects (GH-30577)" went in so it was also unrelated to 
the fix of the initial report.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[Alban Browaeys]

Alban Browaeys  added the comment:

3.10 branch is fixed if I revert e6bb17fe29713368e1fd93d9ac9611017c4f570c 
bpo-46070: _PyGC_Fini() untracks objects (GH-30577). Be it if I revert it atop 
current head 9006b4471cc4d74d0cbf782d841a330b0f46a3d0 or if I test the commit 
before e6bb17fe29713368e1fd93d9ac9611017c4f570c was merged.

As this made no sense with regards to this bug report history that this fix 
broke the branch, I tried v3.10.1 which lacks this fix. Vanilla is broken too. 
Also applying the "_PyGC_Fini untracks objects" upon 3.10.1 does not fix the 
test case crash.

I am puzzled. Will try to bisect the commit that fixed the testcase in the 3.10 
branch before "_PyGC_Fini untracks objects" was merged and after 3.10.1.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[Alban Browaeys]

Alban Browaeys  added the comment:

I did 3.9 branch bisect and commit 52937c26adc35350ca0402070160cf6dc838f359 
bpo-46070: _PyGC_Fini() untracks objects (GH-30577) (GH-30580) is the one that 
broke 3.9 . While with my main branch builds this commit is fine, it is not for 
the 3.9 branches.

Proceeding with 3.10 branch bisect of first bad commit and redoing main branch 
first good commit.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[Alban Browaeys]

Alban Browaeys  added the comment:

By "It is fixed in main branch commit 12c0012cf97d21bc637056983ede0eaf4c0d9c33 
." I mean that this commit is good not that this commit fixes the issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[Alban Browaeys]

Alban Browaeys  added the comment:

sqlite3_crash.py does not crashes on python 3.9 below and equal 3.9.9 and 
python main branch 12c0012cf97d21bc637056983ede0eaf4c0d9c33. I confirm it 
crashes on python 3.9.10, 3.9.11, 3.10 branch commit 
9006b4471cc4d74d0cbf782d841a330b0f46a3d0 .
It is fixed in main branch commit 12c0012cf97d21bc637056983ede0eaf4c0d9c33 .

Currently bisecting both 3.9.9 to 3.9.10 and 3.10 to 3.11 main branch for bad 
to good.

The patches in this bug report are already merged in the 3.10 branch which 
crash.

I cannot reproduce win_py399_crash_reproducer.py which I used as a basis for 
this test case.
The backtrace is the same as the ones from the crashes of the kodi addons (me 
Jellyfin Kodi addon), which is the initial report .
This looks like importing sqlite3 in threads plays badly.

I can reproduce on aarch64 (Odroid C2) LibreElec and builds of cpython on 
Debian stable x86_64 (the extensive testing of the broken interpreters is done 
on x86_64 Debian stable bullseye with a cpython clone and running from 
builddir).

--
nosy: +prahal
Added file: https://bugs.python.org/file50699/sqlite3_crash.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[STINNER Victor]

Change by STINNER Victor :


--
Removed message: https://bugs.python.org/msg411050

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[STINNER Victor]

STINNER Victor  added the comment:

Another measure using the command:

PYTHONHASHSEED=0 ./python -X showrefcount -c pass

I had to run the command 20 times to get a stable value, I don't know why.

main branch: [21981 refs, 5716 blocks]
PR: [21887 refs, 5667 blocks]

=> the PR removes 94 references and 49 memory blocks.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[jokot3]

Change by jokot3 :


--
nosy: +jokot3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[STINNER Victor]

STINNER Victor  added the comment:

I created bpo-46368: "faulthandler: add the ability to dump all interpreters, 
not only the current interpreter".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[STINNER Victor]

STINNER Victor  added the comment:

pyobject_ob_interp.patch: Quick & dirty patch that I wrote to add 
PyObject.ob_interp, store in which interpreter an object has been created.

--
Added file: https://bugs.python.org/file50560/pyobject_ob_interp.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[STINNER Victor]

STINNER Victor  added the comment:

Victor:
> (*) I made the GC state per-interpreter: commit 
> 7247407c35330f3f6292f1d40606b7ba6afd5700 (Nov 20, 2019)

Eric Snow:
> FYI, this was done by me in an earlier comment which we ended up
reverting.  Later you basically un.reverted that.

Well, I recall that your change had to be reverted 2 or 3 times because there 
were many crashes on FreeBSD, and no one understood why it crashed. The root 
cause was bugs related to the GIL and daemon threads. It took me a while (and 
multiple commits) to identify and fix all of them:
https://vstinner.github.io/gil-bugfixes-daemon-threads-python39.html

I decided to split your work into smaller changes to better debug these 
crashes. bpo-36854 contains a few changes, but these changes are based on work 
that I pushed earlier.

For example, there was a tricky bug related to clearing a Python thread state:
https://github.com/python/cpython/commit/9da7430675ceaeae5abeb9c9f7cd552b71b3a93a

Also, once the GC was made per interpreter, we started to discover more and 
more tricky reference leaks:
https://vstinner.github.io/subinterpreter-leaks.html

I spent a significant time to reorder code of Py_Finalize() and 
Py_EndInterpreter() to clear objects earlier or in a different order. Recently, 
I made sure that the free lists can no longer be used after they are cleared. 
It took some notes at:
https://pythondev.readthedocs.io/finalization.html

One of the hardest fix was the commit 9ad58acbe8b90b4d0f2d2e139e38bb5aa32b7fb6 
of bpo-19466. To make this change, first I had to fix a very old bug of 
PyThreadState_Clear() with commit 5804f878e779712e803be927ca8a6df389d82cdf 
(bpo-20526).

Well, it was a long journey and it's not done yet :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[STINNER Victor]

STINNER Victor  added the comment:

It would be nice to add some tests.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 52937c26adc35350ca0402070160cf6dc838f359 by Victor Stinner in 
branch '3.9':
bpo-46070: _PyGC_Fini() untracks objects (GH-30577) (GH-30580)
https://github.com/python/cpython/commit/52937c26adc35350ca0402070160cf6dc838f359


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[miss-islington]

miss-islington  added the comment:


New changeset e6bb17fe29713368e1fd93d9ac9611017c4f570c by Miss Islington (bot) 
in branch '3.10':
bpo-46070: _PyGC_Fini() untracks objects (GH-30577)
https://github.com/python/cpython/commit/e6bb17fe29713368e1fd93d9ac9611017c4f570c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[STINNER Victor]

STINNER Victor  added the comment:

I tested manually my fix GH-30580 using:

* (1) attached  win_py399_crash_reproducer.py
* (2) https://bugs.python.org/issue46070#msg410447 mthod

Without my fix, I can easily reproduce the crash with (1) and (2).

With my fix, I can no longer reproduce the crash with (1) or (2).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +28779
pull_request: https://github.com/python/cpython/pull/30580

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[miss-islington]

Change by miss-islington :


--
pull_requests: +28778
pull_request: https://github.com/python/cpython/pull/30579

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[miss-islington]

Change by miss-islington :


--
pull_requests: +28777
pull_request: https://github.com/python/cpython/pull/30578

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 1a4d1c1c9b08e75e88aeac90901920938f649832 by Victor Stinner in 
branch 'main':
bpo-46070: _PyGC_Fini() untracks objects (GH-30577)
https://github.com/python/cpython/commit/1a4d1c1c9b08e75e88aeac90901920938f649832


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[Eric Snow]

Eric Snow  added the comment:

> (*) I made the GC state per-interpreter: commit 
> 7247407c35330f3f6292f1d40606b7ba6afd5700 (Nov 20, 2019)

FYI, this was done by me in an earlier comment which we ended up
reverting.  Later you basically un.reverted that.

> The bug is that a C function object (_sre.compile) is created in an 
> interpreter, tracked by the GC list of this interpreter, and then it is 
> destroye and untracked in another interpreter.

FWIW, at one point I had a branch that supported sharing read-only
Py_Buffer data.  When the receiving interpreter was done with it I'd
call Py_AddPendingCall() to schedule the cleanup in the "owner"
interpreter.  However, this only worked because I kept track of the
owner.  Adding that pointer to every object wouldn't be feasible but I
suppose there are other things we could do that wouldn't be super
inefficient, like don't worry about it for the main interpreter, use a
hash table (Victor's idea), borrow some of the bits of the PyObject
head to store a flag or even an index into an array (if there are only
a few interpreters), or even make the allocator per-interpreter and
then extrapolate the interpreter based on the object's address.

Regardless, it is still much simpler to make all objects per-interpreter.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[STINNER Victor]

STINNER Victor  added the comment:

This issue has a complex history.

(*) I made the GC state per-interpreter: commit 
7247407c35330f3f6292f1d40606b7ba6afd5700 (Nov 20, 2019)

(*) This change triggered a _PyImport_FixupExtensionObject() bug in 
sub-interpreter, I fixed it with commit 
82c83bd907409c287a5bd0d0f4598f2c0538f34d (Nov 22, 2019)

(*) My _PyImport_FixupExtensionObject() fix introduced bpo-44050 regression, it 
was fixed by commit b9bb74871b27d9226df2dd3fce9d42bda8b43c2b (Oct 5, 2021)

(*) A race condition in the _asyncio extension has been identified and fixed by 
the commit b127e70a8a682fe869c22ce04c379bd85a00db67 (Jan 7, 2021)

(*) I identified a race condition introduced by the per-interpreter GC state 
cahnge: I proposed GH-30577 to fix it.


So far, the GC race condition has only been reproduced on Windows with Python 
3.9 and the _sre exception. On Python 3.10 and newer, it's harder to reproduce 
the crash using stdlib extensions since many of them have been ported to the 
multi-phase initializatioin API.

The GC race condition involves dangling pointers and depends on the memory 
allocator and when GC collections are triggered.

The bug is that a C function object (_sre.compile) is created in an 
interpreter, tracked by the GC list of this interpreter, and then it is 
destroye and untracked in another interpreter. The problem is that the object 
is untracked after the GC list has been destroyed and so "prev" and "next" 
objects of the PyGC_Head structure *can* become dangling pointers.

It's unclear to me what are the "prev" and "next" objects of the C function 
causing the crash (_sre.compile). At least, it seems like it's also used by 
more than one interpreter: it should *not* be done, see bpo-40533.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[STINNER Victor]

STINNER Victor  added the comment:

Oh. I managed to write a simple fix which doesn't require to revert the whole 
"per-interpreter GC" change: GH-30577.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +28776
pull_request: https://github.com/python/cpython/pull/30577

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[STINNER Victor]

STINNER Victor  added the comment:

When the crash occurs, the _sre.compile function is not destroyed in the 
interpreter which created the function.



The crash is related to _sre.compile method. This method is created in 
PyInit__sre() called by "import _sre".

On Windows, the _sre module is not imported at startup. So it's imported first 
in a subinterpreter.

In Python 3.9, the _sre module doesn't use the multiphase initialization API 
and PyModuleDef.m_size = -1. When the module is imported, 
_PyImport_FixupExtensionObject() copies the module dictionary into 
PyModuleDef.m_copy.

In Py_Finalize() and Py_EndInterpreter(), _PyImport_Cleanup() does two things:

* (1) set _sre.__dict__['compile'] to None -> kill the first reference to the 
function
* (2) call _PyInterpreterState_ClearModules() which does 
Py_CLEAR(def->m_base.m_copy), clear the cached copy of the _sre module dict -> 
kill the second reference

I modified Python to add an "ob_interp" member to PyObject to log in which 
interpreter an object is created. I also modified meth_dealloc() to log when 
_sre.compile function is deleted.

Extract of the reformatted output to see what's going on:
---
(...)

(1)
fixup: COPY _sre ModuleDef copy: def=7FFF19209810 interp=01EC1846F2A0

(2)
import: UPDATE(_sre ModuleDef copy): interp=01EC184AB790

(3)
_PyImport_Cleanup: interp=01EC1846F2A0
_PyInterpreterState_ClearModules: PY_CLEAR _sre ModuleDef m_copy: 
def=7FFF19209810 interp=01EC1846F2A0

(4)
_PyImport_Cleanup: interp=01EC184AB790
meth_dealloc(compile): m->ob_interp=01EC1846F2A0, 
interp=01EC184AB790

Windows fatal exception: access violation
(...)
---

Steps:

* (1)

  * interpreter #1 (01EC1846F2A0) creates the _sre.compile function
  * interpreter #1 (01EC1846F2A0) copies _sre module dict into 
PyModuleDef.m_copy
  * at this point, _sre.compile should have 2 references

* (2)

  * interpreter #2 (01EC184AB790) imports _sre: it creates a new module 
object and copies the function from PyModuleDef.m_copy
  * at this point, _sre.compile should have 3 references

* (3)

  * interpreter #1 exit: Py_EndInterpreter() calls _PyImport_Cleanup()
  * at this point, _sre.compile should have 1 reference

* (4)

  * interpreter #2 exit: Py_EndInterpreter() calls _PyImport_Cleanup()
  * the last reference to _sre.compile is deleted: 0 reference
  * meth_dealloc() is called

The first problem is that the function was created in the interpreter #1 but 
deleted in the interpreter #2.

The second problem is that the function is tracked by the GC and it is part of 
the GC list of the interpreter #1. When the interpreter #2 destroys the 
function, the GC list of interpreter #1 is already freed: PyGC_Head contains 
dangling pointers.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[STINNER Victor]

STINNER Victor  added the comment:

I wrote 3 scripts to reproduce the bug in a more reliable way. So I just have 
to type "bisect" and it runs the test 12 times.

(1) bisect.bat:
---
@"C:\vstinner\python\3.9\PCbuild\amd64\python_d.exe" bisect.py
---


(2) bisect.py:
---
import subprocess
import os
import sys

BISECT = False

def run(*args):
print("+ ", ' '.join(args))
env = dict(os.environ)
env['PYTHONFAULTHANDLER'] = '1'
proc = subprocess.run(args, env=env)
exitcode = proc.returncode
if exitcode:
print()
print(f"COMMAND FAILED: {exitcode}")
if BISECT:
print()
print("type: git bisect bad")
sys.exit(exitcode)

python = sys.executable
#script = "win_py399_crash_reproducer.py"
script = "bug.py"
nrun = 12
for i in range(1, nrun+1):
print(f"Run #{i}/{nrun}")
if i % 2:
run(python, script)
else:
run(python, "-X", "dev", script)

if BISECT:
print()
print("Not reproduced")
print()
run("git", "checkout", ".")
run("git", "bisect", "good")
---


(3) win_py399_crash_reproducer.py (import "_sre"):
---
# When this program is run on windows using python 3.9.9 it crashes about 50%
# of the time.

import _testcapi
import threading

code = """
import _sre
print("exit subinterpreter")
"""

def doIt():
_testcapi.run_in_subinterp(code)

tt=[]

for i in range(16):
t = threading.Thread(target=doIt)
t.start()
tt.append(t)

for t in tt:
t.join()
print("exit main")
---


Example:
---
vstinner@DESKTOP-DK7VBIL C:\vstinner\python\3.9>bisect
Run #1/12
+  C:\vstinner\python\3.9\PCbuild\amd64\python_d.exe bug.py
Run #2/12
+  C:\vstinner\python\3.9\PCbuild\amd64\python_d.exe -X dev bug.py
Run #3/12
+  C:\vstinner\python\3.9\PCbuild\amd64\python_d.exe bug.py
Windows fatal exception: access violation
(...)
Current thread 0x0420 (most recent call first):
  File "C:\vstinner\python\3.9\bug.py", line 13 in doIt
  File "C:\vstinner\python\3.9\lib\threading.py", line 910 in run
  File "C:\vstinner\python\3.9\lib\threading.py", line 973 in _bootstrap_inner
  File "C:\vstinner\python\3.9\lib\threading.py", line 930 in _bootstrap
(...)
COMMAND FAILED: 3221225477
---

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[STINNER Victor]

STINNER Victor  added the comment:

I modified PR 30565 (3.10) and PR 30566 (3.9) to fix the ABI. I added 
_PyGC_GetState() which always use PyInterpreterState.gc of the main interpreter.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[Eric Snow]

Eric Snow  added the comment:

> adding a new "gc" member in the _PyRuntimeState structure also causes the ABI 
> CI check to fail.

What if you move it to the end of the struct?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[STINNER Victor]

STINNER Victor  added the comment:

I prepared 3 pull requests to revert the commit 
7247407c35330f3f6292f1d40606b7ba6afd5700:

* PR 30564: main branch
* PR 30565: 3.10 branch
* PR 30566: 3.9 branch

The problem is that the "Check if the ABI has changed" CI job fails in 3.9 and 
3.10 branches.

I recently had the issue for a different revert in bpo-46006: I decided to keep 
the "removed" member, and mark it as "unused". See the commit 
72c260cf0c71eb01eb13100b751e9d5007d00b70 in the 3.10 branch:

struct _Py_unicode_state {
(...)

-PyObject *interned;

+// Unused member kept for ABI backward compatibility with Python 3.10.0:
+// see bpo-46006.
+PyObject *unused_interned;

(...)
}


I can keep the "gc" member in PyInterpreterState, but adding a new "gc" member 
in the _PyRuntimeState structure also causes the ABI CI check to fail.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +28769
pull_request: https://github.com/python/cpython/pull/30566

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +28768
pull_request: https://github.com/python/cpython/pull/30565

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +28767
pull_request: https://github.com/python/cpython/pull/30564

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[STINNER Victor]

STINNER Victor  added the comment:

This bug is hard to reproduce for different reasons:

* It occurs randomly: I need between 1 and 50 attempts to reproduce the bug 
using win_py399_crash_reproducer.py

* So far, the bug was only reproduced on Windows.

* I failed to reproduce the crash on Linux. I tried PYTHONMALLOC=malloc and 
PYTHONMALLOC=malloc_debug with and without 
LD_PRELOAD=/usr/lib64/libjemalloc.so.2 (jemalloc memory allocator).

* The _sre extension has been converted to multi-phase init in Python 3.10. 
"import _sre" is no longer enough to reproduce the crash on Python 3.10 and 
newer.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)

[STINNER Victor]

Change by STINNER Victor :


--
title: [subinterpreters] asyncio crash when importing _asyncio in 
subinterpreter (Python 3.8 regression) -> [subinterpreters] crash when 
importing _sre in subinterpreters in parallel (Python 3.9 regression)

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com