Package: python3.13
Version: 3.13.5-2+deb13u3
Severity: important
Dear Maintainer,
Since upgrading python3.13 from 3.13.5-2+deb13u2 to 3.13.5-2+deb13u3, an
application test suite that uses SQLAlchemy's asyncio ORM (and therefore
greenlet) segfaults reliably. The interpreter dies inside the cyclic GC and
inside pymalloc, at random places, in a different test on every run.
The same code, the same wheels, and the same greenlet version are stable on
upstream CPython 3.13.5 -- the identical upstream version Debian ships -- so
the regression appears to come from the Debian patch set in deb13u3 rather
than from CPython 3.13.5 itself.
Evidence
--------
Each row is a full run of the same test suite (~6850 tests, pytest-xdist -n 30):
Debian 3.13.5-2+deb13u3 5-10 crashed workers per run
Debian 3.13.5-2+deb13u3, gc.disable() clean, 0 crashes
upstream CPython 3.13.5 (identical ver.) clean, 0 crashes
upstream CPython 3.13.12 clean, 0 crashes
Disabling the cyclic garbage collector makes the crashes disappear entirely,
which is what first pointed at the GC.
Backtraces (symbols via debuginfod.debian.net)
----------------------------------------------
Crash 1 -- GC linked list is corrupt. Note the object pointer 0x7dcd499d914f
is not even correctly aligned, so it is not a live object:
#4 _PyGCHead_SET_PREV (gc=..., prev=...) at
../Include/internal/pycore_gc.h:191
#5 _PyObject_GC_UNTRACK (op=0x7dcd499d914f) at
../Include/internal/pycore_object.h:423
#6 listiter_dealloc (self=0x7dcd499d914f) at ../Objects/listobject.c:3950
#7 _PyEval_EvalFrameDefault (...) at ../Python/generated_cases.c.h:2888
Crash 2 and 3 -- pymalloc's free list is corrupt:
#4 pymalloc_alloc (state=..., nbytes=40) at ../Objects/obmalloc.c:2129
#5 _PyObject_Malloc (ctx=..., nbytes=40) at ../Objects/obmalloc.c:2149
#6 PyObject_Malloc (size=40) at ../Objects/obmalloc.c:1316
Other runs died in _PyObject_GC_New and PyObject_GetIter. In every core, the
crashing Python code was running inside a greenlet-switched stack:
#7 greenlet::UserGreenlet::inner_bootstrap (...) at
src/greenlet/greenlet_refs.hpp:245
#8 greenlet::UserGreenlet::g_initialstub (...) at
src/greenlet/TUserGreenlet.cpp:317
#9 greenlet::UserGreenlet::g_switch (...) at
src/greenlet/TUserGreenlet.cpp:185
Suspected cause
---------------
The deb13u3 changelog includes:
- Avoid garbage collecting objects too early when sharing __dict__
(Closes: #1108039)
That is the only GC/object-lifetime change in the u2 -> u3 delta, and the
symptoms are object-lifetime symptoms, so it seems the most likely candidate.
I have not bisected the patch set to confirm this, and I may be wrong about
which patch is responsible -- the load-bearing facts are the four rows in the
table above.
What I ruled out
----------------
- Not application code: the crashes are unchanged with our latest commit
reverted, and were absent before the upgrade.
- Not a stale/ABI-mismatched greenlet: rebuilding greenlet 3.5.3 from source
against the current headers changes nothing. 3.5.3 is the current release.
- Not memory pressure: 61 GB free, no OOM killer activity.
Reproducing it
--------------
I could not reduce this to a small script, which is worth stating plainly.
Neither a long-running server (30k requests, 64-way concurrency, 9 minutes)
nor synthetic greenlet + GC + SQLAlchemy loops (30 parallel processes,
fresh engine and event loop per iteration) crash. It needs the full suite.
The application is open source, so the recipe is:
git clone https://github.com/maziggy/bambuddy
cd bambuddy
python3.13 -m venv venv
./venv/bin/pip install -r requirements.txt -r requirements-dev.txt
cd backend && ../venv/bin/python -m pytest tests/ -n 30 -q
On deb13u3 this produces several "worker crashed" failures within ~75
seconds; the specific tests differ every run. Adding a sitecustomize.py with
gc.disable() on PYTHONPATH makes it pass.
Relevant versions: greenlet 3.5.3, SQLAlchemy 2.0.51, aiosqlite, pytest-xdist.
System
------
Debian 13 (trixie), amd64, kernel 7.0.12-1-pve
python3.13 3.13.5-2+deb13u3
libpython3.13 3.13.5-2+deb13u3
python3-greenlet (from PyPI wheel in venv, 3.5.3)
Thanks for maintaining python3.13.
