Package: python3.13
Version: 3.13.5-2+deb13u5
Severity: important
Tags: patch upstream fixed-upstream trixie
Dear Maintainer,
CPython keeps the frame data stack in 16 KiB chunks. Up to 3.13.12 it
frees a chunk as soon as the last frame in it returns. When a hot call
sits exactly at a chunk boundary, every call maps a new chunk, faults it
in and unmaps it again.
Upstream issue ("Calls across stack chunks perform badly"):
https://github.com/python/cpython/issues/142183
Upstream fixed it by caching one chunk per thread state:
main: https://github.com/python/cpython/pull/145789
3.13: https://github.com/python/cpython/pull/146123
commit 684e7af8cd61bca4914d0ec01b814ff530d5a123, released in
3.13.13
NEWS: "gh-142183: Avoid a pathological case where repeated calls at a
specific stack depth could be significantly slower."
The 3.13 commit changes Python/pystate.c (+26 -4) and
Include/cpython/pystate.h (+2), plus NEWS and Doc/data/python3.13.abi.
Upstream put the new PyThreadState member at the end of the struct, so
the existing layout does not change.
Sid has 3.13.18-1 with the fix; trixie's 3.13.5-2+deb13u5 does not.
I set severity important because ordinary requests of a long-running
application spend a large share of their CPU in the kernel on this, and
the affected loop runs more than 20 times slower; please adjust if you
see it
differently.
How it shows up: a Django application served by uWSGI 2.0.31 has deep
call stacks, so some request handlers loop right at a chunk boundary.
Same x86_64 KVM guest, same code and database, trixie
chroot, both interpreters running pyuwsgi 2.0.31. CPU of one heavy
request, median of three requests without tracing; the mmap counts come
from a separate pass under strace:
trixie python3.13 3.13.5-2+deb13u5: 3880 ms CPU (user 2620, system
1260),
about 349k mmap + 349k munmap of
16 KiB
3.13.18 built from python.org source
in the same chroot: 2360 ms CPU (user 2320, system 40),
78 mmap
The two interpreters also differ in other upstream fixes and in build
options, so not all of the difference is this bug; the reproducer below
isolates it.
The attached chunk-sweep.py (stdlib only) reproduces it: it runs 10000
calls at every extra stack depth from 0 to 159 and prints the depths
with about one
minor page fault per call. The same loop, run by our benchmark with the
trixie /usr/bin/python3.13 in a trixie chroot on two x86_64 KVM guests,
gave:
3.13.5-2+deb13u5, AMD EPYC guest: 3 of 160 depths thrash; worst depth
35.3 ms per 10000 calls against a median of 1.45 ms, 10000 minor
faults, 35.9 ms system time
3.13.5-2+deb13u5, Intel guest: 3 of 160 depths thrash; worst depth
18.6 ms against a median of 0.67 ms, 10000 minor faults
3.13.18, same chroots and guests: 0 of 160; worst depth 1.3 ms (median
1.09 ms) and 0.6 ms (median 0.57 ms), no minor faults
Would you consider the upstream 3.13 commit for a trixie point release?
The same problem exists in bookworm's python3.11, but upstream does not
fix it in 3.11 (security fixes only), so I am not asking for that.
Thanks
Kind regards,
Anton Plotkin
"""Reproducer for CPython gh-142183 (data stack chunk alloc/free thrashing),
stdlib only.
At every extra stack depth from 0 to 159 it runs 10000 calls of a function
whose frame is larger
than the padding frames, and prints the depths where the loop is much slower
and takes about one
minor page fault per call: there every call maps, faults in and unmaps a fresh
16 KiB chunk.
"""
import platform
import resource
import statistics
import sys
import time
CALLS = 10000
DEPTHS = 160
FILLER = tuple(range(24))
def callee():
v0, v1, v2, v3, v4, v5, v6, v7, v8, v9, v10, v11, v12, v13, v14, v15, v16,
v17, v18, v19, v20, v21, v22, v23 = FILLER
return v23
def loop(calls):
for _ in range(calls):
callee()
def at_depth(depth, func, arg):
if depth:
return at_depth(depth - 1, func, arg)
return func(arg)
rows = []
for depth in range(DEPTHS):
before = resource.getrusage(resource.RUSAGE_SELF)
started = time.perf_counter()
at_depth(depth, loop, CALLS)
elapsed = (time.perf_counter() - started) * 1000
after = resource.getrusage(resource.RUSAGE_SELF)
rows.append((elapsed, (after.ru_stime - before.ru_stime) * 1000,
after.ru_minflt - before.ru_minflt, depth))
normal = statistics.median(row[0] for row in rows)
slow = [row for row in rows if row[2] >= CALLS // 2]
print('Python %s on %s %s' % (sys.version.replace('\n', ' '),
platform.system(), platform.machine()))
print('median %.2f ms per %d calls; depths with about one minor fault per call:
%d of %d' % (normal, CALLS, len(slow), DEPTHS))
for elapsed, stime, minflt, depth in slow:
print(' +%3d frames: %7.1f ms (%.0fx median), system %5.1f ms, minor
faults %d' % (depth, elapsed, elapsed / normal, stime, minflt))