Package: python3.13
Version: 3.13.5-2+deb13u5
Severity: important
Tags: patch upstream fixed-upstream trixie

Dear Maintainer,

CPython keeps the frame data stack in 16 KiB chunks. Up to 3.13.12 it frees a chunk as soon as the last frame in it returns. When a hot call sits exactly at a chunk boundary, every call maps a new chunk, faults it in and unmaps it again.

Upstream issue ("Calls across stack chunks perform badly"):

https://github.com/python/cpython/issues/142183

Upstream fixed it by caching one chunk per thread state:

  main: https://github.com/python/cpython/pull/145789
  3.13: https://github.com/python/cpython/pull/146123
        commit 684e7af8cd61bca4914d0ec01b814ff530d5a123, released in 3.13.13
  NEWS: "gh-142183: Avoid a pathological case where repeated calls at a
        specific stack depth could be significantly slower."

The 3.13 commit changes Python/pystate.c (+26 -4) and Include/cpython/pystate.h (+2), plus NEWS and Doc/data/python3.13.abi.

Upstream put the new PyThreadState member at the end of the struct, so the existing layout does not change.

Sid has 3.13.18-1 with the fix; trixie's 3.13.5-2+deb13u5 does not.

I set severity important because ordinary requests of a long-running application spend a large share of their CPU in the kernel on this, and the affected loop runs more than 20 times slower; please adjust if you see it
differently.

How it shows up: a Django application served by uWSGI 2.0.31 has deep call stacks, so some request handlers loop right at a chunk boundary. Same x86_64 KVM guest, same code and database, trixie chroot, both interpreters running pyuwsgi 2.0.31. CPU of one heavy request, median of three requests without tracing; the mmap counts come from a separate pass under strace:

  trixie python3.13 3.13.5-2+deb13u5:  3880 ms CPU (user 2620, system 1260),                                        about 349k mmap + 349k munmap of 16 KiB
  3.13.18 built from python.org source
  in the same chroot:                  2360 ms CPU (user 2320, system 40),
                                       78 mmap

The two interpreters also differ in other upstream fixes and in build options, so not all of the difference is this bug; the reproducer below isolates it.

The attached chunk-sweep.py (stdlib only) reproduces it: it runs 10000 calls at every extra stack depth from 0 to 159 and prints the depths with about one minor page fault per call. The same loop, run by our benchmark with the trixie /usr/bin/python3.13 in a trixie chroot on two x86_64 KVM guests, gave:

  3.13.5-2+deb13u5, AMD EPYC guest: 3 of 160 depths thrash; worst depth
      35.3 ms per 10000 calls against a median of 1.45 ms, 10000 minor
      faults, 35.9 ms system time
  3.13.5-2+deb13u5, Intel guest:    3 of 160 depths thrash; worst depth
      18.6 ms against a median of 0.67 ms, 10000 minor faults
  3.13.18, same chroots and guests: 0 of 160; worst depth 1.3 ms (median
      1.09 ms) and 0.6 ms (median 0.57 ms), no minor faults

Would you consider the upstream 3.13 commit for a trixie point release? The same problem exists in bookworm's python3.11, but upstream does not fix it in 3.11 (security fixes only), so I am not asking for that.

Thanks

Kind regards,

Anton Plotkin
"""Reproducer for CPython gh-142183 (data stack chunk alloc/free thrashing), 
stdlib only.

At every extra stack depth from 0 to 159 it runs 10000 calls of a function 
whose frame is larger
than the padding frames, and prints the depths where the loop is much slower 
and takes about one
minor page fault per call: there every call maps, faults in and unmaps a fresh 
16 KiB chunk.
"""
import platform
import resource
import statistics
import sys
import time

CALLS = 10000
DEPTHS = 160
FILLER = tuple(range(24))


def callee():
    v0, v1, v2, v3, v4, v5, v6, v7, v8, v9, v10, v11, v12, v13, v14, v15, v16, 
v17, v18, v19, v20, v21, v22, v23 = FILLER
    return v23


def loop(calls):
    for _ in range(calls):
        callee()


def at_depth(depth, func, arg):
    if depth:
        return at_depth(depth - 1, func, arg)
    return func(arg)


rows = []
for depth in range(DEPTHS):
    before = resource.getrusage(resource.RUSAGE_SELF)
    started = time.perf_counter()
    at_depth(depth, loop, CALLS)
    elapsed = (time.perf_counter() - started) * 1000
    after = resource.getrusage(resource.RUSAGE_SELF)
    rows.append((elapsed, (after.ru_stime - before.ru_stime) * 1000, 
after.ru_minflt - before.ru_minflt, depth))

normal = statistics.median(row[0] for row in rows)
slow = [row for row in rows if row[2] >= CALLS // 2]
print('Python %s on %s %s' % (sys.version.replace('\n', ' '), 
platform.system(), platform.machine()))
print('median %.2f ms per %d calls; depths with about one minor fault per call: 
%d of %d' % (normal, CALLS, len(slow), DEPTHS))
for elapsed, stime, minflt, depth in slow:
    print('  +%3d frames: %7.1f ms (%.0fx median), system %5.1f ms, minor 
faults %d' % (depth, elapsed, elapsed / normal, stime, minflt))

Reply via email to