If you run WeeWX on Debian 13 (trixie) and your system installed the
python3.13 update from the 13.6 point release (version 3.13.5-2+deb13u3,
rolling out via unattended-upgrades since July 11), be aware that it can
crash weewxd with a segmentation fault.
What it looks like
------------------
weewxd dies with SIGSEGV -- no Python traceback in the log, the process
just exits. systemd shows "Main process exited, code=dumped,
status=11/SEGV". If you look closely at the timing, the crash comes
milliseconds after WeeWX's periodic garbage collection pass (the
"Garbage collected N objects" INFO line, every 3 hours by default).
weewxd is unusually good at tripping this bug precisely because it calls
gc.collect() on that schedule while its other threads keep working.
I run seven WeeWX machines (Raspberry Pi 5, all on trixie). Four crashes
in the ~48 hours after the update reached them; the crash takes down maybe
1 in 25 GC passes, so it looks random and can hit anywhere from hours to
days after a restart. The crashing thread differs every time (sqlite
reads, report generation, image generation), so don't chase whatever
happens to be in your traceback if you have faulthandler enabled -- it's
the victim, not the cause.
The cause
---------
It is not WeeWX and not any extension. Debian's deb13u3 update backported
a garbage-collector fix into Python 3.13.5 but omitted a second upstream
commit that the fix depends on, so a GC pass can free memory that is still
in use. It's tracked as Debian bug #1141977, the missing commit has been
identified by bisection, and the maintainer is engaged, so I'd expect a
fixed package (deb13u4) reasonably soon:
Am I affected?
--------------
dpkg -l python3.13 | grep deb13u3
Two things must both be true: the deb13u3 package is installed AND weewxd
was (re)started after it was installed. A weewxd that has been running
since before the upgrade still uses the old library and is safe until its
next restart.
What to do
----------
1. If your weewxd predates the upgrade, simply don't restart it until the
fixed package arrives.
2. Otherwise, make crashes self-healing while you wait. A systemd
drop-in (systemctl edit weewx) with:
[Service]
Restart=on-failure
RestartSec=10
turns each crash into ~15 seconds of downtime. I also recommend
Environment=PYTHONFAULTHANDLER=1 there -- it logs a full thread dump
on any future native crash, which is invaluable for problems like
this one.
3. Raising gc_interval in weewx.conf reduces exposure but does not
eliminate it (Python's automatic collections use the same broken
code path), so I wouldn't rely on it.
4. When the fixed python3.13 package arrives, remember that a running
weewxd keeps the old library until it is restarted -- do a
"systemctl restart weewx" after the upgrade to actually get off the
broken version.
Downgrading to deb13u2 also works but re-opens the security fixes that
update carried; with the auto-restart in place I don't think it's worth
it.
John
