Re: [weewx-user] Heads up: Debian 13.6 python3.13 update (3.13.5-2+deb13u3) causes random weewxd segfaults

['John Kline' via weewx-user]

Good news: this is fixed.  Debian has released python3.13 3.13.5-2+deb13u4 ("Fix use-after-free in dict.clear() with embedded values.  Resolves a regression in 3.13.5-2+deb13u3").  It reached trixie-updates today. ‘sudo apt update && sudo apt upgrade’ gets you the fix. The breakage showed up all over the archive -- the original report was SQLAlchemy/greenlet test suites segfaulting, and five packages simply stopped building because their build-time test suites crashed ("FTBFS" = fails to build from source):   #1141977  python3.13: segfaults with sqlalchemy/greenlet (original)   #1142056  python3.13: heap corruption (SIGSEGV in cyclic GC)   #1142104  flask-migrate: FTBFS: test failure   #1142105  neutron: FTBFS: test failure   #1142106  python-oslo.cache: FTBFS: segmentation fault   #1142107  qtwebengine-opensource-src: FTBFS: segmentation fault   #1142110  tahoe-lafs: FTBFS: segmentation fault They needed to get this out quickly—and they did. John On Jul 17, 2026, at 5:41 PM, John Kline <j...@johnkline.com> wrote:  It will definitely restart and will run for a while.  How long is not clear.  Perhaps a few days,  Perhaps a week or longer.  Perhaps you won’t see it at all. BTW, they were chasing another bug (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1142056): > Some pipeline jobs failed, because python3.13 interpreter segfaults. This is > a different issue, and it looks more like a bug in python3.13, not in sphinx. Hi. This is probably the python3.13 bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1141977 and at this point I really wonder if there was a bug in sphinx at all (in which case we would not need an upload for trixie at all). That is when they bumped our bug from important to serious.  It is now marked as fixed, but I’m still waiting to hear when they will push it. On Jul 17, 2026, at 4:25 PM, Greg from Oz <ubeau...@gmail.com> wrote: If the system does restart what will happen to weewx? Will it still run for a while and then just stop? On Friday, 17 July 2026 at 08:52:29 UTC+10 Greg from Oz wrote: Thanks John. I have read all of that on the link you gave me but it makes no sense to me at all as I am no programmer. I just use the software. I guess I will wait for the fix to appear.  On Friday, 17 July 2026 at 03:20:24 UTC+10 Vince Skahan wrote: I know John strongly disagrees but this is another reason why I do not do ‘any’ package updates on my LAN-only weewx system. I get it working and leave it alone.  Granted my risk profile permits this for me. I permit ‘no’ incoming traffic to my LAN that the weewx system is on, Running in docker is another way to avoid these kinds of ‘sand shifted under your feet’ problems, making it easier to revert to a previously stable setup. (but wow quite the elaborate investigation figuring out all the gory internals things for this case) On Thursday, July 16, 2026 at 8:13:32 AM UTC-7 John Kline wrote: Hi Greg, I pinged the maintainer (Stefano) on the bug.  Stefano already has a test build of 3.13.5-2+deb13u4.  See his reply here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1141977 for a link to it (if you want to install it).  I have it running on 7 machines and will do the rest after the first weewx gc run in about 2 and 1/2 hours. John On Jul 15, 2026, at 6:48 PM, John Kline <jo...@johnkline.com> wrote:  Well, you can from a Debian snapshot, but I think if I listed instructions on how to do that, I dig myself quite a hole in having to support the process; and, I suspect, some will never undo putting the packages on hold, thus, never get a python upgrade again. On Jul 15, 2026, at 4:05 PM, Greg from Oz <ubea...@gmail.com> wrote: apt-cache policy python3.13 python3.13:   Installed: 3.13.5-2+deb13u3   Candidate: 3.13.5-2+deb13u3   Version table:  *** 3.13.5-2+deb13u3 500         500 http://deb.debian.org/debian trixie/main amd64 Packages         100 /var/lib/dpkg/status Looks like I cannot downgrade...I will just have to now restart the weewx. I do have weewx set up in monit and if it does crash in theory it should restart again. On Wednesday, 15 July 2026 at 23:05:03 UTC+10 John Kline wrote: I believe the maintainers have what they need to push a fix.  I’m hoping that will happen in a week or two. >> At least if it doesn't start I will know what to do It will start just fine.  The issue is, each time weewx requests garbage collection (which, by default, weewx does every 3 hours), there is a small chance of a segment violation soon after.  As such, the symptom will be that weewx stops running for no obvious reason (and nothing in the log after a garbage collection message). It is hard to say how often this might happen for a single instance of weewx running.  The frequency may depend on what extensions you are running (and how many of them).  It is conceivable that some won’t see the issue at all before the fix arrives. I was seeing the issue too often (as I run quite a few weewx instances);  so I resorted to downgrading the python packages and put them on hold while waiting for the update. John On Jul 15, 2026, at 2:53 AM, Greg from Oz <ubea...@gmail.com> wrote:  So when does the new fix arrive? I upgraded the python 2 days ago and weewx has been running for 4 days....I better not restart weewx. At least if it doesn't start I will know what to do...Thanks for the heads up. On Tuesday, 14 July 2026 at 08:17:42 UTC+10 Tom Keffer wrote: Thanks, John, for your detailed report. On Mon, Jul 13, 2026 at 8:09 AM 'John Kline' via weewx-user <weewx...@googlegroups.com> wrote: If you run WeeWX on Debian 13 (trixie) and your system installed the python3.13 update from the 13.6 point release (version 3.13.5-2+deb13u3, rolling out via unattended-upgrades since July 11), be aware that it can crash weewxd with a segmentation fault. What it looks like ------------------ weewxd dies with SIGSEGV -- no Python traceback in the log, the process just exits.  systemd shows "Main process exited, code=dumped, status=11/SEGV".  If you look closely at the timing, the crash comes milliseconds after WeeWX's periodic garbage collection pass (the "Garbage collected N objects" INFO line, every 3 hours by default). weewxd is unusually good at tripping this bug precisely because it calls gc.collect() on that schedule while its other threads keep working. I run seven WeeWX machines (Raspberry Pi 5, all on trixie).  Four crashes in the ~48 hours after the update reached them; the crash takes down maybe 1 in 25 GC passes, so it looks random and can hit anywhere from hours to days after a restart.  The crashing thread differs every time (sqlite reads, report generation, image generation), so don't chase whatever happens to be in your traceback if you have faulthandler enabled -- it's the victim, not the cause. The cause --------- It is not WeeWX and not any extension.  Debian's deb13u3 update backported a garbage-collector fix into Python 3.13.5 but omitted a second upstream commit that the fix depends on, so a GC pass can free memory that is still in use.  It's tracked as Debian bug #1141977, the missing commit has been identified by bisection, and the maintainer is engaged, so I'd expect a fixed package (deb13u4) reasonably soon:   https://bugs.debian.org/1141977 Am I affected? --------------   dpkg -l python3.13 | grep deb13u3 Two things must both be true: the deb13u3 package is installed AND weewxd was (re)started after it was installed.  A weewxd that has been running since before the upgrade still uses the old library and is safe until its next restart. What to do ---------- 1. If your weewxd predates the upgrade, simply don't restart it until the    fixed package arrives. 2. Otherwise, make crashes self-healing while you wait.  A systemd    drop-in (systemctl edit weewx) with:      [Service]      Restart=on-failure      RestartSec=10    turns each crash into ~15 seconds of downtime.  I also recommend    Environment=PYTHONFAULTHANDLER=1 there -- it logs a full thread dump    on any future native crash, which is invaluable for problems like    this one. 3. Raising gc_interval in weewx.conf reduces exposure but does not    eliminate it (Python's automatic collections use the same broken    code path), so I wouldn't rely on it. 4. When the fixed python3.13 package arrives, remember that a running    weewxd keeps the old library until it is restarted -- do a    "systemctl restart weewx" after the upgrade to actually get off the    broken version. Downgrading to deb13u2 also works but re-opens the security fixes that update carried; with the auto-restart in place I don't think it's worth it. John -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/weewx-user/92B83A5B-A968-4531-8DF2-223C1E9D1ED7%40johnkline.com. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/weewx-user/ae5a994b-b4f0-4c6b-aa50-91baedd28a34n%40googlegroups.com. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/weewx-user/a03d6aaa-a55f-4120-83bd-d808c3eb80cfn%40googlegroups.com. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/weewx-user/8a1f1ef1-6eaa-45e1-9f5a-4a4ce6b77e02n%40googlegroups.com. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/weewx-user/0433FB88-A5B2-4673-9231-661838CA6B5C%40johnkline.com.