I know John strongly disagrees but this is another reason why I do not do ‘any’ package updates on my LAN-only weewx system. I get it working and leave it alone. Granted my risk profile permits this for me. I permit ‘no’ incoming traffic to my LAN that the weewx system is on,
Running in docker is another way to avoid these kinds of ‘sand shifted under your feet’ problems, making it easier to revert to a previously stable setup. (but wow quite the elaborate investigation figuring out all the gory internals things for this case) On Thursday, July 16, 2026 at 8:13:32 AM UTC-7 John Kline wrote: > Hi Greg, > > I pinged the maintainer (Stefano) on the bug. Stefano already has a test > build of 3.13.5-2+deb13u4. See his reply here: > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1141977 for a link to > it (if you want to install it). I have it running on 7 machines and will > do the rest after the first weewx gc run in about 2 and 1/2 hours. > > John > > On Jul 15, 2026, at 6:48 PM, John Kline <[email protected]> wrote: > > > > Well, you can from a Debian snapshot, but I think if I listed instructions > on how to do that, I dig myself quite a hole in having to support the > process; and, I suspect, some will never undo putting the packages on hold, > thus, never get a python upgrade again. > > On Jul 15, 2026, at 4:05 PM, Greg from Oz <[email protected]> wrote: > > apt-cache policy python3.13 > python3.13: > Installed: 3.13.5-2+deb13u3 > Candidate: 3.13.5-2+deb13u3 > Version table: > *** 3.13.5-2+deb13u3 500 > 500 http://deb.debian.org/debian trixie/main amd64 Packages > 100 /var/lib/dpkg/status > > Looks like I cannot downgrade...I will just have to now restart the weewx. > I do have weewx set up in monit and if it does crash in theory it should > restart again. > > > On Wednesday, 15 July 2026 at 23:05:03 UTC+10 John Kline wrote: > >> I believe the maintainers have what they need to push a fix. I’m hoping >> that will happen in a week or two. >> >> >> At least if it doesn't start I will know what to do >> >> It will start just fine. The issue is, each time weewx requests garbage >> collection (which, by default, weewx does every 3 hours), there is a small >> chance of a segment violation soon after. As such, the symptom will be >> that weewx stops running for no obvious reason (and nothing in the log >> after a garbage collection message). >> >> It is hard to say how often this might happen for a single instance of >> weewx running. The frequency may depend on what extensions you are running >> (and how many of them). It is conceivable that some won’t see the issue at >> all before the fix arrives. >> >> I was seeing the issue too often (as I run quite a few weewx instances); >> so I resorted to downgrading the python packages and put them on hold >> while waiting for the update. >> >> John >> >> On Jul 15, 2026, at 2:53 AM, Greg from Oz <[email protected]> wrote: >> >> >> >> So when does the new fix arrive? >> I upgraded the python 2 days ago and weewx has been running for 4 >> days....I better not restart weewx. >> At least if it doesn't start I will know what to do...Thanks for the >> heads up. >> >> On Tuesday, 14 July 2026 at 08:17:42 UTC+10 Tom Keffer wrote: >> >>> Thanks, John, for your detailed report. >>> >>> On Mon, Jul 13, 2026 at 8:09 AM 'John Kline' via weewx-user < >>> [email protected]> wrote: >>> >>>> If you run WeeWX on Debian 13 (trixie) and your system installed the >>>> python3.13 update from the 13.6 point release (version 3.13.5-2+deb13u3, >>>> rolling out via unattended-upgrades since July 11), be aware that it can >>>> crash weewxd with a segmentation fault. >>>> >>>> What it looks like >>>> ------------------ >>>> weewxd dies with SIGSEGV -- no Python traceback in the log, the process >>>> just exits. systemd shows "Main process exited, code=dumped, >>>> status=11/SEGV". If you look closely at the timing, the crash comes >>>> milliseconds after WeeWX's periodic garbage collection pass (the >>>> "Garbage collected N objects" INFO line, every 3 hours by default). >>>> weewxd is unusually good at tripping this bug precisely because it calls >>>> gc.collect() on that schedule while its other threads keep working. >>>> >>>> I run seven WeeWX machines (Raspberry Pi 5, all on trixie). Four >>>> crashes >>>> in the ~48 hours after the update reached them; the crash takes down >>>> maybe >>>> 1 in 25 GC passes, so it looks random and can hit anywhere from hours to >>>> days after a restart. The crashing thread differs every time (sqlite >>>> reads, report generation, image generation), so don't chase whatever >>>> happens to be in your traceback if you have faulthandler enabled -- it's >>>> the victim, not the cause. >>>> >>>> The cause >>>> --------- >>>> It is not WeeWX and not any extension. Debian's deb13u3 update >>>> backported >>>> a garbage-collector fix into Python 3.13.5 but omitted a second upstream >>>> commit that the fix depends on, so a GC pass can free memory that is >>>> still >>>> in use. It's tracked as Debian bug #1141977, the missing commit has >>>> been >>>> identified by bisection, and the maintainer is engaged, so I'd expect a >>>> fixed package (deb13u4) reasonably soon: >>>> >>>> https://bugs.debian.org/1141977 >>>> >>>> Am I affected? >>>> -------------- >>>> dpkg -l python3.13 | grep deb13u3 >>>> >>>> Two things must both be true: the deb13u3 package is installed AND >>>> weewxd >>>> was (re)started after it was installed. A weewxd that has been running >>>> since before the upgrade still uses the old library and is safe until >>>> its >>>> next restart. >>>> >>>> What to do >>>> ---------- >>>> 1. If your weewxd predates the upgrade, simply don't restart it until >>>> the >>>> fixed package arrives. >>>> 2. Otherwise, make crashes self-healing while you wait. A systemd >>>> drop-in (systemctl edit weewx) with: >>>> >>>> [Service] >>>> Restart=on-failure >>>> RestartSec=10 >>>> >>>> turns each crash into ~15 seconds of downtime. I also recommend >>>> Environment=PYTHONFAULTHANDLER=1 there -- it logs a full thread dump >>>> on any future native crash, which is invaluable for problems like >>>> this one. >>>> 3. Raising gc_interval in weewx.conf reduces exposure but does not >>>> eliminate it (Python's automatic collections use the same broken >>>> code path), so I wouldn't rely on it. >>>> 4. When the fixed python3.13 package arrives, remember that a running >>>> weewxd keeps the old library until it is restarted -- do a >>>> "systemctl restart weewx" after the upgrade to actually get off the >>>> broken version. >>>> >>>> Downgrading to deb13u2 also works but re-opens the security fixes that >>>> update carried; with the auto-restart in place I don't think it's worth >>>> it. >>>> >>>> John >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "weewx-user" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion visit >>>> https://groups.google.com/d/msgid/weewx-user/92B83A5B-A968-4531-8DF2-223C1E9D1ED7%40johnkline.com >>>> >>>> <https://groups.google.com/d/msgid/weewx-user/92B83A5B-A968-4531-8DF2-223C1E9D1ED7%40johnkline.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "weewx-user" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> >> To view this discussion visit >> https://groups.google.com/d/msgid/weewx-user/ae5a994b-b4f0-4c6b-aa50-91baedd28a34n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/weewx-user/ae5a994b-b4f0-4c6b-aa50-91baedd28a34n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion visit > https://groups.google.com/d/msgid/weewx-user/a03d6aaa-a55f-4120-83bd-d808c3eb80cfn%40googlegroups.com > > <https://groups.google.com/d/msgid/weewx-user/a03d6aaa-a55f-4120-83bd-d808c3eb80cfn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/weewx-user/16e83f39-eac2-4c74-a344-7fb6fca63cd4n%40googlegroups.com.
