I have added retry logic to the database manager in the case where the database is locked. It retries 5 times with exponential backoff. There's also a cheap regression test, but it all seems to work. It wasn't as bad as I thought.
Commit eb99b3f <https://github.com/weewx/weewx/commit/eb99b3fc9e7d73a6b30d519731a20624fd1a2081> On to the shutdown signal. On Tue, Aug 4, 2026 at 4:25 PM 'John Kline' via weewx-user < [email protected]> wrote: > Thanks, Tom. > > On the shutdown signal — a flag checked in the generator loops sounds > right. > > On the database manager, measure twice and cut once. This situation is > rare, and the shutdown signal may be all that’s needed — without an > orphaned report thread surviving into the next engine, it doesn’t compound. > > On Aug 4, 2026, at 3:32 PM, Tom Keffer <[email protected]> wrote: > > > Thanks, John, for your thorough analysis. > > Adding a "please commit suicide" signal to the shutdown mechanism is easy > enough. The report thread would monitor it, probably within the Cheetah > generator loop and the image generator loop. > > As for retrying "database is locked" errors, the database manager code is > the most twisted part of WeeWX. I tread lightly when I get in there, but it > clearly needs a rewrite. Unfortunately, it touches almost everything so it > is likely to be a profound rewrite. Let me think about this. > > -tk > > > > On Tue, Aug 4, 2026 at 3:15 PM 'John Kline' via weewx-user < > [email protected]> wrote: > >> Two things need untangling here, because they’re being conflated. >> >> The abort logic was never removed. weewx 5.4.0, weewx/engine.py, >> StdReport.launch_report_thread(), around line 867: >> >> >> if self.thread and self.thread.is_alive(): >> thread_age = time.time() - self.launch_time >> if thread_age < self.max_wait: >> log.info("Launch of report thread aborted: existing report >> thread still running") >> return >> else: >> log.warning("Previous report thread has been running" >> " %s seconds. Launching report thread anyway.", >> thread_age) >> >> max_wait defaults to 600 (line 847). The behavior you and Matthew >> remember is intact. >> >> >> Why you never see the message. Callbacks for an event run in >> registration order — bind() appends to a list, dispatchEvent() walks it >> (lines 224-241). Services register in the order they’re loaded, and the >> archive service registers for POST_LOOP ahead of the report service. Your >> own log shows the consequence: the archive record was written at 10:05:18, >> which happens inside the archive service’s POST_LOOP handler — so POST_LOOP >> was mid-dispatch when the OperationalError hit at 10:05:23. An exception in >> one callback aborts the whole dispatch. launch_report_thread(), later in >> the chain, was never called. The guard didn’t fail to log; it never ran. >> >> >> That’s also the answer to “why did changing max_wait have no effect”: >> neither branch of that test is being reached. >> >> >> A second report cycle did not start. Two independent reasons. The guard >> logs on both paths — the abort message under 600 seconds, the “Launching >> report thread anyway” warning over it — and neither appears anywhere in >> your log, which plainly isn’t filtering INFO. And the arithmetic forbids it >> anyway: the report thread launched at 10:00:21 and the engine died at >> 10:05:23. It was 302 seconds old. It cannot have reached the 600-second >> threshold, and the restart resets the launch clock, so nothing carries over. >> >> >> If you want this settled by observation instead of argument, py-spy dump >> lists every thread in the process with its stack. Two report engines would >> be two entries. One dump ends the debate. >> >> >> Where weewx could be better. It just isn’t the abort logic. Four >> things, and they compound: >> >> >> 1. There is no way to interrupt a running generator. shutDown() calls >> join(20.0), and that is the entire shutdown mechanism — no flag, no >> exception injected, nothing signalled. The report thread isn’t ignoring a >> request to stop; no request exists. “Unable to shut down StdReport thread” >> is a wait that expired, worded like an attempt that was refused. >> >> 2. The surviving thread becomes untracked. When the engine restarts, the >> new StdReport starts with self.thread = None. The orphan is no longer >> joined, checked, or counted — it is invisible to the very guard we’ve been >> discussing. The protection you’re asking about stops applying to the one >> thread that most needs it. >> >> 3. weewxd restarts the engine in the same process immediately after >> being told the shutdown failed. The orphan survives into the new engine’s >> lifetime, still holding database reads. Exiting the process and letting the >> supervisor restart it would kill the orphan for free. >> >> 4. The result is self-sustaining. The orphan holds reads; the new >> engine’s archive write collides with them; the engine dies with the same >> error; repeat. Your 10:07 restart is iteration two: up at 10:07:44, record >> at 10:07:55, dead at 10:08:00 — and no “Shutting down StdReport thread” >> line this time, because the new engine had no report thread of its own. On >> 1 vCPU, the orphan is also competing with the main loop for your only core. >> >> >> Arguably a fifth: treating “database is locked” as fatal. It’s a >> transient condition; retrying the write would be proportionate. Tearing >> down the engine is what converts a slow report into a crash loop. >> >> >> On beta4. I diffed it against beta3. The scan is unchanged — same >> _archive_latest_numeric, same unbounded query. What changed is the gating: >> a new aqi_source option (auto / local / forecast), with the archive path >> running only when >> >> >> local_aqi_enabled = aqi_enabled and aqi_source in (“auto”, “local”) >> >> >> aqi_source defaults to “auto” — which is in that tuple. On defaults, >> beta4 still runs the scans. So: which did you set? If you set aqi_source = >> forecast, that explains your speedup, and the bug remains live for anyone >> on defaults with a big database. If you’re on auto and it’s still fast, my >> diagnosis is incomplete and something else in beta4 deserves credit. >> >> >> For the skin author: bound the lookback instead of gating the caller. >> The query walks backwards from the newest row — instant when the column has >> current data, a full scan of the table when it doesn’t. A WHERE dateTime >= >> ? window of a day or two gives the identical answer for any station with a >> working PM sensor and costs nothing for one without. Config gating leaves >> the landmine armed for the next person with thirteen years of data who >> never touches aqi_source. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "weewx-user" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion visit >> https://groups.google.com/d/msgid/weewx-user/EFF848BB-B01B-4B73-B7C1-A3E7C06A718A%40johnkline.com >> <https://groups.google.com/d/msgid/weewx-user/EFF848BB-B01B-4B73-B7C1-A3E7C06A718A%40johnkline.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion visit > https://groups.google.com/d/msgid/weewx-user/CAPq0zEC9LT%3DnW6JOQ1-D5iEt20u23up3Z%2B0-nKa4s3y3dbRLgg%40mail.gmail.com > <https://groups.google.com/d/msgid/weewx-user/CAPq0zEC9LT%3DnW6JOQ1-D5iEt20u23up3Z%2B0-nKa4s3y3dbRLgg%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion visit > https://groups.google.com/d/msgid/weewx-user/55F7E600-C97B-4AF1-8ECD-3A75FEC5E9EC%40johnkline.com > <https://groups.google.com/d/msgid/weewx-user/55F7E600-C97B-4AF1-8ECD-3A75FEC5E9EC%40johnkline.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/weewx-user/CAPq0zECQbWeMG8%3DCxNNh9fki1jRm4XtaiMuuGg7xymvJ8-s4%3DA%40mail.gmail.com.
