Very useful data, Cameron. Right now, if you want one year's worth of say, windchill, and it's not in the database, it is calculated one record at a time. For a 5 minute archive interval, that's 100k+ database accesses. It's a general algorithm, but it's slow.
Alternatively, one could write a specialized algorithm for windchill. The sensible thing to do would be to read a year's worth of temperature and wind speed, all in one go --- one database access. Then use the results to calculate the year's worth of windchill. The downside is that it's not general at all: it would only know how to calculate windchill. The upside is that the existing xtypes API can be used right now to do this. You'd have to write extensions for all of your missing synthesized types. How to write a more general algorithm that uses only a single database access? It might be possible to set up a mechanism where an xtype specifies which types it needs from the database first. Then get_series() calls get_scalar() with the results. I'd have to think about that. It would definitely be an extension to the xtypes API. Thanks for setting us on the right course, Cameron. On Thu, Dec 28, 2023 at 3:38 AM 'Cameron D' via weewx-development < [email protected]> wrote: > OK, this "ability to generate synthetic types" was the clue to the > excessive cpu and time. > I started with a copy of my main DB converted to sqlite and trimmed it to > 5 minute intervals and ~100k archive records, so it is close to the > benchmark db. > > I did a bit of (my first ever) python profiling and compared the benchmark > times with the same reports on my DB. > > - benchmark: The busiest weewx function *xtypes.py:128(get_aggregate)* > was called ~16000 times for a total duration of 0.4 seconds. > - my DB: the top function was: *xtypes.py:76(get_scalar)*, called 6 > milllion times, for a duration of 130 seconds, then > *xtypes.py:27(get_scalar)*, was called 35 milllion times, total time > 84 seconds. > > Further down I could see lots of time spent in routines with names based > on *windchill *or *heatindex*. > From the start, I had decided I didn't want to see either of these > parameters, so I had removed them from my DB. But now Seasons is forcing a > years' worth of their recalculation every report. Hence durations of up to > several minutes. > > I gradually trimmed down the skin.conf, and only by removing every trace > of windchill, heatindex and tempfeel could I get the execution times down > from 59 seconds to about 10 seconds - if I left a single reference > remaining then it hung around the 56-59 seconds mark. > > The other, more minor, but still quite significant factor, is that all the > other items listed like extratemp1, extratemp2, etc have been removed from > my db. Referencing them in charts seems to take more cpu to *not *plot > them than to have data and have them plotted. By removing all the chart > references to non-existent columns from my skin I got the elapsed time down > from ~10 seconds to 2.3s. > > It is still refusing to print the times, either to console or the system > log. (but I can see the debug output). The times I quoted were user+system > cpu time consumed, as reported by "time command..." > > > > On Thursday 28 December 2023 at 10:12:54 am UTC+10 Tom Keffer wrote: > > > So, V5 image generation is slower, but then it's also doing more. V5 is > able to generate plots of pure synthetic types, not just types in the > database. That requires consulting the database to see whether it can do > the type. > > -tk > > > -- > You received this message because you are subscribed to the Google Groups > "weewx-development" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/weewx-development/14b64e0a-2550-4e82-b8b9-6184eb888485n%40googlegroups.com > <https://groups.google.com/d/msgid/weewx-development/14b64e0a-2550-4e82-b8b9-6184eb888485n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "weewx-development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/CAPq0zEB86SZ_NYQgzkPXfUOWrZmAVsYPvEgpzdwxmwpVL4COmQ%40mail.gmail.com.
