My assumption would be that any derivable value that you want to plot should be a column in the database. Even if you could speed it up, the calculation it would need to be by a factor approaching 100 to be a useful choice. I think that using derived parameters for only a "current" display should be OK, but as soon as you want to plot it or include it in stats then each report has to process the year's worth.
If I did decide that I wanted to add windchill at a later date, it looks like it would be just a weectl database add-column and then weectl database calc-missing. Does that seem right? On Thursday 28 December 2023 at 11:26:09 pm UTC+10 Tom Keffer wrote: > Very useful data, Cameron. > > Right now, if you want one year's worth of say, windchill, and it's not in > the database, it is calculated one record at a time. For a 5 minute archive > interval, that's 100k+ database accesses. It's a general algorithm, but > it's slow. > > Alternatively, one could write a specialized algorithm for windchill. The > sensible thing to do would be to read a year's worth of temperature and > wind speed, all in one go --- one database access. Then use the results to > calculate the year's worth of windchill. The downside is that it's not > general at all: it would only know how to calculate windchill. The upside > is that the existing xtypes API can be used right now to do this. You'd > have to write extensions for all of your missing synthesized types. > > How to write a more general algorithm that uses only a single database > access? It might be possible to set up a mechanism where an xtype specifies > which types it needs from the database first. Then get_series() calls > get_scalar() with the results. I'd have to think about that. It would > definitely be an extension to the xtypes API. > > Thanks for setting us on the right course, Cameron. > > > > > On Thu, Dec 28, 2023 at 3:38 AM 'Cameron D' via weewx-development < > [email protected]> wrote: > >> OK, this "ability to generate synthetic types" was the clue to the >> excessive cpu and time. >> I started with a copy of my main DB converted to sqlite and trimmed it to >> 5 minute intervals and ~100k archive records, so it is close to the >> benchmark db. >> >> I did a bit of (my first ever) python profiling and compared the >> benchmark times with the same reports on my DB. >> >> - benchmark: The busiest weewx function *xtypes.py:128(get_aggregate)* >> was called ~16000 times for a total duration of 0.4 seconds. >> - my DB: the top function was: *xtypes.py:76(get_scalar)*, called 6 >> milllion times, for a duration of 130 seconds, then >> *xtypes.py:27(get_scalar)*, was called 35 milllion times, total time >> 84 seconds. >> >> Further down I could see lots of time spent in routines with names based >> on *windchill *or *heatindex*. >> From the start, I had decided I didn't want to see either of these >> parameters, so I had removed them from my DB. But now Seasons is forcing a >> years' worth of their recalculation every report. Hence durations of up to >> several minutes. >> >> I gradually trimmed down the skin.conf, and only by removing every trace >> of windchill, heatindex and tempfeel could I get the execution times down >> from 59 seconds to about 10 seconds - if I left a single reference >> remaining then it hung around the 56-59 seconds mark. >> >> The other, more minor, but still quite significant factor, is that all >> the other items listed like extratemp1, extratemp2, etc have been removed >> from my db. Referencing them in charts seems to take more cpu to *not *plot >> them than to have data and have them plotted. By removing all the chart >> references to non-existent columns from my skin I got the elapsed time down >> from ~10 seconds to 2.3s. >> >> It is still refusing to print the times, either to console or the system >> log. (but I can see the debug output). The times I quoted were user+system >> cpu time consumed, as reported by "time command..." >> >> >> >> On Thursday 28 December 2023 at 10:12:54 am UTC+10 Tom Keffer wrote: >> >> >> So, V5 image generation is slower, but then it's also doing more. V5 is >> able to generate plots of pure synthetic types, not just types in the >> database. That requires consulting the database to see whether it can do >> the type. >> >> -tk >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "weewx-development" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/weewx-development/14b64e0a-2550-4e82-b8b9-6184eb888485n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/weewx-development/14b64e0a-2550-4e82-b8b9-6184eb888485n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "weewx-development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/68a6c58f-b458-44fa-ad67-5a174806371fn%40googlegroups.com.
