Gary, I appreciate your response, although much of it is lost on me. However, the idea of a dramatic difference between first pass and subsequent passes isn't happening here. You can tell by the pid that there was a restart, but the times are about the same.
Aug 28 17:58:35 hathaway weewx[1319]: cheetahgenerator: Generated 16 files for report Belchertown in 137.68 seconds Aug 28 18:01:35 hathaway weewx[1319]: cheetahgenerator: Generated 16 files for report Belchertown in 137.63 seconds Aug 28 18:04:38 hathaway weewx[1319]: cheetahgenerator: Generated 16 files for report Belchertown in 137.25 seconds Aug 28 18:08:35 hathaway weewx[25730]: cheetahgenerator: Generated 16 files for report Belchertown in 136.85 seconds Aug 28 18:11:34 hathaway weewx[25730]: cheetahgenerator: Generated 16 files for report Belchertown in 137.22 seconds Aug 28 18:14:35 hathaway weewx[25730]: cheetahgenerator: Generated 16 files for report Belchertown in 138.26 seconds Dave On Tue, Aug 28, 2018 at 7:05 PM gjr80 <gjroder...@gmail.com> wrote: > On Tuesday, 28 August 2018 04:11:36 UTC+10, Pat wrote: >> >> Most of the highcharts code is from @gjr80 so he may be able to speak >> better to it, but looking at how it gets its data, it's using getSqlVectors >> which looks like it gets its data from the archive table and not a day >> summary table. >> >> > I realise this is a bit dated in terms of this thread but I missed this > post. > > The original 'highcharts' extension I shared with Pat used getSQLVectors > throughout which queries the archive and not the daily summaries. From > memory (and my current highcharts extension has morphed substantially from > what I first shared with Pat) the day and week SLEs pulled data directly > from the archive, month and year used an aggregate (hour and day > respectively I think). This was done for much the same reason as the > default WeeWX plots plot an aggregate for month and year plots - there is > just too many data points otherwise. > > Also, on the topic of execution times. My original highcharts extension > used 4 SLEs to produce 4 json format data files using 1 skin with 4 WeeWX > reports; one report each for day, week, month and year. Each of the SLE > made a number of calls to getSQLVectors to gather the data vectors for > the period concerned, These were not done lazily rather they were executed > in full each time the SLE was called. The way that SLEs work within WeeWX > is that all SLEs listed in skin.conf are executed for each report in the > skin. So even with my 4 reports and 4 SLEs there was quite a lot of > unnecessary SLE execution (16 SLE calls in all when only 4 were actually > required) going on each with numerous complex SQL queries of the archive. > If you now magnify that with Pat's extension where there are quite a few > more reports in the skin that use the highcharts SLEs, the underlying > queries in the SLEs are being run many times each report cycle. When > coupled with a database that has a very short archive interval (say sub-5 > minutes) you magnify the time taken to execute these queries due to the > extra data points in a given period. > > I guess the bottom line is that the highcharts SLEs do not scale well in a > WeeWX report context, judicious placements of reports/SLEs in skins can > help. The highcharts SLEs really need to rewritten to be more efficient (eg > lazy evaluation) but that is a biggish task and has follow on consequences > for the supported reports. > > One quick comment on report times, take note of the time taken for the > second and subsequent report passes after a restart. If the total time > taken for the reports is approaching your archive interval you will > possibly have stability issues. The first report pass after a restart > always takes longer, in my experience up to twice as long as second and > subsequent passes. > > Gary > > -- > You received this message because you are subscribed to a topic in the > Google Groups "weewx-user" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/weewx-user/GSrKZEiPqsc/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > weewx-user+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
