Glenn, The change in Toms xtypes.py is the way how the database query is done for max and min aggregation So I assume the old query kept the DB manager very busy (but I did not check it because in the new version the query is fast).
Luc On Saturday, 4 April 2020 10:03:13 UTC-3, Glenn McKechnie wrote: > > Luc, > > It's not weewx increasing in CPU usage, it's the CPU usage for > mariadb. Check on your Synology NAS. > > Maybe it's not the same for yours, but I found this occurring in mine. > > Worked in 3.9.2, failed in weewx 4.0.06b where it sent the CPU through > the roof, with the reports failing in the same manner as yours. > I assumed mariadb had become a choke point, commenting it out (as > you've done) brought things back to 'normal'. > > In fact, looking through my skin.conf shows the note I've left myself. > I see this. > > [[[daywindvec]]] > #-# see note below > #-# aggregate_interval = 300 > [[[[windvec]]]] > label = Wind Vector (5 min avg.) > plot_type = vector > aggregate_type = avg > aggregate_interval = 300 > #[[[[windgustvec]]]] > # label = Gust Wind > # plot_type = vector > ###aggregate_type = max > #-# mysqld goes through the roof if this is enabled > #-# aggregate_interval = 300 > > Obviously I haven't got back to it. Life (and fires) have a knack of > getting in the way sometimes. > > Okay, Quick test here: > CPU for mysqld with no report thread running 0.2% > CPU for mysqld with the (seasons) skin.conf as above 16.2% > Now... > Uncommenting those entries (windgustvec) sends CPU to 163% (114%, 133% > 155% etc) mysqld is seriously struggling with it. I have to revert > those changes and restart mysqld to get it back down again. > > Taking it a little further it's the, not the 'aggregate_interval = > 300' that sets it off it's the 'aggregate_type = max ' > > Changing that to 'aggregate_type = avg' and it's okay, commenting it > out completely is also okay. > > Leaving it uncommented (activated) is poison. > Wait for an archive cycle in this high usage state and the CPU is > still maxed out and my wee_reports fail to cycle. That means that with > my quirky setup it's still running and hasn't finished or been reaped. > Interestingly, when I restart weewx that separate wee_report instance > completes. > > > On 04/04/2020, Lucas Heijst <[email protected] <javascript:>> wrote: > > Tom, Vince, Glenn, > > > > Glenn, > > Yes, I use external mariadb5 databases. > > And no, during the creation of the vector plots the memory use of weewx > is > > not much (2.7 %). > > Changing the aggregate_interval has effect, but is not the main cause. > > > > Tom, Vince, Glenn, > > The main cause of the slow vector calculation is the aggregate_type = > max > > on windgustvec. > > Without the max aggregation all 5 vector plots (6h, day, week, month, > year) > > > > took together 24 seconds. > > > > Generated 76 images for report vproReport in 113.71 seconds > > > > Luc > > > > ===== > > [[[hourwindvec]]] > > [[[[windvec]]]] > > plot_type = vector > > [[[[windgustvec]]]] > > plot_type = vector > > ### aggregate_type = max > > ### aggregate_interval = 900 # == 15 min > > > > [[[daywindvec]]] > > [[[[windvec]]]] > > plot_type = vector > > [[[[windgustvec]]]] > > plot_type = vector > > ### aggregate_type = max > > ### aggregate_interval = 3600 # == 1 hour > > > > [[[weekwindvec]]] > > [[[[windvec]]]] > > plot_type = vector > > [[[[windgustvec]]]] > > plot_type = vector > > ### aggregate_type = max > > > > [[[monthwindvec]]] > > [[[[windvec]]]] > > plot_type = vector > > [[[[windgustvec]]]] > > plot_type = vector > > ### aggregate_type = max > > ### aggregate_interval = 3600 # == 1 hour > > > > [[[yearwindvec]]] > > [[[[windvec]]]] > > plot_type = vector > > [[[[windgustvec]]]] > > plot_type = vector > > ### aggregate_type = max > > ===== > > > > > > > > > > On Friday, 3 April 2020 22:33:50 UTC-3, Glenn McKechnie wrote: > >> > >> (Sigh, and to the list) > >> > >> Luc, > >> > >> I notice a "Launch of report thread aborted: existing report thread > >> still running" > >> > >> Do you use mysql (mariadb)? > >> If you do, does its CPU usage (from top) increase? > >> > >> If so try dropping the aggregate interval = 900 and see if there is a > >> difference > >> > >> On 04/04/2020, Lucas Heijst <[email protected] <javascript:>> wrote: > >> > Tom, > >> > > >> > There was not much info in the syslog, thats why I didnt send it. > >> > I included the syslog this time. > >> > > >> > I was wrong: the reportgenerator didn't hang, only it is VERY slow > >> > > >> > First I generated all other plots: 71 images in 90 seconds > >> > Generated 71 images for report vproReport in 89.58 seconds > >> > > >> > Then I added a 6-hour vector plot. Extra time for that 6h plot: 70 > >> seconds > >> > Generated 72 images for report vproReport in 159.32 seconds > >> > > >> > Then added a 24-hour vector plot. Extra time for that 24h vector plot > >> 266 > >> > seconds > >> > Generated 73 images for report vproReport in 425.79 seconds > >> > > >> > Estimated time for the other vector plots: > >> > week vector plot 31 minutes > >> > month vector plot: 137 minutes > >> > year vector plot: 27 hours > >> > > >> > Luc > >> > > >> > > >> > On Friday, 3 April 2020 20:13:16 UTC-3, Tom Keffer wrote: > >> >> > >> >> Worked fine for me. > >> >> > >> >> Luc: you know better. We need a log! Perhaps the ReportGenerator is > not > >> >> > >> >> finishing before the next report is due? Perhaps the program > >> segfaulted? > >> >> Who knows without a log? > >> >> > >> >> -tk > >> >> [image: image.png] > >> >> > >> >> On Fri, Apr 3, 2020 at 4:09 PM Lucas Heijst <[email protected] > >> >> <javascript:>> wrote: > >> >> > >> >>> Currently running weewx 4.0.0b18. > >> >>> > >> >>> The reportgenerator hangs (it never finishes) during calculating of > >> the > >> >>> section below. > >> >>> > >> >>> [[[hourwindvec]]] > >> >>> [[[[windvec]]]] > >> >>> plot_type = vector > >> >>> [[[[windgustvec]]]] > >> >>> plot_type = vector > >> >>> aggregate_type = max > >> >>> aggregate_interval = 900 # == 15 min > >> >>> > >> >>> Luc > >> >>> > >> >>> -- > >> >>> You received this message because you are subscribed to the Google > >> Groups > >> >>> > >> >>> "weewx-development" group. > >> >>> To unsubscribe from this group and stop receiving emails from it, > send > >> >>> > >> an > >> >>> > >> >>> email to [email protected] <javascript:>. > >> >>> To view this discussion on the web visit > >> >>> > >> > https://groups.google.com/d/msgid/weewx-development/601f87df-cbe7-482c-bf49-722a7a5de7bd%40googlegroups.com > > >> > >> >>> > >> >>> < > >> > https://groups.google.com/d/msgid/weewx-development/601f87df-cbe7-482c-bf49-722a7a5de7bd%40googlegroups.com?utm_medium=email&utm_source=footer> > > > >> > >> > >> >>> . > >> >>> > >> >> > >> > > >> > -- > >> > You received this message because you are subscribed to the Google > >> Groups > >> > "weewx-development" group. > >> > To unsubscribe from this group and stop receiving emails from it, > send > >> an > >> > email to [email protected] <javascript:>. > >> > To view this discussion on the web visit > >> > > >> > https://groups.google.com/d/msgid/weewx-development/afee474f-61cb-4c8d-8bfd-5f1c9a18773f%40googlegroups.com. > > > >> > >> > >> > > >> > >> > >> -- > >> > >> > >> Cheers > >> Glenn > >> > >> rorpi - read only raspberry pi & various weewx addons > >> https://github.com/glennmckechnie > >> > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "weewx-development" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to [email protected] <javascript:>. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/weewx-development/a652d38c-0ed1-4171-9510-5ac601336362%40googlegroups.com. > > > > > > > -- > > > Cheers > Glenn > > rorpi - read only raspberry pi & various weewx addons > https://github.com/glennmckechnie > -- You received this message because you are subscribed to the Google Groups "weewx-development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/0539b3ab-6151-453e-b02d-4010fef0a769%40googlegroups.com.
