Re: [weewx-development] Reportgenerator 'hangs' with vector plot

2020-04-05 Thread Thomas Keffer 
I had the same issue with my webhost. After 5 years of arguing with them, I
gave up and went to AWS Lightsail .
Cheaper, faster, and I can do anything I want with my instance. Much
happier now.

-tk

On Sun, Apr 5, 2020 at 12:38 PM Lucas Heijst  wrote:

> Vince,
>
> My ISP in the Netherlands doesn't allow the use of rsync.
>
> Luc
>
> On Sunday, 5 April 2020 14:45:40 UTC-3, Vince Skahan wrote:
>>
>> On Friday, April 3, 2020 at 6:56:58 PM UTC-7, Vince Skahan wrote:
>>>
>>> On Friday, April 3, 2020 at 5:24:21 PM UTC-7, Lucas Heijst wrote:
>>>
>>> ftpgenerator: Ftp'd 79 files in 89.01 seconds
>>>
>>>
>>> Also switching from ftp to rsync will dramatically improve how long it
>>> takes to upload things to your remote site.
>>>
>>>
>> Luc - just a data point - this is from my Seagate Dockstar which is a
>> 128MB RAM pogoplug, so far slower than even a model-B pi.  Remote host is
>> my Amazon AWS Lightsail instance on Internet.
>>
>> weeutil.rsyncupload: rsync'd 67 files (962749 bytes) in 1.60 seconds
>>
>>
>> Suggest trying to switch to rsync to perhaps spin your processing up a
>> bit.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/b76e7a49-5817-490e-b2a3-f6f5e8a345e9%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEDvMKcdCX0gESigvupQJCmfaV%2By8TK0kj0Bv5CY3wGfXw%40mail.gmail.com.

Re: [weewx-development] Reportgenerator 'hangs' with vector plot

2020-04-05 Thread Vince Skahan 
On Friday, April 3, 2020 at 6:56:58 PM UTC-7, Vince Skahan wrote:
>
> On Friday, April 3, 2020 at 5:24:21 PM UTC-7, Lucas Heijst wrote:
>
> ftpgenerator: Ftp'd 79 files in 89.01 seconds 
>
>
> Also switching from ftp to rsync will dramatically improve how long it 
> takes to upload things to your remote site.
>
>
Luc - just a data point - this is from my Seagate Dockstar which is a 128MB 
RAM pogoplug, so far slower than even a model-B pi.  Remote host is my 
Amazon AWS Lightsail instance on Internet.

weeutil.rsyncupload: rsync'd 67 files (962749 bytes) in 1.60 seconds


Suggest trying to switch to rsync to perhaps spin your processing up a 
bit. 

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/4ebecf1e-44da-4280-9375-f7c97b965e2d%40googlegroups.com.

Re: [weewx-development] Reportgenerator 'hangs' with vector plot

2020-04-04 Thread Greg Troxel 
mwall  writes:

> we started a postgres adapter years ago.  it never made it into weewx core.

Sorry -- I use a lot of programs and my memory was that weewx supported
postgres.  I have used it for various things over the years including
one project that had frequent crashes due to laptop batteries dying and
it did amazingly well in terms of robustness.

> the weewx column names are mixed case, like outTemp and windSpeed.  as i 
> recall, postgres lower-cases everything that is not quoted.  so we would 
> need a general way of dealing with case sensitivity that works with mysql, 
> sqlite, and postgres.

Interesting - I am guessing that postgres is compliant with the SQL
specs, but making a different within-spec choice.

> i'm pretty sure it would not be too difficult for weewx core, but making it 
> general enough to work for search list extensions and cheetah templates 
> could be a problem (i think that is why it never made it into weewx core).


> if you want to hack on it, make a postgresql adapter in bin/weedb/pgsql.py 
> - follow the pattern in sqlite.py or mysql.py in that directory.

I have put your text into my todo list.   But it's large :-(

Thanks for taking the time to straighten me out.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/rmio8s7atmi.fsf%40s1.lexort.com.

Re: [weewx-development] Reportgenerator 'hangs' with vector plot

2020-04-04 Thread mwall 
On Saturday, April 4, 2020 at 9:24:41 AM UTC-4, Greg Troxel wrote:
>
>
> I wonder if people have tried postgresql.  (I'm still on sqlite.) 
>

here is a summary of how identifiers are treated in various databases:

https://github.com/ontop/ontop/wiki/Case-sensitivity-for-SQL-identifiers 

the original discussion in weewx user group:

https://groups.google.com/forum/#!msg/weewx-development/5fzYoRio2aM/P9qgDqHBEjgJ

a related issue that still exists in mysql (when running on windows or 
macos):

https://github.com/weewx/weewx/issues/27


-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/549f14a1-2982-41dd-a68b-d8e0924eeb05%40googlegroups.com.

Re: [weewx-development] Reportgenerator 'hangs' with vector plot

2020-04-04 Thread mwall 
On Saturday, April 4, 2020 at 9:24:41 AM UTC-4, Greg Troxel wrote:
>
> Glenn McKechnie > writes: 
>
> > It's not weewx increasing in CPU usage, it's the CPU usage for 
> > mariadb. Check on your Synology NAS. 
>
> I wonder if people have tried postgresql.  (I'm still on sqlite.) 
>

greg, 

we started a postgres adapter years ago.  it never made it into weewx core.

the weewx column names are mixed case, like outTemp and windSpeed.  as i 
recall, postgres lower-cases everything that is not quoted.  so we would 
need a general way of dealing with case sensitivity that works with mysql, 
sqlite, and postgres.

i'm pretty sure it would not be too difficult for weewx core, but making it 
general enough to work for search list extensions and cheetah templates 
could be a problem (i think that is why it never made it into weewx core).

if you want to hack on it, make a postgresql adapter in bin/weedb/pgsql.py 
- follow the pattern in sqlite.py or mysql.py in that directory.

m

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/3239f908-b392-4855-807d-36d1da319b8f%40googlegroups.com.

Re: [weewx-development] Reportgenerator 'hangs' with vector plot

2020-04-04 Thread Lucas Heijst 
Tom,

The plots look good to me, see attachments.

Luc

On Saturday, 4 April 2020 10:19:06 UTC-3, Tom Keffer wrote:
>
> Before you get too excited, make sure the fix still gives the right 
> answer. It uses a different query strategy.
>
> On Sat, Apr 4, 2020 at 6:17 AM Glenn McKechnie  > wrote:
>
>> Well, that fixed it on this machine!
>>
>> Back to the 16% mysqld usage with both 'aggregate_type = max' and
>> 'aggregate_interval = 300' reinstated in skin.conf
>>
>> Wouldn't have a fix as smooth as that for covid-19 would you?
>>
>>
>>
>> On 04/04/2020, Thomas Keffer > wrote:
>> > Luc, could you try the attached version of xtypes.py and see if it makes
>> > any difference?
>> >
>> > -tk
>> >
>> > On Sat, Apr 4, 2020 at 5:10 AM Lucas Heijst > > wrote:
>> >
>> >> Tom, Vince, Glenn,
>> >>
>> >> Glenn,
>> >> Yes, I use external mariadb5 databases.
>> >> And no, during the creation of the vector plots the memory use of weewx
>> >> is
>> >> not much (2.7 %).
>> >> Changing the aggregate_interval has effect, but is not the main cause.
>> >>
>> >> Tom, Vince, Glenn,
>> >> The main cause of the slow vector calculation is the aggregate_type = 
>> max
>> >> on windgustvec.
>> >> Without the max aggregation all 5 vector plots (6h, day, week, month,
>> >> year) took together 24 seconds.
>> >>
>> >> Generated 76 images for report vproReport in 113.71 seconds
>> >>
>> >> Luc
>> >>
>> >> =
>> >> [[[hourwindvec]]]
>> >> windvec
>> >> plot_type  = vector
>> >> windgustvec
>> >> plot_type  = vector
>> >> ###aggregate_type = max
>> >> ###aggregate_interval = 900# == 15 min
>> >>
>> >> [[[daywindvec]]]
>> >> windvec
>> >> plot_type= vector
>> >> windgustvec
>> >> plot_type= vector
>> >> ###aggregate_type   = max
>> >> ###aggregate_interval   = 3600# == 1 hour
>> >>
>> >> [[[weekwindvec]]]
>> >> windvec
>> >> plot_type= vector
>> >> windgustvec
>> >> plot_type= vector
>> >> ###aggregate_type   = max
>> >>
>> >> [[[monthwindvec]]]
>> >> windvec
>> >> plot_type= vector
>> >> windgustvec
>> >> plot_type= vector
>> >> ###aggregate_type   = max
>> >> ###aggregate_interval   = 3600# == 1 hour
>> >>
>> >> [[[yearwindvec]]]
>> >> windvec
>> >> plot_type= vector
>> >> windgustvec
>> >> plot_type= vector
>> >> ###aggregate_type   = max
>> >> =
>> >>
>> >>
>> >>
>> >>
>> >> On Friday, 3 April 2020 22:33:50 UTC-3, Glenn McKechnie wrote:
>> >>>
>> >>> (Sigh, and to the list)
>> >>>
>> >>> Luc,
>> >>>
>> >>> I notice a "Launch of report thread aborted: existing report thread
>> >>> still running"
>> >>>
>> >>> Do you use mysql (mariadb)?
>> >>> If you do, does its CPU usage (from top) increase?
>> >>>
>> >>> If so try dropping the aggregate interval = 900 and see if there is a
>> >>> difference
>> >>>
>> >>> On 04/04/2020, Lucas Heijst  wrote:
>> >>> > Tom,
>> >>> >
>> >>> > There was not much info in the syslog, thats why I didnt send it.
>> >>> > I included the syslog this time.
>> >>> >
>> >>> > I was wrong: the reportgenerator didn't hang, only it is VERY slow
>> >>> >
>> >>> > First I generated all other plots: 71 images in 90 seconds
>> >>> > Generated 71 images for report vproReport in 89.58 seconds
>> >>> >
>> >>> > Then I added a 6-hour vector plot. Extra time for that 6h plot: 70
>> >>> seconds
>> >>> > Generated 72 images for report vproReport in 159.32 seconds
>> >>> >
>> >>> > Then added a 24-hour vector plot. Extra time for that 24h vector 
>> plot
>> >>> 266
>> >>> > seconds
>> >>> > Generated 73 images for report vproReport in 425.79 seconds
>> >>> >
>> >>> > Estimated time for the other vector plots:
>> >>> > week vector plot 31 minutes
>> >>> > month vector plot: 137 minutes
>> >>> > year vector plot: 27 hours
>> >>> >
>> >>> > Luc
>> >>> >
>> >>> >
>> >>> > On Friday, 3 April 2020 20:13:16 UTC-3, Tom Keffer wrote:
>> >>> >>
>> >>> >> Worked fine for me.
>> >>> >>
>> >>> >> Luc: you know better. We need a log! Perhaps the ReportGenerator is
>> >>> not
>> >>> >> finishing before the next report is due? Perhaps the program
>> >>> segfaulted?
>> >>> >> Who knows without a log?
>> >>> >>
>> >>> >> -tk
>> >>> >> [image: image.png]
>> >>> >>
>> >>> >> On Fri, Apr 3, 2020 at 4:09 PM Lucas Heijst > >>> >> > wrote:
>> >>> >>
>> >>> >>> Currently running weewx 4.0.0b18.
>> >>> >>>
>> >>> >>> The reportgenerator hangs (it never finishes) during calculating 
>> of
>>

Re: [weewx-development] Reportgenerator 'hangs' with vector plot

2020-04-04 Thread Lucas Heijst 
Glenn,

The change in Toms xtypes.py is the way how the database query is done for 
max and min aggregation
So I assume the old query kept the DB manager very busy (but I did not 
check it because in the new version the query is fast).

Luc

On Saturday, 4 April 2020 10:03:13 UTC-3, Glenn McKechnie wrote:
>
> Luc, 
>
> It's not weewx increasing in CPU usage, it's the CPU usage for 
> mariadb. Check on your Synology NAS. 
>
> Maybe it's not the same for yours, but I found this occurring in mine. 
>
> Worked in 3.9.2, failed in weewx 4.0.06b where it sent the CPU through 
> the roof, with the reports failing in the same manner as yours. 
> I assumed mariadb had become a choke point, commenting it out (as 
> you've done) brought things back to 'normal'. 
>
> In fact, looking through my skin.conf shows the note I've left myself. 
> I see this. 
>
> [[[daywindvec]]] 
>#-# see note below 
>#-# aggregate_interval = 300 
> windvec 
> label = Wind Vector (5 min avg.) 
> plot_type = vector 
> aggregate_type = avg 
> aggregate_interval = 300 
> #windgustvec 
> #label = Gust Wind 
> #plot_type = vector 
> ###aggregate_type = max 
> #-# mysqld goes through the roof if this is enabled 
> #-# aggregate_interval = 300 
>
> Obviously I haven't got back to it. Life (and fires) have a knack of 
> getting in the way sometimes. 
>
> Okay, Quick test here: 
> CPU for mysqld with no report thread running  0.2% 
> CPU for mysqld with the (seasons) skin.conf as above 16.2% 
> Now... 
> Uncommenting those entries (windgustvec) sends CPU to 163% (114%, 133% 
> 155% etc) mysqld is seriously struggling with it. I have to revert 
> those changes and restart mysqld to get it back down again. 
>
> Taking it a little further it's the, not the 'aggregate_interval = 
> 300' that sets it off it's the 'aggregate_type = max ' 
>
> Changing that to  'aggregate_type =  avg' and it's okay, commenting it 
> out completely is also okay. 
>
> Leaving it uncommented (activated) is poison. 
> Wait for an archive cycle in this high usage state and the CPU is 
> still maxed out and my wee_reports fail to cycle. That means that with 
> my quirky setup it's still running and hasn't finished or been reaped. 
> Interestingly, when I restart weewx that separate wee_report instance 
> completes. 
>
>
> On 04/04/2020, Lucas Heijst > wrote: 
> > Tom, Vince, Glenn, 
> > 
> > Glenn, 
> > Yes, I use external mariadb5 databases. 
> > And no, during the creation of the vector plots the memory use of weewx 
> is 
> > not much (2.7 %). 
> > Changing the aggregate_interval has effect, but is not the main cause. 
> > 
> > Tom, Vince, Glenn, 
> > The main cause of the slow vector calculation is the aggregate_type = 
> max 
> > on windgustvec. 
> > Without the max aggregation all 5 vector plots (6h, day, week, month, 
> year) 
> > 
> > took together 24 seconds. 
> > 
> > Generated 76 images for report vproReport in 113.71 seconds 
> > 
> > Luc 
> > 
> > = 
> > [[[hourwindvec]]] 
> > windvec 
> > plot_type  = vector 
> > windgustvec 
> > plot_type  = vector 
> > ###aggregate_type = max 
> > ###aggregate_interval = 900# == 15 min 
> > 
> > [[[daywindvec]]] 
> > windvec 
> > plot_type= vector 
> > windgustvec 
> > plot_type= vector 
> > ###aggregate_type   = max 
> > ###aggregate_interval   = 3600# == 1 hour 
> > 
> > [[[weekwindvec]]] 
> > windvec 
> > plot_type= vector 
> > windgustvec 
> > plot_type= vector 
> > ###aggregate_type   = max 
> > 
> > [[[monthwindvec]]] 
> > windvec 
> > plot_type= vector 
> > windgustvec 
> > plot_type= vector 
> > ###aggregate_type   = max 
> > ###aggregate_interval   = 3600# == 1 hour 
> > 
> > [[[yearwindvec]]] 
> > windvec 
> > plot_type= vector 
> > windgustvec 
> > plot_type= vector 
> > ###aggregate_type   = max 
> > = 
> > 
> > 
> > 
> > 
> > On Friday, 3 April 2020 22:33:50 UTC-3, Glenn McKechnie wrote: 
> >> 
> >> (Sigh, and to the list) 
> >> 
> >> Luc, 
> >> 
> >> I notice a "Launch of report thread aborted: existing report thread 
> >> still running" 
> >> 
> >> Do you use mysql (mariadb)? 
> >> If you do, does its CPU usage (from top) increase? 
>

Re: [weewx-development] Reportgenerator 'hangs' with vector plot

2020-04-04 Thread Glenn McKechnie 
On 05/04/2020, Thomas Keffer  wrote:
> Before you get too excited, make sure the fix still gives the right answer.
> It uses a different query strategy.

My first thought was how can I tell without firing up a 3.9.2 version
(Although I think I might have access to one of those :-)

But then if I change the values on this one...
Attached are 3 charts - all at max,  None aggregate interval, 300
aggregate interval and 900 aggregate interval. Seem okay - ish


-- 

Cheers
 Glenn

rorpi - read only raspberry pi & various weewx addons
https://github.com/glennmckechnie

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAAraAzgGEZZXmW2TfrBiSMM%2Bidn8mN87h3%3D%2Bcs6o_m6FjF_WHg%40mail.gmail.com.

Re: [weewx-development] Reportgenerator 'hangs' with vector plot

2020-04-04 Thread Lucas Heijst 
Tom,

YES, it made a huge difference! 
With the new xtypes.py all reports now runned in only 102 seconds!
Generated 76 images for report vproReport in 101.51 seconds
The report was generated with the original skin.conf file with 
aggregate_type set to max and the aggregate_intervals set.

Thanks!

Luc


On Saturday, 4 April 2020 09:41:59 UTC-3, Tom Keffer wrote:
>
> Luc, could you try the attached version of xtypes.py and see if it makes 
> any difference?
>
> -tk
>
> On Sat, Apr 4, 2020 at 5:10 AM Lucas Heijst  > wrote:
>
>> Tom, Vince, Glenn,
>>
>> Glenn,
>> Yes, I use external mariadb5 databases.
>> And no, during the creation of the vector plots the memory use of weewx 
>> is not much (2.7 %).
>> Changing the aggregate_interval has effect, but is not the main cause.
>>
>> Tom, Vince, Glenn,
>> The main cause of the slow vector calculation is the aggregate_type = max 
>> on windgustvec.
>> Without the max aggregation all 5 vector plots (6h, day, week, month, 
>> year) took together 24 seconds.
>>
>> Generated 76 images for report vproReport in 113.71 seconds
>>
>> Luc
>>
>> =
>> [[[hourwindvec]]]
>> windvec
>> plot_type  = vector
>> windgustvec
>> plot_type  = vector
>> ###aggregate_type = max
>> ###aggregate_interval = 900# == 15 min
>>
>> [[[daywindvec]]]
>> windvec
>> plot_type= vector
>> windgustvec
>> plot_type= vector
>> ###aggregate_type   = max
>> ###aggregate_interval   = 3600# == 1 hour
>>
>> [[[weekwindvec]]]
>> windvec
>> plot_type= vector
>> windgustvec
>> plot_type= vector
>> ###aggregate_type   = max
>>
>> [[[monthwindvec]]]
>> windvec
>> plot_type= vector
>> windgustvec
>> plot_type= vector
>> ###aggregate_type   = max
>> ###aggregate_interval   = 3600# == 1 hour
>>
>> [[[yearwindvec]]]
>> windvec
>> plot_type= vector
>> windgustvec
>> plot_type= vector
>> ###aggregate_type   = max
>> =
>>
>>
>>
>>
>> On Friday, 3 April 2020 22:33:50 UTC-3, Glenn McKechnie wrote:
>>>
>>> (Sigh, and to the list) 
>>>
>>> Luc, 
>>>
>>> I notice a "Launch of report thread aborted: existing report thread 
>>> still running" 
>>>
>>> Do you use mysql (mariadb)? 
>>> If you do, does its CPU usage (from top) increase? 
>>>
>>> If so try dropping the aggregate interval = 900 and see if there is a 
>>> difference 
>>>
>>> On 04/04/2020, Lucas Heijst  wrote: 
>>> > Tom, 
>>> > 
>>> > There was not much info in the syslog, thats why I didnt send it. 
>>> > I included the syslog this time. 
>>> > 
>>> > I was wrong: the reportgenerator didn't hang, only it is VERY slow 
>>> > 
>>> > First I generated all other plots: 71 images in 90 seconds 
>>> > Generated 71 images for report vproReport in 89.58 seconds 
>>> > 
>>> > Then I added a 6-hour vector plot. Extra time for that 6h plot: 70 
>>> seconds 
>>> > Generated 72 images for report vproReport in 159.32 seconds 
>>> > 
>>> > Then added a 24-hour vector plot. Extra time for that 24h vector plot 
>>> 266 
>>> > seconds 
>>> > Generated 73 images for report vproReport in 425.79 seconds 
>>> > 
>>> > Estimated time for the other vector plots: 
>>> > week vector plot 31 minutes 
>>> > month vector plot: 137 minutes 
>>> > year vector plot: 27 hours 
>>> > 
>>> > Luc 
>>> > 
>>> > 
>>> > On Friday, 3 April 2020 20:13:16 UTC-3, Tom Keffer wrote: 
>>> >> 
>>> >> Worked fine for me. 
>>> >> 
>>> >> Luc: you know better. We need a log! Perhaps the ReportGenerator is 
>>> not 
>>> >> finishing before the next report is due? Perhaps the program 
>>> segfaulted? 
>>> >> Who knows without a log? 
>>> >> 
>>> >> -tk 
>>> >> [image: image.png] 
>>> >> 
>>> >> On Fri, Apr 3, 2020 at 4:09 PM Lucas Heijst >> >> > wrote: 
>>> >> 
>>> >>> Currently running weewx 4.0.0b18. 
>>> >>> 
>>> >>> The reportgenerator hangs (it never finishes) during calculating of 
>>> the 
>>> >>> section below. 
>>> >>> 
>>> >>> [[[hourwindvec]]] 
>>> >>> windvec 
>>> >>> plot_type  = vector 
>>> >>> windgustvec 
>>> >>> plot_type  = vector 
>>> >>> aggregate_type = max 
>>> >>> aggregate_interval = 900# == 15 min 
>>> >>> 
>>> >>> Luc 
>>> >>> 
>>> >>> -- 
>>> >>> You received this message because you are subscribed to the Google 
>>> Groups 
>>> >>> 
>>> >>> "weewx-development" group. 
>>> >>> To

Re: [weewx-development] Reportgenerator 'hangs' with vector plot

2020-04-04 Thread Greg Troxel 
Glenn McKechnie  writes:

> It's not weewx increasing in CPU usage, it's the CPU usage for
> mariadb. Check on your Synology NAS.

I wonder if people have tried postgresql.  (I'm still on sqlite.)

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/rmieet3crjc.fsf%40s1.lexort.com.

Re: [weewx-development] Reportgenerator 'hangs' with vector plot

2020-04-04 Thread Thomas Keffer 
Before you get too excited, make sure the fix still gives the right answer.
It uses a different query strategy.

On Sat, Apr 4, 2020 at 6:17 AM Glenn McKechnie 
wrote:

> Well, that fixed it on this machine!
>
> Back to the 16% mysqld usage with both 'aggregate_type = max' and
> 'aggregate_interval = 300' reinstated in skin.conf
>
> Wouldn't have a fix as smooth as that for covid-19 would you?
>
>
>
> On 04/04/2020, Thomas Keffer  wrote:
> > Luc, could you try the attached version of xtypes.py and see if it makes
> > any difference?
> >
> > -tk
> >
> > On Sat, Apr 4, 2020 at 5:10 AM Lucas Heijst 
> wrote:
> >
> >> Tom, Vince, Glenn,
> >>
> >> Glenn,
> >> Yes, I use external mariadb5 databases.
> >> And no, during the creation of the vector plots the memory use of weewx
> >> is
> >> not much (2.7 %).
> >> Changing the aggregate_interval has effect, but is not the main cause.
> >>
> >> Tom, Vince, Glenn,
> >> The main cause of the slow vector calculation is the aggregate_type =
> max
> >> on windgustvec.
> >> Without the max aggregation all 5 vector plots (6h, day, week, month,
> >> year) took together 24 seconds.
> >>
> >> Generated 76 images for report vproReport in 113.71 seconds
> >>
> >> Luc
> >>
> >> =
> >> [[[hourwindvec]]]
> >> windvec
> >> plot_type  = vector
> >> windgustvec
> >> plot_type  = vector
> >> ###aggregate_type = max
> >> ###aggregate_interval = 900# == 15 min
> >>
> >> [[[daywindvec]]]
> >> windvec
> >> plot_type= vector
> >> windgustvec
> >> plot_type= vector
> >> ###aggregate_type   = max
> >> ###aggregate_interval   = 3600# == 1 hour
> >>
> >> [[[weekwindvec]]]
> >> windvec
> >> plot_type= vector
> >> windgustvec
> >> plot_type= vector
> >> ###aggregate_type   = max
> >>
> >> [[[monthwindvec]]]
> >> windvec
> >> plot_type= vector
> >> windgustvec
> >> plot_type= vector
> >> ###aggregate_type   = max
> >> ###aggregate_interval   = 3600# == 1 hour
> >>
> >> [[[yearwindvec]]]
> >> windvec
> >> plot_type= vector
> >> windgustvec
> >> plot_type= vector
> >> ###aggregate_type   = max
> >> =
> >>
> >>
> >>
> >>
> >> On Friday, 3 April 2020 22:33:50 UTC-3, Glenn McKechnie wrote:
> >>>
> >>> (Sigh, and to the list)
> >>>
> >>> Luc,
> >>>
> >>> I notice a "Launch of report thread aborted: existing report thread
> >>> still running"
> >>>
> >>> Do you use mysql (mariadb)?
> >>> If you do, does its CPU usage (from top) increase?
> >>>
> >>> If so try dropping the aggregate interval = 900 and see if there is a
> >>> difference
> >>>
> >>> On 04/04/2020, Lucas Heijst  wrote:
> >>> > Tom,
> >>> >
> >>> > There was not much info in the syslog, thats why I didnt send it.
> >>> > I included the syslog this time.
> >>> >
> >>> > I was wrong: the reportgenerator didn't hang, only it is VERY slow
> >>> >
> >>> > First I generated all other plots: 71 images in 90 seconds
> >>> > Generated 71 images for report vproReport in 89.58 seconds
> >>> >
> >>> > Then I added a 6-hour vector plot. Extra time for that 6h plot: 70
> >>> seconds
> >>> > Generated 72 images for report vproReport in 159.32 seconds
> >>> >
> >>> > Then added a 24-hour vector plot. Extra time for that 24h vector plot
> >>> 266
> >>> > seconds
> >>> > Generated 73 images for report vproReport in 425.79 seconds
> >>> >
> >>> > Estimated time for the other vector plots:
> >>> > week vector plot 31 minutes
> >>> > month vector plot: 137 minutes
> >>> > year vector plot: 27 hours
> >>> >
> >>> > Luc
> >>> >
> >>> >
> >>> > On Friday, 3 April 2020 20:13:16 UTC-3, Tom Keffer wrote:
> >>> >>
> >>> >> Worked fine for me.
> >>> >>
> >>> >> Luc: you know better. We need a log! Perhaps the ReportGenerator is
> >>> not
> >>> >> finishing before the next report is due? Perhaps the program
> >>> segfaulted?
> >>> >> Who knows without a log?
> >>> >>
> >>> >> -tk
> >>> >> [image: image.png]
> >>> >>
> >>> >> On Fri, Apr 3, 2020 at 4:09 PM Lucas Heijst  >>> >> > wrote:
> >>> >>
> >>> >>> Currently running weewx 4.0.0b18.
> >>> >>>
> >>> >>> The reportgenerator hangs (it never finishes) during calculating of
> >>> the
> >>> >>> section below.
> >>> >>>
> >>> >>> [[[hourwindvec]]]
> >>> >>> windvec
> >>> >>> plot_type  = vector
> >>> >>> windgustvec
> >>> >>> plot_type  = vector
> >>> >>>

Re: [weewx-development] Reportgenerator 'hangs' with vector plot

2020-04-04 Thread Glenn McKechnie 
Well, that fixed it on this machine!

Back to the 16% mysqld usage with both 'aggregate_type = max' and
'aggregate_interval = 300' reinstated in skin.conf

Wouldn't have a fix as smooth as that for covid-19 would you?



On 04/04/2020, Thomas Keffer  wrote:
> Luc, could you try the attached version of xtypes.py and see if it makes
> any difference?
>
> -tk
>
> On Sat, Apr 4, 2020 at 5:10 AM Lucas Heijst  wrote:
>
>> Tom, Vince, Glenn,
>>
>> Glenn,
>> Yes, I use external mariadb5 databases.
>> And no, during the creation of the vector plots the memory use of weewx
>> is
>> not much (2.7 %).
>> Changing the aggregate_interval has effect, but is not the main cause.
>>
>> Tom, Vince, Glenn,
>> The main cause of the slow vector calculation is the aggregate_type = max
>> on windgustvec.
>> Without the max aggregation all 5 vector plots (6h, day, week, month,
>> year) took together 24 seconds.
>>
>> Generated 76 images for report vproReport in 113.71 seconds
>>
>> Luc
>>
>> =
>> [[[hourwindvec]]]
>> windvec
>> plot_type  = vector
>> windgustvec
>> plot_type  = vector
>> ###aggregate_type = max
>> ###aggregate_interval = 900# == 15 min
>>
>> [[[daywindvec]]]
>> windvec
>> plot_type= vector
>> windgustvec
>> plot_type= vector
>> ###aggregate_type   = max
>> ###aggregate_interval   = 3600# == 1 hour
>>
>> [[[weekwindvec]]]
>> windvec
>> plot_type= vector
>> windgustvec
>> plot_type= vector
>> ###aggregate_type   = max
>>
>> [[[monthwindvec]]]
>> windvec
>> plot_type= vector
>> windgustvec
>> plot_type= vector
>> ###aggregate_type   = max
>> ###aggregate_interval   = 3600# == 1 hour
>>
>> [[[yearwindvec]]]
>> windvec
>> plot_type= vector
>> windgustvec
>> plot_type= vector
>> ###aggregate_type   = max
>> =
>>
>>
>>
>>
>> On Friday, 3 April 2020 22:33:50 UTC-3, Glenn McKechnie wrote:
>>>
>>> (Sigh, and to the list)
>>>
>>> Luc,
>>>
>>> I notice a "Launch of report thread aborted: existing report thread
>>> still running"
>>>
>>> Do you use mysql (mariadb)?
>>> If you do, does its CPU usage (from top) increase?
>>>
>>> If so try dropping the aggregate interval = 900 and see if there is a
>>> difference
>>>
>>> On 04/04/2020, Lucas Heijst  wrote:
>>> > Tom,
>>> >
>>> > There was not much info in the syslog, thats why I didnt send it.
>>> > I included the syslog this time.
>>> >
>>> > I was wrong: the reportgenerator didn't hang, only it is VERY slow
>>> >
>>> > First I generated all other plots: 71 images in 90 seconds
>>> > Generated 71 images for report vproReport in 89.58 seconds
>>> >
>>> > Then I added a 6-hour vector plot. Extra time for that 6h plot: 70
>>> seconds
>>> > Generated 72 images for report vproReport in 159.32 seconds
>>> >
>>> > Then added a 24-hour vector plot. Extra time for that 24h vector plot
>>> 266
>>> > seconds
>>> > Generated 73 images for report vproReport in 425.79 seconds
>>> >
>>> > Estimated time for the other vector plots:
>>> > week vector plot 31 minutes
>>> > month vector plot: 137 minutes
>>> > year vector plot: 27 hours
>>> >
>>> > Luc
>>> >
>>> >
>>> > On Friday, 3 April 2020 20:13:16 UTC-3, Tom Keffer wrote:
>>> >>
>>> >> Worked fine for me.
>>> >>
>>> >> Luc: you know better. We need a log! Perhaps the ReportGenerator is
>>> not
>>> >> finishing before the next report is due? Perhaps the program
>>> segfaulted?
>>> >> Who knows without a log?
>>> >>
>>> >> -tk
>>> >> [image: image.png]
>>> >>
>>> >> On Fri, Apr 3, 2020 at 4:09 PM Lucas Heijst >> >> > wrote:
>>> >>
>>> >>> Currently running weewx 4.0.0b18.
>>> >>>
>>> >>> The reportgenerator hangs (it never finishes) during calculating of
>>> the
>>> >>> section below.
>>> >>>
>>> >>> [[[hourwindvec]]]
>>> >>> windvec
>>> >>> plot_type  = vector
>>> >>> windgustvec
>>> >>> plot_type  = vector
>>> >>> aggregate_type = max
>>> >>> aggregate_interval = 900# == 15 min
>>> >>>
>>> >>> Luc
>>> >>>
>>> >>> --
>>> >>> You received this message because you are subscribed to the Google
>>> Groups
>>> >>>
>>> >>> "weewx-development" group.
>>> >>> To unsubscribe from this group and stop receiving emails from it,
>>> send an
>>> >>>
>>> >>> email to weewx-de...@googlegroups.com .
>>> >>> To view this discussion on the web visit
>>> >>>
>>>

Re: [weewx-development] Reportgenerator 'hangs' with vector plot

2020-04-04 Thread Glenn McKechnie 
Luc,

It's not weewx increasing in CPU usage, it's the CPU usage for
mariadb. Check on your Synology NAS.

Maybe it's not the same for yours, but I found this occurring in mine.

Worked in 3.9.2, failed in weewx 4.0.06b where it sent the CPU through
the roof, with the reports failing in the same manner as yours.
I assumed mariadb had become a choke point, commenting it out (as
you've done) brought things back to 'normal'.

In fact, looking through my skin.conf shows the note I've left myself.
I see this.

[[[daywindvec]]]
   #-# see note below
   #-# aggregate_interval = 300
windvec
label = Wind Vector (5 min avg.)
plot_type = vector
aggregate_type = avg
aggregate_interval = 300
#windgustvec
#label = Gust Wind
#plot_type = vector
###aggregate_type = max
#-# mysqld goes through the roof if this is enabled
#-# aggregate_interval = 300

Obviously I haven't got back to it. Life (and fires) have a knack of
getting in the way sometimes.

Okay, Quick test here:
CPU for mysqld with no report thread running  0.2%
CPU for mysqld with the (seasons) skin.conf as above 16.2%
Now...
Uncommenting those entries (windgustvec) sends CPU to 163% (114%, 133%
155% etc) mysqld is seriously struggling with it. I have to revert
those changes and restart mysqld to get it back down again.

Taking it a little further it's the, not the 'aggregate_interval =
300' that sets it off it's the 'aggregate_type = max '

Changing that to  'aggregate_type =  avg' and it's okay, commenting it
out completely is also okay.

Leaving it uncommented (activated) is poison.
Wait for an archive cycle in this high usage state and the CPU is
still maxed out and my wee_reports fail to cycle. That means that with
my quirky setup it's still running and hasn't finished or been reaped.
Interestingly, when I restart weewx that separate wee_report instance
completes.


On 04/04/2020, Lucas Heijst  wrote:
> Tom, Vince, Glenn,
>
> Glenn,
> Yes, I use external mariadb5 databases.
> And no, during the creation of the vector plots the memory use of weewx is
> not much (2.7 %).
> Changing the aggregate_interval has effect, but is not the main cause.
>
> Tom, Vince, Glenn,
> The main cause of the slow vector calculation is the aggregate_type = max
> on windgustvec.
> Without the max aggregation all 5 vector plots (6h, day, week, month, year)
>
> took together 24 seconds.
>
> Generated 76 images for report vproReport in 113.71 seconds
>
> Luc
>
> =
> [[[hourwindvec]]]
> windvec
> plot_type  = vector
> windgustvec
> plot_type  = vector
> ###aggregate_type = max
> ###aggregate_interval = 900# == 15 min
>
> [[[daywindvec]]]
> windvec
> plot_type= vector
> windgustvec
> plot_type= vector
> ###aggregate_type   = max
> ###aggregate_interval   = 3600# == 1 hour
>
> [[[weekwindvec]]]
> windvec
> plot_type= vector
> windgustvec
> plot_type= vector
> ###aggregate_type   = max
>
> [[[monthwindvec]]]
> windvec
> plot_type= vector
> windgustvec
> plot_type= vector
> ###aggregate_type   = max
> ###aggregate_interval   = 3600# == 1 hour
>
> [[[yearwindvec]]]
> windvec
> plot_type= vector
> windgustvec
> plot_type= vector
> ###aggregate_type   = max
> =
>
>
>
>
> On Friday, 3 April 2020 22:33:50 UTC-3, Glenn McKechnie wrote:
>>
>> (Sigh, and to the list)
>>
>> Luc,
>>
>> I notice a "Launch of report thread aborted: existing report thread
>> still running"
>>
>> Do you use mysql (mariadb)?
>> If you do, does its CPU usage (from top) increase?
>>
>> If so try dropping the aggregate interval = 900 and see if there is a
>> difference
>>
>> On 04/04/2020, Lucas Heijst > wrote:
>> > Tom,
>> >
>> > There was not much info in the syslog, thats why I didnt send it.
>> > I included the syslog this time.
>> >
>> > I was wrong: the reportgenerator didn't hang, only it is VERY slow
>> >
>> > First I generated all other plots: 71 images in 90 seconds
>> > Generated 71 images for report vproReport in 89.58 seconds
>> >
>> > Then I added a 6-hour vector plot. Extra time for that 6h plot: 70
>> seconds
>> > Generated 72 images for report vproReport in 159.32 seconds
>> >
>> > Then added a 24-hour vector plot. Extra time for that

Re: [weewx-development] Reportgenerator 'hangs' with vector plot

2020-04-04 Thread Thomas Keffer 
Luc, could you try the attached version of xtypes.py and see if it makes
any difference?

-tk

On Sat, Apr 4, 2020 at 5:10 AM Lucas Heijst  wrote:

> Tom, Vince, Glenn,
>
> Glenn,
> Yes, I use external mariadb5 databases.
> And no, during the creation of the vector plots the memory use of weewx is
> not much (2.7 %).
> Changing the aggregate_interval has effect, but is not the main cause.
>
> Tom, Vince, Glenn,
> The main cause of the slow vector calculation is the aggregate_type = max
> on windgustvec.
> Without the max aggregation all 5 vector plots (6h, day, week, month,
> year) took together 24 seconds.
>
> Generated 76 images for report vproReport in 113.71 seconds
>
> Luc
>
> =
> [[[hourwindvec]]]
> windvec
> plot_type  = vector
> windgustvec
> plot_type  = vector
> ###aggregate_type = max
> ###aggregate_interval = 900# == 15 min
>
> [[[daywindvec]]]
> windvec
> plot_type= vector
> windgustvec
> plot_type= vector
> ###aggregate_type   = max
> ###aggregate_interval   = 3600# == 1 hour
>
> [[[weekwindvec]]]
> windvec
> plot_type= vector
> windgustvec
> plot_type= vector
> ###aggregate_type   = max
>
> [[[monthwindvec]]]
> windvec
> plot_type= vector
> windgustvec
> plot_type= vector
> ###aggregate_type   = max
> ###aggregate_interval   = 3600# == 1 hour
>
> [[[yearwindvec]]]
> windvec
> plot_type= vector
> windgustvec
> plot_type= vector
> ###aggregate_type   = max
> =
>
>
>
>
> On Friday, 3 April 2020 22:33:50 UTC-3, Glenn McKechnie wrote:
>>
>> (Sigh, and to the list)
>>
>> Luc,
>>
>> I notice a "Launch of report thread aborted: existing report thread
>> still running"
>>
>> Do you use mysql (mariadb)?
>> If you do, does its CPU usage (from top) increase?
>>
>> If so try dropping the aggregate interval = 900 and see if there is a
>> difference
>>
>> On 04/04/2020, Lucas Heijst  wrote:
>> > Tom,
>> >
>> > There was not much info in the syslog, thats why I didnt send it.
>> > I included the syslog this time.
>> >
>> > I was wrong: the reportgenerator didn't hang, only it is VERY slow
>> >
>> > First I generated all other plots: 71 images in 90 seconds
>> > Generated 71 images for report vproReport in 89.58 seconds
>> >
>> > Then I added a 6-hour vector plot. Extra time for that 6h plot: 70
>> seconds
>> > Generated 72 images for report vproReport in 159.32 seconds
>> >
>> > Then added a 24-hour vector plot. Extra time for that 24h vector plot
>> 266
>> > seconds
>> > Generated 73 images for report vproReport in 425.79 seconds
>> >
>> > Estimated time for the other vector plots:
>> > week vector plot 31 minutes
>> > month vector plot: 137 minutes
>> > year vector plot: 27 hours
>> >
>> > Luc
>> >
>> >
>> > On Friday, 3 April 2020 20:13:16 UTC-3, Tom Keffer wrote:
>> >>
>> >> Worked fine for me.
>> >>
>> >> Luc: you know better. We need a log! Perhaps the ReportGenerator is
>> not
>> >> finishing before the next report is due? Perhaps the program
>> segfaulted?
>> >> Who knows without a log?
>> >>
>> >> -tk
>> >> [image: image.png]
>> >>
>> >> On Fri, Apr 3, 2020 at 4:09 PM Lucas Heijst > >> > wrote:
>> >>
>> >>> Currently running weewx 4.0.0b18.
>> >>>
>> >>> The reportgenerator hangs (it never finishes) during calculating of
>> the
>> >>> section below.
>> >>>
>> >>> [[[hourwindvec]]]
>> >>> windvec
>> >>> plot_type  = vector
>> >>> windgustvec
>> >>> plot_type  = vector
>> >>> aggregate_type = max
>> >>> aggregate_interval = 900# == 15 min
>> >>>
>> >>> Luc
>> >>>
>> >>> --
>> >>> You received this message because you are subscribed to the Google
>> Groups
>> >>>
>> >>> "weewx-development" group.
>> >>> To unsubscribe from this group and stop receiving emails from it,
>> send an
>> >>>
>> >>> email to weewx-de...@googlegroups.com .
>> >>> To view this discussion on the web visit
>> >>>
>> https://groups.google.com/d/msgid/weewx-development/601f87df-cbe7-482c-bf49-722a7a5de7bd%40googlegroups.com
>> >>>
>> >>> <
>> https://groups.google.com/d/msgid/weewx-development/601f87df-cbe7-482c-bf49-722a7a5de7bd%40googlegroups.com?utm_medium=email_source=footer>
>>
>> >>> .
>> >>>
>> >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups
>> > "weewx-development" group.

Re: [weewx-development] Reportgenerator 'hangs' with vector plot

2020-04-04 Thread Lucas Heijst 
Tom, Vince, Glenn,

Glenn,
Yes, I use external mariadb5 databases.
And no, during the creation of the vector plots the memory use of weewx is 
not much (2.7 %).
Changing the aggregate_interval has effect, but is not the main cause.

Tom, Vince, Glenn,
The main cause of the slow vector calculation is the aggregate_type = max 
on windgustvec.
Without the max aggregation all 5 vector plots (6h, day, week, month, year) 
took together 24 seconds.

Generated 76 images for report vproReport in 113.71 seconds

Luc

=
[[[hourwindvec]]]
windvec
plot_type  = vector
windgustvec
plot_type  = vector
###aggregate_type = max
###aggregate_interval = 900# == 15 min

[[[daywindvec]]]
windvec
plot_type= vector
windgustvec
plot_type= vector
###aggregate_type   = max
###aggregate_interval   = 3600# == 1 hour

[[[weekwindvec]]]
windvec
plot_type= vector
windgustvec
plot_type= vector
###aggregate_type   = max

[[[monthwindvec]]]
windvec
plot_type= vector
windgustvec
plot_type= vector
###aggregate_type   = max
###aggregate_interval   = 3600# == 1 hour

[[[yearwindvec]]]
windvec
plot_type= vector
windgustvec
plot_type= vector
###aggregate_type   = max
=




On Friday, 3 April 2020 22:33:50 UTC-3, Glenn McKechnie wrote:
>
> (Sigh, and to the list) 
>
> Luc, 
>
> I notice a "Launch of report thread aborted: existing report thread 
> still running" 
>
> Do you use mysql (mariadb)? 
> If you do, does its CPU usage (from top) increase? 
>
> If so try dropping the aggregate interval = 900 and see if there is a 
> difference 
>
> On 04/04/2020, Lucas Heijst > wrote: 
> > Tom, 
> > 
> > There was not much info in the syslog, thats why I didnt send it. 
> > I included the syslog this time. 
> > 
> > I was wrong: the reportgenerator didn't hang, only it is VERY slow 
> > 
> > First I generated all other plots: 71 images in 90 seconds 
> > Generated 71 images for report vproReport in 89.58 seconds 
> > 
> > Then I added a 6-hour vector plot. Extra time for that 6h plot: 70 
> seconds 
> > Generated 72 images for report vproReport in 159.32 seconds 
> > 
> > Then added a 24-hour vector plot. Extra time for that 24h vector plot 
> 266 
> > seconds 
> > Generated 73 images for report vproReport in 425.79 seconds 
> > 
> > Estimated time for the other vector plots: 
> > week vector plot 31 minutes 
> > month vector plot: 137 minutes 
> > year vector plot: 27 hours 
> > 
> > Luc 
> > 
> > 
> > On Friday, 3 April 2020 20:13:16 UTC-3, Tom Keffer wrote: 
> >> 
> >> Worked fine for me. 
> >> 
> >> Luc: you know better. We need a log! Perhaps the ReportGenerator is not 
> >> finishing before the next report is due? Perhaps the program 
> segfaulted? 
> >> Who knows without a log? 
> >> 
> >> -tk 
> >> [image: image.png] 
> >> 
> >> On Fri, Apr 3, 2020 at 4:09 PM Lucas Heijst  >> > wrote: 
> >> 
> >>> Currently running weewx 4.0.0b18. 
> >>> 
> >>> The reportgenerator hangs (it never finishes) during calculating of 
> the 
> >>> section below. 
> >>> 
> >>> [[[hourwindvec]]] 
> >>> windvec 
> >>> plot_type  = vector 
> >>> windgustvec 
> >>> plot_type  = vector 
> >>> aggregate_type = max 
> >>> aggregate_interval = 900# == 15 min 
> >>> 
> >>> Luc 
> >>> 
> >>> -- 
> >>> You received this message because you are subscribed to the Google 
> Groups 
> >>> 
> >>> "weewx-development" group. 
> >>> To unsubscribe from this group and stop receiving emails from it, send 
> an 
> >>> 
> >>> email to weewx-de...@googlegroups.com . 
> >>> To view this discussion on the web visit 
> >>> 
> https://groups.google.com/d/msgid/weewx-development/601f87df-cbe7-482c-bf49-722a7a5de7bd%40googlegroups.com
>  
> >>> 
> >>> <
> https://groups.google.com/d/msgid/weewx-development/601f87df-cbe7-482c-bf49-722a7a5de7bd%40googlegroups.com?utm_medium=email_source=footer>
>  
>
> >>> . 
> >>> 
> >> 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups 
> > "weewx-development" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an 
> > email to weewx-de...@googlegroups.com . 
> > To view this discussion on the web visit 
> > 
>

Re: [weewx-development] Reportgenerator 'hangs' with vector plot

2020-04-03 Thread Lucas Heijst 
Vince,

The following set of weewx programs runs for years now on my raspberry pi 
2B.
1. Vantage driver
2. Meteostick driver
3. Klimalogg drive4
4. Computer monitor.
The programs run from a memory stick.
The data is written to 4 external mariadb5 databases on my Synology NAS.
The uploaded data is send to my webserver in the Netherlands. 
The FTP speed is about 1 file per second.
My ISP doesn’t offer RSYNC (I know how fast this is on my NASes).
Starting with version 4.0.0b17 I have these slow calculations of vector 
plots.

Luc

On Friday, 3 April 2020 22:56:58 UTC-3, Vince Skahan wrote:
>
> On Friday, April 3, 2020 at 5:24:21 PM UTC-7, Lucas Heijst wrote:
>>
>> I included the syslog this time.
>>
>> I was wrong: the reportgenerator didn't hang, only it is VERY slow
>>
>
>
> I still think you are doing too much (and too frequently)  for your model 
> pi, the SD card, and your network.
>
>
> Generated 73 images for report vproReport in 425.79 seconds
>
> ftpgenerator: Ftp'd 79 files in 89.01 seconds 
>
>
> What's your interval period ?
>
> Just the two items above are over 8.5 minutes for one cycle.
>
>
> Also switching from ftp to rsync will dramatically improve how long it 
> takes to upload things to your remote site.
>
> I continue to suggest turning everything off but the base Seasons report 
> to baseline your system, then turn other things one-at-a-time.
>
> Or get a much better computer to run weewx on, and a much better computer 
> to ftp to.
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/3d596c10-5315-48dc-b70a-6008ed6dd31f%40googlegroups.com.

Re: [weewx-development] Reportgenerator 'hangs' with vector plot

2020-04-03 Thread Vince Skahan 
On Friday, April 3, 2020 at 5:24:21 PM UTC-7, Lucas Heijst wrote:
>
> I included the syslog this time.
>
> I was wrong: the reportgenerator didn't hang, only it is VERY slow
>


I still think you are doing too much (and too frequently)  for your model 
pi, the SD card, and your network.


Generated 73 images for report vproReport in 425.79 seconds

ftpgenerator: Ftp'd 79 files in 89.01 seconds 


What's your interval period ?

Just the two items above are over 8.5 minutes for one cycle.


Also switching from ftp to rsync will dramatically improve how long it 
takes to upload things to your remote site.

I continue to suggest turning everything off but the base Seasons report to 
baseline your system, then turn other things one-at-a-time.

Or get a much better computer to run weewx on, and a much better computer 
to ftp to.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/a8fcb81b-e03a-47a6-81a0-2676969c4a42%40googlegroups.com.

Re: [weewx-development] Reportgenerator 'hangs' with vector plot

2020-04-03 Thread Lucas Heijst 
Tom,

There was not much info in the syslog, thats why I didnt send it.
I included the syslog this time.

I was wrong: the reportgenerator didn't hang, only it is VERY slow

First I generated all other plots: 71 images in 90 seconds
Generated 71 images for report vproReport in 89.58 seconds

Then I added a 6-hour vector plot. Extra time for that 6h plot: 70 seconds
Generated 72 images for report vproReport in 159.32 seconds

Then added a 24-hour vector plot. Extra time for that 24h vector plot 266 
seconds
Generated 73 images for report vproReport in 425.79 seconds

Estimated time for the other vector plots:
week vector plot 31 minutes
month vector plot: 137 minutes
year vector plot: 27 hours

Luc


On Friday, 3 April 2020 20:13:16 UTC-3, Tom Keffer wrote:
>
> Worked fine for me. 
>
> Luc: you know better. We need a log! Perhaps the ReportGenerator is not 
> finishing before the next report is due? Perhaps the program segfaulted? 
> Who knows without a log?
>
> -tk
> [image: image.png]
>
> On Fri, Apr 3, 2020 at 4:09 PM Lucas Heijst  > wrote:
>
>> Currently running weewx 4.0.0b18.
>>
>> The reportgenerator hangs (it never finishes) during calculating of the 
>> section below.
>>
>> [[[hourwindvec]]]
>> windvec
>> plot_type  = vector
>> windgustvec
>> plot_type  = vector
>> aggregate_type = max
>> aggregate_interval = 900# == 15 min
>>
>> Luc
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "weewx-development" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to weewx-de...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/weewx-development/601f87df-cbe7-482c-bf49-722a7a5de7bd%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/afee474f-61cb-4c8d-8bfd-5f1c9a18773f%40googlegroups.com.
Apr  3 20:47:12 pi21 vpro[12254] INFO __main__: Initializing weewx version 
4.0.0b18
Apr  3 20:47:12 pi21 vpro[12254] INFO __main__: Using Python 3.7.3 (default, 
Dec 20 2019, 18:57:59) #012[GCC 8.3.0]
Apr  3 20:47:12 pi21 vpro[12254] INFO __main__: Platform 
Linux-4.19.97-v7+-armv7l-with-debian-10.3
Apr  3 20:47:12 pi21 vpro[12254] INFO __main__: Locale is 'en_GB.UTF-8'
Apr  3 20:47:12 pi21 vpro[12254] INFO __main__: PID file is 
/var/run/weewx_vpro.pid
Apr  3 20:47:12 pi21 vpro[12258] INFO __main__: Using configuration file 
/home/weewx/weewx_vpro.conf
Apr  3 20:47:13 pi21 vpro[12258] DEBUG __main__: Initializing engine
Apr  3 20:47:13 pi21 vpro[12258] INFO weewx.engine: Loading station type 
Vantage (user.vantage-312-lh)
Apr  3 20:47:13 pi21 /weewxd: vantage-312-lh: Driver version is 3.1.2
Apr  3 20:47:13 pi21 /weewxd: vantage-312-lh: Opened up serial port /dev/vpro; 
baud 19200; timeout 4.00
Apr  3 20:47:13 pi21 /weewxd: vantage-312-lh: Gentle wake up of console 
successful
Apr  3 20:47:13 pi21 /weewxd: vantage-312-lh: Hardware type is 16
Apr  3 20:47:13 pi21 /weewxd: vantage-312-lh: ISS ID is 2
Apr  3 20:47:13 pi21 /weewxd: vantage-312-lh: Hardware name: Vantage Pro2
Apr  3 20:47:13 pi21 vpro[12258] DEBUG weewx.engine: Loading service 
weewx.engine.StdTimeSynch
Apr  3 20:47:13 pi21 vpro[12258] DEBUG weewx.engine: Finished loading service 
weewx.engine.StdTimeSynch
Apr  3 20:47:13 pi21 vpro[12258] DEBUG weewx.engine: Loading service 
weewx.engine.StdConvert
Apr  3 20:47:13 pi21 vpro[12258] INFO weewx.engine: StdConvert target unit is 
0x10
Apr  3 20:47:13 pi21 vpro[12258] DEBUG weewx.engine: Finished loading service 
weewx.engine.StdConvert
Apr  3 20:47:13 pi21 vpro[12258] DEBUG weewx.engine: Loading service 
weewx.engine.StdCalibrate
Apr  3 20:47:13 pi21 vpro[12258] DEBUG weewx.engine: Finished loading service 
weewx.engine.StdCalibrate
Apr  3 20:47:13 pi21 vpro[12258] DEBUG weewx.engine: Loading service 
weewx.engine.StdQC
Apr  3 20:47:13 pi21 vpro[12258] DEBUG weewx.engine: Finished loading service 
weewx.engine.StdQC
Apr  3 20:47:13 pi21 vpro[12258] DEBUG weewx.engine: Loading service 
weewx.wxservices.StdWXCalculate
Apr  3 20:47:13 pi21 vpro[12258] DEBUG weewx.manager: Daily summary version is 
2.0
Apr  3 20:47:13 pi21 vpro[12258] INFO weewx.wxservices: The following values 
will be calculated: pressure=prefer_hardware, barometer=prefer_hardware, 
altimeter=prefer_hardware, windchill=prefer_hardware, 
heatindex=prefer_hardware, dewpoint=prefer_hardware, 
inDewpoint=prefer_hardware,

Re: [weewx-development] Reportgenerator 'hangs' with vector plot

2020-04-03 Thread Thomas Keffer 
Worked fine for me.

Luc: you know better. We need a log! Perhaps the ReportGenerator is not
finishing before the next report is due? Perhaps the program segfaulted?
Who knows without a log?

-tk
[image: image.png]

On Fri, Apr 3, 2020 at 4:09 PM Lucas Heijst  wrote:

> Currently running weewx 4.0.0b18.
>
> The reportgenerator hangs (it never finishes) during calculating of the
> section below.
>
> [[[hourwindvec]]]
> windvec
> plot_type  = vector
> windgustvec
> plot_type  = vector
> aggregate_type = max
> aggregate_interval = 900# == 15 min
>
> Luc
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-development+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-development/601f87df-cbe7-482c-bf49-722a7a5de7bd%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/CAPq0zEAHtjAMtN12TcY8Em%2B-w1_n%3DD%3DSpypXD7wWJ93-haNtRA%40mail.gmail.com.

[weewx-development] Reportgenerator 'hangs' with vector plot

2020-04-03 Thread Lucas Heijst 
Currently running weewx 4.0.0b18.

The reportgenerator hangs (it never finishes) during calculating of the 
section below.

[[[hourwindvec]]]
windvec
plot_type  = vector
windgustvec
plot_type  = vector
aggregate_type = max
aggregate_interval = 900# == 15 min

Luc

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-development+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-development/601f87df-cbe7-482c-bf49-722a7a5de7bd%40googlegroups.com.