[weewx-development] Re: weewx version 4 can't keep up in time

2020-03-29 Thread Lucas Heijst
Vince, I found one source of a memory leak: the webcam cronjob. In the webcam.debian file a number of variables were used like: WEBCAM_ID=4 CAMERA=picamera$WEBCAM_ID DATETIME=$(date +"%Y-%m-%d %H:%M:%S") EPOCH=$(date +"%s") ... The memory which is used by these variables is not automatically

[weewx-development] Memory leak in StdReport?

2020-03-29 Thread Lucas Heijst
Tom, Currently running weewx 4.0.0b17 and analysing the memory leak in my pi31 system. On the system runs two weewx instances (mben and tfrc) and a 5-minute webcam cronjob. The memory leak of my webcam cronjob task was caused by creating variables like: WEBCAM_ID=3 CAMERA=picamera$WEBCAM_ID

[weewx-development] Re: weewx version 4 can't keep up in time

2020-03-29 Thread Lucas Heijst
Vice, The memory leaks are caused by the weewx reporting (currently weewx 4.0.0b17) I have posted this in a seperate thread. Luc On Sunday, 29 March 2020 08:11:24 UTC-3, Lucas Heijst wrote: > > Vince, > > I found one source of a memory leak: the webcam cronjob. > In the webcam.debian file a

Re: [weewx-development] Memory leak in StdReport?

2020-03-29 Thread Thomas Keffer
1. You are getting memory leaks in 4.0.0 running both Python 2 and Python 3? 2. The biggest architectural differences between v3.9 and v4.0 is in how derived types are handled (V4 uses the new extensible type system) and in how strings are handled (v4 uses unicode), The former is pure Python. The

Re: [weewx-development] Memory leak in StdReport?

2020-03-29 Thread Lucas Heijst
Tom, I I forgot to say these same set of programs and webcam cronjob were running since a year now with weewx 3.9.1, raspian stretch and python2. Now they are running weewx 4.00b16-17, raspbian stretch and python2 or python3 with the memory problems mentioned. Luc On Sunday, 29 March 2020

Re: [weewx-development] Memory leak in StdReport?

2020-03-29 Thread Thomas Keffer
I'm not following. I thought you said you could get rid of the leak by doing the unsets. If so, is there still a problem? On Sun, Mar 29, 2020 at 8:41 AM Lucas Heijst wrote: > Tom, > > Currently running weewx 4.0.0b17 and analysing the memory leak in my pi31 > system. > On the system runs two

Re: [weewx-development] Memory leak in StdReport?

2020-03-29 Thread Thomas Keffer
I'm sorry, Luc, not trying to be difficult, but what is your question? I can't possibly debug your memory leak, especially with the complicated set of drivers and skins you are running. WeeWX itself does not have any memory leaks that I'm aware of. It is 100% Python, and Python uses garbage

Re: [weewx-development] Memory leak in StdReport?

2020-03-29 Thread Lucas Heijst
Tom, I have experienced two types of memory leak. 1. In cronjobs when defining variables without unset them afterwards. 2. With weewx reports (mben with 84+ generated graphs each run). Luc On Sunday, 29 March 2020 19:10:52 UTC-3, Tom Keffer wrote: > > By the way, in my experience, these leaks

Re: [weewx-development] Memory leak in StdReport?

2020-03-29 Thread Lucas Heijst
Gary, The system with weewx 3.9.1 runned with raspbian stretch of March 2019. I did not upgrade raspbian stretch until recently when I started all over with fresh install of the latest version of raspbian stretch. The update and upgrade renewed 64 modules. The tfrec package won’t compile

Re: [weewx-development] Memory leak in StdReport?

2020-03-29 Thread Thomas Keffer
By the way, in my experience, these leaks are caused by underlying "C" libraries or, occasionally, by failure of PIL to release graphics resource (such as font handles). On Sun, Mar 29, 2020 at 3:09 PM Thomas Keffer wrote: > I'm not following. I thought you said you could get rid of the leak by

[weewx-development] Re: Memory leak in StdReport?

2020-03-29 Thread rich T
Luc Don't know if this will be the solution to your issue: https://groups.google.com/forum/#!searchin/weewx-user/memory$20leak$20pil%7Csort:date/weewx-user/qeVSNzy7Zck/k-tAssLwAQAJ Rich > > -- You received this message because you are subscribed to the Google Groups "weewx-development"

Re: [weewx-development] Memory leak in StdReport?

2020-03-29 Thread Lucas Heijst
Gary, You are right, one graph was enough to trigger the memory grow! The strange thing tonight was an huge increase in memory for about one hour (200kB) followed by the ‘normal’ grow without any clue what happened in de syslog. Luc On Sunday, 29 March 2020 22:55:40 UTC-3, gjr80 wrote: > > So

Re: [weewx-development] Memory leak in StdReport?

2020-03-29 Thread Lucas Heijst
Glen, I like the method you use by firing the report in another way but I am not keen in modifying engine.py (each time with a new version of weewx). I could put the report calls in a cron job task, but then they are executed even when weewx is not running. Let me think about it. The timing

Re: [weewx-development] Memory leak in StdReport?

2020-03-29 Thread gjr80
So maybe one of those updated modules is the problem. Running with one plot still calls PIL so if PIL is the leak it will still show. As Vince said you really need to work systematically to find the culprit. PIL seemed to be low hanging fruit to me. Gary -- You received this message because

[weewx-development] Re: Memory leak in StdReport?

2020-03-29 Thread Lucas Heijst
Rich, Two things. My system is using PIL 4.0.0. which seem to cause problems. And.. Recently I changed in all my weewx skin files the path of the font files. Now they point to /home/weewx/bin/user/myfiles/fonts. They used to point to another directory where I copied my ttf fonts. Maybe PIL

Re: [weewx-development] Memory leak in StdReport?

2020-03-29 Thread Glenn McKechnie
On 30/03/2020, Lucas Heijst wrote: [...] > 2. With weewx reports (mben with 84+ generated graphs each run). > > Luc I have a fairly complex set of graphs ( multiple plots, large size, anti-alias turned on) and I have a persistent and steady climb until the OOM killer takes over. If I simplify