Tom, I I forgot to say these same set of programs and webcam cronjob were running since a year now with weewx 3.9.1, raspian stretch and python2. Now they are running weewx 4.00b16-17, raspbian stretch and python2 or python3 with the memory problems mentioned.
Luc On Sunday, 29 March 2020 21:03:03 UTC-3, Lucas Heijst wrote: > > Tom, > > Due to memory grow my system crashed after 9-10 days. So I wrote here that > I would implement a cron job to restart my system after 5 days. > Vice Skahan convinced me to trace down the cause of the memory leaks and > that I did. > > No more, no less. > > Luc > > On Sunday, 29 March 2020 20:54:56 UTC-3, Tom Keffer wrote: >> >> I'm sorry, Luc, not trying to be difficult, but what is your question? I >> can't possibly debug your memory leak, especially with the complicated set >> of drivers and skins you are running. >> >> WeeWX itself does not have any memory leaks that I'm aware of. It is 100% >> Python, and Python uses garbage collection. If there are circular >> references, it can take a while for the interpreter to find unused and >> recycle memory, but WeeWX does not use any circular references. >> >> Here's an example <http://www.threefools.org/weewx/status/index.html> of >> a station that has been running continuously for well over a year. Present >> memory use is VIRT=75060 RES=37736 SHR=7144. So, very long runtimes are >> possible without leaks. >> >> Net-net, if there are leaks, I am pretty confident they are in the >> underlying C libraries. >> >> -tk >> >> On Sun, Mar 29, 2020 at 3:50 PM Lucas Heijst <[email protected]> wrote: >> >>> Tom, >>> >>> I have experienced two types of memory leak. >>> 1. In cronjobs when defining variables without unset them afterwards. >>> 2. With weewx reports (mben with 84+ generated graphs each run). >>> >>> Luc >>> >>> On Sunday, 29 March 2020 19:10:52 UTC-3, Tom Keffer wrote: >>>> >>>> By the way, in my experience, these leaks are caused by underlying "C" >>>> libraries or, occasionally, by failure of PIL to release graphics resource >>>> (such as font handles). >>>> >>>> On Sun, Mar 29, 2020 at 3:09 PM Thomas Keffer <[email protected]> wrote: >>>> >>>>> I'm not following. I thought you said you could get rid of the leak by >>>>> doing the unsets. If so, is there still a problem? >>>>> >>>>> >>>>> On Sun, Mar 29, 2020 at 8:41 AM Lucas Heijst <[email protected]> >>>>> wrote: >>>>> >>>>>> Tom, >>>>>> >>>>>> Currently running weewx 4.0.0b17 and analysing the memory leak in my >>>>>> pi31 system. >>>>>> On the system runs two weewx instances (mben and tfrc) and a 5-minute >>>>>> webcam cronjob. >>>>>> >>>>>> The memory leak of my webcam cronjob task was caused by creating >>>>>> variables like: >>>>>> WEBCAM_ID=3 >>>>>> CAMERA=picamera$WEBCAM_ID >>>>>> DATETIME=$(date +"%Y-%m-%d %H:%M:%S") >>>>>> EPOCH=$(date +"%s") >>>>>> ... >>>>>> >>>>>> The used memory is freed by: >>>>>> unset WEBCAM_ID >>>>>> unset CAMERA >>>>>> unset DATETIME >>>>>> unset EPOCH >>>>>> ... >>>>>> >>>>>> >>>>>> The memory leak of the weewx instances mben and tfrc are caused by >>>>>> the weewx reporting tasks, see data below: >>>>>> >>>>>> ----- >>>>>> used time >>>>>> mem >>>>>> ----- ----- >>>>>> 73348 10:30 >>>>>> 95504 11:07 >>>>>> 48576 11:38 no webcam, no mben, no tfrc (no increase of memory) >>>>>> 56136 11:41 mben without modbus read and modbus service >>>>>> 57384 11:43 >>>>>> 57876 11:44 >>>>>> 58360 11:45 >>>>>> 64836 11:47 used memory increased with 6476 due to mben31 report >>>>>> 64572 11:48 >>>>>> 64560 11:49 >>>>>> 64548 11:50 >>>>>> 66240 11:51 used memory increased with 1692 due to mben31 report >>>>>> 55808 11:53 mben without mben31 report >>>>>> 56092 11:54 >>>>>> 55828 11:55 >>>>>> 55808 11:56 >>>>>> 56000 11:57 >>>>>> 56000 11:58 >>>>>> 56516 11:59 >>>>>> 56772 12:00 >>>>>> 56164 12:02 mben now with modbus service (fully functional; without >>>>>> reporting) >>>>>> 56436 12:03 >>>>>> 55920 12:04 >>>>>> 76056 12:09 mben without mben31 report; tfrc started without tfrc31 >>>>>> report >>>>>> 76388 12:10 >>>>>> 77084 12:11 >>>>>> 77248 12:12 >>>>>> 77432 12:13 >>>>>> 77312 12:14 >>>>>> 76908 12:15 >>>>>> 76976 12:16 >>>>>> 77268 12:21 >>>>>> 77248 12:24 hardly any increase of used memory the last 15 minutes >>>>>> ----- >>>>>> >>>>>> When all reports are disabled, the memory leak on my system is >>>>>> practically zero. >>>>>> >>>>>> Attached the skin settings of mben31. >>>>>> >>>>>> Any clue how reduce this memory leak? >>>>>> >>>>>> Luc >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "weewx-development" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/weewx-development/d91724fe-da46-4b4e-9913-59f55aa91657%40googlegroups.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/weewx-development/d91724fe-da46-4b4e-9913-59f55aa91657%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "weewx-development" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/weewx-development/b5c12aeb-c8e1-489e-b457-04d2f8783ca0%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/weewx-development/b5c12aeb-c8e1-489e-b457-04d2f8783ca0%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "weewx-development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/f0a096da-16c4-48cc-86e6-13e9f18fba96%40googlegroups.com.
