[weewx-development] Re: weewx version 4 can't keep up in time
Vice, The memory leaks are caused by the weewx reporting (currently weewx 4.0.0b17) I have posted this in a seperate thread. Luc On Sunday, 29 March 2020 08:11:24 UTC-3, Lucas Heijst wrote: > > Vince, > > I found one source of a memory leak: the webcam cronjob. > In the webcam.debian file a number of variables were used like: > WEBCAM_ID=4 > CAMERA=picamera$WEBCAM_ID > DATETIME=$(date +"%Y-%m-%d %H:%M:%S") > EPOCH=$(date +"%s") > ... > > The memory which is used by these variables is not automatically freed > after the cronjob task ends. > You need to unset them like: > unset WEBCAM_ID > unset CAMERA > unset DATETIME > unset EPOCH > ... > > Now the memory leak of the cronjob is minimal. > > Luc > > > > On Friday, 27 March 2020 23:34:22 UTC-3, Lucas Heijst wrote: >> >> Vince, >> >> These are the steps I will do: >> 1. Stop mben, this will also stop cmon. Tfrc runs and the web photos are >> taken starting at 06:00 tomorrow morning. >> 2. When they are stable I will move cmon to tfrc. >> 3. When tfrc, web photo and cmon are stable I will start the mben driver >> but stop the mben service which handles the data of the mben driver. >> >> Luc >> >> On Friday, 27 March 2020 19:14:18 UTC-3, Vince Skahan wrote: >>> >>> On Friday, March 27, 2020 at 12:45:22 PM UTC-7, Lucas Heijst wrote: Vince, As I stated earlier, I have no knowledge how to get program modbusenergy stable. I only can monitor the difference in memory leak with different settings. Restarting a program doesn't change the used memory. >>> >>> Disable it. We need to see which of the many things you are running is >>> the unstable thing. >>> >> -- You received this message because you are subscribed to the Google Groups "weewx-development" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-development+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/eac3f14d-599c-4149-81f1-2d35e7946f54%40googlegroups.com.
[weewx-development] Re: weewx version 4 can't keep up in time
Vince, I found one source of a memory leak: the webcam cronjob. In the webcam.debian file a number of variables were used like: WEBCAM_ID=4 CAMERA=picamera$WEBCAM_ID DATETIME=$(date +"%Y-%m-%d %H:%M:%S") EPOCH=$(date +"%s") ... The memory which is used by these variables is not automatically freed after the cronjob task ends. You need to unset them like: unset WEBCAM_ID unset CAMERA unset DATETIME unset EPOCH ... Now the memory leak of the cronjob is minimal. Luc On Friday, 27 March 2020 23:34:22 UTC-3, Lucas Heijst wrote: > > Vince, > > These are the steps I will do: > 1. Stop mben, this will also stop cmon. Tfrc runs and the web photos are > taken starting at 06:00 tomorrow morning. > 2. When they are stable I will move cmon to tfrc. > 3. When tfrc, web photo and cmon are stable I will start the mben driver > but stop the mben service which handles the data of the mben driver. > > Luc > > On Friday, 27 March 2020 19:14:18 UTC-3, Vince Skahan wrote: >> >> On Friday, March 27, 2020 at 12:45:22 PM UTC-7, Lucas Heijst wrote: >>> >>> Vince, >>> >>> As I stated earlier, I have no knowledge how to get program modbusenergy >>> stable. >>> I only can monitor the difference in memory leak with different settings. >>> Restarting a program doesn't change the used memory. >>> >> >> Disable it. We need to see which of the many things you are running is >> the unstable thing. >> > -- You received this message because you are subscribed to the Google Groups "weewx-development" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-development+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/df470038-c175-4f49-ba90-a9356c408862%40googlegroups.com.
[weewx-development] Re: weewx version 4 can't keep up in time
Vince, These are the steps I will do: 1. Stop mben, this will also stop cmon. Tfrc runs and the web photos are taken starting at 06:00 tomorrow morning. 2. When they are stable I will move cmon to tfrc. 3. When tfrc, web photo and cmon are stable I will start the mben driver but stop the mben service which handles the data of the mben driver. Luc On Friday, 27 March 2020 19:14:18 UTC-3, Vince Skahan wrote: > > On Friday, March 27, 2020 at 12:45:22 PM UTC-7, Lucas Heijst wrote: >> >> Vince, >> >> As I stated earlier, I have no knowledge how to get program modbusenergy >> stable. >> I only can monitor the difference in memory leak with different settings. >> Restarting a program doesn't change the used memory. >> > > Disable it. We need to see which of the many things you are running is > the unstable thing. > -- You received this message because you are subscribed to the Google Groups "weewx-development" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-development+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/ca36afa2-0cab-4d23-9dca-12097abf407c%40googlegroups.com.
[weewx-development] Re: weewx version 4 can't keep up in time
On Friday, March 27, 2020 at 12:45:22 PM UTC-7, Lucas Heijst wrote: > > Vince, > > As I stated earlier, I have no knowledge how to get program modbusenergy > stable. > I only can monitor the difference in memory leak with different settings. > Restarting a program doesn't change the used memory. > Disable it. We need to see which of the many things you are running is the unstable thing. -- You received this message because you are subscribed to the Google Groups "weewx-development" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-development+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/c2abfd58-2cf9-4ce1-b167-6fd8118aa193%40googlegroups.com.
[weewx-development] Re: weewx version 4 can't keep up in time
Vince, As I stated earlier, I have no knowledge how to get program modbusenergy stable. I only can monitor the difference in memory leak with different settings. Restarting a program doesn't change the used memory. Luc On Friday, 27 March 2020 16:32:56 UTC-3, Vince Skahan wrote: > > On Friday, March 27, 2020 at 11:47:34 AM UTC-7, Lucas Heijst wrote > >> Program modbusenergy has a memory leak of 80 KB per 24 h (nothing new), >> so the memory will be full after 7.5 days. >> It looks like the program starts and stops a program twice per second, >> because each time I perform 'ps aux' the count is higher (2 counts per >> second). >> Currently the counter has reach a maximum of 32316 and does't increase >> anymore. >> >> I have too little knowledge of linux and python to debug the memory leak, >> so as a work-around my system is rebooted each 5th day. >> >> >> > I'd reiterate my earlier suggestion.. > > - disable everything except one thing > - run it and get that stable > - add one thing at a time > - get 'that' stable > - then add the next thing > - and so on > > Alternately. > > - start with your as-is > - disable one thing at a time > - see if that fixes things for a few days > - if not, disable the next thing > - and so on (ie, same as above, just by subtracting services) > > From your skeletal description it certainly sounds like modbusenergy might > be your problem. Can you disable that first for a few days and see if > things stabilize ? > > You should never have to reboot a Linux system for stability reasons. My > ancient 128 MB RAM Seagate dockstar running my primary weewx has been up > for 3.3 years, although I reset weewx occasionally when I update versions > that of course. > > > -- You received this message because you are subscribed to the Google Groups "weewx-development" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-development+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/6c485b26-a93f-4c88-b5d0-b188a28c1206%40googlegroups.com.
[weewx-development] Re: weewx version 4 can't keep up in time
On Friday, March 27, 2020 at 11:47:34 AM UTC-7, Lucas Heijst wrote > Program modbusenergy has a memory leak of 80 KB per 24 h (nothing new), so > the memory will be full after 7.5 days. > It looks like the program starts and stops a program twice per second, > because each time I perform 'ps aux' the count is higher (2 counts per > second). > Currently the counter has reach a maximum of 32316 and does't increase > anymore. > > I have too little knowledge of linux and python to debug the memory leak, > so as a work-around my system is rebooted each 5th day. > > > I'd reiterate my earlier suggestion.. - disable everything except one thing - run it and get that stable - add one thing at a time - get 'that' stable - then add the next thing - and so on Alternately. - start with your as-is - disable one thing at a time - see if that fixes things for a few days - if not, disable the next thing - and so on (ie, same as above, just by subtracting services) >From your skeletal description it certainly sounds like modbusenergy might be your problem. Can you disable that first for a few days and see if things stabilize ? You should never have to reboot a Linux system for stability reasons. My ancient 128 MB RAM Seagate dockstar running my primary weewx has been up for 3.3 years, although I reset weewx occasionally when I update versions that of course. -- You received this message because you are subscribed to the Google Groups "weewx-development" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-development+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/0f366208-e9b1-46ee-949a-e8048a0721e0%40googlegroups.com.
[weewx-development] Re: weewx version 4 can't keep up in time
Gary, The programs modbusenergy.py, tfrc.py and cmon.py are modified to use module logging instead of syslog. Everything still works OK. Program modbusenergy has a memory leak of 80 KB per 24 h (nothing new), so the memory will be full after 7.5 days. It looks like the program starts and stops a program twice per second, because each time I perform 'ps aux' the count is higher (2 counts per second). Currently the counter has reach a maximum of 32316 and does't increase anymore. I have too little knowledge of linux and python to debug the memory leak, so as a work-around my system is rebooted each 5th day. Luc On Thursday, 26 March 2020 14:24:27 UTC-3, Lucas Heijst wrote: > > Thanks Gary, > > In no means I wanted to be hars, sorry about that. I am Dutch and we > usualy say things with less diplomacy than others. > Currently I am gathering the information you asked for. > > The good news is that the following combination seems to work OK: > - > RPI 3B+ > weewx version 4.0.0b16 > Python 2.7.13 > Platform Linux-4.19.66-v7+-armv7l-with-debian-9.11 (latest version of > raspbian jessie) > - > The weewx programs modbusenergy.py, tfrc.py and cmon.py (used with weewx > 3.9.2) were not modified so far. > The difference between version 3.9.2 and 4.0.0.b16 for the data in > /var/log/syslog is the debug logging which will not be suppressed with > setting ‘debug = 0’ for the program messages generated with: > def logmsg(dst, msg): > syslog.syslog(dst, 'ModbusEnergy: %s' % msg) > I will let this configuration run for 24 hours and then modify the three > user.py files to use logging: > log = logging.getLogger(__name__) instead of syslog. > I will keep you informed. > > Luc > > On Thursday, 26 March 2020 04:47:37 UTC-3, gjr80 wrote: >> >> On Thursday, 26 March 2020 13:21:55 UTC+10, Lucas Heijst wrote: >> >>> I think the fact that weewx can't keep up in time to handle the 1-minute >>> archive data is the reason that the memory is filled. >>> >> >> I think you are being a little harsh here Luc. From what i can read of >> the log extracts you are running a rather complex setup (two instances of >> WeeWX and at least three databases). I can think of a number of scenarios >> that could cause the symptoms you are seeing and none involve WeeWX not >> 'keeping up'. >> >> >>> Each archive record contains 90 real values (doubles) and these values >>> are stored in memory (I suppose) as long as they are not written to the >>> database. >>> >> One hour before the low-memory crash weewx was 90 1-minute archive >>> records behind. >>> >> >> Might help to have a little more info if you want any meaningful advice >> other than maybe this or maybe that. >> >> Are you running with software archive record generation? If so then WeeWX >> does not store any archive records (other than at most two, the last one >> that is being processed and the current one that is being accumulated). >> WeeWX accumulates loop packets and as soon as WeeWX receives a loop packet >> that belongs in the next archive period WeeWX synthesises an archive record >> and emits a NEW_ARCHIVE_RECORD event which causes that archive record to be >> archived and otherwise processed by the WeeWX services. The loop packet >> from the next archive period (and subsequent loop packets) are accumulated >> to create the next archive record. And the process repeats. So on a system >> with a driver that normally emits loop packets every 4 seconds, but for >> some reason these loop packets are being blocked within the driver and >> released much less frequently, WeeWX will only emit an archive record once >> it has received a loop packet timestamped in the next archive period. So >> what you can see happening is archive records being emitted and saved to >> database but at intervals much longer than the archive interval, in many >> cases the interval between these archive records being emitted increases as >> more and more loop packets are delayed. Ultimately an out of memory error >> can occur, if things haven't ground to a halt beforehand. >> >> How about providing some config details? A debug log from startup would >> show almost everything needed to gicve us a more complete picture of your >> setup. What about your driver, does it interrogate hardware in its own >> thread or in the WeeWX thread? If it is not in its own thread it is >> possible that it is blocking and causing delays. What happens when you run >> WeeWX directly, do you see loop packets appearing, does the difference in >> successive loop packet timestamps align with the frequency they appear on >> the console? (ie if the loop packets are timestamped every four seconds do >> they appear on the console every fours seconds or every 15 seconds, 30 >> seconds, 60 seconds?) If there is a discrepancy that is a sure sign that >> something is blocking. How about posting a copy of the driver? >> >> Hard to say a great deal without more logs, more config
[weewx-development] Re: weewx version 4 can't keep up in time
On Thursday, 26 March 2020 13:21:55 UTC+10, Lucas Heijst wrote: > I think the fact that weewx can't keep up in time to handle the 1-minute > archive data is the reason that the memory is filled. > I think you are being a little harsh here Luc. From what i can read of the log extracts you are running a rather complex setup (two instances of WeeWX and at least three databases). I can think of a number of scenarios that could cause the symptoms you are seeing and none involve WeeWX not 'keeping up'. > Each archive record contains 90 real values (doubles) and these values are > stored in memory (I suppose) as long as they are not written to the > database. > One hour before the low-memory crash weewx was 90 1-minute archive records > behind. > Might help to have a little more info if you want any meaningful advice other than maybe this or maybe that. Are you running with software archive record generation? If so then WeeWX does not store any archive records (other than at most two, the last one that is being processed and the current one that is being accumulated). WeeWX accumulates loop packets and as soon as WeeWX receives a loop packet that belongs in the next archive period WeeWX synthesises an archive record and emits a NEW_ARCHIVE_RECORD event which causes that archive record to be archived and otherwise processed by the WeeWX services. The loop packet from the next archive period (and subsequent loop packets) are accumulated to create the next archive record. And the process repeats. So on a system with a driver that normally emits loop packets every 4 seconds, but for some reason these loop packets are being blocked within the driver and released much less frequently, WeeWX will only emit an archive record once it has received a loop packet timestamped in the next archive period. So what you can see happening is archive records being emitted and saved to database but at intervals much longer than the archive interval, in many cases the interval between these archive records being emitted increases as more and more loop packets are delayed. Ultimately an out of memory error can occur, if things haven't ground to a halt beforehand. How about providing some config details? A debug log from startup would show almost everything needed to gicve us a more complete picture of your setup. What about your driver, does it interrogate hardware in its own thread or in the WeeWX thread? If it is not in its own thread it is possible that it is blocking and causing delays. What happens when you run WeeWX directly, do you see loop packets appearing, does the difference in successive loop packet timestamps align with the frequency they appear on the console? (ie if the loop packets are timestamped every four seconds do they appear on the console every fours seconds or every 15 seconds, 30 seconds, 60 seconds?) If there is a discrepancy that is a sure sign that something is blocking. How about posting a copy of the driver? Hard to say a great deal without more logs, more config details and more driver details. Gary -- You received this message because you are subscribed to the Google Groups "weewx-development" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-development+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/402c8712-b979-4d29-a056-2b745068f1e0%40googlegroups.com.
[weewx-development] Re: weewx version 4 can't keep up in time
Vince, I think the fact that weewx can't keep up in time to handle the 1-minute archive data is the reason that the memory is filled.. Each archive record contains 90 real values (doubles) and these values are stored in memory (I suppose) as long as they are not written to the database. One hour before the low-memory crash weewx was 90 1-minute archive records behind. Luc On Thursday, 26 March 2020 00:03:52 UTC-3, Vince Skahan wrote: > > On Wednesday, March 25, 2020 at 12:06:30 PM UTC-7, Lucas Heijst wrote: >> >> Not sure. See the attached cpu data graph. >> The empty parts in the graph are a result of the linux system looping >> (see: crash log) >> >> >>> > The log showed the oom_killer which is more likely your problem. You > must have a memory leak somewhere. > > I'd probably suggest the usual steps. Run just the simulator and verify > weewx is ok. Then add things one by one and try to see which one is the > culprit. > > You can grab my 'mem' extension from > https://github.com/vinceskahan/vds-weewx-v3-mem-extension if you need a > lightweight thing to watch weewx memory usage. It's a bit lighter weight > than cmon/pmon/etc. I think. > > > -- You received this message because you are subscribed to the Google Groups "weewx-development" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-development+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/11926381-42cf-4617-b2f7-860a72467d55%40googlegroups.com.
[weewx-development] Re: weewx version 4 can't keep up in time
On Wednesday, March 25, 2020 at 12:06:30 PM UTC-7, Lucas Heijst wrote: > > Not sure. See the attached cpu data graph. > The empty parts in the graph are a result of the linux system looping > (see: crash log) > > >> The log showed the oom_killer which is more likely your problem. You must have a memory leak somewhere. I'd probably suggest the usual steps. Run just the simulator and verify weewx is ok. Then add things one by one and try to see which one is the culprit. You can grab my 'mem' extension from https://github.com/vinceskahan/vds-weewx-v3-mem-extension if you need a lightweight thing to watch weewx memory usage. It's a bit lighter weight than cmon/pmon/etc. I think. -- You received this message because you are subscribed to the Google Groups "weewx-development" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-development+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/ac2cf9e9-a69e-47ca-8e29-65826eba2e29%40googlegroups.com.
[weewx-development] Re: weewx version 4 can't keep up in time
Sounds like something is blocking waiting for i/o to complete perhaps (???) -- You received this message because you are subscribed to the Google Groups "weewx-development" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-development+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/7baff891-8b26-4c47-a5ff-05144f8f6539%40googlegroups.com.
