Glenn,
I have chosen for another approach.
The MyReport service (see below) is currently embedded in tfrc.py and
modbusenergy.py.
Service MyReport opens a subproces which spawns the report command.
Output on syslog:
=====
Mar 30 10:25:01 pi31b CRON[1395]: (root) CMD
(/home/weewx/util/init.d/webcam.debian 1> /var/log/webcam.txt 2>
/var/log/webcam.err)
Mar 30 10:25:04 pi31b mben[595] INFO user.modbusenergy: huidige P1 1209 P2
147 P3 1209 PTot 2565 ETotEbsBer 68133.9
Mar 30 10:25:05 pi31b mben[595] INFO weewx.manager: Added record 2020-03-30
10:25:00 -03 (1585574700) to database 'weewx_mben31'
Mar 30 10:25:05 pi31b mben[595] INFO weewx.manager: Added record 2020-03-30
10:25:00 -03 (1585574700) to daily summary in 'weewx_mben31'
Mar 30 10:25:05 pi31b mben[595] INFO user.modbusenergy: Start MyReport for
/home/weewx/weewx_mben.conf
Mar 30 10:25:06 pi31b wee_reports[1422] INFO weewx.engine: StdConvert
target unit is 0x10
Mar 30 10:25:06 pi31b wee_reports[1422] INFO weewx.wxservices: The
following values will be calculated: pressure=prefer_hardware,
barometer=prefer_hardware, altimeter=prefer_hardware,
windchill=prefer_hardware, heatindex=prefer_hardware,
dewpoint=prefer_hardware, inDewpoint=prefer_hardware,
rainRate=prefer_hardware
Mar 30 10:25:06 pi31b wee_reports[1422] INFO weewx.wxservices: The
following algorithms will be used for calculations: altimeter=aaASOS,
maxSolarRad=RS
Mar 30 10:25:06 pi31b wee_reports[1422] INFO user.modbusenergy: service
version is 0.12
Mar 30 10:25:06 pi31b wee_reports[1422] INFO user.modbusenergy:
mben_service: debug_mben = 1
Mar 30 10:25:06 pi31b wee_reports[1422] INFO user.modbusenergy:
mben_service: binding=mbendevices31_binding
Mar 30 10:25:06 pi31b wee_reports[1422] INFO weewx.engine: Archive will use
data binding wx_binding
Mar 30 10:25:06 pi31b wee_reports[1422] INFO weewx.engine: Record
generation will be attempted in 'hardware'
Mar 30 10:25:06 pi31b wee_reports[1422] INFO weewx.engine: Using archive
interval of 60 seconds (specified by hardware)
Mar 30 10:25:06 pi31b wee_reports[1422] INFO user.modbusenergy: MyReport
service version is 0.12
Mar 30 10:25:07 pi31b wee_reports[1422] INFO weewx.cheetahgenerator:
Generated 1 files for report mben31 in 0.63 seconds
Mar 30 10:25:18 pi31b tfrc[542] INFO weewx.manager: Added record 2020-03-30
10:25:18 -03 (1585574718) to database 'cmon31'
Mar 30 10:25:18 pi31b tfrc[542] INFO weewx.manager: Added record 2020-03-30
10:25:18 -03 (1585574718) to daily summary in 'cmon31'
Mar 30 10:25:19 pi31b tfrc[542] INFO weewx.manager: Added record 2020-03-30
10:25:00 -03 (1585574700) to database 'weewx_tfrc31'
Mar 30 10:25:19 pi31b tfrc[542] INFO weewx.manager: Added record 2020-03-30
10:25:00 -03 (1585574700) to daily summary in 'weewx_tfrc31'
Mar 30 10:25:19 pi31b tfrc[542] INFO user.tfrc: Start MyReport for
/home/weewx/weewx_tfrc.
Mar 30 10:25:20 pi31b wee_reports[1452] INFO weewx.engine: StdConvert
target unit is 0x10
Mar 30 10:25:20 pi31b wee_reports[1452] INFO weewx.wxservices: The
following values will be calculated: pressure=prefer_hardware,
barometer=prefer_hardware, altimeter=prefer_hardware,
windchill=prefer_hardware, heatindex=prefer_hardware,
dewpoint=prefer_hardware, inDewpoint=prefer_hardware,
rainRate=prefer_hardware
Mar 30 10:25:20 pi31b wee_reports[1452] INFO weewx.wxservices: The
following algorithms will be used for calculations: altimeter=aaASOS,
maxSolarRad=RS
Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.cmon: service version is
0.18
Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.cmon: sysinfo: Linux
pi31b 4.19.66-v7+ #1253 SMP Thu Aug 15 11:49:46 BST 2019 armv7l
Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.cmon: cpuinfo: CPU
variant: 0x0
Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.cmon: cconfpuinfo: CPU
architecture: 7
Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.cmon: cpuinfo: CPU part:
0xd03
Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.cmon: cpuinfo: CPU
implementer: 0x41
Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.cmon: cpuinfo: BogoMIPS:
76.80
Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.cmon: cpuinfo: Revision:
a02082
Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.cmon: cpuinfo: Serial:
00000000496ca922
Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.cmon: cpuinfo: model
name: ARMv7 Processor rev 4 (v7l)
Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.cmon: cpuinfo: Hardware:
BCM2835
Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.cmon: cpuinfo: CPU
revision: 4
Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.cmon: cpuinfo: processor:
3
Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.cmon: cpuinfo: Features:
half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae
evtstrm crc32
Mar 30 10:25:20 pi31b wee_reports[1452] INFO weewx.engine: Archive will use
data binding wx_binding
Mar 30 10:25:20 pi31b wee_reports[1452] INFO weewx.engine: Record
generation will be attempted in 'hardware'
Mar 30 10:25:20 pi31b wee_reports[1452] INFO weewx.engine: Using archive
interval of 300 seconds (specified by hardware)
Mar 30 10:25:20 pi31b wee_reports[1452] INFO user.tfrc: MyReport service
version is 0.4
Mar 30 10:25:21 pi31b wee_reports[1452] INFO weewx.cheetahgenerator:
Generated 1 files for report tfrc31 in 0.54 seconds
Mar 30 10:25:29 pi31b wee_reports[1452] INFO weewx.imagegenerator:
Generated 5 images for report tfrc31 in 7.17 seconds
Mar 30 10:25:29 pi31b wee_reports[1422] INFO weewx.imagegenerator:
Generated 84 images for report mben31 in 21.51 seconds
Mar 30 10:25:29 pi31b mben[595] INFO user.modbusenergy: MyReport for
/home/weewx/weewx_mben.conf finished
Mar 30 10:25:39 pi31b mben[595] INFO weewx.manager: Added record 2020-03-30
10:25:39 -03 (1585574739) to database 'mbendevices31'
Mar 30 10:25:39 pi31b mben[595] INFO weewx.manager: Added record 2020-03-30
10:25:39 -03 (1585574739) to daily summary in 'mbendevices31'
Mar 30 10:25:55 pi31b wee_reports[1452] INFO weewx.reportengine:
ftpgenerator: Ftp'd 2 files in 26.36 seconds
Mar 30 10:25:55 pi31b tfrc[542] INFO user.tfrc: MyReport for
/home/weewx/weewx_tfrc.conf finished
Mar 30 10:26:04 pi31b mben[595] INFO user.modbusenergy: huidige P1 178 P2
133 P3 155 PTot 465 ETotEbsBer 68134.0
Mar 30 10:26:04 pi31b mben[595] INFO weewx.manager: Added record 2020-03-30
10:26:00 -03 (1585574760) to database 'weewx_mben31'
Mar 30 10:26:05 pi31b mben[595] INFO weewx.manager: Added record 2020-03-30
10:26:00 -03 (1585574760) to daily summary in 'weewx_mben31'
=====
Luc
*Modified files:*
In weewx_tfrc.conf:
...
report_services = weewx.engine.StdPrint, user.tfrc.MyReport
In weewx_mben.conf:
...
report_services = weewx.engine.StdPrint, user.modbusenergy.MyReport
In tfrc.py and modbusenergy.py:
...
import subprocess
from subprocess import Popen,PIPE
...
DRIVER_NAME = 'TFRC'
DRIVER_VERSION = '0.4'
MY_REPORT_CONF = '/home/weewx/weewx_tfrc.conf' # (for tfrc.py)
or
MY_REPORT_CONF = '/home/weewx/weewx_mben.conf' # (for modbusenergy.py)
...
############################## Service ##############################
class MyReport(StdService):
""" MyReport replaces StdReport"""
def __init__(self, engine, config_dict):
super(MyReport, self).__init__(engine, config_dict)
loginf("MyReport service version is %s" % DRIVER_VERSION)
self.bind(weewx.NEW_ARCHIVE_RECORD, self.handle_new_archive)
def handle_new_archive(self, event):
loginf("Start MyReport for %s" % MY_REPORT_CONF)
#start and process things, then wait
reportArg = "--config=" + MY_REPORT_CONF
p = subprocess.Popen(["/home/weewx/bin/wee_reports", reportArg],
stdout=PIPE)
p.communicate() #now wait plus that you can send commands to process
loginf("MyReport for %s finished" % MY_REPORT_CONF)
############################## Conf Editor ##############################
On Monday, 30 March 2020 08:30:09 UTC-3, Glenn McKechnie wrote:
>
> Luc,
>
> (Bugger, and this one to the list)
> NB:
> The script lines wordwrap with my mailer, some may need.. straightening?
>
> On 30/03/2020, Lucas Heijst <[email protected] <javascript:>> wrote:
> > Glen,
> >
> > I like the method you use by firing the report in another way but I am
> not
> > keen in modifying engine.py (each time with a new version of weewx).
> > I could put the report calls in a cron job task, but then they are
> executed
> >
> > even when weewx is not running.
> > Let me think about it. The timing when the report is started is not a
> > problem I expect. I have runned many times the report by hand while the
> > weewx instances were running somewhere in their cycli.
>
>
> The following is a description of bypassing the internal weewx report
> generation, and replacing it with a seperate instance.
> It looks complicated but in actual fact, it's pretty simple and runs
> flawlessly here.
>
> To recap...
> The objective is to remove any memory problems that occur during the
> report cycle (whatever the cause) by ensuring that the report cycle is
> seperated from the main weewx instance. This allows it to release all
> its memory upon completion. It starts afresh each time it's called.
> The reason for doing it this way is to ensure that the reports are
> generated at the same time weewx would have done them, the theory
> being that there is less chance of interference/breakage of weewx.
>
>
> It consists of ...
> 1. patching engine.py, and tweaking weewx.conf
> 2. adding a call to rsyslog
> 3. creating a calling script for rsyslog to action.
> 4. creating a wee_report calling script, that gets out of the way of
> rsyslog.
>
> The following expands on those 4 points.
>
> 1.
> $ diff -Naur /home/weewx/bin/weewx/engine.py.org
> /home/weewx/bin/weewx/engine.py
> --- /home/weewx/bin/weewx/engine.py.org 2020-03-30 17:49:40.760732824
> +1100
> +++ /home/weewx/bin/weewx/engine.py 2020-01-26 03:47:05.588871567
> +1100
> @@ -602,6 +602,11 @@
> raise ValueError("Unknown station record generation
> value %s" % self.record_generation)
> self.old_accumulator = None
>
> + # Glenn McKechnie - flag to run wee_reports
> + # also turn off StdReport
> + # report_services = weewx.engine.StdPrint # ,
> weewx.engine.StdReport
> + log.info("engine: weewx loop has finished")
> +
> # Set the time of the next break loop:
> self.end_archive_delay_ts = self.end_archive_period_ts +
> self.archive_delay
>
> Those 5 lines that start with a "+" are the only change that's made to
> engine.py. It doesn't break anything. If it's not there the following
> won't work and no independent report run will be performed.
> It's only one-line that actually matters, the comments are there to
> aid my memory when I need to, very occasionally, find and revisit it.
> If it is there and you don't use it, your logs suffer the indignity of
> a pointless message.
> If it is there and weewx is not running, then wee_reports will not be
> called by the scripts, as distinct from a CRON entry which would only
> stay silent if it could detect a stopped weewx.
>
> The other change is to weewx.conf where the report_services line has
> the weewx.engine.StdReport entry commented out.
> report_services = weewx.engine.StdPrint # , weewx.engine.StdReport
>
> 2.
> I've redirected my syslog output to a weewx specific log file so I use
> the rsyslog.d/weewx.conf example file. The next step is to add the
> following additional lines to the top of that rsyslog.d conf file.
> $ /etc/rsyslog.d/weewx.conf
> #
> # This will break rsyslog if the called command takes too long to
> complete.
> # Therefore, the script that this points to has to be simplistic. In fact
> # this points to a one-line call to the real script before it
> backgrounds/completes.
> # This script is mode 0644
> # source https://www.slideshare.net/rainergerhards1/writing-rsyslog-p
> #
> module(load="omprog")
> if $rawmsg contains 'weewx loop has finished' then
> action(type="omprog"
> binary="/home/weewx/bin/user/runreports.sh")
> #
>
> 3.
> The script that is being called consists of six lines, but only one
> truly matters and that's the call to run_weereports.sh
> $ /home/weewx/bin/user/runreports.sh
> #!/bin/sh
> # script called from rsyslog.conf or equivalent ... rsyslog.d/weewx.conf
> `bash /home/weewx/bin/user/run_weereports.sh &`
> exit 0
> # This script needs to be executable
> # run chmod 0755 /home/weewx/bin/user/runreports.sh
>
> It calls /home/weewx/bin/user/run_weereports.sh and then drops into
> the background and thereby returns control to the rsyslog daemon. It
> may not look like much but this script is an important intermediate
> step. If this was the actual script that invoked wee_reports it would
> take too long to complete and you'd find rsyslog would break. (ie:-
> you'd start getting log messages to your terminal and nothing to any
> of the log files!)
>
> 4.
> Now, The following is the actual call to wee_reports, run_weereports.sh
> This is mine, yours can be whatever you like. I write a PID file and
> with that I can decide to run wee_reports every archive , or every
> alternate archive interval. It doesn't matter, just so long as it runs
> wee_reports once and that you know what's going on.
> This has syslog entries, you can comment them out once you've finished
> with them.
>
> $ /home/weewx/bin/user/run_weereports.sh
> #!/bin/bash
> # script called from /home/weewx/bin/user/runreports.sh
> pid_file='/var/run/runreports.pid'
> tymestamp=$(date +%d%m%Y-%H:%M)
>
> if [ -f $pid_file ]
> then
> `rm -f $pid_file`
> logger --id --tag weewx_weatherpi "Skipped this weewx report by
> removing $pid_file"
> else
> $(/home/weewx/bin/wee_reports --config=/home/weewx/weatherpi.conf &
> echo $! > $pid_file)
> logger --id --tag weewx_weatherpi "Ran wee_reports manually,
> created $pid_file"
> # comment the following to skip alternate wee_report runs
> # `rm -f $pid_file`
> fi
> # This script can be made executable if you wish
> # run chmod 0755 /home/weewx/bin/user/run_weereports.sh
>
> exit 0
>
> Putting it all together...
>
> Change anything in the scripts that fits your setup. My weatherpi.conf
> is probably the equivalent of your weewx.conf. My directories may not
> match yours. Tweak to suit.
>
> I'd suggest starting with the last script, run it and make sure it's
> doing what you want. (running wee_reports)
>
> Next, create the second (calling ) script and make sure that it does
> what you expect it to. (running the above script, that then runs
> wee_reports)
>
> Then I'd add the rsyslog modifications and restart rsyslog. Nothing
> spectacular should happen, except that it restarts and stays running.
> Ah... If you don't have seperate weewx logging, then that addition I
> mention (the 12 lines including module(load="omprog") ) goes into
> your /etc/rsyslog.conf file after the #### RULES #### header (It's
> been a while since I did it that way, I'm pretty sure that was the
> spot.)
>
> The next step is to test it by running the command...
> logger "weewx loop has finished"
> and that simulates what the (not yet) modified engine.py will utter.
> When that message hits syslog wee_reports should fire up and do its
> thing. (ie:- the above calling script runs and that executes the
> script that actually fires up wee_reports)
>
> Up to this point, the dominoes have been placed but only a manual step
> can set them in motion.
> So, if all the above works as intended then modify engine.py, then
> weewx.conf and you've set the trigger. Restart weewx and it should
> fire. Review your reports. Watch your logs, there should be a series
> of wee_report entries occur at the appropriate point (ie: same as
> before) in the weewx cycle.
>
> Run it for a while and check your memory usage and it should be flat
> lining. That's if StdReports was the cause of your memory leak (which
> is probably in the version of PIL or other C lib as Tom notes.)
>
> If you still get a message about the report cycle being blocked by a
> previous run (I was fairly certain that message still occured but as I
> think about it , doubt enters. Hmmm.) then there is a serious problem
> within the actual report run. I've had something similar with mysql
> hitting the roof due to a ... Okay, I should go back and revisit that
> issue. :-(
>
> That's it. Hopefully I've missed nothing, and that it's of use.
>
>
> > On Sunday, 29 March 2020 23:32:44 UTC-3, Glenn McKechnie wrote:
> >>
> >> On 30/03/2020, Lucas Heijst <[email protected] <javascript:>> wrote:
> >> [...]
> >> > 2. With weewx reports (mben with 84+ generated graphs each run).
> >> >
> >> > Luc
> >>
> >> I have a fairly complex set of graphs ( multiple plots, large size,
> >> anti-alias turned on) and I have a persistent and steady climb until
> >> the OOM killer takes over. If I simplify the graphing then it holds up
> >> very well, but there's no fun in that!
> >>
> >> Rather than a Cronjob to force a set restart, I've turned off
> StdReport
> >> report_services = weewx.engine.StdPrint # , weewx.engine.StdReport
> >> and then once engine.py has finished its post_loop(self, _event):
> >> function, it writes a log message to loginfo (rsyslog).
> >> syslog picks up that message and then fires off an instance of
> >> wee_reports.
> >>
> >> The advantage of this is that...
> >> 1. The Reports are called at the correct point in the weewx cycle.
> >> 2. Once wee_reports finishes it closes completely (assuming another
> >> problem isn't holding it open) and completely frees the memory it's
> >> used.
> >> 3. Weewx stays up and I don't waste anymore time looking for a memory
> >> leak that I can do nothing about, or is way above my skill set. I've
> >> chased PIL versions and run memory profilers but never had a suitable
> >> Eureka moment.
> >>
> >> The disadvantage is that...
> >> 1. weeWX uptime is reported as zero
> >> 2. engine.py needs a one line patch (a suitable loginfo message)
> >> 3. rsyslog is easy but a set procedure needs following. One that's
> >> simple in hindsight.
> >>
> >> now, in a week, weewx might use 2 Meg. It basically flatlines with this
> >> action.
> >>
> >> If you want I'll track down the scripts, post them and outline the full
> >> logic.
>
> --
>
> Cheers
> Glenn
>
> rorpi - read only raspberry pi & various weewx addons
> https://github.com/glennmckechnie
>
--
You received this message because you are subscribed to the Google Groups
"weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-development/b9b3eaba-d711-484a-9b77-28be3bfef0ce%40googlegroups.com.
