Very clever! Thanks!! I’ll give it a try in a few hours.
Regards, Garry Lockyer C: +1.250.689.0686 E: [email protected] > On Jan 20, 2021, at 06:24, [email protected] <[email protected]> wrote: > > Note, it is growing - just much slower. > >> On Wednesday, 20 January 2021 at 09:13:35 UTC-5 [email protected] wrote: >> Garry, >> I might have a workaround for you. Wrap your html file in a template. >> something like this. >> index.html.tmp (this is in the belchertown skin) >> #include index_hook_after_charts.inc >> >> index_hook_after_charts.inc (this is the 'wrapper', naturally if you 'owned' >> index.html.tmpl you would not need it) >> #include raw generated.html >> >> generated.html (the file you generate - name as you want) >> your html >> >> I ran this on my loop 10,000 times and saw no appreciable memory increase >> and performed better too. >> rich >> >>> On Tuesday, 19 January 2021 at 22:45:42 UTC-5 [email protected] wrote: >>> Many thanks to all for poking at this! >>> >>> >>> Regards, >>> >>> Garry Lockyer >>> C: +1.250.689.0686 >>> E: [email protected] >>> >>> >>>>> On Jan 19, 2021, at 19:38, vince <[email protected]> wrote: >>>>> >>>> Just a followup - I took belchertown out of the picture and used my >>>> minimal demo-skin and added 'one line' that #include(d) a file generated >>>> by Garry's service, and it 'is' leaking memory albeit less quickly than >>>> the belchertown example..... >>> >>>> >>>> I'll let it run overnight and we'll see what it looks like after 12 more >>>> hours running. The VM is set to only 1GB RAM but I'm hoping if it runs >>>> out it'll just go into swap and stay up. OS is debian 10 under >>>> Vagrant/VirtualBox with 4.3.0 installed using python3 and setup.py >>>> >>>> Some mem.sdb output (date/time, units, interval, mem_size, mem_rss, >>>> mem_share)... >>>> >>>> 1611109216 16 5 141.0625 71.734375 12.2734375 >>>> 1611109516 16 5 142.5625 75.65234375 12.3359375 >>>> 1611109816 16 5 144.0625 79.65625 12.3359375 >>>> 1611110116 16 5 145.5625 84.5234375 12.3359375 >>>> 1611110416 16 5 147.24609375 88.0859375 12.3359375 >>>> 1611110716 16 5 148.49609375 91.43359375 12.3359375 >>>> 1611111016 16 5 149.99609375 95.421875 12.3359375 >>>> 1611111316 16 5 151.49609375 100.3359375 12.3359375 >>>> 1611111616 16 5 152.99609375 103.8359375 12.3359375 >>>> 1611111916 16 5 154.49609375 107.51171875 12.3359375 >>>> 1611112216 16 5 155.99609375 111.5 12.3359375 >>>> 1611112516 16 5 157.24609375 116.0859375 12.3359375 >>>> 1611112816 16 5 158.74609375 119.5859375 12.3359375 >>>> 1611113116 16 5 160.24609375 123.24609375 12.3359375 >>>> 1611113416 16 5 161.74609375 127.234375 12.3359375 >>>> 1611113716 16 5 163.24609375 132.11328125 12.3359375 >>>> >>>> >>>> >>>>> On Tuesday, January 19, 2021 at 5:21:31 PM UTC-8 [email protected] wrote: >>>>> Ok, my standalone testcase was wrong. I think I have a valid one here, >>>>> https://github.com/bellrichm/experiments >>>>> It just calls Cheetah.Template.Template in a loop. If the include file >>>>> modified attribute does not change, all seems stable. If it changes, >>>>> memory usage seems to grow. >>>>> rich >>>>> >>>>>> On Tuesday, 19 January 2021 at 19:16:46 UTC-5 [email protected] wrote: >>>>>> I'm not quite following this discussion. >>>>>> >>>>>> With Cheetah, you can compile the template and save the results. I >>>>>> believe this is what monitors the changed status of #include files. But, >>>>>> that's not the way WeeWX uses templates. It compiles, evaluates, then >>>>>> throws the results away. >>>>>> >>>>>> If you use the "raw" directive in an #include, then the file will not be >>>>>> parsed. So, how can any $ directives work? >>>>>> >>>>>> Or, does Belchertown work differently? >>>>>> >>>>>> -tk >>>>>> >>>>>>> On Tue, Jan 19, 2021 at 3:55 PM [email protected] <[email protected]> >>>>>>> wrote: >>>>>>> I agree that something does not seem quite right with Cheetah’s >>>>>>> ‘monitoring’ of include files. I took a different approach. >>>>>>> First, I wrote a simple WeeWX service that would perform the ‘touch’ >>>>>>> command against the large file and a simple template that included the >>>>>>> large file. When the file modified attribute changed, I observed the >>>>>>> memory growth, otherwise all was stable. Also, when the file was not >>>>>>> ‘touched’ the Cheetah processing was noticeably faster. >>>>>>> Next I wrote a standalone program to run a loop that ran the touch >>>>>>> command and invoke Cheetah. I ran it 3 times. Each time around loop >>>>>>> 90, the memory growth stopped and the processing was fast as though >>>>>>> Cheetah was using cached data. >>>>>>> I am back to running under WeeWX to see if it will level off like it >>>>>>> does standalone. I am also wading through the Cheetah code and >>>>>>> documentation. As of now, I thoroughly confused. >>>>>>> rich >>>>>>> >>>>>>>>> On Tuesday, 19 January 2021 at 15:55:38 UTC-5 [email protected] >>>>>>>>> wrote: >>>>>>>>> I wasn't happy leaving this is as is so I did some more testing. >>>>>>>>> >>>>>>>>> I looked in /home/weewx/bin/user/belchertown.py for any mention of >>>>>>>>> the 4 include files it uses to add content - nothing there. >>>>>>>>> >>>>>>>>> So, I looked in /home/weewx/skins/Belchertown/index.html.templ and >>>>>>>>> found 4 lines that reference the include files - they are of the form: >>>>>>>>> >>>>>>>>> #if >>>>>>>>> os.path.exists("index_hook_after_station_info.inc") >>>>>>>>> <!-- Start of index_hook_after_station_info row --> >>>>>>>>> <div class="row index-hook-after-station-info >>>>>>>>> border-bottom"> >>>>>>>>> #include "index_hook_after_station_info.inc" >>>>>>>>> </div> >>>>>>>>> <!-- End of index_hook_after_station_info row --> >>>>>>>>> #end if >>>>>>>>> That caused me to look up the Cheetah '#include' directive: >>>>>>>>> >>>>>>>>> "7.6 #include >>>>>>>>> Syntax: >>>>>>>>> >>>>>>>>> #include [raw] FILENAME_EXPR #include [raw] source=STRING_EXPR >>>>>>>>> The #include directive is used to include text from outside the >>>>>>>>> template definition. The text can come from an external file or from >>>>>>>>> a $placeholder variable. When working with external files, Cheetah >>>>>>>>> will monitor for changes to the included file and update as necessary. >>>>>>>>> >>>>>>>>> This example demonstrates its use with external files: >>>>>>>>> >>>>>>>>> #include "includeFileName.txt" >>>>>>>>> The content of "includeFileName.txt" will be parsed for Cheetah >>>>>>>>> syntax. >>>>>>>>> And this example demonstrates use with $placeholder variables: >>>>>>>>> >>>>>>>>> #include source=$myParseText >>>>>>>>> The value of $myParseText will be parsed for Cheetah syntax. This is >>>>>>>>> not the same as simply placing the $placeholder tag ``$myParseText'' >>>>>>>>> in the template definition. In the latter case, the value of >>>>>>>>> $myParseText would not be parsed. >>>>>>>>> By default, included text will be parsed for Cheetah tags. The >>>>>>>>> argument ``raw'' can be used to suppress the parsing. >>>>>>>>> >>>>>>>>> #include raw "includeFileName.txt" #include raw source=$myParseText >>>>>>>>> Cheetah wraps each chunk of #include text inside a nested Template >>>>>>>>> object. Each nested template has a copy of the main template's >>>>>>>>> searchList. However, #set variables are visible across includes only >>>>>>>>> if the defined using the #set global keyword. >>>>>>>>> >>>>>>>>> All directives must be balanced in the include file. That is, if you >>>>>>>>> start a #for or #if block inside the include, you must end it in the >>>>>>>>> same include. (This is unlike PHP, which allows unbalanced constructs >>>>>>>>> in include files.)" >>>>>>>>> >>>>>>>>> I decided to insert the 'raw' argument into the 4 '#include' >>>>>>>>> directives to see if anything changed. >>>>>>>>> >>>>>>>>> After about 1 hour. memory usage did not grow above about 65 MB! >>>>>>>>> >>>>>>>>> I just removed the 'raw' argument and restarted WeeWX. After just a >>>>>>>>> few minutes, memory usage is almost 300 MB and growing! >>>>>>>>> >>>>>>>>> As before, the station website is at: >>>>>>>>> https://lockyer.ca/weather/OsoyoosLakeNorthEast and the mem graphs >>>>>>>>> are at https://lockyer.ca/weather/OsoyoosLakeNorthEast/mem. >>>>>>>>> >>>>>>>>> So, I think I have conclusively shown that the problem is within >>>>>>>>> Cheetah and is related to it parsing an include file. >>>>>>>>> >>>>>>>>> I'm going to re-insert the 'raw' arguments and carry on. >>>>>>>>> >>>>>>>>> Should I report this issue to the Cheetah folks or is there someone >>>>>>>>> closer to the Cheetah developers that can better do that? >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Garry >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Monday, January 18, 2021 at 9:27:42 AM UTC-8 [email protected] >>>>>>>>>> wrote: >>>>>>>>>> Thanks! >>>>>>>>>> >>>>>>>>>> I think I've done all that I can to isolate the problem. >>>>>>>>>> >>>>>>>>>> I'm going to get back to finishing up development and add a cron >>>>>>>>>> entry to restart WeeWX every 24 hours in the wee hours of the >>>>>>>>>> morning. >>>>>>>>>> >>>>>>>>>> I will be re-deploying a Davis Pro 2 system very soon (within the >>>>>>>>>> week) and then deploying another Davis Pro 2 system shortly after >>>>>>>>>> that. And sometime after than, a couple of WeatherFlow Tempest >>>>>>>>>> systems. >>>>>>>>>> >>>>>>>>>> I'll post everything on GitHub soonest. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>> Garry >>>>>>>>>> PS: Re: Ubuntu Desktop on RPi, I've been using it for development >>>>>>>>>> for the last few days and it has not crashed or hung, so I think I'm >>>>>>>>>> going to stay with it. Only issue is wireless mouse lag (Raspberry >>>>>>>>>> Pi OS had same/similar problem) and I've ordered a Logitech MK540 >>>>>>>>>> wireless keyboard/mouse combo to see if newer hardwaere works better >>>>>>>>>> (than my dated Microsoft mouse). >>>>>>>>>>> On Sunday, January 17, 2021 at 5:40:15 PM UTC-8 vince wrote: >>>>>>>>>>> Restarted with your add-on plus belchertown both enabled - the RSS >>>>>>>>>>> is nominally growing at 4 MB per 5-minute archive period. >>>>>>>>>>> >>>>>>>>>>> 1610928917 139.53125 63.97265625 15.453125 >>>>>>>>>>> 1610929217 143.02734375 70.41796875 15.4921875 >>>>>>>>>>> 1610929517 144.46484375 74.43359375 15.4921875 >>>>>>>>>>> 1610929817 145.96484375 78.48046875 15.55078125 >>>>>>>>>>> 1610930117 147.46484375 82.65625 15.58203125 >>>>>>>>>>> 1610930417 149.09375 86.69921875 15.58203125 >>>>>>>>>>> 1610930717 150.34375 90.55078125 15.58203125 >>>>>>>>>>> 1610931017 151.84375 94.203125 15.24609375 >>>>>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "weewx-user" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>>> an email to [email protected]. >>>>>> >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/weewx-user/bfd80498-e70b-4a26-9f76-6a922439a1ben%40googlegroups.com. >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "weewx-user" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to [email protected]. >>> >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/weewx-user/4916786b-3bb2-49a6-93b1-87a60a822dbdn%40googlegroups.com. >>>> <Screen Shot 2021-01-19 at 7.33.46 PM.png> > > -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/weewx-user/71e3dde7-0174-42d2-908e-1d6ea27b42ban%40googlegroups.com. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/BDFA0BEF-AC3F-4EDB-A2E2-C7D113D8A9C6%40gmail.com.
