Indeed. Everybody else is equally surprised. But it's very reproducible and repeatable.
On Tuesday, January 19, 2021 at 5:14:20 PM UTC-8 Tom Keffer wrote: > OK, got it. > > Still, I'm surprised that memory would grow with a #include, raw, or not. > As I say, compiled templates are thrown away after every use. > > On Tue, Jan 19, 2021 at 5:08 PM Garry A Lockyer <[email protected]> > wrote: > >> With apologies if you know much of this. . . >> >> The Belchertown skin allows you to add content to the main page it emits >> each archive cycle via 4 “hook” files. If the hook files are in >> Belchertown’s skin folder, their content is included in ‘index.html’. >> >> I first learned about this feature ‘cause it is used by a >> WeeWX/Belchertown extension to fetch aviation weather observations (METARs) >> and forecasts (TAFS). That extension is a search list extension and >> eventually I used that mechanism to create my own extension to fetch METARs >> and TAFs, and extended it to fetch weather information (current conditions >> and forecasts) from Environment Canada and NOAA’s National Weather Service, >> Storm Prediction Center and National Hurricane Service, as well as road >> information from 511 Alberta, Rive BC and Washington State Department of >> Transportation. >> >> I deployed a Davis Pro 2 weather station with WeeWX, Belcherton and my >> extension and it crashed after about 3 days. After much debugging and >> rewriting (including converting my Belchertown specific search list >> extension to a skin independent service to create the include files, using >> Ubuntu instead of Raspberry Pi OS, and creating a much smaller/simpler >> program to demonstrate the problem), I narrowed the problem down to the >> Cheetah generator, specifically the parsing of included files by the >> Cheetah generator. >> >> With “out of the box” WeeWX and Belchertown, If the include files are >> recreated (each archive cycle) and the Belchertown skin is enabled, WeeWX >> memory usage will grow every cycle. If the ‘raw’ argument is added to the >> ‘include’ directives in Belcherotwn’s ‘index.html.tmpl’ memory usage >> stabilizes. The bigger the include files, the faster memory usage grows. >> >> My extension does not use $ directives in the include files it creates so >> losing that functionality is not a problem for me but obviously it could be >> an issue for other extensions or include file generators. >> >> I don’t think the Cheetah generator should cause increasing memory usage >> because it’s asked to include content, whether it’s asked to parse it or >> not. I can live with modifying ‘index.html.tmpl’ but hope the underlying >> problem gets fixed. >> >> I’ve entered as ‘issue’ against Belchertown so that its author is aware, >> can possibly provide a workaround in his code and/or “escalate” the issue >> to the Cheetah folks. >> >> Regards, >> >> Garry Lockyer >> C: +1.250.689.0686 <(250)%20689-0686> >> E: [email protected] >> >> >> On Jan 19, 2021, at 16:16, Tom Keffer <[email protected]> wrote: >> >> >> I'm not quite following this discussion. >> >> With Cheetah, you can compile the template and save the results. I >> believe this is what monitors the changed status of #include files. But, >> that's not the way WeeWX uses templates. It compiles, evaluates, then >> throws the results away. >> >> If you use the "raw" directive in an #include, then the file will not be >> parsed. So, how can any $ directives work? >> >> Or, does Belchertown work differently? >> >> -tk >> >> On Tue, Jan 19, 2021 at 3:55 PM [email protected] <[email protected]> >> wrote: >> >>> I agree that something does not seem quite right with Cheetah’s >>> ‘monitoring’ of include files. I took a different approach. >>> First, I wrote a simple WeeWX service that would perform the ‘touch’ >>> command against the large file and a simple template that included the >>> large file. When the file modified attribute changed, I observed the memory >>> growth, otherwise all was stable. Also, when the file was not ‘touched’ the >>> Cheetah processing was noticeably faster. >>> Next I wrote a standalone program to run a loop that ran the touch >>> command and invoke Cheetah. I ran it 3 times. Each time around loop 90, >>> the memory growth stopped and the processing was fast as though Cheetah was >>> using cached data. >>> I am back to running under WeeWX to see if it will level off like it >>> does standalone. I am also wading through the Cheetah code and >>> documentation. As of now, I thoroughly confused. >>> rich >>> >>> On Tuesday, 19 January 2021 at 15:55:38 UTC-5 [email protected] wrote: >>> >>>> I wasn't happy leaving this is as is so I did some more testing. >>>> >>>> I looked in /home/weewx/bin/user/belchertown.py for any mention of the >>>> 4 include files it uses to add content - nothing there. >>>> >>>> So, I looked in /home/weewx/skins/Belchertown/index.html.templ and >>>> found 4 lines that reference the include files - they are of the form: >>>> >>>> #if os.path.exists("index_hook_after_station_info.inc") >>>> <!-- Start of index_hook_after_station_info row --> >>>> <div class="row index-hook-after-station-info >>>> border-bottom"> >>>> #include "index_hook_after_station_info.inc" >>>> </div> >>>> <!-- End of index_hook_after_station_info row --> >>>> #end if >>>> That caused me to look up the Cheetah '#include' directive: >>>> >>>> "7.6 #include >>>> >>>> Syntax: >>>> #include [raw] FILENAME_EXPR #include [raw] source=STRING_EXPR >>>> >>>> The #include directive is used to include text from outside the >>>> template definition. The text can come from an external file or from >>>> a $placeholder variable. When working with external files, Cheetah will >>>> monitor for changes to the included file and update as necessary. >>>> >>>> This example demonstrates its use with external files: >>>> #include "includeFileName.txt" >>>> The content of "includeFileName.txt" will be parsed for Cheetah syntax. >>>> >>>> And this example demonstrates use with $placeholder variables: >>>> #include source=$myParseText >>>> The value of $myParseText will be parsed for Cheetah syntax. This is >>>> not the same as simply placing the $placeholder tag ``$myParseText'' in >>>> the >>>> template definition. In the latter case, the value of $myParseText would >>>> not be parsed. >>>> >>>> By default, included text will be parsed for Cheetah tags. The argument >>>> ``raw'' can be used to suppress the parsing. >>>> >>>> #include raw "includeFileName.txt" #include raw source=$myParseText >>>> >>>> Cheetah wraps each chunk of #include text inside a >>>> nested Template object. Each nested template has a copy of the main >>>> template's searchList. However, #set variables are visible across includes >>>> only if the defined using the #set global keyword. >>>> >>>> All directives must be balanced in the include file. That is, if you >>>> start a #for or #if block inside the include, you must end it in the same >>>> include. (This is unlike PHP, which allows unbalanced constructs in >>>> include >>>> files.)" >>>> >>>> I decided to insert the 'raw' argument into the 4 '#include' directives >>>> to see if anything changed. >>>> >>>> After about 1 hour. memory usage did not grow above about 65 MB! >>>> >>>> I just removed the 'raw' argument and restarted WeeWX. After just a >>>> few minutes, memory usage is almost 300 MB and growing! >>>> >>>> As before, the station website is at: >>>> https://lockyer.ca/weather/OsoyoosLakeNorthEast and the mem graphs are >>>> at https://lockyer.ca/weather/OsoyoosLakeNorthEast/mem. >>>> >>>> So, I think I have conclusively shown that the problem is within >>>> Cheetah and is related to it parsing an include file. >>>> >>>> I'm going to re-insert the 'raw' arguments and carry on. >>>> >>>> Should I report this issue to the Cheetah folks or is there someone >>>> closer to the Cheetah developers that can better do that? >>>> >>>> Regards, >>>> >>>> Garry >>>> >>>> >>>> On Monday, January 18, 2021 at 9:27:42 AM UTC-8 [email protected] >>>> wrote: >>>> >>>>> Thanks! >>>>> >>>>> I think I've done all that I can to isolate the problem. >>>>> >>>>> I'm going to get back to finishing up development and add a cron entry >>>>> to restart WeeWX every 24 hours in the wee hours of the morning. >>>>> >>>>> I will be re-deploying a Davis Pro 2 system very soon (within the >>>>> week) and then deploying another Davis Pro 2 system shortly after that. >>>>> And sometime after than, a couple of WeatherFlow Tempest systems. >>>>> >>>>> I'll post everything on GitHub soonest. >>>>> >>>>> Regards, >>>>> >>>>> Garry >>>>> PS: Re: Ubuntu Desktop on RPi, I've been using it for development for >>>>> the last few days and it has not crashed or hung, so I think I'm going to >>>>> stay with it. Only issue is wireless mouse lag (Raspberry Pi OS had >>>>> same/similar problem) and I've ordered a Logitech MK540 wireless >>>>> keyboard/mouse combo to see if newer hardwaere works better (than my >>>>> dated >>>>> Microsoft mouse). >>>>> On Sunday, January 17, 2021 at 5:40:15 PM UTC-8 vince wrote: >>>>> >>>>>> Restarted with your add-on plus belchertown both enabled - the RSS is >>>>>> nominally growing at 4 MB per 5-minute archive period. >>>>>> >>>>>> 1610928917 139.53125 63.97265625 15.453125 >>>>>> 1610929217 143.02734375 70.41796875 15.4921875 >>>>>> 1610929517 144.46484375 74.43359375 15.4921875 >>>>>> 1610929817 145.96484375 78.48046875 15.55078125 >>>>>> 1610930117 147.46484375 82.65625 15.58203125 >>>>>> 1610930417 149.09375 86.69921875 15.58203125 >>>>>> 1610930717 150.34375 90.55078125 15.58203125 >>>>>> 1610931017 151.84375 94.203125 15.24609375 >>>>>> >>>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "weewx-user" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/weewx-user/bfd80498-e70b-4a26-9f76-6a922439a1ben%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/weewx-user/bfd80498-e70b-4a26-9f76-6a922439a1ben%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "weewx-user" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/weewx-user/CAPq0zEBLSWhzeEFOyksrx64Pro2M0F2skgOCePZ0DUWSEoH7rw%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/weewx-user/CAPq0zEBLSWhzeEFOyksrx64Pro2M0F2skgOCePZ0DUWSEoH7rw%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> -- >> You received this message because you are subscribed to the Google Groups >> "weewx-user" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/weewx-user/7F58A737-4F41-4C62-8C7F-A5CA7678C672%40gmail.com >> >> <https://groups.google.com/d/msgid/weewx-user/7F58A737-4F41-4C62-8C7F-A5CA7678C672%40gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/de524d16-4cd3-4f7c-a0bd-8162fe24f963n%40googlegroups.com.
