With apologies if you know much of this. . . The Belchertown skin allows you to add content to the main page it emits each archive cycle via 4 “hook” files. If the hook files are in Belchertown’s skin folder, their content is included in ‘index.html’.
I first learned about this feature ‘cause it is used by a WeeWX/Belchertown extension to fetch aviation weather observations (METARs) and forecasts (TAFS). That extension is a search list extension and eventually I used that mechanism to create my own extension to fetch METARs and TAFs, and extended it to fetch weather information (current conditions and forecasts) from Environment Canada and NOAA’s National Weather Service, Storm Prediction Center and National Hurricane Service, as well as road information from 511 Alberta, Rive BC and Washington State Department of Transportation. I deployed a Davis Pro 2 weather station with WeeWX, Belcherton and my extension and it crashed after about 3 days. After much debugging and rewriting (including converting my Belchertown specific search list extension to a skin independent service to create the include files, using Ubuntu instead of Raspberry Pi OS, and creating a much smaller/simpler program to demonstrate the problem), I narrowed the problem down to the Cheetah generator, specifically the parsing of included files by the Cheetah generator. With “out of the box” WeeWX and Belchertown, If the include files are recreated (each archive cycle) and the Belchertown skin is enabled, WeeWX memory usage will grow every cycle. If the ‘raw’ argument is added to the ‘include’ directives in Belcherotwn’s ‘index.html.tmpl’ memory usage stabilizes. The bigger the include files, the faster memory usage grows. My extension does not use $ directives in the include files it creates so losing that functionality is not a problem for me but obviously it could be an issue for other extensions or include file generators. I don’t think the Cheetah generator should cause increasing memory usage because it’s asked to include content, whether it’s asked to parse it or not. I can live with modifying ‘index.html.tmpl’ but hope the underlying problem gets fixed. I’ve entered as ‘issue’ against Belchertown so that its author is aware, can possibly provide a workaround in his code and/or “escalate” the issue to the Cheetah folks. Regards, Garry Lockyer C: +1.250.689.0686 E: [email protected] > On Jan 19, 2021, at 16:16, Tom Keffer <[email protected]> wrote: > > > I'm not quite following this discussion. > > With Cheetah, you can compile the template and save the results. I believe > this is what monitors the changed status of #include files. But, that's not > the way WeeWX uses templates. It compiles, evaluates, then throws the results > away. > > If you use the "raw" directive in an #include, then the file will not be > parsed. So, how can any $ directives work? > > Or, does Belchertown work differently? > > -tk > >> On Tue, Jan 19, 2021 at 3:55 PM [email protected] <[email protected]> >> wrote: >> I agree that something does not seem quite right with Cheetah’s ‘monitoring’ >> of include files. I took a different approach. >> First, I wrote a simple WeeWX service that would perform the ‘touch’ command >> against the large file and a simple template that included the large file. >> When the file modified attribute changed, I observed the memory growth, >> otherwise all was stable. Also, when the file was not ‘touched’ the Cheetah >> processing was noticeably faster. >> Next I wrote a standalone program to run a loop that ran the touch command >> and invoke Cheetah. I ran it 3 times. Each time around loop 90, the memory >> growth stopped and the processing was fast as though Cheetah was using >> cached data. >> I am back to running under WeeWX to see if it will level off like it does >> standalone. I am also wading through the Cheetah code and documentation. >> As of now, I thoroughly confused. >> rich >> >>> On Tuesday, 19 January 2021 at 15:55:38 UTC-5 [email protected] wrote: >>> I wasn't happy leaving this is as is so I did some more testing. >>> >>> I looked in /home/weewx/bin/user/belchertown.py for any mention of the 4 >>> include files it uses to add content - nothing there. >>> >>> So, I looked in /home/weewx/skins/Belchertown/index.html.templ and found 4 >>> lines that reference the include files - they are of the form: >>> >>> #if os.path.exists("index_hook_after_station_info.inc") >>> <!-- Start of index_hook_after_station_info row --> >>> <div class="row index-hook-after-station-info >>> border-bottom"> >>> #include "index_hook_after_station_info.inc" >>> </div> >>> <!-- End of index_hook_after_station_info row --> >>> #end if >>> That caused me to look up the Cheetah '#include' directive: >>> >>> "7.6 #include >>> Syntax: >>> >>> #include [raw] FILENAME_EXPR #include [raw] source=STRING_EXPR >>> The #include directive is used to include text from outside the template >>> definition. The text can come from an external file or from a $placeholder >>> variable. When working with external files, Cheetah will monitor for >>> changes to the included file and update as necessary. >>> >>> This example demonstrates its use with external files: >>> >>> #include "includeFileName.txt" >>> The content of "includeFileName.txt" will be parsed for Cheetah syntax. >>> And this example demonstrates use with $placeholder variables: >>> >>> #include source=$myParseText >>> The value of $myParseText will be parsed for Cheetah syntax. This is not >>> the same as simply placing the $placeholder tag ``$myParseText'' in the >>> template definition. In the latter case, the value of $myParseText would >>> not be parsed. >>> By default, included text will be parsed for Cheetah tags. The argument >>> ``raw'' can be used to suppress the parsing. >>> >>> #include raw "includeFileName.txt" #include raw source=$myParseText >>> Cheetah wraps each chunk of #include text inside a nested Template object. >>> Each nested template has a copy of the main template's searchList. However, >>> #set variables are visible across includes only if the defined using the >>> #set global keyword. >>> >>> All directives must be balanced in the include file. That is, if you start >>> a #for or #if block inside the include, you must end it in the same >>> include. (This is unlike PHP, which allows unbalanced constructs in include >>> files.)" >>> >>> I decided to insert the 'raw' argument into the 4 '#include' directives to >>> see if anything changed. >>> >>> After about 1 hour. memory usage did not grow above about 65 MB! >>> >>> I just removed the 'raw' argument and restarted WeeWX. After just a few >>> minutes, memory usage is almost 300 MB and growing! >>> >>> As before, the station website is at: >>> https://lockyer.ca/weather/OsoyoosLakeNorthEast and the mem graphs are at >>> https://lockyer.ca/weather/OsoyoosLakeNorthEast/mem. >>> >>> So, I think I have conclusively shown that the problem is within Cheetah >>> and is related to it parsing an include file. >>> >>> I'm going to re-insert the 'raw' arguments and carry on. >>> >>> Should I report this issue to the Cheetah folks or is there someone closer >>> to the Cheetah developers that can better do that? >>> >>> Regards, >>> >>> Garry >>> >>> >>> >>>> On Monday, January 18, 2021 at 9:27:42 AM UTC-8 [email protected] wrote: >>>> Thanks! >>>> >>>> I think I've done all that I can to isolate the problem. >>>> >>>> I'm going to get back to finishing up development and add a cron entry to >>>> restart WeeWX every 24 hours in the wee hours of the morning. >>>> >>>> I will be re-deploying a Davis Pro 2 system very soon (within the week) >>>> and then deploying another Davis Pro 2 system shortly after that. And >>>> sometime after than, a couple of WeatherFlow Tempest systems. >>>> >>>> I'll post everything on GitHub soonest. >>>> >>>> Regards, >>>> >>>> Garry >>>> PS: Re: Ubuntu Desktop on RPi, I've been using it for development for the >>>> last few days and it has not crashed or hung, so I think I'm going to stay >>>> with it. Only issue is wireless mouse lag (Raspberry Pi OS had >>>> same/similar problem) and I've ordered a Logitech MK540 wireless >>>> keyboard/mouse combo to see if newer hardwaere works better (than my dated >>>> Microsoft mouse). >>>>> On Sunday, January 17, 2021 at 5:40:15 PM UTC-8 vince wrote: >>>>> Restarted with your add-on plus belchertown both enabled - the RSS is >>>>> nominally growing at 4 MB per 5-minute archive period. >>>>> >>>>> 1610928917 139.53125 63.97265625 15.453125 >>>>> 1610929217 143.02734375 70.41796875 15.4921875 >>>>> 1610929517 144.46484375 74.43359375 15.4921875 >>>>> 1610929817 145.96484375 78.48046875 15.55078125 >>>>> 1610930117 147.46484375 82.65625 15.58203125 >>>>> 1610930417 149.09375 86.69921875 15.58203125 >>>>> 1610930717 150.34375 90.55078125 15.58203125 >>>>> 1610931017 151.84375 94.203125 15.24609375 >>>>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "weewx-user" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/weewx-user/bfd80498-e70b-4a26-9f76-6a922439a1ben%40googlegroups.com. > > -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/weewx-user/CAPq0zEBLSWhzeEFOyksrx64Pro2M0F2skgOCePZ0DUWSEoH7rw%40mail.gmail.com. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/7F58A737-4F41-4C62-8C7F-A5CA7678C672%40gmail.com.
