Ok thank you Garry On Fri, Jan 22, 2021, 7:33 PM Garry A Lockyer <[email protected]> wrote:
> I’ll assume your talking to me. > > There are no proposed changes to any existing software - at least not to > WeeWX or the Belchertown skin. Any changes would likely be modifications > to the Cheetah generator so that it does not consume ever increasing > amounts of memory, but I know nothing technical about the Cheetah generator > so can’t say if such modifications are possible. > > That being said . . . > > *IF* someone uses the “index hook” feature of the Belchertown skin *AND* > the hook include files are regenerated frequently, such as every archive > period, they should be aware that they may experience ever increasing > memory usage. The bigger the include files, the greater the increase in > memory usage. > > The problem can be avoided by creating “static” index hook files that > reference (with a ‘#include raw “filename” directive) the file containing > the HTML to be included which can be periodically regenerated. > > I intend to implement the workaround in my WeeWX extension such that a > user does not have to worry about about the technicalities but that code > isn’t written yet. > > Regards, > > Garry Lockyer > C: +1.250.689.0686 > E: [email protected] > > > On Jan 22, 2021, at 17:14, Timothy L <[email protected]> wrote: > > > Would it be possible to post a step by step instruction for your changes > if not too much trouble? > > On Fri, Jan 22, 2021, 3:01 PM Garry A Lockyer <[email protected]> > wrote: > >> The workaround is to have “static” index hook files that each just have 1 >> Cheetah directive in them. That directive has the ‘raw’ argument. For my >> extension, I will implement the work around, using >> “index-hook-after-station-info.inc” (and assuming I got the file name >> correct!) as the example, such that: >> >> index-hook-after-station-info.inc contains: >> >> #include raw “WXFeeds-index-hook-after-station-info.inc” >> >> and is not re-created. >> >> My extension will generate “WXFeeds-index-hook-after-station-info.inc” >> each archive cycle. >> >> I don’t think any changes to Belchertown are required other than perhaps >> a warning that memory usage can increase if the hooks are used without the >> workaround (until the Cheetah generator is fixed). >> >> Thanks for your time, attention and the Belchertown skin! >> >> Regards, >> >> Garry Lockyer >> C: +1.250.689.0686 >> E: [email protected] >> >> >> On Jan 22, 2021, at 12:00, Pat <[email protected]> wrote: >> >> I'm tracking this thread a bit slowly... Is the raw portion within the >> #include custom to your setup - or is this something that should be added >> to the Belchertown skin? >> >> On Thursday, January 21, 2021 at 12:47:26 PM UTC-5 [email protected] >> wrote: >> >>> It looks like the workaround Rich suggested works. >>> >>> A couple of notes: >>> >>> 1. the generated filename must be in quotes (#include raw "generated >>> file name") >>> - the Cheetah generator will throw an error if not. >>> >>> 2. There must not be 'raw' arguments in index.html.tmpl >>> - I left them in after previous debugging >>> - if they are in, the Cheetah generator will not parse the generated >>> file and the Cheetah directive will >>> be display in the output HTML. >>> >>> I will be putting a control in my extension to enable/disable this >>> workaround. >>> >>> Again, thanks to Rich and others for helping with this. >>> >>> Regards, >>> >>> Garry >>> >>> On Wednesday, January 20, 2021 at 10:16:26 AM UTC-8 [email protected] >>> wrote: >>> >>>> Thanks to Vince, Rich and all who took the time to investigate this >>>> problem. >>>> >>>> Happy that it’s not WeeWX or Belchertown, two fine programs! >>>> >>>> I hope to release my WXFeeds service extension within the next few days. >>>> >>>> Regards, >>>> >>>> Garry >>>> >>>> On Wednesday, January 20, 2021 at 8:19:32 AM UTC-8 vince wrote: >>>> >>>>> Well FWIW, it's not Belchertown. I took my minimalist demo-skin and >>>>> aded the one include line in it and boy does it grow..... >>>>> >>>>> Two tests with a restart that's obvious. >>>>> >>>>> On Wednesday, January 20, 2021 at 7:07:23 AM UTC-8 [email protected] >>>>> wrote: >>>>> >>>>>> I haven’t made any changes yet but in thinking about your workaround, >>>>>> it will be interesting to see if Cheetah actuality processes the included >>>>>> include file or if it skips it because the outer include file’s >>>>>> attributes >>>>>> are static. >>>>>> >>>>>> If Cheetah doesn’t process the inner include file, it still should be >>>>>> a way to inject the ‘raw’ argument without having to edit >>>>>> ‘index.html.tmpl’. That is, my code would generate both the outer and >>>>>> inner include files each cycle, with the outer include file containing >>>>>> the >>>>>> ‘raw’ argument as you suggest. >>>>>> >>>>>> Regards, >>>>>> >>>>>> Garry Lockyer >>>>>> C: +1.250.689.0686 <(250)%20689-0686> >>>>>> E: [email protected] >>>>>> >>>>>> >>>>>> On Jan 20, 2021, at 06:24, [email protected] <[email protected]> >>>>>> wrote: >>>>>> >>>>>> Note, it is growing - just much slower. >>>>>> >>>>>> >>>>>> >>>>>> On Wednesday, 20 January 2021 at 09:13:35 UTC-5 [email protected] >>>>>> wrote: >>>>>> >>>>>>> Garry, >>>>>>> I might have a workaround for you. Wrap your html file in a >>>>>>> template. something like this. >>>>>>> index.html.tmp (this is in the belchertown skin) >>>>>>> #include index_hook_after_charts.inc >>>>>>> >>>>>>> index_hook_after_charts.inc (this is the 'wrapper', naturally if you >>>>>>> 'owned' index.html.tmpl you would not need it) >>>>>>> #include raw generated.html >>>>>>> >>>>>>> generated.html (the file you generate - name as you want) >>>>>>> your html >>>>>>> >>>>>>> I ran this on my loop 10,000 times and saw no appreciable memory >>>>>>> increase and performed better too. >>>>>>> rich >>>>>>> >>>>>>> On Tuesday, 19 January 2021 at 22:45:42 UTC-5 [email protected] >>>>>>> wrote: >>>>>>> >>>>>>>> Many thanks to all for poking at this! >>>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Garry Lockyer >>>>>>>> C: +1.250.689.0686 <(250)%20689-0686> >>>>>>>> E: [email protected] >>>>>>>> >>>>>>>> >>>>>>>> On Jan 19, 2021, at 19:38, vince <[email protected]> wrote: >>>>>>>> >>>>>>>> Just a followup - I took belchertown out of the picture and used >>>>>>>> my minimal demo-skin and added 'one line' that #include(d) a file >>>>>>>> generated >>>>>>>> by Garry's service, and it 'is' leaking memory albeit less quickly >>>>>>>> than the >>>>>>>> belchertown example..... >>>>>>>> >>>>>>>> >>>>>>>> I'll let it run overnight and we'll see what it looks like after 12 >>>>>>>> more hours running. The VM is set to only 1GB RAM but I'm hoping if >>>>>>>> it >>>>>>>> runs out it'll just go into swap and stay up. OS is debian 10 under >>>>>>>> Vagrant/VirtualBox with 4.3.0 installed using python3 and setup.py >>>>>>>> >>>>>>>> Some mem.sdb output (date/time, units, interval, mem_size, mem_rss, >>>>>>>> mem_share)... >>>>>>>> >>>>>>>> 1611109216 16 5 141.0625 71.734375 12.2734375 >>>>>>>> 1611109516 16 5 142.5625 75.65234375 12.3359375 >>>>>>>> 1611109816 16 5 144.0625 79.65625 12.3359375 >>>>>>>> 1611110116 16 5 145.5625 84.5234375 12.3359375 >>>>>>>> 1611110416 16 5 147.24609375 88.0859375 12.3359375 >>>>>>>> 1611110716 16 5 148.49609375 91.43359375 12.3359375 >>>>>>>> 1611111016 16 5 149.99609375 95.421875 12.3359375 >>>>>>>> 1611111316 16 5 151.49609375 100.3359375 12.3359375 >>>>>>>> 1611111616 16 5 152.99609375 103.8359375 12.3359375 >>>>>>>> 1611111916 16 5 154.49609375 107.51171875 12.3359375 >>>>>>>> 1611112216 16 5 155.99609375 111.5 12.3359375 >>>>>>>> 1611112516 16 5 157.24609375 116.0859375 12.3359375 >>>>>>>> 1611112816 16 5 158.74609375 119.5859375 12.3359375 >>>>>>>> 1611113116 16 5 160.24609375 123.24609375 12.3359375 >>>>>>>> 1611113416 16 5 161.74609375 127.234375 12.3359375 >>>>>>>> 1611113716 16 5 163.24609375 132.11328125 12.3359375 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Tuesday, January 19, 2021 at 5:21:31 PM UTC-8 [email protected] >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Ok, my standalone testcase was wrong. I think I have a valid one >>>>>>>>> here, https://github.com/bellrichm/experiments >>>>>>>>> It just calls Cheetah.Template.Template in a loop. If the include >>>>>>>>> file modified attribute does not change, all seems stable. If it >>>>>>>>> changes, >>>>>>>>> memory usage seems to grow. >>>>>>>>> rich >>>>>>>>> >>>>>>>>> On Tuesday, 19 January 2021 at 19:16:46 UTC-5 [email protected] >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> I'm not quite following this discussion. >>>>>>>>>> >>>>>>>>>> With Cheetah, you can compile the template and save the results. >>>>>>>>>> I believe this is what monitors the changed status of #include >>>>>>>>>> files. But, >>>>>>>>>> that's not the way WeeWX uses templates. It compiles, evaluates, then >>>>>>>>>> throws the results away. >>>>>>>>>> >>>>>>>>>> If you use the "raw" directive in an #include, then the file will >>>>>>>>>> not be parsed. So, how can any $ directives work? >>>>>>>>>> >>>>>>>>>> Or, does Belchertown work differently? >>>>>>>>>> >>>>>>>>>> -tk >>>>>>>>>> >>>>>>>>>> On Tue, Jan 19, 2021 at 3:55 PM [email protected] < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> I agree that something does not seem quite right with Cheetah’s >>>>>>>>>>> ‘monitoring’ of include files. I took a different approach. >>>>>>>>>>> First, I wrote a simple WeeWX service that would perform the >>>>>>>>>>> ‘touch’ command against the large file and a simple template that >>>>>>>>>>> included >>>>>>>>>>> the large file. When the file modified attribute changed, I >>>>>>>>>>> observed the >>>>>>>>>>> memory growth, otherwise all was stable. Also, when the file was not >>>>>>>>>>> ‘touched’ the Cheetah processing was noticeably faster. >>>>>>>>>>> Next I wrote a standalone program to run a loop that ran the >>>>>>>>>>> touch command and invoke Cheetah. I ran it 3 times. Each time >>>>>>>>>>> around loop >>>>>>>>>>> 90, the memory growth stopped and the processing was fast as though >>>>>>>>>>> Cheetah >>>>>>>>>>> was using cached data. >>>>>>>>>>> I am back to running under WeeWX to see if it will level off >>>>>>>>>>> like it does standalone. I am also wading through the Cheetah code >>>>>>>>>>> and >>>>>>>>>>> documentation. As of now, I thoroughly confused. >>>>>>>>>>> rich >>>>>>>>>>> >>>>>>>>>>> On Tuesday, 19 January 2021 at 15:55:38 UTC-5 >>>>>>>>>>> [email protected] wrote: >>>>>>>>>>> >>>>>>>>>>>> I wasn't happy leaving this is as is so I did some more testing. >>>>>>>>>>>> >>>>>>>>>>>> I looked in /home/weewx/bin/user/belchertown.py for any mention >>>>>>>>>>>> of the 4 include files it uses to add content - nothing there. >>>>>>>>>>>> >>>>>>>>>>>> So, I looked in /home/weewx/skins/Belchertown/index.html.templ >>>>>>>>>>>> and found 4 lines that reference the include files - they are of >>>>>>>>>>>> the form: >>>>>>>>>>>> >>>>>>>>>>>> #if >>>>>>>>>>>> os.path.exists("index_hook_after_station_info.inc") >>>>>>>>>>>> <!-- Start of index_hook_after_station_info row >>>>>>>>>>>> --> >>>>>>>>>>>> <div class="row index-hook-after-station-info >>>>>>>>>>>> border-bottom"> >>>>>>>>>>>> #include "index_hook_after_station_info.inc" >>>>>>>>>>>> </div> >>>>>>>>>>>> <!-- End of index_hook_after_station_info row >>>>>>>>>>>> --> >>>>>>>>>>>> #end if >>>>>>>>>>>> That caused me to look up the Cheetah '#include' directive: >>>>>>>>>>>> >>>>>>>>>>>> "7.6 #include >>>>>>>>>>>> >>>>>>>>>>>> Syntax: >>>>>>>>>>>> #include [raw] FILENAME_EXPR #include [raw] source=STRING_EXPR >>>>>>>>>>>> >>>>>>>>>>>> The #include directive is used to include text from outside the >>>>>>>>>>>> template definition. The text can come from an external file or >>>>>>>>>>>> from >>>>>>>>>>>> a $placeholder variable. When working with external files, Cheetah >>>>>>>>>>>> will >>>>>>>>>>>> monitor for changes to the included file and update as necessary. >>>>>>>>>>>> >>>>>>>>>>>> This example demonstrates its use with external files: >>>>>>>>>>>> #include "includeFileName.txt" >>>>>>>>>>>> The content of "includeFileName.txt" will be parsed for Cheetah >>>>>>>>>>>> syntax. >>>>>>>>>>>> >>>>>>>>>>>> And this example demonstrates use with $placeholder variables: >>>>>>>>>>>> #include source=$myParseText >>>>>>>>>>>> The value of $myParseText will be parsed for Cheetah syntax. >>>>>>>>>>>> This is not the same as simply placing the $placeholder tag >>>>>>>>>>>> ``$myParseText'' in the template definition. In the latter case, >>>>>>>>>>>> the value >>>>>>>>>>>> of $myParseText would not be parsed. >>>>>>>>>>>> >>>>>>>>>>>> By default, included text will be parsed for Cheetah tags. The >>>>>>>>>>>> argument ``raw'' can be used to suppress the parsing. >>>>>>>>>>>> >>>>>>>>>>>> #include raw "includeFileName.txt" #include raw >>>>>>>>>>>> source=$myParseText >>>>>>>>>>>> >>>>>>>>>>>> Cheetah wraps each chunk of #include text inside a >>>>>>>>>>>> nested Template object. Each nested template has a copy of the main >>>>>>>>>>>> template's searchList. However, #set variables are visible across >>>>>>>>>>>> includes >>>>>>>>>>>> only if the defined using the #set global keyword. >>>>>>>>>>>> >>>>>>>>>>>> All directives must be balanced in the include file. That is, >>>>>>>>>>>> if you start a #for or #if block inside the include, you must end >>>>>>>>>>>> it in the >>>>>>>>>>>> same include. (This is unlike PHP, which allows unbalanced >>>>>>>>>>>> constructs in >>>>>>>>>>>> include files.)" >>>>>>>>>>>> >>>>>>>>>>>> I decided to insert the 'raw' argument into the 4 '#include' >>>>>>>>>>>> directives to see if anything changed. >>>>>>>>>>>> >>>>>>>>>>>> After about 1 hour. memory usage did not grow above about 65 MB! >>>>>>>>>>>> >>>>>>>>>>>> I just removed the 'raw' argument and restarted WeeWX. After >>>>>>>>>>>> just a few minutes, memory usage is almost 300 MB and growing! >>>>>>>>>>>> >>>>>>>>>>>> As before, the station website is at: >>>>>>>>>>>> https://lockyer.ca/weather/OsoyoosLakeNorthEast and the mem >>>>>>>>>>>> graphs are at >>>>>>>>>>>> https://lockyer.ca/weather/OsoyoosLakeNorthEast/mem. >>>>>>>>>>>> >>>>>>>>>>>> So, I think I have conclusively shown that the problem is >>>>>>>>>>>> within Cheetah and is related to it parsing an include file. >>>>>>>>>>>> >>>>>>>>>>>> I'm going to re-insert the 'raw' arguments and carry on. >>>>>>>>>>>> >>>>>>>>>>>> Should I report this issue to the Cheetah folks or is there >>>>>>>>>>>> someone closer to the Cheetah developers that can better do that? >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> >>>>>>>>>>>> Garry >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Monday, January 18, 2021 at 9:27:42 AM UTC-8 >>>>>>>>>>>> [email protected] wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Thanks! >>>>>>>>>>>>> >>>>>>>>>>>>> I think I've done all that I can to isolate the problem. >>>>>>>>>>>>> >>>>>>>>>>>>> I'm going to get back to finishing up development and add a >>>>>>>>>>>>> cron entry to restart WeeWX every 24 hours in the wee hours of >>>>>>>>>>>>> the morning. >>>>>>>>>>>>> >>>>>>>>>>>>> I will be re-deploying a Davis Pro 2 system very soon (within >>>>>>>>>>>>> the week) and then deploying another Davis Pro 2 system shortly >>>>>>>>>>>>> after >>>>>>>>>>>>> that. And sometime after than, a couple of WeatherFlow Tempest >>>>>>>>>>>>> systems. >>>>>>>>>>>>> >>>>>>>>>>>>> I'll post everything on GitHub soonest. >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> >>>>>>>>>>>>> Garry >>>>>>>>>>>>> PS: Re: Ubuntu Desktop on RPi, I've been using it for >>>>>>>>>>>>> development for the last few days and it has not crashed or hung, >>>>>>>>>>>>> so I >>>>>>>>>>>>> think I'm going to stay with it. Only issue is wireless mouse lag >>>>>>>>>>>>> (Raspberry Pi OS had same/similar problem) and I've ordered a >>>>>>>>>>>>> Logitech >>>>>>>>>>>>> MK540 wireless keyboard/mouse combo to see if newer hardwaere >>>>>>>>>>>>> works better >>>>>>>>>>>>> (than my dated Microsoft mouse). >>>>>>>>>>>>> On Sunday, January 17, 2021 at 5:40:15 PM UTC-8 vince wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Restarted with your add-on plus belchertown both enabled - >>>>>>>>>>>>>> the RSS is nominally growing at 4 MB per 5-minute archive period. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1610928917 139.53125 63.97265625 15.453125 >>>>>>>>>>>>>> 1610929217 143.02734375 70.41796875 15.4921875 >>>>>>>>>>>>>> 1610929517 144.46484375 74.43359375 15.4921875 >>>>>>>>>>>>>> 1610929817 145.96484375 78.48046875 15.55078125 >>>>>>>>>>>>>> 1610930117 147.46484375 82.65625 15.58203125 >>>>>>>>>>>>>> 1610930417 149.09375 86.69921875 15.58203125 >>>>>>>>>>>>>> 1610930717 150.34375 90.55078125 15.58203125 >>>>>>>>>>>>>> 1610931017 151.84375 94.203125 15.24609375 >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>> Google Groups "weewx-user" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>> >>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>> https://groups.google.com/d/msgid/weewx-user/bfd80498-e70b-4a26-9f76-6a922439a1ben%40googlegroups.com >>>>>>>>>>> <https://groups.google.com/d/msgid/weewx-user/bfd80498-e70b-4a26-9f76-6a922439a1ben%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>>> . >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "weewx-user" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/weewx-user/4916786b-3bb2-49a6-93b1-87a60a822dbdn%40googlegroups.com >>>>>>>> <https://groups.google.com/d/msgid/weewx-user/4916786b-3bb2-49a6-93b1-87a60a822dbdn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> <Screen Shot 2021-01-19 at 7.33.46 PM.png> >>>>>>>> >>>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "weewx-user" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/weewx-user/71e3dde7-0174-42d2-908e-1d6ea27b42ban%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/weewx-user/71e3dde7-0174-42d2-908e-1d6ea27b42ban%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>>> -- >> You received this message because you are subscribed to the Google Groups >> "weewx-user" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/weewx-user/9d1ce1c1-3017-43f2-b66f-13c5637748e6n%40googlegroups.com >> <https://groups.google.com/d/msgid/weewx-user/9d1ce1c1-3017-43f2-b66f-13c5637748e6n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> -- >> You received this message because you are subscribed to the Google Groups >> "weewx-user" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/weewx-user/772028EA-5967-47A8-A130-5864076C10DD%40gmail.com >> <https://groups.google.com/d/msgid/weewx-user/772028EA-5967-47A8-A130-5864076C10DD%40gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/weewx-user/CAGW8XMQMwDDpKmfMgan0jkGtG1uOxBXg3LzW8cRFH%2BpYMd3XwA%40mail.gmail.com > <https://groups.google.com/d/msgid/weewx-user/CAGW8XMQMwDDpKmfMgan0jkGtG1uOxBXg3LzW8cRFH%2BpYMd3XwA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/weewx-user/04E005A6-BF4B-40A8-AE3F-0B3E80292402%40gmail.com > <https://groups.google.com/d/msgid/weewx-user/04E005A6-BF4B-40A8-AE3F-0B3E80292402%40gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAGW8XMSwGnnqdC5kc22BviwEDWVEu-GJaBWvAjNUMGeVKA1W4g%40mail.gmail.com.
