It looks like the workaround Rich suggested works.
A couple of notes:
1. the generated filename must be in quotes (#include raw "generated file
name")
- the Cheetah generator will throw an error if not.
2. There must not be 'raw' arguments in index.html.tmpl
- I left them in after previous debugging
- if they are in, the Cheetah generator will not parse the generated
file and the Cheetah directive will
be display in the output HTML.
I will be putting a control in my extension to enable/disable this
workaround.
Again, thanks to Rich and others for helping with this.
Regards,
Garry
On Wednesday, January 20, 2021 at 10:16:26 AM UTC-8 [email protected]
wrote:
> Thanks to Vince, Rich and all who took the time to investigate this
> problem.
>
> Happy that it’s not WeeWX or Belchertown, two fine programs!
>
> I hope to release my WXFeeds service extension within the next few days.
>
> Regards,
>
> Garry
>
> On Wednesday, January 20, 2021 at 8:19:32 AM UTC-8 vince wrote:
>
>> Well FWIW, it's not Belchertown. I took my minimalist demo-skin and
>> aded the one include line in it and boy does it grow.....
>>
>> Two tests with a restart that's obvious.
>>
>> On Wednesday, January 20, 2021 at 7:07:23 AM UTC-8 [email protected]
>> wrote:
>>
>>> I haven’t made any changes yet but in thinking about your workaround, it
>>> will be interesting to see if Cheetah actuality processes the included
>>> include file or if it skips it because the outer include file’s attributes
>>> are static.
>>>
>>> If Cheetah doesn’t process the inner include file, it still should be a
>>> way to inject the ‘raw’ argument without having to edit ‘index.html.tmpl’.
>>> That is, my code would generate both the outer and inner include files
>>> each cycle, with the outer include file containing the ‘raw’ argument as
>>> you suggest.
>>>
>>> Regards,
>>>
>>> Garry Lockyer
>>> C: +1.250.689.0686 <(250)%20689-0686>
>>> E: [email protected]
>>>
>>>
>>> On Jan 20, 2021, at 06:24, [email protected] <[email protected]> wrote:
>>>
>>> Note, it is growing - just much slower.
>>>
>>>
>>>
>>> On Wednesday, 20 January 2021 at 09:13:35 UTC-5 [email protected] wrote:
>>>
>>>> Garry,
>>>> I might have a workaround for you. Wrap your html file in a template.
>>>> something like this.
>>>> index.html.tmp (this is in the belchertown skin)
>>>> #include index_hook_after_charts.inc
>>>>
>>>> index_hook_after_charts.inc (this is the 'wrapper', naturally if you
>>>> 'owned' index.html.tmpl you would not need it)
>>>> #include raw generated.html
>>>>
>>>> generated.html (the file you generate - name as you want)
>>>> your html
>>>>
>>>> I ran this on my loop 10,000 times and saw no appreciable memory
>>>> increase and performed better too.
>>>> rich
>>>>
>>>> On Tuesday, 19 January 2021 at 22:45:42 UTC-5 [email protected]
>>>> wrote:
>>>>
>>>>> Many thanks to all for poking at this!
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> Garry Lockyer
>>>>> C: +1.250.689.0686 <(250)%20689-0686>
>>>>> E: [email protected]
>>>>>
>>>>>
>>>>> On Jan 19, 2021, at 19:38, vince <[email protected]> wrote:
>>>>>
>>>>> Just a followup - I took belchertown out of the picture and used my
>>>>> minimal demo-skin and added 'one line' that #include(d) a file generated
>>>>> by
>>>>> Garry's service, and it 'is' leaking memory albeit less quickly than the
>>>>> belchertown example.....
>>>>>
>>>>>
>>>>> I'll let it run overnight and we'll see what it looks like after 12
>>>>> more hours running. The VM is set to only 1GB RAM but I'm hoping if it
>>>>> runs out it'll just go into swap and stay up. OS is debian 10 under
>>>>> Vagrant/VirtualBox with 4.3.0 installed using python3 and setup.py
>>>>>
>>>>> Some mem.sdb output (date/time, units, interval, mem_size, mem_rss,
>>>>> mem_share)...
>>>>>
>>>>> 1611109216 16 5 141.0625 71.734375 12.2734375
>>>>> 1611109516 16 5 142.5625 75.65234375 12.3359375
>>>>> 1611109816 16 5 144.0625 79.65625 12.3359375
>>>>> 1611110116 16 5 145.5625 84.5234375 12.3359375
>>>>> 1611110416 16 5 147.24609375 88.0859375 12.3359375
>>>>> 1611110716 16 5 148.49609375 91.43359375 12.3359375
>>>>> 1611111016 16 5 149.99609375 95.421875 12.3359375
>>>>> 1611111316 16 5 151.49609375 100.3359375 12.3359375
>>>>> 1611111616 16 5 152.99609375 103.8359375 12.3359375
>>>>> 1611111916 16 5 154.49609375 107.51171875 12.3359375
>>>>> 1611112216 16 5 155.99609375 111.5 12.3359375
>>>>> 1611112516 16 5 157.24609375 116.0859375 12.3359375
>>>>> 1611112816 16 5 158.74609375 119.5859375 12.3359375
>>>>> 1611113116 16 5 160.24609375 123.24609375 12.3359375
>>>>> 1611113416 16 5 161.74609375 127.234375 12.3359375
>>>>> 1611113716 16 5 163.24609375 132.11328125 12.3359375
>>>>>
>>>>>
>>>>>
>>>>> On Tuesday, January 19, 2021 at 5:21:31 PM UTC-8 [email protected]
>>>>> wrote:
>>>>>
>>>>>> Ok, my standalone testcase was wrong. I think I have a valid one
>>>>>> here, https://github.com/bellrichm/experiments
>>>>>> It just calls Cheetah.Template.Template in a loop. If the include
>>>>>> file modified attribute does not change, all seems stable. If it
>>>>>> changes,
>>>>>> memory usage seems to grow.
>>>>>> rich
>>>>>>
>>>>>> On Tuesday, 19 January 2021 at 19:16:46 UTC-5 [email protected] wrote:
>>>>>>
>>>>>>> I'm not quite following this discussion.
>>>>>>>
>>>>>>> With Cheetah, you can compile the template and save the results. I
>>>>>>> believe this is what monitors the changed status of #include files.
>>>>>>> But,
>>>>>>> that's not the way WeeWX uses templates. It compiles, evaluates, then
>>>>>>> throws the results away.
>>>>>>>
>>>>>>> If you use the "raw" directive in an #include, then the file will
>>>>>>> not be parsed. So, how can any $ directives work?
>>>>>>>
>>>>>>> Or, does Belchertown work differently?
>>>>>>>
>>>>>>> -tk
>>>>>>>
>>>>>>> On Tue, Jan 19, 2021 at 3:55 PM [email protected] <[email protected]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I agree that something does not seem quite right with Cheetah’s
>>>>>>>> ‘monitoring’ of include files. I took a different approach.
>>>>>>>> First, I wrote a simple WeeWX service that would perform the
>>>>>>>> ‘touch’ command against the large file and a simple template that
>>>>>>>> included
>>>>>>>> the large file. When the file modified attribute changed, I observed
>>>>>>>> the
>>>>>>>> memory growth, otherwise all was stable. Also, when the file was not
>>>>>>>> ‘touched’ the Cheetah processing was noticeably faster.
>>>>>>>> Next I wrote a standalone program to run a loop that ran the touch
>>>>>>>> command and invoke Cheetah. I ran it 3 times. Each time around loop
>>>>>>>> 90,
>>>>>>>> the memory growth stopped and the processing was fast as though
>>>>>>>> Cheetah was
>>>>>>>> using cached data.
>>>>>>>> I am back to running under WeeWX to see if it will level off like
>>>>>>>> it does standalone. I am also wading through the Cheetah code and
>>>>>>>> documentation. As of now, I thoroughly confused.
>>>>>>>> rich
>>>>>>>>
>>>>>>>> On Tuesday, 19 January 2021 at 15:55:38 UTC-5 [email protected]
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I wasn't happy leaving this is as is so I did some more testing.
>>>>>>>>>
>>>>>>>>> I looked in /home/weewx/bin/user/belchertown.py for any mention of
>>>>>>>>> the 4 include files it uses to add content - nothing there.
>>>>>>>>>
>>>>>>>>> So, I looked in /home/weewx/skins/Belchertown/index.html.templ and
>>>>>>>>> found 4 lines that reference the include files - they are of the form:
>>>>>>>>>
>>>>>>>>> #if
>>>>>>>>> os.path.exists("index_hook_after_station_info.inc")
>>>>>>>>> <!-- Start of index_hook_after_station_info row -->
>>>>>>>>> <div class="row index-hook-after-station-info
>>>>>>>>> border-bottom">
>>>>>>>>> #include "index_hook_after_station_info.inc"
>>>>>>>>> </div>
>>>>>>>>> <!-- End of index_hook_after_station_info row -->
>>>>>>>>> #end if
>>>>>>>>> That caused me to look up the Cheetah '#include' directive:
>>>>>>>>>
>>>>>>>>> "7.6 #include
>>>>>>>>>
>>>>>>>>> Syntax:
>>>>>>>>> #include [raw] FILENAME_EXPR #include [raw] source=STRING_EXPR
>>>>>>>>>
>>>>>>>>> The #include directive is used to include text from outside the
>>>>>>>>> template definition. The text can come from an external file or from
>>>>>>>>> a $placeholder variable. When working with external files, Cheetah
>>>>>>>>> will
>>>>>>>>> monitor for changes to the included file and update as necessary.
>>>>>>>>>
>>>>>>>>> This example demonstrates its use with external files:
>>>>>>>>> #include "includeFileName.txt"
>>>>>>>>> The content of "includeFileName.txt" will be parsed for Cheetah
>>>>>>>>> syntax.
>>>>>>>>>
>>>>>>>>> And this example demonstrates use with $placeholder variables:
>>>>>>>>> #include source=$myParseText
>>>>>>>>> The value of $myParseText will be parsed for Cheetah syntax. This
>>>>>>>>> is not the same as simply placing the $placeholder tag
>>>>>>>>> ``$myParseText'' in
>>>>>>>>> the template definition. In the latter case, the value of
>>>>>>>>> $myParseText
>>>>>>>>> would not be parsed.
>>>>>>>>>
>>>>>>>>> By default, included text will be parsed for Cheetah tags. The
>>>>>>>>> argument ``raw'' can be used to suppress the parsing.
>>>>>>>>>
>>>>>>>>> #include raw "includeFileName.txt" #include raw
>>>>>>>>> source=$myParseText
>>>>>>>>>
>>>>>>>>> Cheetah wraps each chunk of #include text inside a
>>>>>>>>> nested Template object. Each nested template has a copy of the main
>>>>>>>>> template's searchList. However, #set variables are visible across
>>>>>>>>> includes
>>>>>>>>> only if the defined using the #set global keyword.
>>>>>>>>>
>>>>>>>>> All directives must be balanced in the include file. That is, if
>>>>>>>>> you start a #for or #if block inside the include, you must end it in
>>>>>>>>> the
>>>>>>>>> same include. (This is unlike PHP, which allows unbalanced constructs
>>>>>>>>> in
>>>>>>>>> include files.)"
>>>>>>>>>
>>>>>>>>> I decided to insert the 'raw' argument into the 4 '#include'
>>>>>>>>> directives to see if anything changed.
>>>>>>>>>
>>>>>>>>> After about 1 hour. memory usage did not grow above about 65 MB!
>>>>>>>>>
>>>>>>>>> I just removed the 'raw' argument and restarted WeeWX. After just
>>>>>>>>> a few minutes, memory usage is almost 300 MB and growing!
>>>>>>>>>
>>>>>>>>> As before, the station website is at:
>>>>>>>>> https://lockyer.ca/weather/OsoyoosLakeNorthEast and the mem
>>>>>>>>> graphs are at https://lockyer.ca/weather/OsoyoosLakeNorthEast/mem.
>>>>>>>>>
>>>>>>>>> So, I think I have conclusively shown that the problem is within
>>>>>>>>> Cheetah and is related to it parsing an include file.
>>>>>>>>>
>>>>>>>>> I'm going to re-insert the 'raw' arguments and carry on.
>>>>>>>>>
>>>>>>>>> Should I report this issue to the Cheetah folks or is there
>>>>>>>>> someone closer to the Cheetah developers that can better do that?
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Garry
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Monday, January 18, 2021 at 9:27:42 AM UTC-8
>>>>>>>>> [email protected] wrote:
>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>>
>>>>>>>>>> I think I've done all that I can to isolate the problem.
>>>>>>>>>>
>>>>>>>>>> I'm going to get back to finishing up development and add a cron
>>>>>>>>>> entry to restart WeeWX every 24 hours in the wee hours of the
>>>>>>>>>> morning.
>>>>>>>>>>
>>>>>>>>>> I will be re-deploying a Davis Pro 2 system very soon (within the
>>>>>>>>>> week) and then deploying another Davis Pro 2 system shortly after
>>>>>>>>>> that.
>>>>>>>>>> And sometime after than, a couple of WeatherFlow Tempest systems.
>>>>>>>>>>
>>>>>>>>>> I'll post everything on GitHub soonest.
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>> Garry
>>>>>>>>>> PS: Re: Ubuntu Desktop on RPi, I've been using it for development
>>>>>>>>>> for the last few days and it has not crashed or hung, so I think I'm
>>>>>>>>>> going
>>>>>>>>>> to stay with it. Only issue is wireless mouse lag (Raspberry Pi OS
>>>>>>>>>> had
>>>>>>>>>> same/similar problem) and I've ordered a Logitech MK540 wireless
>>>>>>>>>> keyboard/mouse combo to see if newer hardwaere works better (than my
>>>>>>>>>> dated
>>>>>>>>>> Microsoft mouse).
>>>>>>>>>> On Sunday, January 17, 2021 at 5:40:15 PM UTC-8 vince wrote:
>>>>>>>>>>
>>>>>>>>>>> Restarted with your add-on plus belchertown both enabled - the
>>>>>>>>>>> RSS is nominally growing at 4 MB per 5-minute archive period.
>>>>>>>>>>>
>>>>>>>>>>> 1610928917 139.53125 63.97265625 15.453125
>>>>>>>>>>> 1610929217 143.02734375 70.41796875 15.4921875
>>>>>>>>>>> 1610929517 144.46484375 74.43359375 15.4921875
>>>>>>>>>>> 1610929817 145.96484375 78.48046875 15.55078125
>>>>>>>>>>> 1610930117 147.46484375 82.65625 15.58203125
>>>>>>>>>>> 1610930417 149.09375 86.69921875 15.58203125
>>>>>>>>>>> 1610930717 150.34375 90.55078125 15.58203125
>>>>>>>>>>> 1610931017 151.84375 94.203125 15.24609375
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "weewx-user" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to [email protected].
>>>>>>>>
>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/weewx-user/bfd80498-e70b-4a26-9f76-6a922439a1ben%40googlegroups.com
>>>>>>>>
>>>>>>>> <https://groups.google.com/d/msgid/weewx-user/bfd80498-e70b-4a26-9f76-6a922439a1ben%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>> .
>>>>>>>>
>>>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "weewx-user" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to [email protected].
>>>>>
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/weewx-user/4916786b-3bb2-49a6-93b1-87a60a822dbdn%40googlegroups.com
>>>>>
>>>>> <https://groups.google.com/d/msgid/weewx-user/4916786b-3bb2-49a6-93b1-87a60a822dbdn%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>> <Screen Shot 2021-01-19 at 7.33.46 PM.png>
>>>>>
>>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "weewx-user" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>>
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/weewx-user/71e3dde7-0174-42d2-908e-1d6ea27b42ban%40googlegroups.com
>>>
>>> <https://groups.google.com/d/msgid/weewx-user/71e3dde7-0174-42d2-908e-1d6ea27b42ban%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/636375d5-7a48-4455-8212-b04eb43f341en%40googlegroups.com.
