Garry,
I ran your test extension for a bit in my Ubuntu VM. Doesn't appear to be
leaking.
16:42:24 {'mem_size': 338.53515625, 'mem_rss': 32.953125, 'mem_share':
11.28125}
16:47:38 {'mem_size': 423.80859375, 'mem_rss': 47.75, 'mem_share':
11.796875}
16:52:53 {'mem_size': 424.4140625, 'mem_rss': 48.2265625, 'mem_share':
11.796875}
16:57:24 {'mem_size': 426.46875, 'mem_rss': 50.19921875, 'mem_share':
11.796875}
17:02:32 {'mem_size': 426.46875, 'mem_rss': 50.21875, 'mem_share':
11.796875}
17:07:20 {'mem_size': 427.1796875, 'mem_rss': 50.9921875, 'mem_share':
11.796875}
17:12:27 {'mem_size': 427.1796875, 'mem_rss': 50.9921875, 'mem_share':
11.796875}
17:17:26 {'mem_size': 427.1796875, 'mem_rss': 50.9921875, 'mem_share':
11.796875}
17:22:24 {'mem_size': 427.1796875, 'mem_rss': 50.9921875, 'mem_share':
11.796875}
17:27:19 {'mem_size': 424.6640625, 'mem_rss': 48.71875, 'mem_share':
11.796875}
17:32:27 {'mem_size': 424.6640625, 'mem_rss': 48.71875, 'mem_share':
11.796875}
17:37:17 {'mem_size': 424.5859375, 'mem_rss': 48.7109375, 'mem_share':
11.796875}
17:42:21 {'mem_size': 424.58203125, 'mem_rss': 48.70703125, 'mem_share':
11.796875}
17:47:24 {'mem_size': 424.984375, 'mem_rss': 49.140625, 'mem_share':
11.796875}
17:52:21 {'mem_size': 424.984375, 'mem_rss': 49.140625, 'mem_share':
11.796875}
17:57:15 {'mem_size': 425.05078125, 'mem_rss': 49.26953125, 'mem_share':
11.796875}
18:02:11 {'mem_size': 425.05078125, 'mem_rss': 49.26953125, 'mem_share':
11.796875}
18:07:13 {'mem_size': 425.1171875, 'mem_rss': 49.3359375, 'mem_share':
11.796875}
18:12:14 {'mem_size': 425.1171875, 'mem_rss': 49.3359375, 'mem_share':
11.796875}
18:17:12 {'mem_size': 425.21875, 'mem_rss': 49.4375, 'mem_share':
11.796875}
18:22:10 {'mem_size': 425.09765625, 'mem_rss': 49.31640625, 'mem_share':
11.796875}
18:27:13 {'mem_size': 425.0390625, 'mem_rss': 49.2578125, 'mem_share':
11.796875}
18:32:15 {'mem_size': 425.0390625, 'mem_rss': 49.2578125, 'mem_share':
11.796875}
18:37:15 {'mem_size': 425.03125, 'mem_rss': 49.25, 'mem_share': 11.796875}
18:42:24 {'mem_size': 425.01953125, 'mem_rss': 49.2578125, 'mem_share':
11.796875}
18:47:14 {'mem_size': 425.0078125, 'mem_rss': 49.24609375, 'mem_share':
11.796875}
18:52:13 {'mem_size': 425.0078125, 'mem_rss': 49.24609375, 'mem_share':
11.796875}
18:57:15 {'mem_size': 425.61328125, 'mem_rss': 49.8515625, 'mem_share':
11.796875}
19:02:19 {'mem_size': 425.0078125, 'mem_rss': 49.2734375, 'mem_share':
11.796875}
When I have some time, I'll install on my test pi along with the
Belchertown skin and Vince's mem extension and see if I see anything.
Yes, I enjoy a good memory 'puzzle'.
rich
On Saturday, 9 January 2021 at 16:52:08 UTC-5 garrya...@gmail.com wrote:
> Many, many thanks!
>
> Before I wrote the service to investigate the problem, I wrote a
> standalone program to eliminate all WeeWX components. It did NOT exhibit
> the problem.
>
> Problem only seems to show up under WeeWX/Belchertown *and* if the include
> file changes / is recreated.
>
> Anyway, I really appreciate you looking into my problem. I’m actively
> working on a skin-independent service to read many RSS/Atom feeds and will
> utilize your suggestions above.
>
> Regards,
>
> Garry Lockyer
> C: +1.250.689.0686 <(250)%20689-0686>
> E: ga...@lockyer.ca
>
>
> On Jan 9, 2021, at 12:39, bell...@gmail.com <bell...@gmail.com> wrote:
>
>
>
> Garry,
> I took your WxFeedsMemory.py and made some modifications.
> First, I changed the logging to simple print calls. this was just a
> convenience for me to easily see what was going on.
> Second, I stole some code from Vince’s mem extension,
> https://github.com/vinceskahan/vds-weewx-v3-mem-extension. I did this
> because I am used to the data it returns. (Note, I highly recommend this
> when debugging memory.)
> Third, I added code so I could call your extension repeatedly in a loop.
>
> When I called new_archive_record 100 times there was a slight increase in
> memory, but nothing alarming. But it was slow, approximately 2 minutes a
> call.
>
> So, I commented out the flush and fsync calls and called
> new_archive_record 10,000 times. The slight increase seemed to stop around
> 6,000 calls.
>
> I’m wondering if the amount of time it takes to run is causing problems
> when run as a service. I plan to uncomment the code and run it as an
> extension. I’ll let you know what I see.
> rich
>
> On Thursday, 7 January 2021 at 13:00:40 UTC-5 garrya...@gmail.com wrote:
>
>> After many hours of troubleshooting and testing, I think I have an idea
>> what's happening.
>>
>> Background: I want to use weewx with the Belchertown skin and my
>> extension that reads numerous RSS/Atom feeds on a Pi Zero, with either a
>> Davis Pro V2 or Weatherflow weather station. I will eventually have four
>> weather stations in my area. I thought I had things working reasonably
>> well so deployed one station. It stopped working after about 3 days. I
>> can't access that system remotely so I built a very similar system (the
>> next one to deploy) and it falso ailed due to lack of memory. Weewx memory
>> usage on my development system also increases over time - it can grow to
>> over 1 GB in a few hours!
>>
>> My extension was an extension to Belchetown based on the METAR
>> extension. The extension created Belchertown index hook include files each
>> archive cycle. While researching this problem I re-read the weewx
>> customization guide and noted that extensions should NOT be dependent on
>> other extensions so I decided to re-write my extension as a service (which
>> looks like a much cleaner solution) without any dependencies on Belchertown
>> (other than include file names). All I've done so far is create a service
>> (WxFeedsMemoryTest.py) to test weewx/Belchertown memory consumption.
>>
>> The Problem: weewx/Bechertown memory usage increases over time. It
>> starts out at about 45 MB and grows at about 3MB per archive period/cycle
>> (using my test case). A 512 MB Pi will exhaust memory within a few days.
>>
>> It appears that the problem is associated with the creation of the
>> Belchertown include files while weewx/Belchertown is running:
>>
>> - if the include file is 'static' as in not (re)created while
>> weewx/Belchertown is running, memory usage is static - it does not grow
>> beyond about 50 MB.
>>
>> - if the include file is 'dynamic' as in (re)created while
>> weewx/Belchetown is running, memory usage increases.
>> - if the include file is created once, and becomes 'static', memory
>> usage increases and then stabilizes.
>> - if the include file is recreated continuously (such as on each
>> archive cycle), memory usage increases each cycle.
>>
>> It does not appear to matter if the include file is created directly, or
>> created as a temporary file and then copied or renamed.
>>
>> The attached service (WxFeedsMemoryTest.py) can be used to demonstrate
>> the problem. Please see installation and use instructions within the
>> WxFeedsMemoryTest.py.
>>
>> I'm going to continue to work on moving my extension from "an extension
>> to an extension" to a service in the hope that this memory problem can be
>> resolved.
>>
>> With apologies in advance if I'm doing something to cause the problem,
>> please review, advise and let me know what I can do to avoid the problem.
>>
>> Regards,
>>
>> Garry
>> On Thursday, December 31, 2020 at 7:05:15 PM UTC-8 garrya...@gmail.com
>> wrote:
>>
>>> Got MemoryError after about 9 hours after restart. Have removed cmon by
>>> commenting out any mention of cmon in weewx.conf and restarted.
>>>
>>> Regards,
>>>
>>> Garry Lockyer
>>> Former DEC Product Support Engineer :^)
>>> Kepner-Tregoe Trained :^))
>>> C: +1.250.689.0686 <(250)%20689-0686>
>>> E: ga...@lockyer.ca
>>>
>>>
>>> On Dec 31, 2020, at 11:44, vince <vince...@gmail.com> wrote:
>>>
>>> On Thursday, December 31, 2020 at 11:39:39 AM UTC-8 garrya...@gmail.com
>>> wrote:
>>>
>>>> Re: editing the Belchertown skin, nope haven’t touched it, *other than
>>>> interfacing with it via the include files (as generated by my
>>>> BelchertownWxFeeds extension)*. When all the endpoints (for testing)
>>>> are enabled index.html is about 1.8MB, so perhaps that’s causing the
>>>> problem. I can easily reduce / eliminate endpoints and prefer to do that
>>>> before eliminating the Belchertown skin.
>>>>
>>>>
>>> There it is. You touched it :-)
>>>
>>> Usual debugging rules apply. Reset it to a baseline unmodified config.
>>> Add in changes one-by-one. If it goes sideways, revert to the last known
>>> good and reverify that it stays good.
>>>
>>>
>>> --
>>>
>>> You received this message because you are subscribed to the Google
>>> Groups "weewx-user" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to weewx-user+...@googlegroups.com.
>>>
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/weewx-user/57a5bfc7-d0b4-4419-b178-6342564642edn%40googlegroups.com
>>>
>>> <https://groups.google.com/d/msgid/weewx-user/57a5bfc7-d0b4-4419-b178-6342564642edn%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+...@googlegroups.com.
>
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/d5e41e91-5dab-41bc-bc34-682826ba4753n%40googlegroups.com
>
> <https://groups.google.com/d/msgid/weewx-user/d5e41e91-5dab-41bc-bc34-682826ba4753n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/9d827197-21fe-4238-86a1-58bd001d4b66n%40googlegroups.com.
