Garry, I took your WxFeedsMemory.py and made some modifications. First, I changed the logging to simple print calls. this was just a convenience for me to easily see what was going on. Second, I stole some code from Vince’s mem extension, https://github.com/vinceskahan/vds-weewx-v3-mem-extension. I did this because I am used to the data it returns. (Note, I highly recommend this when debugging memory.) Third, I added code so I could call your extension repeatedly in a loop.
When I called new_archive_record 100 times there was a slight increase in memory, but nothing alarming. But it was slow, approximately 2 minutes a call. So, I commented out the flush and fsync calls and called new_archive_record 10,000 times. The slight increase seemed to stop around 6,000 calls. I’m wondering if the amount of time it takes to run is causing problems when run as a service. I plan to uncomment the code and run it as an extension. I’ll let you know what I see. rich On Thursday, 7 January 2021 at 13:00:40 UTC-5 [email protected] wrote: > After many hours of troubleshooting and testing, I think I have an idea > what's happening. > > Background: I want to use weewx with the Belchertown skin and my extension > that reads numerous RSS/Atom feeds on a Pi Zero, with either a Davis Pro > V2 or Weatherflow weather station. I will eventually have four weather > stations in my area. I thought I had things working reasonably well so > deployed one station. It stopped working after about 3 days. I can't > access that system remotely so I built a very similar system (the next one > to deploy) and it falso ailed due to lack of memory. Weewx memory usage on > my development system also increases over time - it can grow to over 1 GB > in a few hours! > > My extension was an extension to Belchetown based on the METAR extension. > The extension created Belchertown index hook include files each archive > cycle. While researching this problem I re-read the weewx customization > guide and noted that extensions should NOT be dependent on other extensions > so I decided to re-write my extension as a service (which looks like a much > cleaner solution) without any dependencies on Belchertown (other than > include file names). All I've done so far is create a service > (WxFeedsMemoryTest.py) to test weewx/Belchertown memory consumption. > > The Problem: weewx/Bechertown memory usage increases over time. It starts > out at about 45 MB and grows at about 3MB per archive period/cycle (using > my test case). A 512 MB Pi will exhaust memory within a few days. > > It appears that the problem is associated with the creation of the > Belchertown include files while weewx/Belchertown is running: > > - if the include file is 'static' as in not (re)created while > weewx/Belchertown is running, memory usage is static - it does not grow > beyond about 50 MB. > > - if the include file is 'dynamic' as in (re)created while > weewx/Belchetown is running, memory usage increases. > - if the include file is created once, and becomes 'static', memory > usage increases and then stabilizes. > - if the include file is recreated continuously (such as on each archive > cycle), memory usage increases each cycle. > > It does not appear to matter if the include file is created directly, or > created as a temporary file and then copied or renamed. > > The attached service (WxFeedsMemoryTest.py) can be used to demonstrate the > problem. Please see installation and use instructions within the > WxFeedsMemoryTest.py. > > I'm going to continue to work on moving my extension from "an extension to > an extension" to a service in the hope that this memory problem can be > resolved. > > With apologies in advance if I'm doing something to cause the problem, > please review, advise and let me know what I can do to avoid the problem. > > Regards, > > Garry > On Thursday, December 31, 2020 at 7:05:15 PM UTC-8 [email protected] > wrote: > >> Got MemoryError after about 9 hours after restart. Have removed cmon by >> commenting out any mention of cmon in weewx.conf and restarted. >> >> Regards, >> >> Garry Lockyer >> Former DEC Product Support Engineer :^) >> Kepner-Tregoe Trained :^)) >> C: +1.250.689.0686 <(250)%20689-0686> >> E: [email protected] >> >> >> On Dec 31, 2020, at 11:44, vince <[email protected]> wrote: >> >> On Thursday, December 31, 2020 at 11:39:39 AM UTC-8 [email protected] >> wrote: >> >>> Re: editing the Belchertown skin, nope haven’t touched it, *other than >>> interfacing with it via the include files (as generated by my >>> BelchertownWxFeeds extension)*. When all the endpoints (for testing) >>> are enabled index.html is about 1.8MB, so perhaps that’s causing the >>> problem. I can easily reduce / eliminate endpoints and prefer to do that >>> before eliminating the Belchertown skin. >>> >>> >> There it is. You touched it :-) >> >> Usual debugging rules apply. Reset it to a baseline unmodified config. >> Add in changes one-by-one. If it goes sideways, revert to the last known >> good and reverify that it stays good. >> >> >> -- >> >> You received this message because you are subscribed to the Google Groups >> "weewx-user" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/weewx-user/57a5bfc7-d0b4-4419-b178-6342564642edn%40googlegroups.com >> >> <https://groups.google.com/d/msgid/weewx-user/57a5bfc7-d0b4-4419-b178-6342564642edn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/d5e41e91-5dab-41bc-bc34-682826ba4753n%40googlegroups.com.
