- If you disable Belchertown and 'enable' your custom code that generates
the include file, does memory grow ?
- Yes. I just started at test run with WXFeedsMemoryTest.py (attached
here) and Belchertown enabled.
The station website is at:
https://lockyer.ca/weather/OsoyoosLakeNorthEast and the mem graphs are
at: https://lockyer.ca/weather/OsoyoosLakeNorthEast/mem.
- You also haven't told us what hardware you're on. Is it a pi ? What os
? What version ?
- From an earlier post:
RPi 4 - 8 GB
Ubuntu Raspberry Pi OS 20.10
Python3 - 3.8.6
WeeWX 4.3
Belchertown 1.2
- Problem was first noticed on RPi Zero W with Raspberry Pi OS Lite.
- Problem was reproduced on RPi 4 - 4 & 8 GB with Raspberry Pi OS
Desktop.
- My extension was reduced to minimum code needed to reproduce the
problem - that code is attached.
- Your log says you're running python3. Have you tried the same thing on a
python2 system ?
- No. I have no interest in python2. :^))
- Can you duplicate the problem using just the Simulator driver and your
addition so we can try to replicate it ?
- The run I just started (about 11:30 AM PST) is using the Simulator
(rather than WeatherFlow), and memory usage
is growing. I'll let it fun for 1-2 hours and then disable Belchertown.
Regards,
Garry
Sorry but I was not able to attach a file to this message! I was able to
previously under Raspberry Pi OS so maybe it's related to Ubuntu?
Here's the code for WXFeedsMemoryTest.py:
######################################################################################################
# WXFeedsMemoryTest.py
#
# Copyright (c) 2021 Garry Lockyer - [email protected]
# See the file LICENSE.txt for your rights.
#
# Based on alarm.py by:
# Copyright (c) 2009-2019 Tom Keffer <[email protected]>
# See the file LICENSE.txt for your rights.
"""
To use this service, add the following to the weewx configuration file:
[WXFeedsMemoryTest]
# No options are required or processed at this time.
This service requests data from an RSS feed:
feedURL =
"https://511.alberta.ca/api/v2/get/winterroads?format=json&lang=en";
and creates an include file that the Belchertown Skin can include.
*******************************************************************************
To enable this service:
1) Copy this file to the WeWX bin/user directory - /home/weewx/bin/user if
WeeWX was
installed with setup.py. See https://bit.ly/33YHsqX for where your user
directory is located.
2) Modify the weewx configuration file by adding this service to the option
"report_services", located in section [Engine][[Services]].
[Engine]
[[Services]]
...
report_services = weewx.engine.StdPrint, weewx.engine.StdReport,
user.WXFeedsMemoryTest.WXFeedsMemoryTest
# I want WXFeedsMemoryTest to run before StdReport so I actually use:
report_services = weewx.engine.StdPrint,
user.WXFeedsMemoryTest.WXFeedsMemoryTest, weewx.engine.StdReport
"""
import weewx
from weewx.engine import StdService
import logging
import os
import sys
import gc
import requests
import shutil
# Inherit from the base class StdService:
class WXFeedsMemoryTest(StdService):
"""
A WeeWX service extension to demonstrate that WeeWX with the
Belchertown skin
memory usage grows after each archive period.
All it does is get one RSS feed and produce one Belchertown include
file.
Other than creating an include in skins/Belchertown there are no
dependencies
on any skin.
"""
def __init__(self, engine, config_dict):
# Pass the initialization information on to my superclass:
super(WXFeedsMemoryTest, self).__init__(engine, config_dict)
self.log = logging.getLogger( __name__ )
assert self.log != None
self.log.info( "WXFeedsMemoryTest::__init__(). . ." )
self.bind( weewx.NEW_ARCHIVE_RECORD, self.new_archive_record )
return
def new_archive_record(self, event):
"""Gets called on a new archive record event."""
self.log.info( "WXFeedsMemoryTest::new_archive_record(). . ." )
self.log.info( "Process ID: %d" % os.getpid() )
# This URL is for Alberta 511's Road Conditions RSS feed.
feedURL =
"https://511.alberta.ca/api/v2/get/winterroads?format=json&lang=en";
# The include file is used by the Belchertown Skin.
#
# The problem:
#
# If the include file is static, as in it is not re-created while
# Belchertown is enabled, weewx memory usage does not grow.
#
# If the include file is dynamic, as in it is re-created while
# Belchertown is enabled, weewx memory usage increases
# after each file re-creation.
#
# If this service is enabled AND Belchertown is disabled, memory
# usage does not grow,
includeFileName =
"/home/weewx/skins/Belchertown/index_hook_after_charts.inc"
tempFileName = "/home/weewx/skins/Belchertown/WXFeedsTemp.inc"
# The include file can be created DIRECTLY or INDIRECTLY:
#
# DIRECTLY: File is opened, written to and closed.
#
# INDIRECTLY: A temp file is opened, written to and closed and then
# renamed or copied to the Belchertown skin filename. During
# testing, the thinking was the problem was associated with the
# actual writing of the file and that the problem might go away if
# the file was created and then renamed or copied. It did not.
directFileCreation = True # If True, include file is created
directly, if False, include is create as a
# 'temporary' file and then copied or
renamed.
# Does not seem to affect problem.
copyFile = True # For indirect include file creation:
# If True, temporary file is copied, if
False, temporary file is renamed.
# Does not seem to affect problem.
recreateFile = True # If True, recreate the include file,
if False, do not recreate the include file.
#
# Recreating the include file appears
to cause weewx memory usage to grow.
#
# If True, weewx memory usage will
increase on each archive cycle.
#
# If False and the include file does
not exist when weewx/Belchertown starts, it
# will be created the first time this
method is called but not on subsequent calls.
# Memory usage will stabilize.
assert gc.isenabled() == True
self.log.info( "WXFeedsMemoryTest:" )
self.log.info( "Memory use before gc.collect():" )
x, y, z = gc.get_count()
self.log.info( "gc.get_count(): %d, %d, %d" % ( x, y, z ) )
self.log.info( "sys.getallocatedblocks(): %d" %
sys.getallocatedblocks() )
#sys._clear_type_cache()
gc.collect()
self.log.info( "Memory use after gc.collect():" )
x, y, z = gc.get_count()
self.log.info( "gc.get_count(): %d, %d, %d" % ( x, y, z ) )
self.log.info( "sys.getallocatedblocks(): %d" %
sys.getallocatedblocks() )
self.log.info( "" )
try:
session = requests.Session()
assert session != None
response = session.get( feedURL )
assert response != None
except:
self.log.info("Alberta511RoadConditions: requests.get()
returned error: %d!" % response.status_code)
if response.status_code == requests.codes.ok :
roadConditions = response.json()
self.log.info( "Alberta511RoadConditions: Received %d entries."
% len( roadConditions ) )
if os.path.exists( includeFileName ) == True and \
recreateFile == False:
return
if os.path.exists( includeFileName ) == True:
os.remove( includeFileName )
if os.path.exists( tempFileName ) == True:
os.remove( tempFileName )
if directFileCreation == True:
file = open( includeFileName, "wt", 1 )
else:
file = open( tempFileName, "wt", 1 )
assert file != None
for roadCondition in roadConditions:
HTML = "<table>\n"
\
+ "<thead>\n"
\
+ "<tr>\n"
\
+ "<th></th>\n"
\
+ "<th></th>\n"
\
+ "</tr>\n"
\
+ "</thead>\n"
\
+ "<tbody>\n"
\
+ "<tr>\n"
\
+ "<td><b>Roadway Name: </b></td>\n"
\
+ "<td>%s</td>\n" % roadCondition.get(
"RoadwayName" ) \
+ "</tr>\n"
\
+ "<tr>\n"
\
+ "<td><b>Location
Description: </b></td>\n" \
+ "<td>%s</td>\n" % roadCondition.get(
"LocationDescription" ) \
+ "</tr>\n"
\
+ "<tr>\n"
\
+ "<td><b>Primary
Condition: </b></td>\n" \
+ "<td>%s</td>\n" % roadCondition.get(
"Primary Condition" ) \
+ "</tr>\n"
\
+ "<tr>\n"
\
+ "<td><b>Id: </b></td>\n"
\
+ "<td>%s</td>\n" % roadCondition.get( "Id"
) \
+ "</tr>\n"
\
+ "<tr>\n"
\
+ "<td><b>Area Name: </b></td>\n"
\
+ "<td>%s</td>\n" % roadCondition.get(
"AreaName" ) \
+ "</tr>\n"
\
+ "<tr>\n"
\
+ "<td><b>Last Updated: </b></td>\n"
\
+ "<td>%s</td>\n" % roadCondition.get(
"LastUpdated" ) \
+ "</tr>\n"
\
+ "</tbody>\n"
\
+ "</table>\n"
\
+ "<hr>\n"
file.write( HTML )
file.flush()
os.fsync( file.fileno() )
del roadConditions
file.close()
del file
if directFileCreation == False:
# The include file should have been deleted above but
# so that we don't get any exceptions because the destination
# already exists, we'll delete it here, before a copy or rename.
if os.path.exists( includeFileName ) == True:
os.remove( includeFileName )
if copyFile == True:
shutil.copy2( tempFileName, includeFileName )
else:
os.replace( tempFileName, includeFileName )
response.close()
del response
session.close()
del session
self.log.info( "WXFeedsMemoryTest:" )
self.log.info( "Garbage collection after writing file:" )
self.log.info( "Memory use before gc.collect():" )
x, y, z = gc.get_count()
self.log.info( "gc.get_count(): %d, %d, %d" % ( x, y, z ) )
self.log.info( "sys.getallocatedblocks(): %d" %
sys.getallocatedblocks() )
#sys._clear_type_cache()
gc.collect()
self.log.info( "Memory use after gc.collect():" )
x, y, z = gc.get_count()
self.log.info( "gc.get_count(): %d, %d, %d" % ( x, y, z ) )
self.log.info( "sys.getallocatedblocks(): %d" %
sys.getallocatedblocks() )
self.log.info( "" )
return
#########################################################################################
On Sunday, January 17, 2021 at 9:28:24 AM UTC-8 vince wrote:
> On Saturday, January 9, 2021 at 8:02:51 PM UTC-8 [email protected]
> wrote:
>
>> My experience is that when I recreate an include file for Belchertown on
>> each archive cycle, as an extension to Belchertown or as a service, memory
>> usage increases with each cycle and eventually errors start.
>>
>
> I continue to think you need to concentrate on what your code is doing and
> your os configuration.
>
> - If you disable Belchertown and 'enable' your custom code that
> generates the include file, does memory grow ?
> - You also haven't told us what hardware you're on. Is it a pi ?
> What os ? What version ?
> - Your log says you're running python3. Have you tried the same thing
> on a python2 system ?
> - Can you duplicate the problem using just the Simulator driver and
> your addition so we can try to replicate it ?
>
>
>
>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/a6397145-c226-4a42-a8f3-1ee7b3d86aa5n%40googlegroups.com.
