I'm running on: raspbian stretch, python 2.7.13, and WeeWX 3.9.1.
I was noticing a steady increase in memory usage. I narrowed it down to one
service I had installed. This service binds to the loop event and makes
over 10 calls to getSql. I updated the service to open and close the
connection on each loop event. This seems to have stopped the memory
increase.
Next, I wrote a small program that does the same steps. Create a
connection, get a cursor, execute a select, fetchone, close the cursor. But
this isn't exibiting the memory increase.
Next I wrote a wrapper class for the Connection class and subclassed the
Cursor class, like WeeWX does. Running with these, I am again seeing the
memory increase.
I don't see how the 'wrapper' classes should matter... Granted at 10+ calls
per loop, this is aproximately 240 calls a minute - so it would add up
fast; but I would think others would be seeing something too. I did see
this thread about the forecast service,
https://groups.google.com/forum/#!topic/weewx-user/H4GVpoI5l70/discussion
It looks to make a lot of calls to getSql, but binds to archive, so it
would happen a lot slower. Not sure if there was any resolution.
I've attached my test program. It can call the sqlite classes directly or
use the 'wrapper' classes I stole from WeeWx. I may be all wet, but I'm
seeing something strange in my environment... I'll keep digging.
-rich
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/3753360b-d4c7-44b3-82ca-a8140514b310%40googlegroups.com.
#!/usr/bin/python2
"""Testing memory"""
import gc
import os
import resource
import sqlite3
import six
VERSION ='0.0.1'
DATABASE = '/home/pi/weewx-weather/archive-replica/weewx.sdb'
STATEMENT = 'SELECT * from archive'
LOOP_AMOUNT = 100000
WRAP_CLASS = True
class Connection(object):
""" connection """
def __init__(self, database_name=''):
"""Initialize an instance of Connection."""
connection = sqlite3.connect(database_name)
self.connection = connection
def cursor(self):
"""Return a cursor object."""
return Cursor(self.connection)
def close(self):
"""close"""
self.connection.close()
class Cursor(sqlite3.Cursor):
"""A wrapper around the sqlite cursor object"""
# The sqlite3 cursor object is very full featured. We need only turn
# the sqlite exceptions into weedb exceptions.
def __init__(self, *args, **kwargs):
sqlite3.Cursor.__init__(self, *args, **kwargs)
def execute(self, *args, **kwargs):
"""execute"""
return sqlite3.Cursor.execute(self, *args, **kwargs)
def fetchone(self):
"""fecth one row"""
return sqlite3.Cursor.fetchone(self)
def __enter__(self):
return self
def __exit__(self, etyp, einst, etb): # @UnusedVariable
# It is not an error to close a sqlite3 cursor multiple times,
# so there's no reason to guard it with a "try" clause:
self.close()
def loop(conn):
"""loop"""
print_mem()
i = 0
while i < LOOP_AMOUNT:
cursor = conn.cursor()
cursor.execute(STATEMENT)
result = cursor.fetchone()
# print(result)
cursor.close()
i += 1
print_mem()
ngc = gc.collect()
print("Garbage collected %d objects" % ngc)
print_mem()
print("press enter to continue.")
if six.PY2:
raw_input()
else:
input()
def print_mem():
"""print mem usage"""
try:
page_size = resource.getpagesize()
pid = os.getpid()
procfile = "/proc/%s/statm" % pid
try:
mem_tuple = open(procfile).read().split()
except (IOError, ):
return
# Unpack the tuple:
(size, resident, share, text, lib, data, dt) = mem_tuple
mb = 1024 * 1024
mem_size = float(size) * page_size / mb
mem_rss = float(resident) * page_size / mb
mem_share = float(share) * page_size / mb
print("mem_size: %f mem_rss: %f mem_share: %f" % (mem_size, mem_rss, mem_share))
except (ValueError, IOError, KeyError) as e:
print('memory_info failed: %s' % e)
if WRAP_CLASS:
loop(Connection(DATABASE))
else:
loop(sqlite3.Connection(DATABASE))
print_mem()
