Hi,
To trap bad data I have a derivative of alarm.py that has been sucessully
running for years on Python 2.7 and weewx 3.x. I upgraded to weewx 4 and
Python 3 a month or so ago and it has been running quite happily until a
few days ago.
The bad data parameters have been triggered and badData.py has been called.
The line that causes the issue is;
if eval(self.expression, None, record):
and the error is;
TypeError: '<' not supported between instances of 'NoneType' and 'float'
*FYI I'm a Python novice and copy from working code then tweek to what I
need and iterate to a result!*
I looked at Alarm.py in the examples directory and Customisation guide and
the offending line is the same as in my badData.py. So I'm assuming that
syntax is OK for Python3
The error message is objecting to comparing a float to a None. I'm not sure
why this is an issue now when it's run for several years.
My bad data test is for a temp under 10 or wind gust over 100
In weewx.conf:
badData_test = outTemp < 10.0 or windGust > 100
as it complaining about the < then it's the outTemp variable that's coming
through as non numeric.
Lokking in the archive table around the time of the crash I can see a null
record for outTemp and windGust
sqlite> select datetime, outtemp, windgust from archive where datetime >
1594530300 and datetime < 1594531800;
1594530600|43.34|0.0
1594530900||
1594531319|43.52|0.0
1594531619|43.52|0.0
I have had null values whilst running python2
I'm not sure where to go from here, help!
*Configuration:*
Maplin/Fine Offset weather station been running since 2013
Raspberry PI 1B running Buster
Python 3.7.3
*1 . badData.py - attached*
*2. extracts from Weewx.conf:*
##############################################################################
# This section configures the internal weewx engine.
[Engine]
[[Services]]
# This section specifies the services that should be run. They are
# grouped by type, and the order of services within each group
# determines the order in which the services will be run.
prep_services = weewx.engine.StdTimeSynch,
data_services = ,
process_services = weewx.engine.StdConvert,
weewx.engine.StdCalibrate, weewx.engine.StdQC,
weewx.wxservices.StdWXCalculate
archive_services = weewx.engine.StdArchive,
user.forecast.ZambrettiForecast, user.forecast.WUForecast,
user.forecast.NWSFor$
restful_services = weewx.restx.StdWunderground, weewx.restx.StdWOW,
weewx.restx.StdPWSweather, weewx.restx.StdCWOP, weewx.r$
report_services = weewx.engine.StdPrint, weewx.engine.StdReport,
user.alarm.MyAlarm, *user.badData.MyBadData*
[Alarm]
expression = " inHumidity > 90.0"
time_wait = 86400
smtp_host = xxxx
smtp_user = xxxx
smtp_password = xxxx
mailto = xxxx
from = xxxx
subject = Humidity over 90!
* badData_test = outTemp < 10.0 or windGust > 100*
badData_subject = Weather Station spike detected
*3. extrract from syslog at time of the crash*
Jul 12 06:10:29 weepi weewx[513] INFO weewx.restx: WOW: Published record
2020-07-12 06:10:00 BST (1594530600)
Jul 12 06:10:29 weepi weewx[513] INFO weewx.restx: Wunderground-PWS:
Published record 2020-07-12 06:10:00 BST (1594530600)
Jul 12 06:10:45 weepi weewx[513] INFO weewx.cheetahgenerator: Generated 8
files for report SeasonsReport in 14.87 seconds
Jul 12 06:10:52 weepi weewx[513] INFO weewx.imagegenerator: Generated 15
images for report SeasonsReport in 7.48 seconds
Jul 12 06:10:52 weepi weewx[513] INFO weewx.reportengine: Copied 0 files to
/home/weewx/public_html
Jul 12 06:10:54 weepi weewx[513] INFO weewx.cheetahgenerator: Generated 6
files for report SmartphoneReport in 0.84 seconds
Jul 12 06:10:56 weepi weewx[513] INFO weewx.imagegenerator: Generated 6
images for report SmartphoneReport in 2.36 seconds
Jul 12 06:10:56 weepi weewx[513] INFO weewx.reportengine: Copied 0 files to
/home/weewx/public_html/smartphone
Jul 12 06:10:57 weepi weewx[513] INFO weewx.cheetahgenerator: Generated 1
files for report MobileReport in 0.38 seconds
Jul 12 06:10:59 weepi weewx[513] INFO weewx.imagegenerator: Generated 4
images for report MobileReport in 1.66 seconds
Jul 12 06:10:59 weepi weewx[513] INFO weewx.reportengine: Copied 0 files to
/home/weewx/public_html/mobile
Jul 12 06:11:03 weepi weewx[513] INFO weewx.cheetahgenerator: Generated 3
files for report exfoliation in 3.55 seconds
Jul 12 06:11:10 weepi weewx[513] INFO weewx.imagegenerator: Generated 15
images for report exfoliation in 6.91 seconds
Jul 12 06:11:10 weepi weewx[513] INFO weewx.reportengine: Copied 0 files to
/home/weewx/public_html/exfoliation
Jul 12 06:15:31 weepi weewx[513] INFO weewx.manager: Added record
2020-07-12 06:15:00 BST (1594530900) to database 'weewx.sdb'
Jul 12 06:15:31 weepi weewx[513] INFO weewx.manager: Added record
2020-07-12 06:15:00 BST (1594530900) to daily summary in 'weewx.sdb'
Jul 12 06:15:31 weepi /weewxd: forecast: MainThread: Zambretti: starting
thread
Jul 12 06:15:31 weepi /weewxd: forecast: MainThread: UKMO: starting thread
Jul 12 06:15:31 weepi weewx[513] INFO weewx.engine: Main loop exiting.
Shutting engine down.
Jul 12 06:15:31 weepi weewx[513] INFO weewx.engine: Shutting down StdReport
thread
Jul 12 06:15:31 weepi weewx[513] INFO weewx.restx: WOW: Published record
2020-07-12 06:15:00 BST (1594530900)
Jul 12 06:15:31 weepi weewx[513] INFO weewx.restx: Wunderground-PWS:
Published record 2020-07-12 06:15:00 BST (1594530900)
Jul 12 06:15:31 weepi weewx[513] CRITICAL __main__: Caught unrecoverable
exception:
Jul 12 06:15:31 weepi weewx[513] CRITICAL __main__: **** '<' not
supported between instances of 'NoneType' and 'float'
Jul 12 06:15:31 weepi weewx[513] CRITICAL __main__: **** Traceback
(most recent call last):
Jul 12 06:15:31 weepi weewx[513] CRITICAL __main__: **** File
"/home/weewx/bin/weewx/engine.py", line 195, in run
Jul 12 06:15:31 weepi weewx[513] CRITICAL __main__: ****
self.dispatchEvent(weewx.Event(weewx.CHECK_LOOP, packet=packet))
Jul 12 06:15:31 weepi weewx[513] CRITICAL __main__: **** File
"/home/weewx/bin/weewx/engine.py", line 224, in dispatchEvent
Jul 12 06:15:31 weepi weewx[513] CRITICAL __main__: ****
callback(event)
Jul 12 06:15:31 weepi weewx[513] CRITICAL __main__: **** File
"/home/weewx/bin/weewx/engine.py", line 578, in check_loop
Jul 12 06:15:31 weepi weewx[513] CRITICAL __main__: **** raise
BreakLoop
Jul 12 06:15:31 weepi weewx[513] CRITICAL __main__: ****
weewx.engine.BreakLoop
Jul 12 06:15:31 weepi weewx[513] CRITICAL __main__: ****
Jul 12 06:15:31 weepi weewx[513] CRITICAL __main__: **** During
handling of the above exception, another exception occurred:
Jul 12 06:15:31 weepi weewx[513] CRITICAL __main__: ****
Jul 12 06:15:32 weepi weewx[513] CRITICAL __main__: **** Traceback
(most recent call last):
Jul 12 06:15:32 weepi weewx[513] CRITICAL __main__: **** File
"/home/weewx/bin/weewxd", line 154, in main
Jul 12 06:15:32 weepi weewx[513] CRITICAL __main__: ****
engine.run()
Jul 12 06:15:32 weepi weewx[513] CRITICAL __main__: **** File
"/home/weewx/bin/weewx/engine.py", line 202, in run
Jul 12 06:15:32 weepi weewx[513] CRITICAL __main__: ****
self.dispatchEvent(weewx.Event(weewx.POST_LOOP))
Jul 12 06:15:32 weepi weewx[513] CRITICAL __main__: **** File
"/home/weewx/bin/weewx/engine.py", line 224, in dispatchEvent
Jul 12 06:15:32 weepi weewx[513] CRITICAL __main__: ****
callback(event)
Jul 12 06:15:32 weepi weewx[513] CRITICAL __main__: **** File
"/home/weewx/bin/weewx/engine.py", line 588, in post_loop
Jul 12 06:15:32 weepi weewx[513] CRITICAL __main__: ****
self._software_catchup()
Jul 12 06:15:32 weepi weewx[513] CRITICAL __main__: **** File
"/home/weewx/bin/weewx/engine.py", line 658, in _software_catchup
Jul 12 06:15:32 weepi weewx[513] CRITICAL __main__: ****
origin='software'))
Jul 12 06:15:32 weepi weewx[513] CRITICAL __main__: **** File
"/home/weewx/bin/weewx/engine.py", line 224, in dispatchEvent
Jul 12 06:15:32 weepi weewx[513] CRITICAL __main__: ****
callback(event)
Jul 12 06:15:32 weepi weewx[513] CRITICAL __main__: **** File
"/home/weewx/bin/user/badData.py", line 94, in newArchiveRecord
Jul 12 06:15:32 weepi weewx[513] CRITICAL __main__: **** if
eval(self.expression, None, record): # NOTE 2
Jul 12 06:15:32 weepi weewx[513] CRITICAL __main__: **** File
"<string>", line 1, in <module>
Jul 12 06:15:32 weepi weewx[513] CRITICAL __main__: **** TypeError:
'<' not supported between instances of 'NoneType' and 'float'
Jul 12 06:15:32 weepi weewx[513] CRITICAL __main__: **** Exiting.
Jul 12 06:17:01 weepi CRON[32233]: (root) CMD ( cd / && run-parts
--report /etc/cron.hourly)
Jul 12 06:17:33 weepi systemd[1]: Starting Daily apt upgrade and clean
activities...
Jul 12 06:17:40 weepi systemd[1]: apt-daily-upgrade.service: Succeeded.
Thanks in Advance
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/d77920b8-55ec-487b-9d21-579edf931e74o%40googlegroups.com.
#
# Copyright (c) 2009 Tom Keffer <[email protected]>
#
# See the file LICENSE.txt for your full rights.
#
# $Id: alarm.py 2749 2014-11-29 18:15:24Z tkeffer $
#
""" Based on Alarm example but testing for bad data
********************************************************************************
To use this alarm, add the following somewhere in your configuration file
weewx.conf:
[Alarm]
expression = "outTemp < 40.0" # Not used by badData
time_wait = 3600 # Not used by badData
smtp_host = smtp.mymailserver.com
smtp_user = myusername
smtp_password = mypassword
mailto = [email protected], [email protected]
from = [email protected]
subject = "Alarm message from weewx!" # Not used by badData
badData_test = outTemp < 10.0 # only used by badData
badData_subject = Weather Station spike detected # only used by badData
********************************************************************************
To specify that this new service be loaded and run, it must be added to the
configuration option "report_services", located in sub-section [Engine][[Services]].
[Engine]
[[Services]]
...
report_services = weewx.engine.StdPrint, weewx.engine.StdReport, user.alarm2.MyBadData
********************************************************************************
If you wish to use both this example and the lowBattery.py example, simply merge
the two configuration options together under [Alarm] and add both services to
report_services.
********************************************************************************
"""
import time
import smtplib
from email.mime.text import MIMEText
import threading
import syslog
import weewx
from weewx.engine import StdService
from weeutil.weeutil import timestamp_to_string, option_as_list
# Inherit from the base class StdService:
class MyBadData(StdService):
"""Custom service that sounds an alarm if an arbitrary expression evaluates true"""
def __init__(self, engine, config_dict):
# Pass the initialization information on to my superclass:
super(MyBadData, self).__init__(engine, config_dict)
# This will hold the time when the last alarm message went out:
self.last_msg_ts = 0
try:
# Dig the needed options out of the configuration dictionary.
# If a critical option is missing, an exception will be raised and
# the alarm will not be set.
self.expression = config_dict['Alarm']['badData_test']
#self.time_wait = int(config_dict['Alarm'].get('time_wait', 3600))
self.smtp_host = config_dict['Alarm']['smtp_host']
self.smtp_user = config_dict['Alarm'].get('smtp_user')
self.smtp_password = config_dict['Alarm'].get('smtp_password')
self.SUBJECT = config_dict['Alarm'].get('badData_subject', "Suspected data spike - Alarm from weewx")
self.FROM = config_dict['Alarm'].get('from', '[email protected]')
self.TO = option_as_list(config_dict['Alarm']['mailto'])
syslog.syslog(syslog.LOG_INFO, "alarm: Alarm set for expression: '%s'" % self.expression)
# If we got this far, it's ok to start intercepting events:
self.bind(weewx.NEW_ARCHIVE_RECORD, self.newArchiveRecord) # NOTE 1
except KeyError as e:
syslog.syslog(syslog.LOG_INFO, "alarm: No alarm set. %s" % e)
def newArchiveRecord(self, event):
"""Gets called on a new archive record event."""
# Get the new archive record:
record = event.record
# Evaluate the expression in the context of the event archive record.
# Sound the alarm if it evaluates true:
if eval(self.expression, None, record): # NOTE 2
# Sound the alarm!
# Launch in a separate thread so it doesn't block the main LOOP thread:
t = threading.Thread(target = MyBadData.soundTheAlarm, args=(self, record))
t.start()
# Record when the message went out:
self.last_msg_ts = time.time()
def soundTheAlarm(self, rec):
"""This function is called when the given expression evaluates True."""
# Get the time and convert to a string:
t_str = timestamp_to_string(rec['dateTime'])
# Log it in the system log:
syslog.syslog(syslog.LOG_INFO, "alarm: Alarm expression \"%s\" evaluated True at %s" % (self.expression, t_str))
# Form the message text:
msg_text = "Alarm expression \"%s\" evaluated True at %s\nRecord:\n%s" % (self.expression, t_str, str(rec))
# Convert to MIME:
msg = MIMEText(msg_text)
# Fill in MIME headers:
msg['Subject'] = self.SUBJECT
msg['From'] = self.FROM
msg['To'] = ','.join(self.TO)
# Create an instance of class SMTP for the given SMTP host:
s = smtplib.SMTP(self.smtp_host)
try:
# Some servers (eg, gmail) require encrypted transport.
# Be prepared to catch an exception if the server
# doesn't support it.
s.ehlo()
s.starttls()
s.ehlo()
syslog.syslog(syslog.LOG_DEBUG, " **** using encrypted transport")
except smtplib.SMTPException:
syslog.syslog(syslog.LOG_DEBUG, " **** using unencrypted transport")
try:
# If a username has been given, assume that login is required for this host:
if self.smtp_user:
s.login(self.smtp_user, self.smtp_password)
syslog.syslog(syslog.LOG_DEBUG, " **** logged in with user name %s" % (self.smtp_user,))
# Send the email:
s.sendmail(msg['From'], self.TO, msg.as_string())
# Log out of the server:
s.quit()
except Exception as e:
syslog.syslog(syslog.LOG_ERR, "alarm: SMTP mailer refused message with error %s" % (e,))
raise
# Log sending the email:
syslog.syslog(syslog.LOG_INFO, " **** email sent to: %s" % self.TO)
if __name__ == '__main__':
"""This section is used for testing the code. """
import sys
import configobj
from optparse import OptionParser
usage_string ="""Usage:
alarm2.py config_path
Arguments:
config_path: Path to weewx.conf"""
parser = OptionParser(usage=usage_string)
(options, args) = parser.parse_args()
if len(args) < 1:
sys.stderr.write("Missing argument(s).\n")
sys.stderr.write(parser.parse_args(["--help"]))
exit()
config_path = args[0]
weewx.debug = 1
try :
config_dict = configobj.ConfigObj(config_path, file_error=True)
except IOError:
print ("Unable to open configuration file ", config_path)
exit()
if 'Alarm' not in config_dict:
print >>sys.stderr, "No [Alarm] section in the configuration file %s" % config_path
exit(1)
engine = None
alarm = MyBadData(engine, config_dict)
rec = {'extraTemp1': 1.0,
'outTemp' : 38.2,
'dateTime' : int(time.time())}
event = weewx.Event(weewx.NEW_ARCHIVE_RECORD, record=rec)
alarm.newArchiveRecord(event)
