Re: [weewx-user] MQTT lagging
I am running weewx and Belchertown on an older rpi 3 model B version 1.2. Cheetah takes around 20 seconds to generate the Belchertown reports. Time for new hardware? On Sunday, November 1, 2020 at 10:32:27 AM UTC-8 vince wrote: > On Sunday, November 1, 2020 at 9:52:51 AM UTC-8, T Reid wrote: >> >> > My guess is that your weewx instance is CPU bound and really is >> running behind. >> >> You may be on to something. I watched top for a while. CPU load on the >> python2 process runs under 10% when MQTT is posting loop records. But >> every five minutes, when the archive records and web pages are being >> generated, the CPU load of the python2 process jumps up over 100%. >> > > > I might have missed it, but what kind of box are you running on ? An old > model-B pi with a slow SD card, a 4GB pi4 with a SSD, etc. ??? > > My experience is that Belchertown is pretty heavyweight. Look at the time > it takes the archive interval skins to run. Disable all except the > standard or seasons skin and do a test to baseline your system. Then flip > back to Belchertown and see if things bog down again. > > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/02cc932b-838f-40c2-8a71-914bcef91fc6n%40googlegroups.com.
Re: [weewx-user] MQTT lagging
> My guess is that your weewx instance is CPU bound and really is running behind. You may be on to something. I watched top for a while. CPU load on the python2 process runs under 10% when MQTT is posting loop records. But every five minutes, when the archive records and web pages are being generated, the CPU load of the python2 process jumps up over 100%. In /var/log/messages, the MQTT records are posting on time, but fall behind by 10 seconds or more when the archive records and web pages are generated. The MQTT records then catch up again almost immediately after the archive activity dies down. I may have to dig around in the log files a bit more to find lags that it did not recover from. On Saturday, October 31, 2020 at 6:47:04 AM UTC-7 Greg Troxel wrote: > > T Reid writes: > > > As I feared, the problem is not fixed. The lag between the posting of > MQTT > > records by weewx and the current time has wandered all over the place > over > > the past two days. It has gone from dead on time to 1.5 hours behind, > and > > everything in between. Last night it was doing okay, this morning it was > > 80 minutes behind, and now it is 20 minutes behind. There is no rhyme or > > reason to it. The only constant is that weewx can't seem to post MQTT > > records on time on a consistent basis. > > My guess is that your weewx instance is CPU bound and really is running > behind. > > Is your archive interval 5 minutes, or something longer? If it's > shorter than 5 minutes, then don't do that! > > > I suggested restarting the broker because if the problem was in the > network going to the broker, then the reconnect would start fresh. But > you see old records newly appearing after the broker restart, which more > or less proves that weewx is newly posting them after the restart. > > So look with ps, top, and so on about CPU usage. Check if your html > skins are being generated on time, and transferred to your web server > etc., or if those also are slow. > > Try to simplify your setup in terms of skins and generating things, and > see if that helps. If it does, that supports the "not enough CPU time" > theory. > > Back off from loop to archive in mqtt, and see if that is reliably > timely. I'm not saying you should run that way, or have to, but it's an > interesting data point. > > I am running weewx on a Raspberry Pi 3 (NetBSD 9) with a VP2, and every > 5-minute archive interval: > > generate 2 skins > rsync them > post aggregate data to mqtt > > and I do not publish loop data. I just looked and the weewx python > process has used 17 hours of CPU time in 16 days. > > > - I looked at the traffic with mosquitto_sub, and the incoming > > weather data right now is from 45 minutes ago. > > It would be interesting to know if the inter-message arrival rate is the > same as the intended transmit rate, or if it catches up some between > archive intervals and then has a bigger lag over them. > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/63939b20-1f02-42bb-8aff-0b60048dda4en%40googlegroups.com.
Re: [weewx-user] MQTT lagging
As I feared, the problem is not fixed. The lag between the posting of MQTT records by weewx and the current time has wandered all over the place over the past two days. It has gone from dead on time to 1.5 hours behind, and everything in between. Last night it was doing okay, this morning it was 80 minutes behind, and now it is 20 minutes behind. There is no rhyme or reason to it. The only constant is that weewx can't seem to post MQTT records on time on a consistent basis. On Wednesday, October 28, 2020 at 9:25:32 PM UTC-7 T Reid wrote: > Greg, I am not ready to declare victory yet, but after restarting my > mosquitto broker, I let weewx keep running and overnight the MQTT loop > records gradually caught up! Not all at once, but over time you could see > the delay shrinking. All day today, the display has been pretty much spot > on. I would love to understand what caused this in the first place, and > why restarting the MQTT broker would fix it. Meanwhile, I have my fingers > crossed that the delay does not come back. > > On Tuesday, October 27, 2020 at 10:19:40 PM UTC-7 T Reid wrote: > >> Thank you for your very thorough response, Greg. In response to your >> questions and suggestions: >> >>- I am publishing aggregate data. Here is the MQTT section of my >>weeewx.conf: >> >> [[MQTT]] >> server_url = mqtts://user:pass...@mqtt.example.com:8883/ >> <http://user:passw...@mqtt.example.com:8883/> >> topic = weather/belmont >> binding = archive, loop >> aggregation = aggregate >> [[[tls]]] >> ca_certs = /etc/ssl/certs/ca-certificates.crt >> >>- I looked at the traffic with mosquitto_sub, and the incoming >>weather data right now is from 45 minutes ago. >>- I'm afraid that I don't know what running "netstat ... to look for >>full output queues" means. >>- What am I looking for in the logs? I see new "INFO weewx.restx: >>MQTT" entries every two seconds or so. Each entry says that it has >>published a record with a time stamp from about 45 minutes ago. >>- What I am seeing on my weather page using the Belchertown skin are >>charts built using current archive records (updated every five minutes), >>but "station observations" at the top of the page using MQTT delivered >>loop records that reflect weather data from 45 minutes ago. Earlier >> today, >>the lag was only 30 minutes. The lag grows over time. >>- I restarted the broker, and the MQTT weather data on the web page, >>on mosquitto_sub, and in the weewx logs is still running 45 minutes >>behind. No change. >>- I run my own broker using mosquitto running on a droplet on Digital >>Ocean. My copy of weewx is running on a raspberry pi that is hooked up >> to >>the datalogger on my Davis Vantage Pro2 console with a USB cable. >> >> To my eyes, it looks like weewx is running further and further behind in >> publishing loop records over MQTT, which makes no sense to me. >> >> On Sunday, October 25, 2020 at 7:43:08 AM UTC-7 Greg Troxel wrote: >> >>> >>> T Reid writes: >>> >>> > I am using the MQTT add-in to pipe real time loop entries from my >>> Davis >>> > Vantage Pro into the Belchertown skin, using the standard instructions >>> for >>> > that skin. It generally works great. But every couple of days the MQTT >>> > feed will fall behind, sometimes by as much as an hour or more. To fix >>> it, >>> > I have to restart weewx. Any idea what might be happening? I am >>> running >>> > weewx 4.1.1 with Belchertown 1.1. >>> >>> My suggestions: >>> >>> explain if you are publishing aggregate or individual data >>> >>> Use mosquitto_sub (from mosquitto) on the broker to watch the traffic. >>> >>> Run netstat on the weewx host to look for full output queues >>> >>> read the weewx logs >>> >>> explain what you are seeing more precisely than saying "fall behind". >>> >>> next time it fails, restart your broker instead of weewx and see what >>> happens. That will cause the weewx->broker connection to get shut >>> down and weewx/mqtt will open a new one. >>> >>> if you are using some cloud broker, run your own >>> >>> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/cfcbc844-e7b6-40aa-b25a-f62fee48789cn%40googlegroups.com.
Re: [weewx-user] MQTT lagging
Greg, I am not ready to declare victory yet, but after restarting my mosquitto broker, I let weewx keep running and overnight the MQTT loop records gradually caught up! Not all at once, but over time you could see the delay shrinking. All day today, the display has been pretty much spot on. I would love to understand what caused this in the first place, and why restarting the MQTT broker would fix it. Meanwhile, I have my fingers crossed that the delay does not come back. On Tuesday, October 27, 2020 at 10:19:40 PM UTC-7 T Reid wrote: > Thank you for your very thorough response, Greg. In response to your > questions and suggestions: > >- I am publishing aggregate data. Here is the MQTT section of my >weeewx.conf: > > [[MQTT]] > server_url = mqtts://user:pass...@mqtt.example.com:8883/ > <http://user:passw...@mqtt.example.com:8883/> > topic = weather/belmont > binding = archive, loop > aggregation = aggregate > [[[tls]]] > ca_certs = /etc/ssl/certs/ca-certificates.crt > >- I looked at the traffic with mosquitto_sub, and the incoming weather >data right now is from 45 minutes ago. >- I'm afraid that I don't know what running "netstat ... to look for >full output queues" means. >- What am I looking for in the logs? I see new "INFO weewx.restx: >MQTT" entries every two seconds or so. Each entry says that it has >published a record with a time stamp from about 45 minutes ago. >- What I am seeing on my weather page using the Belchertown skin are >charts built using current archive records (updated every five minutes), >but "station observations" at the top of the page using MQTT delivered >loop records that reflect weather data from 45 minutes ago. Earlier > today, >the lag was only 30 minutes. The lag grows over time. >- I restarted the broker, and the MQTT weather data on the web page, >on mosquitto_sub, and in the weewx logs is still running 45 minutes >behind. No change. >- I run my own broker using mosquitto running on a droplet on Digital >Ocean. My copy of weewx is running on a raspberry pi that is hooked up to >the datalogger on my Davis Vantage Pro2 console with a USB cable. > > To my eyes, it looks like weewx is running further and further behind in > publishing loop records over MQTT, which makes no sense to me. > > On Sunday, October 25, 2020 at 7:43:08 AM UTC-7 Greg Troxel wrote: > >> >> T Reid writes: >> >> > I am using the MQTT add-in to pipe real time loop entries from my Davis >> > Vantage Pro into the Belchertown skin, using the standard instructions >> for >> > that skin. It generally works great. But every couple of days the MQTT >> > feed will fall behind, sometimes by as much as an hour or more. To fix >> it, >> > I have to restart weewx. Any idea what might be happening? I am running >> > weewx 4.1.1 with Belchertown 1.1. >> >> My suggestions: >> >> explain if you are publishing aggregate or individual data >> >> Use mosquitto_sub (from mosquitto) on the broker to watch the traffic. >> >> Run netstat on the weewx host to look for full output queues >> >> read the weewx logs >> >> explain what you are seeing more precisely than saying "fall behind". >> >> next time it fails, restart your broker instead of weewx and see what >> happens. That will cause the weewx->broker connection to get shut >> down and weewx/mqtt will open a new one. >> >> if you are using some cloud broker, run your own >> >> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/340a1353-0726-455a-81f3-77e3b0688f69n%40googlegroups.com.
Re: [weewx-user] MQTT lagging
Thank you for your very thorough response, Greg. In response to your questions and suggestions: - I am publishing aggregate data. Here is the MQTT section of my weeewx.conf: [[MQTT]] server_url = mqtts://user:passw...@mqtt.example.com:8883/ topic = weather/belmont binding = archive, loop aggregation = aggregate [[[tls]]] ca_certs = /etc/ssl/certs/ca-certificates.crt - I looked at the traffic with mosquitto_sub, and the incoming weather data right now is from 45 minutes ago. - I'm afraid that I don't know what running "netstat ... to look for full output queues" means. - What am I looking for in the logs? I see new "INFO weewx.restx: MQTT" entries every two seconds or so. Each entry says that it has published a record with a time stamp from about 45 minutes ago. - What I am seeing on my weather page using the Belchertown skin are charts built using current archive records (updated every five minutes), but "station observations" at the top of the page using MQTT delivered loop records that reflect weather data from 45 minutes ago. Earlier today, the lag was only 30 minutes. The lag grows over time. - I restarted the broker, and the MQTT weather data on the web page, on mosquitto_sub, and in the weewx logs is still running 45 minutes behind. No change. - I run my own broker using mosquitto running on a droplet on Digital Ocean. My copy of weewx is running on a raspberry pi that is hooked up to the datalogger on my Davis Vantage Pro2 console with a USB cable. To my eyes, it looks like weewx is running further and further behind in publishing loop records over MQTT, which makes no sense to me. On Sunday, October 25, 2020 at 7:43:08 AM UTC-7 Greg Troxel wrote: > > T Reid writes: > > > I am using the MQTT add-in to pipe real time loop entries from my Davis > > Vantage Pro into the Belchertown skin, using the standard instructions > for > > that skin. It generally works great. But every couple of days the MQTT > > feed will fall behind, sometimes by as much as an hour or more. To fix > it, > > I have to restart weewx. Any idea what might be happening? I am running > > weewx 4.1.1 with Belchertown 1.1. > > My suggestions: > > explain if you are publishing aggregate or individual data > > Use mosquitto_sub (from mosquitto) on the broker to watch the traffic. > > Run netstat on the weewx host to look for full output queues > > read the weewx logs > > explain what you are seeing more precisely than saying "fall behind". > > next time it fails, restart your broker instead of weewx and see what > happens. That will cause the weewx->broker connection to get shut > down and weewx/mqtt will open a new one. > > if you are using some cloud broker, run your own > > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/75afa842-2d50-4e1f-96ff-21799bc73b5cn%40googlegroups.com.
[weewx-user] MQTT lagging
I am using the MQTT add-in to pipe real time loop entries from my Davis Vantage Pro into the Belchertown skin, using the standard instructions for that skin. It generally works great. But every couple of days the MQTT feed will fall behind, sometimes by as much as an hour or more. To fix it, I have to restart weewx. Any idea what might be happening? I am running weewx 4.1.1 with Belchertown 1.1. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/07ec5fb8-eec2-4267-9c1a-db468ea29895n%40googlegroups.com.
Re: [weewx-user] New forecasting service using NDFD
No luck. The Databases section has a single entry for all forecast data. So nothing to add there. On Saturday, November 9, 2019 at 3:58:18 PM UTC-8, T Reid wrote: > > I did add it to the Forecast section in weewx.conf, and to the Services > list, but haven't tried the Databases section. I will try that and report > back. > > On Saturday, November 9, 2019 at 3:07:10 PM UTC-8, p q wrote: >> >> >> Configuration >> >>To enable forecasting, add a [Forecast] section to weewx.conf, add a >>section to [Databases] to indicate where forecast data should be >> stored, >>then append user.forecast.XXXForecast to the service list for each >>forecasting method that should be enabled. >> >> Have you tried this? >> >> >>> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/2ebf8d2f-1d59-4183-a120-004a26d6147f%40googlegroups.com.
Re: [weewx-user] New forecasting service using NDFD
I did add it to the Forecast section in weewx.conf, and to the Services list, but haven't tried the Databases section. I will try that and report back. On Saturday, November 9, 2019 at 3:07:10 PM UTC-8, p q wrote: > > > Configuration > >To enable forecasting, add a [Forecast] section to weewx.conf, add a >section to [Databases] to indicate where forecast data should be stored, >then append user.forecast.XXXForecast to the service list for each >forecasting method that should be enabled. > > Have you tried this? > > >> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/19707692-ff95-4725-b445-e0f4c0c5c4ce%40googlegroups.com.
[weewx-user] New forecasting service using NDFD
I need help implementing a new forecasting service on weewx using the
NWS National Digital Forecast Database
(https://graphical.weather.gov/xml/). This should allow for much more
precise local forecasting than the existing NWS forecast module in weewx.
I wrote the attached class and pasted it into forecast.py. But cannot
figure out how to get forecast.py to actually run forecasts using this
class. Any suggestions? Thanks.
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/c924906c-9acd-4e0a-a49c-767a9c514e1b%40googlegroups.com.
import xml.etree.ElementTree as ET
# -
# National Digital Forecast Database
#
# Forecasts from the National Digital Forecast Database. Returns xml data.
#
# For documentation, see:
# https://graphical.weather.gov/xml/
#
# 12-hour:
# pop12 - likelihood of measurable precipitation (1/100 inch)
# maxt - temperature in degrees F
# mint - temperature in degrees F
#
# 6-hour:
# qpf - quantitative precipitation forecast; amount or range in inches
# wgust - only displayed if gusts exceed windspd by 10 mph
#
# 3-hour:
# temp - degrees F
# dew - degrees F
# rh - relative humidity %
# wdir - 8 compass points
# wspd - miles per hour
# sky - sky coverage
# icons - url for weather condition icon
# -
NDFD_KEY = 'NDFD'
NDFD_DEFAULT_URL = 'https://graphical.weather.gov/xml/SOAP_server/ndfdXMLclient.php?whichClient='
class NDFDForecast(Forecast):
def __init__(self, engine, config_dict):
super(NDFDForecast, self).__init__(engine, config_dict, NDFD_KEY, interval=10800)
d = config_dict.get('Forecast', {}).get(NDFD_KEY, {})
self.url = d.get('url', NDFD_DEFAULT_URL)
self.max_tries = int(d.get('max_tries', 3))
self.location = d.get('location', None)
self.units = d.get('units', 'e')
errmsg = []
if self.location is None:
errmsg.append('location is not specified')
if errmsg:
for e in errmsg:
logerr("%s: %s" % (NDFD_KEY, e))
logerr('%s: forecast will not be run' % NDFD_KEY)
return
loginf('%s: interval=%s max_age=%s location=%s' % (NDFD_KEY, self.interval, self.max_age, self.location))
self._bind()
def get_forecast(self, dummy_event):
"""Return a parsed forecast."""
text = self.download(location=self.location, url=self.url, units=self.units, max_tries=self.max_tries)
if text is None:
logerr('%s: no forecast data for %s from %s' % (NDFD_KEY, self.location, self.url))
return None
if self.save_raw:
self.save_raw_forecast(text, basename='ndfd-raw')
records, msgs = self.parse(text, location=self.location)
if self.save_failed and len(msgs) > 0:
self.save_failed_forecast(text, basename='ndfd-fail', msgs=msgs)
loginf('%s: got %d forecast records' % (NDFD_KEY, len(records)))
return records
@staticmethod
def download(location, url=NDFD_DEFAULT_URL, units='e', max_tries=3):
"""Download a forecast from the National Digital Forecast Database
location - location for which the forecast is required, either as
a five-digit zipcode or in the format lat,lon.
url - URL to the forecast service. if anything other than the
default is specified, that entire URL is used. if the default
is specified, it is used as the base and other items are added
to it.
units - units to be used in the forecast, either 'e' for US or 'm' for metric.
max_tries - how many times to try before giving up
"""
if url == NDFD_DEFAULT_URL:
begDate = datetime.datetime.utcnow()
begStr = begDate.strftime("%Y-%m-%dT00%%3A00%%3A00")
endDate = begDate.replace(year=begDate.year+1)
endStr = endDate.strftime("%Y-%m-%dT00%%3A00%%3A00")
optString = '=time-series=' + begStr + '=' endStr + '=' + units +'=maxt=mint=temp=qpf=pop12=dew=wspd=wdir=sky=icons=rh=wgust=Submit'
locPoint = None
locZip = re.match(r'^\d{5}$', location.strip())
if locZip is not None:
u = url + 'LatLonListZipCode=' + str(locZip) + '=Submit'
request = urllib2.Request(u)
loginf("%s: downloading latitude and longitude for zipcode %s" % (NDFD_KEY, locZip))
for count in range(max_tries):
try:
response = urllib2.urlopen(request)
tree = ET.parse(response)
