try 10 minutes even as the interval. Be honest - since weewx is storing
AVERAGES anyway, there is probably little difference between a 5 minute
interval and a 10 minute interval except that your servers will have a
fraction of the workload and the stations will suffer fewer dropouts and
losses of data which to me would be a win-win situation!! You may as
well try it and I think you will be very pleasantly sur[rised at the
result of using either 10 or even 15 minute archive intervals!!
On Thursday, 17 May 2018 15:18:06 UTC+3, Hrmeteohub Pljusak wrote:
Setting 5-min archiving interval made great difference on WMR88 - I
had only one drop-out in last four hours. Already I have made it a
default in the firmware, for all stations. I'll test this on WMR88
for several days and then see how it goes.
I the mean time I will focus probably on Davis stations. It is hard
to say if archiving interval of 5 minutes made a difference so far,
since the errors did happen only 1-2 times each day.
I would really like to work with just one type of station, but our
volunteers have all kind of stations, from very old to brand new,
and expect us to support them all.
You are right on the money with a need for new API.
Thank you for the advice.
On Thu, May 17, 2018 at 1:16 PM, Andrew Milner
<[email protected] <javascript:>> wrote:
I seriously recommend that you focus on one particular model of
weather station first - eg FineOffset (use software records
rather than hardware - see weewx hardware manual for more on the
options) and maybe Davis - use hardware records and once again
see weewx hardware manual for options). In both cases I would
suggest use a 5 minute archive interval. For WU do not use
rapid fire until you are happy things are ok.
For the rest of your setup it would probably be simplest for you
to have arranged for your server to have accepted input via a
restless API and built a restless service to upload directly to
your site - like weewx does to WU and other hosts - but that is
far more suitable for the development forum rather than this
forum as it is outside the scope of current weewx.
On Thursday, 17 May 2018 10:49:39 UTC+3, Hrmeteohub Pljusak wrote:
Dear Andrew,
First, thank you and thank Thomas and everybody else for
producing weewx. I apologize once more for coming harsh in
my first post.
It appears that you are quite right that I am forcing weewx
into something it was not made to do. Perhaps maybe not all
of it, but I suspect it is the way I am doing thins more to
be blame in some instances.
Also, we are doing that with several weather stations, at
locations apart, with so many different co-factors. Seems
like a good recipe for headache.
As luck will have it, weewx is now my tool of choice. You
are very much correct when you say that I have many problems
on a number of fronts simultaneously. Unfortunately. It
looks a bit chaotic when I have presented it all at once.
Please accept my exclamations as expression of anger onto
my/our choices and problems.
Thank you for giving me info on WU. I did not know that.
If nothing goes through the database, than the problem must
be on "my side", CSV reporting and sending to pljusak.com
<http://pljusak.com> server.
Some of the problems were expected, like intermittent
sending of data from WMR stations to weewx. I was combing
through data and testing parts of the process for Davis and
FineOffset stations for weeks now, and simply can not put my
finger on the problem. No such event was seen in weeks for
WS2300, WMR100, WMR200 stations in weeks of testing.
Please do not read the rest of my post if you do not have
too much time on your hands. As Thomas wrote, I tend to rant.
Let me give some perspective to HRmeteohub project.
The goal of the project is to send data from weather
stations to crowd sourcing web page pljusak.com
<http://pljusak.com>, as well as to WU, AWEKAS, and so on.
The project was started by amateur meteorologists and
enthusiasts in Croatia, and it runs for about ten years now.
At first phase, all users have used some kind of windows
based software. In that (first) phase, our job was to hand
out the templates for PC software that users were using and
users would use FTP to send the graphs and data to server.
It worked, but what a mess (and security headaches).
In second phase we started using Open2300 in small devices
such as NSLU and wifi routers. About six years ago this has
transformed in customized OpenWRT firmware for 5-6 different
weather station types. Hence my references to wview. At that
time our team stabilized on one sysadmin, one programmer,
and me in charge of building the firmware, testing,
documening as well as supporting users together with sysadmin.
However, hardware producers have been changing their routers
and weather stations, combined with changes in OpenWRT -
making our task almost impossible.
At this time about 40 users use HRmeteohub firmware, and
some would like to, but supported wifi routers are not on
the market anymore. Pljusak.com is NGO run by volunteers,
all the software, firmware and services are provided for
free and all income from advertising is used to maintain the
server and keep NGO functioning. The web page is now about
5-6 years old and ready for overhaul. Basically the server
produces something like this:
And each station has got it's page:
This is where weewx comes on stage. We have been testing it
on RaspberryPi with good results, but it is rather too
complicated for our average meteo-enthusiast to install and
setup R-pi with weewx and our scripts for sending the data
to pljusak.com <http://pljusak.com>. Plus, R-pi (version 2B
at that time) wirth power source, wifi card, box was a bit
more expensive than wifi router. Our users could can buy
second-hand PC with old windows on it for the same price. It
(r-pi) also did not work as reliably as wifi routers. At
that time, putting Python and weewx in small, cheap wifi
router was simply not possible, but in time, OpenWRT merged
with LEDE, routers got faster with more Flash and RAM, and
suddenly it become a viable option.
After about six months of tries and errors, the firmware
proved to be stable. I can not recall single crash of weewx
or firmware in last two months.
Unfortunately, Python is big beast for wifi routers with
16MB flash, and 64Mb RAM, so we could not install the
graphing, but settle down for archiving the data in devices
RAM (default option) or USB stick (works well, but still
under testing). Every two hours one script tests internet
connection and how much RAM database occupies and if it
reaches critical level the user is notified and the database
is cleared. Yes, I know that is not nice thing, but the user
has got it's data backed up on server. Generally this
happens depending on archiving interval, how much RAM device
has got and if there are other services needing more RAM.
We use weewx to read the measurements from weather station,
and archive it in database. Sending data to WU, CWOP, AWEKAS
and others is done using weewx extensions.We have not come
up with extension for our pljusak.com <http://pljusak.com>
so far as none of us is proficient in Python.
What we do with archived data in weewx.sdb is rather simple.
CSV extension dumps last measurement to text. Every 5
minutes we send it to server, parse it, put it in database
and make a graph. We have tried sending one last measurement
from database or sending 2-3 of them, but above mentioned
problems persisted. The (raw, unparsed) data sent from
router is visible on the web page, so we can see if what
arrived at server before it hits database and graphs). We
can also see csv.txt on router through it's Luci web page
and confirm that the data dumped from weewx.sdb and on
server are a match.
OK. I guess I have taken you enough of your time.
I'll be back when I have some solutions for above mentioned
problems.
On Thursday, May 17, 2018 at 7:33:24 AM UTC+2, Andrew Milner
wrote:
Nothing goes through the data and changes the data
contained in the archive.
The plotting engine when drawing graphplots (creating
the .png files) does, I believe, try and smooth (and
scale) the data for the plots.
Have you compared your graphs with the standard weewx
report graphs/plots??
On Thursday, 17 May 2018 08:21:39 UTC+3, Hrmeteohub
Pljusak wrote:
Thank you.
I will try extending the archive interval for
stations other that Davis Vantage series. However, I
believe that weewx uses archive interval from VP
station, as it is set to "record_generation = hardware".
Is there some function that goes through archive
data in weewx and does "the smoothing"? I ask that
because I have compared local data with data that
are on WU and our remote server, and the values with
errors did not match on all three locations. At the
end of the day, I could not see errors on WU while
database dumps to CSV file contained 1-2 error per
day. The csv file was done with extenstion set up:
[CSV]
filename = /tmp/data.txt
binding = archive
At first I thought that these errors are perhaps
only in loop data, or errors on USB communication,
but after several weeks I could not catch them in
logs with matching timestamps.
On Thursday, May 17, 2018 at 7:02:27 AM UTC+2,
Andrew Milner wrote:
…. the choices seem to include:
1. Use weewx database and make your graphing
package do the smoothing (in the way I presume
wview does)
2. Use weewx database and weewx graphs - rather
than making your own graphs
3. Increase archive interval to ensure every
archive record contains at least one reading for
each sensor - I think some sensors only output
every 15 minutes (=900 seconds)
4. Explain what you feel to be wrong
or
5. Return to wview
As a further point you seem to be jumping around
many different station types and drivers.
Stations of different types emit different data
and weewx will cope with each. However the
config file needs to be set up for each station
type with regard to hardware/software
preferences, archive interval, hardware/software
record creation etc. HOWEVER - it is overriding
within weewx that an archive record contains the
average value for a reading during an archive
period. If there is no reading there is no
value to store. In the hgardware manual there
is information regarding each station type - and
for example the config options for a fine offset
station would involve periodic polling at about
60 second intervals (no point in more frequently
as station does not emit any faster) and
software record creation rather than hardware ….
and so on. The options to use for fine offset
are not the same options you would use with a
davis station, so it is not possible to just
change drivers when changing station types.
My FineOffset has run for several years without
issues. Using QC values in your config helps
reduce spikes
On Thursday, 17 May 2018 07:39:05 UTC+3, Thomas
Keffer wrote:
Your rant would be a lot more useful if it
offered some constructive examples of where
you think weewx is failing. Instead, there's
just vague generalities like "erratic
measurements from Davis VantagePro2..."
I flatly do not believe this. The VP2 driver
is nearly 10 years old, has been extremely
well vetted, and used by thousands without
any problems. Most likely the problem is in
your configuration.
The WMR100 driver is not as mature, but is
still used by many people. I would not be
surprised if it had some problems, but your
inchoate rant isn't going to put us any
closer to solving them.
I am sorry you are disappointed. If you had
paid anything for Weewx, I would be happy to
refund your money. But you didn't.
Stop whining and offer a pull request if you
think there's a problem.
-tk
On Wed, May 16, 2018, 2:38 PM Hrmeteohub
Pljusak <[email protected]> wrote:
I am not positive that my questions have
not been answered. If so, please point
me to location where I can read how to
mitigate the problems. Thank you!
Recently I have switched from wview to
weewx 3.7.1. I must say I got rather
disappointed with problems I have
encountered.
First, Oregon Scientific WMR88/100/200
stations started to produce erratic
graphs on WU and other web pages, with
number of incomplete measurements. The
weewx documentation gently describes
that as "station problem". That is not
acceptable, or nice, but just a dirty
walk-around-the-problem.
Yes, I am aware of the way that stations
"blasts data" from sensors to station,
but wview did away with it quite gently.
Also, other (read windows-based)
software works nicely with OS stations.
Weewx simply gives us nothing (null),
and lets the user deal with it. Well
guess what!
This should be corrected.
The graph below is from WU.
Just to ilustrate, this is an example of
the data from weewx.sdb:
dateTime,ET,altimeter,appTemp,barometer,cloudbase,dewpoint,hourRain,humidex,inDewpoint,inHumidity,inTemp,inTempBatteryStatus,interval,maxSolarRad,outHumidity,outTemp,outTempBatteryStatus,pressure,rain,rain24,rainBatteryStatus,rainRate,rainTotal,usUnits,windBatteryStatus,windDir,windGust,windGustDir,windSpeed,windrun
1526378490.0,None,1008.47595669,18.7382035985,1008.41843898,1330.2378329,9.93516495581,0.0,20.4483894915,51.3461294561,46.0,23.0,0.0,1,None,55.0,19.2,0.0,988.0,0.0,6.35,0.0,0.0,2.41,17,0.0,337.5,0.700003479692,None,0.700003479692,0.0
1526378610.0,None,None,18.174962293,None,1364.37850406,9.66136186735,20.3234925844,51.3461294561,46.0,23.0,0.0,1,None,54.0,19.2,0.0,None,17,0.0,225.0,1.40000695938,None,1.40000695938,0.0420002087815
1526378670.0,None,1008.47595669,18.384962815,1008.41843898,1364.37850406,9.66136186735,20.3234925844,51.3461294561,46.0,23.0,0.0,1,None,54.0,19.2,0.0,988.0,17,0.0,315.0,1.10000546809,None,1.10000546809,0.126000626344
1526378730.0,None,1008.47595669,None,None,None,None,None,None,1,None,988.0,17,0.0,225.0,0.900004473889,None,0.900004473889,0.19200095443
1526378850.0,None,1008.47595669,None,None,None,None,0.0,None,51.3461294561,46.0,23.0,0.0,1,None,988.0,0.0,6.35,0.0,0.0,2.41,17,0.0,315.0,1.10000546809,None,1.10000546809,0.246001222863
1526378910.0,None,1008.47595669,18.6649635109,1008.41843898,1364.37850406,9.66136186735,20.3234925844,None,1,None,54.0,19.2,0.0,988.0,17,0.0,225.0,0.700003479692,None,0.700003479692,0.312001550948
1526379030.0,None,1008.47595669,None,None,None,None,0.0,None,51.3461294561,46.0,23.0,0.0,1,None,988.0,0.0,6.35,0.0,0.0,2.41,17,0.35400175973
1526379150.0,None,1008.47595669,None,None,None,None,0.0,None,51.3461294561,46.0,23.0,0.0,1,None,988.0,0.0,6.35,0.0,0.0,2.41,17,0.35400175973
1526379210.0,None,1008.47595669,None,None,None,None,None,51.3461294561,46.0,23.0,0.0,1,None,988.0,17,0.0,250.773234946,0.900004473889,None,0.750003728241,0.35400175973
1526379330.0,None,None,18.2996293732,None,1365.28742772,9.75407243522,0.0,20.4655511807,51.3461294561,46.0,23.0,0.0,1,None,54.0,19.3,0.0,None,0.0,6.35,0.0,0.0,2.41,17,0.0,292.5,1.40000695938,None,1.40000695938,0.399001983424
1526379390.0,None,1008.47595669,18.0196286772,1008.41138549,1365.28742772,9.75407243522,0.0,20.4655511807,51.3461294561,46.0,23.0,0.0,1,None,54.0,19.3,0.0,988.0,0.0,6.35,0.0,0.0,2.41,17,0.0,292.5,1.80000894778,None,1.80000894778,0.483002400987
1526379450.0,None,1008.47595669,None,None,None,None,None,None,1,None,988.0,17,0.0,292.5,2.50001242747,None,2.50001242747,0.591002937854
Each "None" means that weewx did not get
the data from station. Well, wview and
every other software works around this,
and so should weewx!
Furthermore, weewx tends to have simply
wrong - erratic measurements in archive
from Davis VantagePro, Pro2 and Vue
stations. Same problem can be found on
much cheaper and lower quality
FineOffset WH1080/2080 stations. Note
that this is data that is not seen on WU
- I don't know why - Perhaps WU or weewx
does some clean-up of the data later.
This graph is drawn from data as it is
copied from table "archive" in 5 minute
intervals.
weewx 3.7.1. is installed on
OpenWRT/LEDE 17.0.1 The data is read in
5 minute intervals from weewx database
(archive, not loop), and written to
database on separate server and (using
weewx) sent to WU. Data is read from WMR
and WH stations every 90 seconds, from
Davis stations every 60 seconds (archive
interval).
The database example is made using CVS
export weewx extension.
The weewx.conf is as follows:
# WEEWX CONFIGURATION FILE
# Copyright (c) 2009-2015 Tom Keffer
<tkeffer>
# See the file LICENSE.txt for your rights.
#
debug = 0
WEEWX_ROOT = /home/weewx
socket_timeout = 20
version = 3.7.1
[Station]
location = STATIONNAME
latitude = 432423412
longitude = 3245143214
altitude = 175, meter # Choose
'foot' or 'meter' for unit
station_type = WMR100
rain_year_start = 1
# Start of week (0=Monday, 6=Sunday)
week_start = 0
station_url = http://www.pljusak.com
##############################################################################
[WMR100]
# This section is for the Oregon
Scientific WMR100
# The driver to use
driver = weewx.drivers.wmr100
# The station model, e.g., WMR100,
WMR100N, WMRS200
model = WMR100
[StdRESTful]
# To guard against parsing errors,
put your password(s) in quotes:
[[StationRegistry]]
# To register this weather
station with weewx, set this to true
register_this_station = false
[[AWEKAS]]
enable = false
username = replace_me
password = replace_me
[[CWOP]]
# and specify the station ID
(e.g., CW1234).
enable = false
station = replace_me
# If this is an APRS (radio
amateur) station, uncomment
# the following and replace with a
passcode (e.g., 12345).
#passcode = replace_me (APRS
stations only)
[[PWSweather]]
enable = false
station = replace_me
password = replace_me
[[WOW]]
enable = false
station = replace_me
password = replace_me
[[Wunderground]]
enable = true
station = ISTATIONNAME1
password = PASSOWRD
rapidfire = False
[StdReport]
SKIN_ROOT = skins
HTML_ROOT = /tmp
# This module is not used
[[simple]]
HTML_ROOT = /tmp/simple
skin = simple
[StdConvert]
# DO NOT MODIFY THIS VALUE UNLESS
YOU KNOW WHAT YOU ARE DOING!
target_unit = METRICWX # Options
are 'US', 'METRICWX', or 'METRIC'
[StdCalibrate]
[[Corrections]]
# For each type, an arbitrary
calibration expression can be given.
# It should be in the units
defined in the StdConvert section.
# Example:
foo = foo + 0.2
[StdQC]
[[MinMax]]
barometer = 900, 1400, mbar
outTemp = -30, 50, degree_C
inTemp = -30, 50, degree_C
outHumidity = 0, 100
inHumidity = 0, 100
windSpeed = 0, 300, km_per_hour
pressure = 900, 1100, mbar
[StdWXCalculate]
[[Calculations]]
# Derived quantities are
calculated by this service. Possible
values are:
# hardware - use the
value provided by hardware
# software - use the
value calculated by weewx
# prefer_hardware - use value
provide by hardware if available,
#
otherwise use value calculated by weewx
pressure = prefer_hardware
barometer = prefer_hardware
altimeter = prefer_hardware
windchill = hardware
heatindex = hardware
dewpoint = prefer_hardware
inDewpoint = prefer_hardware
rainRate = hardware
[StdTimeSynch]
clock_check = 14400
max_drift = 5
[StdArchive]
# If the station hardware supports
data logging then the archive interval
# will be downloaded from the
station. Otherwise, specify it (in seconds).
archive_interval = 90
# If possible, new archive records
are downloaded from the station
# hardware. If the hardware does
not support this, then new archive
# records will be generated in
software.
# Set the following to "software"
to force software record generation.
record_generation = software
# Whether to include LOOP data in
hi/low statistics
loop_hilo = True
# The data binding used to save
archive records
data_binding = wx_binding
[DataBindings]
[[wx_binding]]
database = archive_sqlite
table_name = archive
manager =
weewx.wxmanager.WXDaySummaryManager
schema = schemas.wview.schema
[Databases]
[[archive_sqlite]]
database_type = SQLite
database_name = weewx.sdb
[DatabaseTypes]
[[SQLite]]
driver = weedb.sqlite
SQLITE_ROOT = /tmp
[Engine]
[[Services]]
prep_services =
weewx.engine.StdTimeSynch
data_services = ,
process_services =
weewx.engine.StdConvert,
weewx.engine.StdCalibrate,
weewx.engine.StdQC,
weewx.wxservices.StdWXCalculate,
user.csv.CSV
archive_services =
weewx.engine.StdArchive
restful_services =
weewx.restx.StdStationRegistry,
weewx.restx.StdWunderground,
weewx.restx.StdPWSweather,
weewx.restx.StdCWOP, weewx.restx.StdWOW,
weewx.restx.StdAWEKAS
report_services =
weewx.engine.StdPrint
[CSV]
filename = /tmp/data.txt
binding = archive
--
You received this message because you
are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop
receiving emails from it, send an email
to [email protected].
For more options, visit
https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to a topic
in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/weewx-user/HXHjLqxOJxE/unsubscribe
<https://groups.google.com/d/topic/weewx-user/HXHjLqxOJxE/unsubscribe>.
To unsubscribe from this group and all its topics, send an email
to [email protected] <javascript:>.
For more options, visit https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to a topic in the
Google Groups "weewx-user" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/weewx-user/HXHjLqxOJxE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
[email protected]
<mailto:[email protected]>.
For more options, visit https://groups.google.com/d/optout.
