I seriously recommend that you focus on one particular model of weather 
station first - eg FineOffset (use software records rather than hardware - 
see weewx hardware manual for more on the options) and maybe Davis - use 
hardware records and once again see weewx hardware manual for options).  In 
both cases I would suggest use a 5 minute archive interval.  For WU do not 
use rapid fire until you are happy things are ok.

For the rest of your setup it would probably be simplest for you to have 
arranged for your server to have accepted input via a restless API and 
built a restless service to upload directly to your site - like weewx does 
to WU and other hosts - but that is far more suitable for the development 
forum rather than this forum as it is outside the scope of current weewx.



On Thursday, 17 May 2018 10:49:39 UTC+3, Hrmeteohub Pljusak wrote:
>
> Dear Andrew,
>
> First, thank you and thank Thomas and everybody else for producing weewx. 
> I apologize once more for coming harsh in my first post.
>
> It appears that you are quite right that I am forcing weewx into something 
> it was not made to do. Perhaps maybe not all of it, but I suspect it is the 
> way I am doing thins more to be blame in some instances.
> Also, we are doing that with several weather stations, at locations apart, 
> with so many different co-factors. Seems like a good recipe for headache.
> As luck will have it, weewx is now my tool of choice. You are very much 
> correct when you say that I have many problems on a number of fronts 
> simultaneously. Unfortunately. It looks a bit chaotic when I have presented 
> it all at once. Please accept my exclamations as expression of anger onto 
> my/our choices and problems. 
>
> Thank you for giving me info on WU. I did not know that.
> If nothing goes through the database, than the problem must be on "my 
> side", CSV reporting and sending to pljusak.com server.
>
> Some of the problems were expected, like intermittent sending of data from 
> WMR stations to weewx. I was combing through data and testing parts of the 
> process for Davis and FineOffset stations for weeks now, and simply can not 
> put my finger on the problem. No such event was seen in weeks for WS2300, 
> WMR100, WMR200 stations in weeks of testing.
>
> Please do not read the rest of my post if you do not have too much time on 
> your hands. As Thomas wrote, I tend to rant.
>
>
> Let me give some perspective to HRmeteohub project. 
>
> The goal of the project is to send data from weather stations to crowd 
> sourcing web page pljusak.com, as well as to WU, AWEKAS, and so on. The 
> project was started by amateur meteorologists and enthusiasts in Croatia, 
> and it runs for about ten years now. At first phase, all users have used 
> some kind of windows based software. In that (first) phase, our job was to 
> hand out the templates for PC software that users were using and users 
> would use FTP to send the graphs and data to server. It worked, but what a 
> mess (and security headaches). 
> In second phase we started using Open2300 in small devices such as NSLU 
> and wifi routers. About six years ago this has transformed in customized 
> OpenWRT firmware for 5-6 different weather station types. Hence my 
> references to wview. At that time our team stabilized on one sysadmin, one 
> programmer, and me in charge of building the firmware, testing, documening 
> as well as supporting users together with sysadmin.
> However, hardware producers have been changing their routers and weather 
> stations, combined with changes in OpenWRT - making our task almost 
> impossible.
>
> At this time about 40 users use HRmeteohub firmware, and some would like 
> to, but supported wifi routers are not on the market anymore. Pljusak.com 
> is NGO run by volunteers, all the software, firmware and services are 
> provided for free and all income from advertising is used to maintain the 
> server and keep NGO functioning. The web page is now about 5-6 years old 
> and ready for overhaul. Basically the server produces something like this:
>
> And each station has got it's page:
>
> This is where weewx comes on stage. We have been testing it on RaspberryPi 
> with good results, but it is rather too complicated for our average 
> meteo-enthusiast to install and setup R-pi with weewx and our scripts for 
> sending the data to pljusak.com. Plus, R-pi (version 2B at that time) 
> wirth power source, wifi card, box was a bit more expensive than wifi 
> router. Our users could can buy second-hand PC with old windows on it for 
> the same price. It (r-pi) also did not work as reliably as wifi routers. At 
> that time, putting Python and weewx in small, cheap wifi router was simply 
> not possible, but in time, OpenWRT merged with LEDE, routers got faster 
> with more Flash and RAM, and suddenly it become a viable option.
> After about six months of tries and errors, the firmware proved to be 
> stable. I can not recall single crash of weewx or firmware in last two 
> months. 
> Unfortunately, Python is big beast for wifi routers with 16MB flash, and 
> 64Mb RAM, so we could not install the graphing, but settle down for 
> archiving the data in devices RAM (default option) or USB stick (works 
> well, but still under testing). Every two hours one script tests internet 
> connection and how much RAM database occupies and if it reaches critical 
> level the user is notified and the database is cleared. Yes, I know that is 
> not nice thing, but the user has got it's data backed up on server. 
> Generally this happens depending on archiving interval, how much RAM device 
> has got and if there are other services needing more RAM.
>
> We use weewx to read the measurements from weather station, and archive it 
> in database. Sending data to WU, CWOP, AWEKAS and others is done using 
> weewx extensions.We have not come up with extension for our pljusak.com 
> so far as none of us is proficient in Python.
> What we do with archived data in weewx.sdb is rather simple. CSV extension 
> dumps last measurement to text. Every 5 minutes we send it to server, parse 
> it, put it in database and make a graph. We have tried sending one last 
> measurement from database or sending 2-3 of them, but above mentioned 
> problems persisted. The (raw, unparsed) data sent from router is visible on 
> the web page, so we can see if what arrived at server before it hits 
> database and graphs). We can also see csv.txt on router through it's Luci 
> web page and confirm that the data dumped from weewx.sdb and on server are 
> a match.
>
> OK. I guess I have taken you enough of your time.
>
> I'll be back when I have some solutions for above mentioned problems.
>
> On Thursday, May 17, 2018 at 7:33:24 AM UTC+2, Andrew Milner wrote:
>>
>> Nothing goes through the data and changes the data contained in the 
>> archive.
>>
>> The plotting engine when drawing graphplots (creating the .png files) 
>> does, I believe, try and smooth (and scale) the data for the plots.
>>
>> Have you compared your graphs with the standard weewx report 
>> graphs/plots??
>>
>>
>>
>> On Thursday, 17 May 2018 08:21:39 UTC+3, Hrmeteohub Pljusak wrote:
>>>
>>> Thank you.
>>> I will try extending the archive interval for stations other that Davis 
>>> Vantage series. However, I believe that weewx uses archive interval from VP 
>>> station, as it is set to "record_generation = hardware".
>>>
>>> Is there some function that goes through archive data in weewx and does 
>>> "the smoothing"? I ask that because I have compared local data with data 
>>> that are on WU and our remote server, and the values with errors did not 
>>> match on all three locations. At the end of the day, I could not see errors 
>>> on WU while database dumps to CSV file contained 1-2 error per day. The csv 
>>> file was done with extenstion set up:
>>> [CSV]
>>>     filename = /tmp/data.txt
>>>     binding = archive
>>>
>>> At first I thought that these errors are perhaps only in loop data, or 
>>> errors on USB communication, but after several weeks I could not catch them 
>>> in logs with matching timestamps.
>>>
>>>
>>>
>>> On Thursday, May 17, 2018 at 7:02:27 AM UTC+2, Andrew Milner wrote:
>>>>
>>>> …. the choices seem to include:
>>>> 1.  Use weewx database and make your graphing package do the smoothing 
>>>> (in the way I presume wview does)
>>>> 2.  Use weewx database and weewx graphs - rather than making your own 
>>>> graphs
>>>> 3.  Increase archive interval to ensure every archive record contains 
>>>> at least one reading for each sensor - I think some sensors only output 
>>>> every 15 minutes (=900 seconds)
>>>> 4.  Explain what you feel to be wrong
>>>> or
>>>> 5.  Return to wview
>>>>
>>>> As a further point you seem to be jumping around many different station 
>>>> types and drivers.  Stations of different types emit different data and 
>>>> weewx will cope with each.  However the config file needs to be set up for 
>>>> each station type with regard to hardware/software preferences, archive 
>>>> interval, hardware/software record creation etc.  HOWEVER - it is 
>>>> overriding within weewx that an archive record contains the average value 
>>>> for a reading during an archive period.  If there is no reading there is 
>>>> no 
>>>> value to store.  In the hgardware manual there is information regarding 
>>>> each station type - and for example the config options for a fine offset 
>>>> station would involve periodic polling at about 60 second intervals (no 
>>>> point in more frequently as station does not emit any faster) and software 
>>>> record creation rather than hardware …. and so on.  The options to use for 
>>>> fine offset are not the same options you would use with a davis station, 
>>>> so 
>>>> it is not possible to just change drivers when changing station types.
>>>>
>>>>
>>>>
>>>> My FineOffset has run for several years without issues.  Using QC 
>>>> values in your config helps reduce spikes
>>>> On Thursday, 17 May 2018 07:39:05 UTC+3, Thomas Keffer wrote:
>>>>>
>>>>> ​Your rant would be a lot more useful if it offered ​some constructive 
>>>>> examples of where you think weewx is failing. Instead, there's just vague 
>>>>> generalities like "erratic measurements from Davis VantagePro2..." 
>>>>>
>>>>> I flatly do not believe this. The VP2 driver is nearly 10 years old, 
>>>>> has been extremely well vetted, and used by thousands without any 
>>>>> problems. 
>>>>> Most likely the problem is in your configuration. 
>>>>>
>>>>> The WMR100 driver is not as mature, but is still used by many people. 
>>>>> I would not be surprised if it had some problems, but your inchoate rant 
>>>>> isn't going to put us any closer to solving them.
>>>>>
>>>>> I am sorry you are disappointed. If you had paid anything for Weewx, I 
>>>>> would be happy to refund your money. But you didn't. 
>>>>>
>>>>> Stop whining and offer a pull request if you think there's a problem.
>>>>>
>>>>> -tk
>>>>>
>>>>>
>>>>> On Wed, May 16, 2018, 2:38 PM Hrmeteohub Pljusak <hrmet...@gmail.com> 
>>>>> wrote:
>>>>>
>>>>>> I am not positive that my questions have not been answered. If so, 
>>>>>> please point me to location where I can read how to mitigate the 
>>>>>> problems. 
>>>>>> Thank you!
>>>>>>
>>>>>> Recently I have switched from wview to weewx 3.7.1. I must say I got 
>>>>>> rather disappointed with problems I have encountered.
>>>>>>
>>>>>> First, Oregon Scientific WMR88/100/200 stations started to produce 
>>>>>> erratic graphs on WU and other web pages, with number of incomplete 
>>>>>> measurements. The weewx documentation gently describes that as "station 
>>>>>> problem". That is not acceptable, or nice, but just a dirty 
>>>>>> walk-around-the-problem.
>>>>>> Yes, I am aware of the way that stations "blasts data" from sensors 
>>>>>> to station, but wview did away with it quite gently. Also, other (read 
>>>>>> windows-based) software works nicely with OS stations. Weewx simply 
>>>>>> gives 
>>>>>> us nothing (null), and lets the user deal with it. Well guess what! 
>>>>>> This should be corrected.
>>>>>>
>>>>>> The graph below is from WU.
>>>>>> Just to ilustrate, this is an example of the data from weewx.sdb:
>>>>>>
>>>>>> dateTime,ET,altimeter,appTemp,barometer,cloudbase,dewpoint,hourRain,humidex,inDewpoint,inHumidity,inTemp,inTempBatteryStatus,interval,maxSolarRad,outHumidity,outTemp,outTempBatteryStatus,pressure,rain,rain24,rainBatteryStatus,rainRate,rainTotal,usUnits,windBatteryStatus,windDir,windGust,windGustDir,windSpeed,windrun
>>>>>>
>>>>>> 1526378490.0,None,1008.47595669,18.7382035985,1008.41843898,1330.2378329,9.93516495581,0.0,20.4483894915,51.3461294561,46.0,23.0,0.0,1,None,55.0,19.2,0.0,988.0,0.0,6.35,0.0,0.0,2.41,17,0.0,337.5,0.700003479692,None,0.700003479692,0.0
>>>>>>
>>>>>> 1526378610.0,None,None,18.174962293,None,1364.37850406,9.66136186735,20.3234925844,51.3461294561,46.0,23.0,0.0,1,None,54.0,19.2,0.0,None,17,0.0,225.0,1.40000695938,None,1.40000695938,0.0420002087815
>>>>>>
>>>>>> 1526378670.0,None,1008.47595669,18.384962815,1008.41843898,1364.37850406,9.66136186735,20.3234925844,51.3461294561,46.0,23.0,0.0,1,None,54.0,19.2,0.0,988.0,17,0.0,315.0,1.10000546809,None,1.10000546809,0.126000626344
>>>>>>
>>>>>> 1526378730.0,None,1008.47595669,None,None,None,None,None,None,1,None,988.0,17,0.0,225.0,0.900004473889,None,0.900004473889,0.19200095443
>>>>>>
>>>>>> 1526378850.0,None,1008.47595669,None,None,None,None,0.0,None,51.3461294561,46.0,23.0,0.0,1,None,988.0,0.0,6.35,0.0,0.0,2.41,17,0.0,315.0,1.10000546809,None,1.10000546809,0.246001222863
>>>>>>
>>>>>> 1526378910.0,None,1008.47595669,18.6649635109,1008.41843898,1364.37850406,9.66136186735,20.3234925844,None,1,None,54.0,19.2,0.0,988.0,17,0.0,225.0,0.700003479692,None,0.700003479692,0.312001550948
>>>>>>
>>>>>> 1526379030.0,None,1008.47595669,None,None,None,None,0.0,None,51.3461294561,46.0,23.0,0.0,1,None,988.0,0.0,6.35,0.0,0.0,2.41,17,0.35400175973
>>>>>>
>>>>>> 1526379150.0,None,1008.47595669,None,None,None,None,0.0,None,51.3461294561,46.0,23.0,0.0,1,None,988.0,0.0,6.35,0.0,0.0,2.41,17,0.35400175973
>>>>>>
>>>>>> 1526379210.0,None,1008.47595669,None,None,None,None,None,51.3461294561,46.0,23.0,0.0,1,None,988.0,17,0.0,250.773234946,0.900004473889,None,0.750003728241,0.35400175973
>>>>>>
>>>>>> 1526379330.0,None,None,18.2996293732,None,1365.28742772,9.75407243522,0.0,20.4655511807,51.3461294561,46.0,23.0,0.0,1,None,54.0,19.3,0.0,None,0.0,6.35,0.0,0.0,2.41,17,0.0,292.5,1.40000695938,None,1.40000695938,0.399001983424
>>>>>>
>>>>>> 1526379390.0,None,1008.47595669,18.0196286772,1008.41138549,1365.28742772,9.75407243522,0.0,20.4655511807,51.3461294561,46.0,23.0,0.0,1,None,54.0,19.3,0.0,988.0,0.0,6.35,0.0,0.0,2.41,17,0.0,292.5,1.80000894778,None,1.80000894778,0.483002400987
>>>>>>
>>>>>> 1526379450.0,None,1008.47595669,None,None,None,None,None,None,1,None,988.0,17,0.0,292.5,2.50001242747,None,2.50001242747,0.591002937854
>>>>>>
>>>>>> Each "None" means that weewx did not get the data from station. Well, 
>>>>>> wview and every other software works around this, and so should weewx!
>>>>>>
>>>>>> Furthermore, weewx tends to have simply wrong - erratic measurements 
>>>>>> in archive from Davis VantagePro, Pro2 and Vue stations. Same problem 
>>>>>> can 
>>>>>> be found on much cheaper and lower quality FineOffset WH1080/2080 
>>>>>> stations. 
>>>>>> Note that this is data that is not seen on WU - I don't know why - 
>>>>>> Perhaps 
>>>>>> WU or weewx does some clean-up of the data later. This graph is drawn 
>>>>>> from 
>>>>>> data as it is copied from table "archive" in 5 minute intervals.
>>>>>>
>>>>>> weewx 3.7.1. is installed on OpenWRT/LEDE 17.0.1 The data is read in 
>>>>>> 5 minute intervals from weewx database (archive, not loop), and written 
>>>>>> to 
>>>>>> database on separate server and (using weewx) sent to WU. Data is read 
>>>>>> from 
>>>>>> WMR and WH stations every 90 seconds, from Davis stations every 60 
>>>>>> seconds 
>>>>>> (archive interval).
>>>>>> The database example is made using CVS export weewx extension.
>>>>>>
>>>>>> The weewx.conf is as follows:
>>>>>> # WEEWX CONFIGURATION FILE
>>>>>> # Copyright (c) 2009-2015 Tom Keffer <tkeffer>
>>>>>> # See the file LICENSE.txt for your rights.
>>>>>> #
>>>>>>
>>>>>> debug = 0
>>>>>> WEEWX_ROOT = /home/weewx
>>>>>> socket_timeout = 20
>>>>>> version = 3.7.1
>>>>>>
>>>>>> [Station]
>>>>>>     
>>>>>>     location = STATIONNAME
>>>>>>     latitude = 432423412
>>>>>>     longitude = 3245143214
>>>>>>     altitude = 175, meter    # Choose 'foot' or 'meter' for unit
>>>>>>     station_type = WMR100
>>>>>>     rain_year_start = 1
>>>>>>     # Start of week (0=Monday, 6=Sunday)
>>>>>>     week_start = 0
>>>>>>     station_url = http://www.pljusak.com
>>>>>>
>>>>>>
>>>>>> ##############################################################################
>>>>>>
>>>>>> [WMR100]
>>>>>>     # This section is for the Oregon Scientific WMR100
>>>>>>     
>>>>>>     # The driver to use
>>>>>>     driver = weewx.drivers.wmr100
>>>>>>     
>>>>>>     # The station model, e.g., WMR100, WMR100N, WMRS200
>>>>>>     model = WMR100
>>>>>>
>>>>>>
>>>>>> [StdRESTful]
>>>>>>     # To guard against parsing errors, put your password(s) in quotes:
>>>>>>     
>>>>>>     [[StationRegistry]]
>>>>>>         # To register this weather station with weewx, set this to 
>>>>>> true
>>>>>>         register_this_station = false
>>>>>>     [[AWEKAS]]
>>>>>>         enable = false
>>>>>>         username = replace_me
>>>>>>         password = replace_me
>>>>>>     [[CWOP]]
>>>>>>         # and specify the station ID (e.g., CW1234).
>>>>>>         enable = false
>>>>>>         station = replace_me
>>>>>>     # If this is an APRS (radio amateur) station, uncomment
>>>>>>     # the following and replace with a passcode (e.g., 12345).
>>>>>>     #passcode = replace_me (APRS stations only)
>>>>>>     [[PWSweather]]
>>>>>>         enable = false
>>>>>>         station = replace_me
>>>>>>         password = replace_me
>>>>>>     [[WOW]]
>>>>>>         enable = false
>>>>>>         station = replace_me
>>>>>>         password = replace_me
>>>>>>     [[Wunderground]]
>>>>>>         enable = true
>>>>>>         station = ISTATIONNAME1
>>>>>>         password = PASSOWRD
>>>>>>         rapidfire = False
>>>>>>
>>>>>> [StdReport]
>>>>>>
>>>>>>     SKIN_ROOT = skins
>>>>>>     HTML_ROOT = /tmp 
>>>>>>     # This module is not used 
>>>>>>    [[simple]]
>>>>>>         HTML_ROOT = /tmp/simple
>>>>>>         skin = simple
>>>>>>
>>>>>> [StdConvert]
>>>>>>     # DO NOT MODIFY THIS VALUE UNLESS YOU KNOW WHAT YOU ARE DOING!
>>>>>>     target_unit = METRICWX    # Options are 'US', 'METRICWX', or 
>>>>>> 'METRIC'
>>>>>>
>>>>>> [StdCalibrate]
>>>>>>     
>>>>>>     [[Corrections]]
>>>>>>         # For each type, an arbitrary calibration expression can be 
>>>>>> given. 
>>>>>>         # It should be in the units defined in the StdConvert section.
>>>>>>         # Example:
>>>>>>         foo = foo + 0.2
>>>>>>
>>>>>> [StdQC]
>>>>>>     
>>>>>>     [[MinMax]]
>>>>>>         barometer = 900, 1400, mbar
>>>>>>         outTemp = -30, 50, degree_C
>>>>>>         inTemp = -30, 50, degree_C
>>>>>>         outHumidity = 0, 100
>>>>>>         inHumidity = 0, 100
>>>>>>         windSpeed = 0, 300, km_per_hour
>>>>>>         pressure = 900, 1100, mbar
>>>>>>
>>>>>> [StdWXCalculate]
>>>>>>     
>>>>>>     [[Calculations]]
>>>>>>         # Derived quantities are calculated by this service. Possible 
>>>>>> values are:
>>>>>>         #  hardware        - use the value provided by hardware
>>>>>>         #  software        - use the value calculated by weewx
>>>>>>         #  prefer_hardware - use value provide by hardware if 
>>>>>> available,
>>>>>>         #                      otherwise use value calculated by weewx
>>>>>>         
>>>>>>         pressure = prefer_hardware
>>>>>>         barometer = prefer_hardware
>>>>>>         altimeter = prefer_hardware
>>>>>>         windchill = hardware
>>>>>>         heatindex = hardware
>>>>>>         dewpoint = prefer_hardware
>>>>>>         inDewpoint = prefer_hardware
>>>>>>         rainRate = hardware
>>>>>>
>>>>>> [StdTimeSynch]
>>>>>>     clock_check = 14400
>>>>>>     max_drift = 5
>>>>>>
>>>>>> [StdArchive]
>>>>>>     # If the station hardware supports data logging then the archive 
>>>>>> interval
>>>>>>     # will be downloaded from the station. Otherwise, specify it (in 
>>>>>> seconds).
>>>>>>     archive_interval = 90
>>>>>>     # If possible, new archive records are downloaded from the station
>>>>>>     # hardware. If the hardware does not support this, then new 
>>>>>> archive
>>>>>>     # records will be generated in software.
>>>>>>     # Set the following to "software" to force software record 
>>>>>> generation.
>>>>>>     record_generation = software
>>>>>>     # Whether to include LOOP data in hi/low statistics
>>>>>>     loop_hilo = True
>>>>>>     # The data binding used to save archive records
>>>>>>     data_binding = wx_binding
>>>>>>
>>>>>> [DataBindings]
>>>>>>     [[wx_binding]]
>>>>>>         database = archive_sqlite
>>>>>>         table_name = archive
>>>>>>         manager = weewx.wxmanager.WXDaySummaryManager
>>>>>>         schema = schemas.wview.schema
>>>>>>
>>>>>> [Databases]
>>>>>>     [[archive_sqlite]]
>>>>>>         database_type = SQLite
>>>>>>         database_name = weewx.sdb
>>>>>>
>>>>>> [DatabaseTypes]
>>>>>>     [[SQLite]]
>>>>>>         driver = weedb.sqlite
>>>>>>         SQLITE_ROOT = /tmp
>>>>>>
>>>>>> [Engine]
>>>>>>     
>>>>>>     [[Services]]
>>>>>>         prep_services = weewx.engine.StdTimeSynch
>>>>>>         data_services = ,
>>>>>>         process_services = weewx.engine.StdConvert, 
>>>>>> weewx.engine.StdCalibrate, weewx.engine.StdQC, 
>>>>>> weewx.wxservices.StdWXCalculate, user.csv.CSV
>>>>>>         archive_services = weewx.engine.StdArchive
>>>>>>         restful_services = weewx.restx.StdStationRegistry, 
>>>>>> weewx.restx.StdWunderground, weewx.restx.StdPWSweather, 
>>>>>> weewx.restx.StdCWOP, weewx.restx.StdWOW, weewx.restx.StdAWEKAS
>>>>>>         report_services = weewx.engine.StdPrint
>>>>>>
>>>>>> [CSV]
>>>>>>     filename = /tmp/data.txt
>>>>>>     binding = archive
>>>>>>
>>>>>>
>>>>>> -- 
>>>>>> You received this message because you are subscribed to the Google 
>>>>>> Groups "weewx-user" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it, 
>>>>>> send an email to weewx-user+...@googlegroups.com.
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to