I did not quite follow your posting. I do not see how the buffer size in the VP can overload weewx!! Weewx is designed to run 24/7 and only do catchup processing/recovery in the event of power failures, so under normal usage I would not have thought there would be a problem. Have you trierd ther VP with software record generation or only hardware? When you change the archive interval in weewx did you also change it in the VP station. What interval is weewx using (see log at startup of weewx)
If lengthening the archive interval solved the wmr problem then as I see it you have a simple choice - use the larger archive interval or don't use wmr hardware!! I would have thought that only the stations which experience problems and issues would need to use the larger interval - not necessarily every station in your network. As has been said many times in this forum comments like 'VP struggles' do not tell us anything - we (and you) need to look at the log, maybe run with debug enabled and so forth in order to understand and more importantly see what is going on within weewx - any error messages and so forth. On Tuesday, 22 May 2018 15:14:13 UTC+3, Hrmeteohub Pljusak wrote: > > Huh, I have waited two days to see how my WMR and Davis are going with > aforemntioned change. While WMR performs butifully, with not a single > drop-out, Davis VP still struggles. I have to dig into that next. > As far as lengthening the archive interval, this is not solely my > decision. There are many stations involved, and I have to see if others > agree with it. I'll try, and then the we'll make a decision. > > One thing that bothers me are constraints of the router that runs the > installed software. Python is a beast, and after everything is set up, > there is not much memory left on the RAM. Think 400MHz processor, 60MB > total RAM. I suspect that is why archive interval is having a strong > impact on performance in Davis stations as they have large archive > capacity and may overload the router, either on the startup or collide > with some other housekeeping that OS is doing. The situation is very > different from e.g. Raspberry Pi with +10x RAM. > Also, I think I have to dig deeper in the process how I am reporting the > info to our page, as you have suggested. I am thin on Python knowledge > so I will need help from someone for that. > > I'll keep you posted, and thank you very much! > > > On 17.5.2018. 15:34, Andrew Milner wrote: > > try 10 minutes even as the interval. Be honest - since weewx is storing > > AVERAGES anyway, there is probably little difference between a 5 minute > > interval and a 10 minute interval except that your servers will have a > > fraction of the workload and the stations will suffer fewer dropouts and > > losses of data which to me would be a win-win situation!! You may as > > well try it and I think you will be very pleasantly sur[rised at the > > result of using either 10 or even 15 minute archive intervals!! > > > > On Thursday, 17 May 2018 15:18:06 UTC+3, Hrmeteohub Pljusak wrote: > > > > Setting 5-min archiving interval made great difference on WMR88 - I > > had only one drop-out in last four hours. Already I have made it a > > default in the firmware, for all stations. I'll test this on WMR88 > > for several days and then see how it goes. > > I the mean time I will focus probably on Davis stations. It is hard > > to say if archiving interval of 5 minutes made a difference so far, > > since the errors did happen only 1-2 times each day. > > > > I would really like to work with just one type of station, but our > > volunteers have all kind of stations, from very old to brand new, > > and expect us to support them all. > > You are right on the money with a need for new API. > > > > Thank you for the advice. > > > > On Thu, May 17, 2018 at 1:16 PM, Andrew Milner > > <[email protected] <javascript:>> wrote: > > > > I seriously recommend that you focus on one particular model of > > weather station first - eg FineOffset (use software records > > rather than hardware - see weewx hardware manual for more on the > > options) and maybe Davis - use hardware records and once again > > see weewx hardware manual for options). In both cases I would > > suggest use a 5 minute archive interval. For WU do not use > > rapid fire until you are happy things are ok. > > > > For the rest of your setup it would probably be simplest for you > > to have arranged for your server to have accepted input via a > > restless API and built a restless service to upload directly to > > your site - like weewx does to WU and other hosts - but that is > > far more suitable for the development forum rather than this > > forum as it is outside the scope of current weewx. > > > > > > > > On Thursday, 17 May 2018 10:49:39 UTC+3, Hrmeteohub Pljusak > wrote: > > > > Dear Andrew, > > > > First, thank you and thank Thomas and everybody else for > > producing weewx. I apologize once more for coming harsh in > > my first post. > > > > It appears that you are quite right that I am forcing weewx > > into something it was not made to do. Perhaps maybe not all > > of it, but I suspect it is the way I am doing thins more to > > be blame in some instances. > > Also, we are doing that with several weather stations, at > > locations apart, with so many different co-factors. Seems > > like a good recipe for headache. > > As luck will have it, weewx is now my tool of choice. You > > are very much correct when you say that I have many problems > > on a number of fronts simultaneously. Unfortunately. It > > looks a bit chaotic when I have presented it all at once. > > Please accept my exclamations as expression of anger onto > > my/our choices and problems. > > > > Thank you for giving me info on WU. I did not know that. > > If nothing goes through the database, than the problem must > > be on "my side", CSV reporting and sending to pljusak.com > > <http://pljusak.com> server. > > > > Some of the problems were expected, like intermittent > > sending of data from WMR stations to weewx. I was combing > > through data and testing parts of the process for Davis and > > FineOffset stations for weeks now, and simply can not put my > > finger on the problem. No such event was seen in weeks for > > WS2300, WMR100, WMR200 stations in weeks of testing. > > > > Please do not read the rest of my post if you do not have > > too much time on your hands. As Thomas wrote, I tend to > rant. > > > > > > Let me give some perspective to HRmeteohub project. > > > > The goal of the project is to send data from weather > > stations to crowd sourcing web page pljusak.com > > <http://pljusak.com>, as well as to WU, AWEKAS, and so on. > > The project was started by amateur meteorologists and > > enthusiasts in Croatia, and it runs for about ten years now. > > At first phase, all users have used some kind of windows > > based software. In that (first) phase, our job was to hand > > out the templates for PC software that users were using and > > users would use FTP to send the graphs and data to server. > > It worked, but what a mess (and security headaches). > > In second phase we started using Open2300 in small devices > > such as NSLU and wifi routers. About six years ago this has > > transformed in customized OpenWRT firmware for 5-6 different > > weather station types. Hence my references to wview. At that > > time our team stabilized on one sysadmin, one programmer, > > and me in charge of building the firmware, testing, > > documening as well as supporting users together with > sysadmin. > > However, hardware producers have been changing their routers > > and weather stations, combined with changes in OpenWRT - > > making our task almost impossible. > > > > At this time about 40 users use HRmeteohub firmware, and > > some would like to, but supported wifi routers are not on > > the market anymore. Pljusak.com is NGO run by volunteers, > > all the software, firmware and services are provided for > > free and all income from advertising is used to maintain the > > server and keep NGO functioning. The web page is now about > > 5-6 years old and ready for overhaul. Basically the server > > produces something like this: > > > > And each station has got it's page: > > > > This is where weewx comes on stage. We have been testing it > > on RaspberryPi with good results, but it is rather too > > complicated for our average meteo-enthusiast to install and > > setup R-pi with weewx and our scripts for sending the data > > to pljusak.com <http://pljusak.com>. Plus, R-pi (version 2B > > at that time) wirth power source, wifi card, box was a bit > > more expensive than wifi router. Our users could can buy > > second-hand PC with old windows on it for the same price. It > > (r-pi) also did not work as reliably as wifi routers. At > > that time, putting Python and weewx in small, cheap wifi > > router was simply not possible, but in time, OpenWRT merged > > with LEDE, routers got faster with more Flash and RAM, and > > suddenly it become a viable option. > > After about six months of tries and errors, the firmware > > proved to be stable. I can not recall single crash of weewx > > or firmware in last two months. > > Unfortunately, Python is big beast for wifi routers with > > 16MB flash, and 64Mb RAM, so we could not install the > > graphing, but settle down for archiving the data in devices > > RAM (default option) or USB stick (works well, but still > > under testing). Every two hours one script tests internet > > connection and how much RAM database occupies and if it > > reaches critical level the user is notified and the database > > is cleared. Yes, I know that is not nice thing, but the user > > has got it's data backed up on server. Generally this > > happens depending on archiving interval, how much RAM device > > has got and if there are other services needing more RAM. > > > > We use weewx to read the measurements from weather station, > > and archive it in database. Sending data to WU, CWOP, AWEKAS > > and others is done using weewx extensions.We have not come > > up with extension for our pljusak.com <http://pljusak.com> > > so far as none of us is proficient in Python. > > What we do with archived data in weewx.sdb is rather simple. > > CSV extension dumps last measurement to text. Every 5 > > minutes we send it to server, parse it, put it in database > > and make a graph. We have tried sending one last measurement > > from database or sending 2-3 of them, but above mentioned > > problems persisted. The (raw, unparsed) data sent from > > router is visible on the web page, so we can see if what > > arrived at server before it hits database and graphs). We > > can also see csv.txt on router through it's Luci web page > > and confirm that the data dumped from weewx.sdb and on > > server are a match. > > > > OK. I guess I have taken you enough of your time. > > > > I'll be back when I have some solutions for above mentioned > > problems. > > > > On Thursday, May 17, 2018 at 7:33:24 AM UTC+2, Andrew Milner > > wrote: > > > > Nothing goes through the data and changes the data > > contained in the archive. > > > > The plotting engine when drawing graphplots (creating > > the .png files) does, I believe, try and smooth (and > > scale) the data for the plots. > > > > Have you compared your graphs with the standard weewx > > report graphs/plots?? > > > > > > > > On Thursday, 17 May 2018 08:21:39 UTC+3, Hrmeteohub > > Pljusak wrote: > > > > Thank you. > > I will try extending the archive interval for > > stations other that Davis Vantage series. However, I > > believe that weewx uses archive interval from VP > > station, as it is set to "record_generation = > hardware". > > > > Is there some function that goes through archive > > data in weewx and does "the smoothing"? I ask that > > because I have compared local data with data that > > are on WU and our remote server, and the values with > > errors did not match on all three locations. At the > > end of the day, I could not see errors on WU while > > database dumps to CSV file contained 1-2 error per > > day. The csv file was done with extenstion set up: > > [CSV] > > filename = /tmp/data.txt > > binding = archive > > > > At first I thought that these errors are perhaps > > only in loop data, or errors on USB communication, > > but after several weeks I could not catch them in > > logs with matching timestamps. > > > > > > > > On Thursday, May 17, 2018 at 7:02:27 AM UTC+2, > > Andrew Milner wrote: > > > > …. the choices seem to include: > > 1. Use weewx database and make your graphing > > package do the smoothing (in the way I presume > > wview does) > > 2. Use weewx database and weewx graphs - rather > > than making your own graphs > > 3. Increase archive interval to ensure every > > archive record contains at least one reading for > > each sensor - I think some sensors only output > > every 15 minutes (=900 seconds) > > 4. Explain what you feel to be wrong > > or > > 5. Return to wview > > > > As a further point you seem to be jumping around > > many different station types and drivers. > > Stations of different types emit different data > > and weewx will cope with each. However the > > config file needs to be set up for each station > > type with regard to hardware/software > > preferences, archive interval, hardware/software > > record creation etc. HOWEVER - it is overriding > > within weewx that an archive record contains the > > average value for a reading during an archive > > period. If there is no reading there is no > > value to store. In the hgardware manual there > > is information regarding each station type - and > > for example the config options for a fine offset > > station would involve periodic polling at about > > 60 second intervals (no point in more frequently > > as station does not emit any faster) and > > software record creation rather than hardware …. > > and so on. The options to use for fine offset > > are not the same options you would use with a > > davis station, so it is not possible to just > > change drivers when changing station types. > > > > > > > > My FineOffset has run for several years without > > issues. Using QC values in your config helps > > reduce spikes > > On Thursday, 17 May 2018 07:39:05 UTC+3, Thomas > > Keffer wrote: > > > > Your rant would be a lot more useful if it > > offered some constructive examples of where > > you think weewx is failing. Instead, there's > > just vague generalities like "erratic > > measurements from Davis VantagePro2..." > > > > I flatly do not believe this. The VP2 driver > > is nearly 10 years old, has been extremely > > well vetted, and used by thousands without > > any problems. Most likely the problem is in > > your configuration. > > > > The WMR100 driver is not as mature, but is > > still used by many people. I would not be > > surprised if it had some problems, but your > > inchoate rant isn't going to put us any > > closer to solving them. > > > > I am sorry you are disappointed. If you had > > paid anything for Weewx, I would be happy to > > refund your money. But you didn't. > > > > Stop whining and offer a pull request if you > > think there's a problem. > > > > -tk > > > > > > On Wed, May 16, 2018, 2:38 PM Hrmeteohub > > Pljusak <[email protected]> wrote: > > > > I am not positive that my questions have > > not been answered. If so, please point > > me to location where I can read how to > > mitigate the problems. Thank you! > > > > Recently I have switched from wview to > > weewx 3.7.1. I must say I got rather > > disappointed with problems I have > > encountered. > > > > First, Oregon Scientific WMR88/100/200 > > stations started to produce erratic > > graphs on WU and other web pages, with > > number of incomplete measurements. The > > weewx documentation gently describes > > that as "station problem". That is not > > acceptable, or nice, but just a dirty > > walk-around-the-problem. > > Yes, I am aware of the way that stations > > "blasts data" from sensors to station, > > but wview did away with it quite gently. > > Also, other (read windows-based) > > software works nicely with OS stations. > > Weewx simply gives us nothing (null), > > and lets the user deal with it. Well > > guess what! > > This should be corrected. > > > > The graph below is from WU. > > Just to ilustrate, this is an example of > > the data from weewx.sdb: > > > dateTime,ET,altimeter,appTemp,barometer,cloudbase,dewpoint,hourRain,humidex,inDewpoint,inHumidity,inTemp,inTempBatteryStatus,interval,maxSolarRad,outHumidity,outTemp,outTempBatteryStatus,pressure,rain,rain24,rainBatteryStatus,rainRate,rainTotal,usUnits,windBatteryStatus,windDir,windGust,windGustDir,windSpeed,windrun > > > > > 1526378490.0,None,1008.47595669,18.7382035985,1008.41843898,1330.2378329,9.93516495581,0.0,20.4483894915,51.3461294561,46.0,23.0,0.0,1,None,55.0,19.2,0.0,988.0,0.0,6.35,0.0,0.0,2.41,17,0.0,337.5,0.700003479692,None,0.700003479692,0.0 > > > > > 1526378610.0,None,None,18.174962293,None,1364.37850406,9.66136186735,20.3234925844,51.3461294561,46.0,23.0,0.0,1,None,54.0,19.2,0.0,None,17,0.0,225.0,1.40000695938,None,1.40000695938,0.0420002087815 > > > > > 1526378670.0,None,1008.47595669,18.384962815,1008.41843898,1364.37850406,9.66136186735,20.3234925844,51.3461294561,46.0,23.0,0.0,1,None,54.0,19.2,0.0,988.0,17,0.0,315.0,1.10000546809,None,1.10000546809,0.126000626344 > > > > > 1526378730.0,None,1008.47595669,None,None,None,None,None,None,1,None,988.0,17,0.0,225.0,0.900004473889,None,0.900004473889,0.19200095443 > > > > > 1526378850.0,None,1008.47595669,None,None,None,None,0.0,None,51.3461294561,46.0,23.0,0.0,1,None,988.0,0.0,6.35,0.0,0.0,2.41,17,0.0,315.0,1.10000546809,None,1.10000546809,0.246001222863 > > > > > 1526378910.0,None,1008.47595669,18.6649635109,1008.41843898,1364.37850406,9.66136186735,20.3234925844,None,1,None,54.0,19.2,0.0,988.0,17,0.0,225.0,0.700003479692,None,0.700003479692,0.312001550948 > > > > > 1526379030.0,None,1008.47595669,None,None,None,None,0.0,None,51.3461294561,46.0,23.0,0.0,1,None,988.0,0.0,6.35,0.0,0.0,2.41,17,0.35400175973 > > > > > 1526379150.0,None,1008.47595669,None,None,None,None,0.0,None,51.3461294561,46.0,23.0,0.0,1,None,988.0,0.0,6.35,0.0,0.0,2.41,17,0.35400175973 > > > > > 1526379210.0,None,1008.47595669,None,None,None,None,None,51.3461294561,46.0,23.0,0.0,1,None,988.0,17,0.0,250.773234946,0.900004473889,None,0.750003728241,0.35400175973 > > > > > 1526379330.0,None,None,18.2996293732,None,1365.28742772,9.75407243522,0.0,20.4655511807,51.3461294561,46.0,23.0,0.0,1,None,54.0,19.3,0.0,None,0.0,6.35,0.0,0.0,2.41,17,0.0,292.5,1.40000695938,None,1.40000695938,0.399001983424 > > > > > 1526379390.0,None,1008.47595669,18.0196286772,1008.41138549,1365.28742772,9.75407243522,0.0,20.4655511807,51.3461294561,46.0,23.0,0.0,1,None,54.0,19.3,0.0,988.0,0.0,6.35,0.0,0.0,2.41,17,0.0,292.5,1.80000894778,None,1.80000894778,0.483002400987 > > > > > 1526379450.0,None,1008.47595669,None,None,None,None,None,None,1,None,988.0,17,0.0,292.5,2.50001242747,None,2.50001242747,0.591002937854 > > > > > > Each "None" means that weewx did not get > > the data from station. Well, wview and > > every other software works around this, > > and so should weewx! > > > > Furthermore, weewx tends to have simply > > wrong - erratic measurements in archive > > from Davis VantagePro, Pro2 and Vue > > stations. Same problem can be found on > > much cheaper and lower quality > > FineOffset WH1080/2080 stations. Note > > that this is data that is not seen on WU > > - I don't know why - Perhaps WU or weewx > > does some clean-up of the data later. > > This graph is drawn from data as it is > > copied from table "archive" in 5 minute > > intervals. > > > > weewx 3.7.1. is installed on > > OpenWRT/LEDE 17.0.1 The data is read in > > 5 minute intervals from weewx database > > (archive, not loop), and written to > > database on separate server and (using > > weewx) sent to WU. Data is read from WMR > > and WH stations every 90 seconds, from > > Davis stations every 60 seconds (archive > > interval). > > The database example is made using CVS > > export weewx extension. > > > > The weewx.conf is as follows: > > # WEEWX CONFIGURATION FILE > > # Copyright (c) 2009-2015 Tom Keffer > > <tkeffer> > > # See the file LICENSE.txt for your > rights. > > # > > > > debug = 0 > > WEEWX_ROOT = /home/weewx > > socket_timeout = 20 > > version = 3.7.1 > > > > [Station] > > > > location = STATIONNAME > > latitude = 432423412 > > longitude = 3245143214 > > altitude = 175, meter # Choose > > 'foot' or 'meter' for unit > > station_type = WMR100 > > rain_year_start = 1 > > # Start of week (0=Monday, > 6=Sunday) > > week_start = 0 > > station_url = > http://www.pljusak.com > > > > > ############################################################################## > > > > > > [WMR100] > > # This section is for the Oregon > > Scientific WMR100 > > > > # The driver to use > > driver = weewx.drivers.wmr100 > > > > # The station model, e.g., WMR100, > > WMR100N, WMRS200 > > model = WMR100 > > > > > > [StdRESTful] > > # To guard against parsing errors, > > put your password(s) in quotes: > > > > [[StationRegistry]] > > # To register this weather > > station with weewx, set this to true > > register_this_station = false > > [[AWEKAS]] > > enable = false > > username = replace_me > > password = replace_me > > [[CWOP]] > > # and specify the station ID > > (e.g., CW1234). > > enable = false > > station = replace_me > > # If this is an APRS (radio > > amateur) station, uncomment > > # the following and replace with a > > passcode (e.g., 12345). > > #passcode = replace_me (APRS > > stations only) > > [[PWSweather]] > > enable = false > > station = replace_me > > password = replace_me > > [[WOW]] > > enable = false > > station = replace_me > > password = replace_me > > [[Wunderground]] > > enable = true > > station = ISTATIONNAME1 > > password = PASSOWRD > > rapidfire = False > > > > [StdReport] > > > > SKIN_ROOT = skins > > HTML_ROOT = /tmp > > # This module is not used > > [[simple]] > > HTML_ROOT = /tmp/simple > > skin = simple > > > > [StdConvert] > > # DO NOT MODIFY THIS VALUE UNLESS > > YOU KNOW WHAT YOU ARE DOING! > > target_unit = METRICWX # Options > > are 'US', 'METRICWX', or 'METRIC' > > > > [StdCalibrate] > > > > [[Corrections]] > > # For each type, an arbitrary > > calibration expression can be given. > > # It should be in the units > > defined in the StdConvert section. > > # Example: > > foo = foo + 0.2 > > > > [StdQC] > > > > [[MinMax]] > > barometer = 900, 1400, mbar > > outTemp = -30, 50, degree_C > > inTemp = -30, 50, degree_C > > outHumidity = 0, 100 > > inHumidity = 0, 100 > > windSpeed = 0, 300, km_per_hour > > pressure = 900, 1100, mbar > > > > [StdWXCalculate] > > > > [[Calculations]] > > # Derived quantities are > > calculated by this service. Possible > > values are: > > # hardware - use the > > value provided by hardware > > # software - use the > > value calculated by weewx > > # prefer_hardware - use value > > provide by hardware if available, > > # > > otherwise use value calculated by weewx > > > > pressure = prefer_hardware > > barometer = prefer_hardware > > altimeter = prefer_hardware > > windchill = hardware > > heatindex = hardware > > dewpoint = prefer_hardware > > inDewpoint = prefer_hardware > > rainRate = hardware > > > > [StdTimeSynch] > > clock_check = 14400 > > max_drift = 5 > > > > [StdArchive] > > # If the station hardware supports > > data logging then the archive interval > > # will be downloaded from the > > station. Otherwise, specify it (in > seconds). > > archive_interval = 90 > > # If possible, new archive records > > are downloaded from the station > > # hardware. If the hardware does > > not support this, then new archive > > # records will be generated in > > software. > > # Set the following to "software" > > to force software record generation. > > record_generation = software > > # Whether to include LOOP data in > > hi/low statistics > > loop_hilo = True > > # The data binding used to save > > archive records > > data_binding = wx_binding > > > > [DataBindings] > > [[wx_binding]] > > database = archive_sqlite > > table_name = archive > > manager = > > weewx.wxmanager.WXDaySummaryManager > > schema = schemas.wview.schema > > > > [Databases] > > [[archive_sqlite]] > > database_type = SQLite > > database_name = weewx.sdb > > > > [DatabaseTypes] > > [[SQLite]] > > driver = weedb.sqlite > > SQLITE_ROOT = /tmp > > > > [Engine] > > > > [[Services]] > > prep_services = > > weewx.engine.StdTimeSynch > > data_services = , > > process_services = > > weewx.engine.StdConvert, > > weewx.engine.StdCalibrate, > > weewx.engine.StdQC, > > weewx.wxservices.StdWXCalculate, > > user.csv.CSV > > archive_services = > > weewx.engine.StdArchive > > restful_services = > > weewx.restx.StdStationRegistry, > > weewx.restx.StdWunderground, > > weewx.restx.StdPWSweather, > > weewx.restx.StdCWOP, weewx.restx.StdWOW, > > weewx.restx.StdAWEKAS > > report_services = > > weewx.engine.StdPrint > > > > [CSV] > > filename = /tmp/data.txt > > binding = archive > > > > > > -- > > You received this message because you > > are subscribed to the Google Groups > > "weewx-user" group. > > To unsubscribe from this group and stop > > receiving emails from it, send an email > > to [email protected]. > > For more options, visit > > https://groups.google.com/d/optout > > <https://groups.google.com/d/optout>. > > > > -- > > You received this message because you are subscribed to a topic > > in the Google Groups "weewx-user" group. > > To unsubscribe from this topic, visit > > > https://groups.google.com/d/topic/weewx-user/HXHjLqxOJxE/unsubscribe > > < > https://groups.google.com/d/topic/weewx-user/HXHjLqxOJxE/unsubscribe>. > > To unsubscribe from this group and all its topics, send an email > > to [email protected] <javascript:>. > > For more options, visit https://groups.google.com/d/optout > > <https://groups.google.com/d/optout>. > > > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "weewx-user" group. > > To unsubscribe from this topic, visit > > https://groups.google.com/d/topic/weewx-user/HXHjLqxOJxE/unsubscribe. > > To unsubscribe from this group and all its topics, send an email to > > [email protected] <javascript:> > > <mailto:[email protected] <javascript:>>. > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
