Yes. That's certainly possible. -tk
On Tue, May 7, 2019 at 1:35 PM engolling <[email protected]> wrote: > Hello together, > > it's me again with hopefully a last question... > As you know I'm augmenting some extra data from my service to the WeeWx > archive records. But if archive records are loaded of the stations logger > this data (which is in the future) is also augmented. > Is it possible to read the timestamp of the archive record beeing saved at > the moment and then decide if data should be augmented or not? > Im thinking of a code like this: > if dateTime_AugmentedData - event.record[dateTime] < 300: > --> augment_data > > > > > > Am Mittwoch, 3. April 2019 00:54:17 UTC+2 schrieb gjr80: >> >> Good that you have tracked down the issue. Running hardware record >> generation on a Davis station has it advantages, though maybe not so >> advantageous on emulated hardware. >> >> Regards plotting rain data from two sensors. The current WeeWX plot >> engine will certainly allow you to plot two obs in a bar graph plot, though >> I expect the outcome will be one bar plotted over the top of the other. Not >> really what you are seeking. There is no option to plot the bars adjacent >> to each other. It may be possible to modify the plot engine though I do not >> expect it would be a simple modification, the plot engine is fairly >> rudimentary. You may need to use an external charting package which should >> easily do what you want, though this too will not be a turn key solution as >> you will need to generate the necessary data for the package as well as >> integrate it into your web page(s). >> >> Gary >> >> On Wednesday, 3 April 2019 03:44:09 UTC+10, engolling wrote: >>> >>> Hi Gary, >>> >>> again thanks for your extended reply. >>> Meanwhile I added a lot of debugging code in the vantage driver and the >>> WeatherDuino code and was able to find out the reason what happened. >>> Their is a buffer on the flash chip which is written before storing the >>> data in the flash page. And this buffer always had the outdated data of the >>> page before in the last entries. >>> This lead to this behaviour that the actual written page contained wrong >>> data, but the page before and the page after was pretty fine... >>> >>> It took me quite some time to find the system behind this error. >>> I'm in touch with the developper of the WeatherDuino and this issue will >>> be fixed in the next firmware release so there will be full compatibility >>> with WeeWx. >>> >>> Now that erverything is working I have a last question - do you, or >>> anybody else, think it is possible to draw a bar graph having the rain bars >>> of two sensors right beside? >>> I did not find anything in the manual and if I define the diagramm like >>> in the line / dot graphs the two bars overlay, so that the bar of the >>> second rain sensor is not visible, if it is a bit smaller. >>> >>> Thank you for all your replies. >>> >>> Regards, >>> engolling >>> >>> Am Dienstag, 2. April 2019 16:01:11 UTC+2 schrieb gjr80: >>>> >>>> Sorry for being a little tardy in replying but I wanted to sit down and >>>> work my way through the Davis protocol. It seems everything is being >>>> followed, I do not know if the lack of a final acknowledgement is critical >>>> or not but what I would say is that the Davis driver works with real Davis >>>> hardware without issue. >>>> >>>> One thing I did wonder, the DMPAFT command downloads all archive >>>> records after a given timestamp. Archive records are downloaded a page at a >>>> time and a page contains 5 archive records. When we look at the earlier log >>>> extracts you have provided there seems to be groups of 5 archive records >>>> being processed (1 page?) but of course all that are already in the archive >>>> are rejected, for example: >>>> >>>> Mar 18 22:45:18 WeatherDuinoPi weewx[12220]: manager: Unable to add >>>> record 2019-03-18 22:43:00 CET (1552945380) to database 'weewx.sdb': >>>> UNIQUE constraint failed: archive.dateTime >>>> Mar 18 22:45:20 WeatherDuinoPi weewx[12220]: manager: Added record 2019 >>>> -03-18 22:44:00 CET (1552945440) to database 'weewx.sdb' >>>> Mar 18 22:45:20 WeatherDuinoPi weewx[12220]: manager: Added record 2019 >>>> -03-18 22:44:00 CET (1552945440) to daily summary in 'weewx.sdb' >>>> Mar 18 22:45:22 WeatherDuinoPi weewx[12220]: manager: Unable to add >>>> record 2019-03-18 22:40:00 CET (1552945200) to database 'weewx.sdb': >>>> UNIQUE constraint failed: archive.dateTime >>>> Mar 18 22:45:24 WeatherDuinoPi weewx[12220]: manager: Unable to add >>>> record 2019-03-18 22:41:00 CET (1552945260) to database 'weewx.sdb': >>>> UNIQUE constraint failed: archive.dateTime >>>> Mar 18 22:45:25 WeatherDuinoPi weewx[12220]: manager: Unable to add >>>> record 2019-03-18 22:42:00 CET (1552945320) to database 'weewx.sdb': >>>> UNIQUE constraint failed: archive.dateTime >>>> >>>> I did not see anywhere where you ran WeeWX directly >>>> <http://weewx.com/docs/usersguide.htm#Running_directly> and provided a >>>> few minutes of console output. Maybe that may shed more light on the >>>> problem. Failing that you may need to put some logging in the vantage >>>> driver to see exactly what the driver is reading and then sending to WeeWX. >>>> >>>> Gary >>>> >>>> >>>> On Wednesday, 27 March 2019 07:44:59 UTC+10, engolling wrote: >>>>> >>>>> Hi Gary, >>>>> >>>>> I tried to debug the system and therefore I built a "serial sniffer" >>>>> in form of an Arduino Mega passing through the data between serial0 and >>>>> serial1 and printing all messages out on serial2. >>>>> I used HTERM to display it and I will provide some screenshots for a >>>>> better visibility. >>>>> Here we see a typical archive request: >>>>> >>>>> 1. DMPAFT\n - request of WeeWx >>>>> 2. WeatherDuino acknowledge with 0x06 >>>>> 3. WeeWx sends Date, Time and Checksum >>>>> 4. WeatherDuino acknowledge with 0x06 >>>>> 5. WeatherDuino sends number of pages and checksum >>>>> 6. WeeWx acknowledge >>>>> 7. WeatherDuino sends page and overhead - in sum 267 bytes as expected. >>>>> *8. WeeWx sends LineFeed, but should acknowledge, send a CRC failure >>>>> or skip - that is strange...* >>>>> >>>>> See documentation line 35 and 36 >>>>> >>>>> https://www.davisinstruments.com/support/weather/download/VantageSerialProtocolDocs_v261.pdf >>>>> >>>>> 1 Minute and 360ms later the next DMPAFT command is sent. >>>>> >>>>> This example is starting today at 19:05:16 local time - this is >>>>> written into the logfile. >>>>> Mar 26 19:05:17 WeatherDuinoPi weewx[17760]: manager: Added record >>>>> 2019-03-26 19:04:00 CET (1553623440) to database 'weewx.sdb' >>>>> Mar 26 19:05:17 WeatherDuinoPi weewx[17760]: manager: Added record >>>>> 2019-03-26 19:04:00 CET (1553623440) to daily summary in 'weewx.sdb' >>>>> Mar 26 19:05:19 WeatherDuinoPi weewx[17760]: manager: Unable to add >>>>> record 2019-03-26 19:00:00 CET (1553623200) to database 'weewx.sdb': >>>>> UNIQUE constraint failed: archive.dateTime >>>>> Mar 26 19:05:21 WeatherDuinoPi weewx[17760]: manager: Unable to add >>>>> record 2019-03-26 19:01:00 CET (1553623260) to database 'weewx.sdb': >>>>> UNIQUE constraint failed: archive.dateTime >>>>> Mar 26 19:05:31 WeatherDuinoPi weewx[17760]: cheetahgenerator: >>>>> Generated 9 files for report StandardReport in 10.25 seconds >>>>> Mar 26 19:06:19 WeatherDuinoPi weewx[17760]: manager: Added record >>>>> 2019-03-26 19:05:00 CET (1553623500) to database 'weewx.sdb' >>>>> Mar 26 19:06:20 WeatherDuinoPi weewx[17760]: manager: Added record >>>>> 2019-03-26 19:05:00 CET (1553623500) to daily summary in 'weewx.sdb' >>>>> Mar 26 19:06:23 WeatherDuinoPi weewx[17760]: manager: Unable to add >>>>> record 2019-03-26 19:01:00 CET (1553623260) to database 'weewx.sdb': >>>>> UNIQUE constraint failed: archive.dateTime >>>>> >>>>> In my opinion the WeatherDuino is acting as described in the protocol >>>>> description. It is also a bit strange that WeeWx tries to add the "older" >>>>> records now to the database which are also saved in the same page of the >>>>> flash. But as it does not acknowledge the received page something strange >>>>> is going on at this point. >>>>> >>>>> Maybe you or somebody else owning an original Davis station has an >>>>> idea. >>>>> >>>>> Regards, >>>>> engolling >>>>> >>>>> >>>>> Am Montag, 25. März 2019 14:47:08 UTC+1 schrieb gjr80: >>>>>> >>>>>> The vantage driver is somewhat complex. How the vantage driver >>>>>> operates depends on whether hardware or software record generation is in >>>>>> use. If software record generation is in use then the vantage driver >>>>>> obtains loop packets from the console (using the LOOP command) and the >>>>>> loop >>>>>> packets are then passed to WeeWX. This goes on continuously. WeeWX >>>>>> accumulates the loop packets and at the end of each archive interval >>>>>> WeeWX >>>>>> synthesises an archive record from the accumulated loop packets. This >>>>>> archive record is then further processed by various services, saved to >>>>>> database and used as necessary in report generation. >>>>>> >>>>>> If hardware record generation is in use then the same loop packet >>>>>> processes occur as for software record generation (ie the driver passes >>>>>> loop packets to WeeWX and WeeWX accumulates them) but at the end of the >>>>>> archive interval instead of WeeWX synthesising an archive record from the >>>>>> accumulated loop packets, WeeWX asks the driver to obtain an archive >>>>>> record >>>>>> from the console. The driver uses the DMPAFT command to obtain the >>>>>> archive >>>>>> record and this archive record is passed back to WeeWX. This hardware >>>>>> archive record is further processed by various services, saved to >>>>>> database >>>>>> and used as necessary in report generation. >>>>>> >>>>>> As far as I know the DMPAFT command is only used at the end of the >>>>>> archive interval (eg every 5 minutes) and not very few seconds. I think >>>>>> to >>>>>> dig any further into the issue would take some detailed analysis of how >>>>>> the >>>>>> WeatherDuino responds to various commands. I suspect that somewhere the >>>>>> behaviour when the WeatherDuino receives the DMPAFT command is slightly >>>>>> different to the Davis console/logger. You should be able to instrument >>>>>> the >>>>>> vantage driver to log anything from raw data from the console/logger >>>>>> through to the decoded observations and packets/records. The good thing >>>>>> is >>>>>> the Davis protocols are documented and in the public domain so it should >>>>>> be >>>>>> easy to compare actual and expected responses. >>>>>> >>>>>> Gary >>>>>> >>>>>> On Sunday, 24 March 2019 09:03:28 UTC+10, engolling wrote: >>>>>>> >>>>>>> Hi Gary, >>>>>>> I checked the emulation of the WeatherDuino in the code and from my >>>>>>> point of view it seems to be implemented as in the protocol description. >>>>>>> Nevertheless it might differ in any behaviour which is not exactly >>>>>>> defined in the description. >>>>>>> >>>>>>> It's very hard for me to understand the python code of the vantage >>>>>>> driver since I am not familiar with the language. >>>>>>> I estimate that the vantage driver make the weewx engine to save an >>>>>>> archive record each time it gets one from the hardware. >>>>>>> According to my observations und the upper preconditions it seems >>>>>>> that archive records are received in the same frequency as the LOOP >>>>>>> packets. >>>>>>> This would mean that the vantage driver polls with a DMPAFT command >>>>>>> each view seconds. >>>>>>> Do you have any knowledge about this matter and do you know how it >>>>>>> should work correctly? >>>>>>> >>>>>>> To get this issue solved I think it would be necessary to get a >>>>>>> serial sniffer in place to be able to see the communications between >>>>>>> both >>>>>>> devices. >>>>>>> Can I simply add some logs in the source of the vantage driver, like >>>>>>> I did in my data service? >>>>>>> >>>>>>> Regards, >>>>>>> engolling >>>>>>> >>>>>>> Am Freitag, 22. März 2019 00:48:49 UTC+1 schrieb gjr80: >>>>>>>> >>>>>>>> It seems that the WeatherDuino emulation of the Davis logger may >>>>>>>> still have some issues. It's possible that the WeeWX vantage driver has >>>>>>>> some issues but the driver is mature and has been in use for many years >>>>>>>> communicating with actual Davis hardware without issue. One thing I >>>>>>>> have >>>>>>>> noticed with some of the other PWS software packages is that they seem >>>>>>>> to >>>>>>>> operate using the vantage loop packets only, whereas WeeWX has the >>>>>>>> option >>>>>>>> of operating with loop packets only (record_generation=software) >>>>>>>> or a combination of loop packets and hardware archive records ( >>>>>>>> record_generation=hardware). That may explain why no one else sees >>>>>>>> this issue. I am certain I have seen another WeeWX user who was using >>>>>>>> the >>>>>>>> WeatherDuino as a Davis emulator and there were some clear issues with >>>>>>>> the >>>>>>>> emulator. Unfortunately I cannot find that thread. >>>>>>>> >>>>>>>> You can obtain extra debug info in the log by setting debug=1 in >>>>>>>> weewx.conf, though for the vantage driver I doubt this will give >>>>>>>> you too much more information. The vantage driver does not have the >>>>>>>> debug >>>>>>>> capabilities that some other driver have so that you can see individual >>>>>>>> commands and responses. >>>>>>>> >>>>>>>> Gary >>>>>>>> >>>>>>>> On Wednesday, 20 March 2019 08:26:33 UTC+10, engolling wrote: >>>>>>>>> >>>>>>>>> Hi Gary, >>>>>>>>> >>>>>>>>> I disabled my data service and the errors still were there. See >>>>>>>>> the logfile in attachment. >>>>>>>>> >>>>>>>>> So I assumed that there is a problem with the vantage driver in >>>>>>>>> combination with my hardware. >>>>>>>>> So I changed in weewx.conf to >>>>>>>>> # If possible, new archive records are downloaded from the >>>>>>>>> station >>>>>>>>> # hardware. If the hardware does not support this, then new >>>>>>>>> archive >>>>>>>>> # records will be generated in software. >>>>>>>>> # Set the following to "software" to force software record >>>>>>>>> generation. >>>>>>>>> record_generation = software >>>>>>>>> this solves the problem with the logging errors. >>>>>>>>> >>>>>>>>> With the "unable to add records" problem solved I finished my data >>>>>>>>> service - also in attachment - which seems to be working now :). >>>>>>>>> >>>>>>>>> Back to the issue with the archive records - I read in the >>>>>>>>> customization guide, that in case of the record generation is set >>>>>>>>> hardware, >>>>>>>>> the archive records are directly genereted by the hardware. So I >>>>>>>>> assume, >>>>>>>>> that my hardware emitts an archive record to often. >>>>>>>>> As I have seen in the vantage protocol descriptin the archive data >>>>>>>>> can be requested via the DMPAFT command and sending the timestamp >>>>>>>>> of the last received archive record. >>>>>>>>> I think somewhere is a slight different behaviour between my >>>>>>>>> WeatherDuino and an original Davis device - because the developper of >>>>>>>>> the >>>>>>>>> WeatherDuino says the emulation is compatible with the original Davis >>>>>>>>> software and nearly each other weather software. I can confirm this. >>>>>>>>> Maybe it is necessary to use some kind of port sniffer to check >>>>>>>>> the communication or do you know if there are additional debug >>>>>>>>> outputs of >>>>>>>>> the driver available. >>>>>>>>> >>>>>>>>> Best regards, >>>>>>>>> engolling >>>>>>>>> >>>>>>>>> Am Dienstag, 19. März 2019 00:57:16 UTC+1 schrieb gjr80: >>>>>>>>>> >>>>>>>>>> On Tuesday, 19 March 2019 07:57:06 UTC+10, engolling wrote: >>>>>>>>>>> >>>>>>>>>>> Hi Gary, >>>>>>>>>>> thanks for your patience. >>>>>>>>>>> >>>>>>>>>>> This is not normal operation. Because something it triggering an >>>>>>>>>>>> archive record to be saved every few seconds the >>>>>>>>>>>> NEW_ARCHIVE_RECORD event >>>>>>>>>>>> is also triggered which causes your service to fire as well. The >>>>>>>>>>>> issue here >>>>>>>>>>>> is not your service but whatever is causing this archive record to >>>>>>>>>>>> be >>>>>>>>>>>> written every few seconds. I suggest we step back a bit and get a >>>>>>>>>>>> clear >>>>>>>>>>>> picture of how your system is configured and what it is running. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I was wondering where these errors where coming from and I >>>>>>>>>>> already tried to get more information about it. I found somewhere a >>>>>>>>>>> statement that is not too bad and might happen if a very narrow >>>>>>>>>>> saving >>>>>>>>>>> interval is used. But now I increasingly understand... >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Errors involving 'Unable to add record' are not good and are a >>>>>>>>>> sign something is wrong, at the very least you are wasting processor >>>>>>>>>> time, >>>>>>>>>> something you probably should avoid with such a short archive >>>>>>>>>> interval. A >>>>>>>>>> short archive interval should not be the cause in itself >>>>>>>>>> >>>>>>>>>> On Tuesday, 19 March 2019 08:21:12 UTC+10, engolling wrote: >>>>>>>>>> >>>>>>>>>>> One more thing - I do not own a original davis station - I'm >>>>>>>>>>> using a WeatherDuino, emulating a davis station for import of >>>>>>>>>>> "standard" >>>>>>>>>>> data. >>>>>>>>>>> https://www.meteocercal.info/forum/index.php >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> OK, not saying that is the issue but it is another complicating >>>>>>>>>> factor. Your weewx.conf looks fine and I don't see anything in >>>>>>>>>> your service that would be causing the archive record problem. Let's >>>>>>>>>> step >>>>>>>>>> right back to basics and disable your service so that you have WeeWX >>>>>>>>>> running just the vantage driver to talk the WeatherDuino. To do this: >>>>>>>>>> >>>>>>>>>> 1. edit weewx.conf, and change: >>>>>>>>>> >>>>>>>>>> [Engine] >>>>>>>>>> >>>>>>>>>> [[Services]] >>>>>>>>>> # This section specifies the services that should be >>>>>>>>>> run. They are >>>>>>>>>> # grouped by type, and the order of services within each >>>>>>>>>> group >>>>>>>>>> # determines the order in which the services will be run. >>>>>>>>>> prep_services = weewx.engine.StdTimeSynch >>>>>>>>>> data_services = user.WeeWx_WeatherDuino_Logger_plugin. >>>>>>>>>> WeeWxService, >>>>>>>>>> process_services = weewx.engine.StdConvert, weewx.engine. >>>>>>>>>> StdCalibrate, weewx.engine.StdQC, weewx.wxservices.StdWXCalculate >>>>>>>>>> >>>>>>>>>> to >>>>>>>>>> >>>>>>>>>> [Engine] >>>>>>>>>> >>>>>>>>>> [[Services]] >>>>>>>>>> # This section specifies the services that should be >>>>>>>>>> run. They are >>>>>>>>>> # grouped by type, and the order of services within each >>>>>>>>>> group >>>>>>>>>> # determines the order in which the services will be run. >>>>>>>>>> prep_services = weewx.engine.StdTimeSynch >>>>>>>>>> data_services = , # >>>>>>>>>> user.WeeWx_WeatherDuino_Logger_plugin.WeeWxService, >>>>>>>>>> process_services = weewx.engine.StdConvert, weewx.engine. >>>>>>>>>> StdCalibrate, weewx.engine.StdQC, weewx.wxservices.StdWXCalculate >>>>>>>>>> >>>>>>>>>> 2. save weewx.conf >>>>>>>>>> >>>>>>>>>> 3. restart WeeWX and let it run for 10 minutes or so then take a >>>>>>>>>> copy of the log from when WeeWX was restarted and post it here, make >>>>>>>>>> sure >>>>>>>>>> you capture the full WeeWX startup >>>>>>>>>> >>>>>>>>>> This should show us whether the WeatherDuino/vantage driver is >>>>>>>>>> causing the problem or if it is your service. >>>>>>>>>> >>>>>>>>>> Gary >>>>>>>>>> >>>>>>>>>> -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/weewx-user/94baea9a-ed38-4e52-9ea1-6ccdb5689960%40googlegroups.com > <https://groups.google.com/d/msgid/weewx-user/94baea9a-ed38-4e52-9ea1-6ccdb5689960%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zEAX-uGo2WicdKc24E9fr6Qu2PXhnWJ_bqpo%3DQELPz46pg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
