Sorry for being a little tardy in replying but I wanted to sit down and work my way through the Davis protocol. It seems everything is being followed, I do not know if the lack of a final acknowledgement is critical or not but what I would say is that the Davis driver works with real Davis hardware without issue.
One thing I did wonder, the DMPAFT command downloads all archive records after a given timestamp. Archive records are downloaded a page at a time and a page contains 5 archive records. When we look at the earlier log extracts you have provided there seems to be groups of 5 archive records being processed (1 page?) but of course all that are already in the archive are rejected, for example: Mar 18 22:45:18 WeatherDuinoPi weewx[12220]: manager: Unable to add record 2019-03-18 22:43:00 CET (1552945380) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime Mar 18 22:45:20 WeatherDuinoPi weewx[12220]: manager: Added record 2019-03- 18 22:44:00 CET (1552945440) to database 'weewx.sdb' Mar 18 22:45:20 WeatherDuinoPi weewx[12220]: manager: Added record 2019-03- 18 22:44:00 CET (1552945440) to daily summary in 'weewx.sdb' Mar 18 22:45:22 WeatherDuinoPi weewx[12220]: manager: Unable to add record 2019-03-18 22:40:00 CET (1552945200) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime Mar 18 22:45:24 WeatherDuinoPi weewx[12220]: manager: Unable to add record 2019-03-18 22:41:00 CET (1552945260) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime Mar 18 22:45:25 WeatherDuinoPi weewx[12220]: manager: Unable to add record 2019-03-18 22:42:00 CET (1552945320) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime I did not see anywhere where you ran WeeWX directly <http://weewx.com/docs/usersguide.htm#Running_directly> and provided a few minutes of console output. Maybe that may shed more light on the problem. Failing that you may need to put some logging in the vantage driver to see exactly what the driver is reading and then sending to WeeWX. Gary On Wednesday, 27 March 2019 07:44:59 UTC+10, engolling wrote: > > Hi Gary, > > I tried to debug the system and therefore I built a "serial sniffer" in > form of an Arduino Mega passing through the data between serial0 and > serial1 and printing all messages out on serial2. > I used HTERM to display it and I will provide some screenshots for a > better visibility. > Here we see a typical archive request: > > 1. DMPAFT\n - request of WeeWx > 2. WeatherDuino acknowledge with 0x06 > 3. WeeWx sends Date, Time and Checksum > 4. WeatherDuino acknowledge with 0x06 > 5. WeatherDuino sends number of pages and checksum > 6. WeeWx acknowledge > 7. WeatherDuino sends page and overhead - in sum 267 bytes as expected. > *8. WeeWx sends LineFeed, but should acknowledge, send a CRC failure or > skip - that is strange...* > > See documentation line 35 and 36 > > https://www.davisinstruments.com/support/weather/download/VantageSerialProtocolDocs_v261.pdf > > 1 Minute and 360ms later the next DMPAFT command is sent. > > This example is starting today at 19:05:16 local time - this is written > into the logfile. > Mar 26 19:05:17 WeatherDuinoPi weewx[17760]: manager: Added record 2019-03 > -26 19:04:00 CET (1553623440) to database 'weewx.sdb' > Mar 26 19:05:17 WeatherDuinoPi weewx[17760]: manager: Added record 2019-03 > -26 19:04:00 CET (1553623440) to daily summary in 'weewx.sdb' > Mar 26 19:05:19 WeatherDuinoPi weewx[17760]: manager: Unable to add > record 2019-03-26 19:00:00 CET (1553623200) to database 'weewx.sdb': > UNIQUE constraint failed: archive.dateTime > Mar 26 19:05:21 WeatherDuinoPi weewx[17760]: manager: Unable to add > record 2019-03-26 19:01:00 CET (1553623260) to database 'weewx.sdb': > UNIQUE constraint failed: archive.dateTime > Mar 26 19:05:31 WeatherDuinoPi weewx[17760]: cheetahgenerator: Generated 9 > files for report StandardReport in 10.25 seconds > Mar 26 19:06:19 WeatherDuinoPi weewx[17760]: manager: Added record 2019-03 > -26 19:05:00 CET (1553623500) to database 'weewx.sdb' > Mar 26 19:06:20 WeatherDuinoPi weewx[17760]: manager: Added record 2019-03 > -26 19:05:00 CET (1553623500) to daily summary in 'weewx.sdb' > Mar 26 19:06:23 WeatherDuinoPi weewx[17760]: manager: Unable to add > record 2019-03-26 19:01:00 CET (1553623260) to database 'weewx.sdb': > UNIQUE constraint failed: archive.dateTime > > In my opinion the WeatherDuino is acting as described in the protocol > description. It is also a bit strange that WeeWx tries to add the "older" > records now to the database which are also saved in the same page of the > flash. But as it does not acknowledge the received page something strange > is going on at this point. > > Maybe you or somebody else owning an original Davis station has an idea. > > Regards, > engolling > > > Am Montag, 25. März 2019 14:47:08 UTC+1 schrieb gjr80: >> >> The vantage driver is somewhat complex. How the vantage driver operates >> depends on whether hardware or software record generation is in use. If >> software record generation is in use then the vantage driver obtains loop >> packets from the console (using the LOOP command) and the loop packets are >> then passed to WeeWX. This goes on continuously. WeeWX accumulates the loop >> packets and at the end of each archive interval WeeWX synthesises an >> archive record from the accumulated loop packets. This archive record is >> then further processed by various services, saved to database and used as >> necessary in report generation. >> >> If hardware record generation is in use then the same loop packet >> processes occur as for software record generation (ie the driver passes >> loop packets to WeeWX and WeeWX accumulates them) but at the end of the >> archive interval instead of WeeWX synthesising an archive record from the >> accumulated loop packets, WeeWX asks the driver to obtain an archive record >> from the console. The driver uses the DMPAFT command to obtain the archive >> record and this archive record is passed back to WeeWX. This hardware >> archive record is further processed by various services, saved to database >> and used as necessary in report generation. >> >> As far as I know the DMPAFT command is only used at the end of the >> archive interval (eg every 5 minutes) and not very few seconds. I think to >> dig any further into the issue would take some detailed analysis of how the >> WeatherDuino responds to various commands. I suspect that somewhere the >> behaviour when the WeatherDuino receives the DMPAFT command is slightly >> different to the Davis console/logger. You should be able to instrument the >> vantage driver to log anything from raw data from the console/logger >> through to the decoded observations and packets/records. The good thing is >> the Davis protocols are documented and in the public domain so it should be >> easy to compare actual and expected responses. >> >> Gary >> >> On Sunday, 24 March 2019 09:03:28 UTC+10, engolling wrote: >>> >>> Hi Gary, >>> I checked the emulation of the WeatherDuino in the code and from my >>> point of view it seems to be implemented as in the protocol description. >>> Nevertheless it might differ in any behaviour which is not exactly >>> defined in the description. >>> >>> It's very hard for me to understand the python code of the vantage >>> driver since I am not familiar with the language. >>> I estimate that the vantage driver make the weewx engine to save an >>> archive record each time it gets one from the hardware. >>> According to my observations und the upper preconditions it seems that >>> archive records are received in the same frequency as the LOOP packets. >>> This would mean that the vantage driver polls with a DMPAFT command >>> each view seconds. >>> Do you have any knowledge about this matter and do you know how it >>> should work correctly? >>> >>> To get this issue solved I think it would be necessary to get a serial >>> sniffer in place to be able to see the communications between both devices. >>> Can I simply add some logs in the source of the vantage driver, like I >>> did in my data service? >>> >>> Regards, >>> engolling >>> >>> Am Freitag, 22. März 2019 00:48:49 UTC+1 schrieb gjr80: >>>> >>>> It seems that the WeatherDuino emulation of the Davis logger may still >>>> have some issues. It's possible that the WeeWX vantage driver has some >>>> issues but the driver is mature and has been in use for many years >>>> communicating with actual Davis hardware without issue. One thing I have >>>> noticed with some of the other PWS software packages is that they seem to >>>> operate using the vantage loop packets only, whereas WeeWX has the option >>>> of operating with loop packets only (record_generation=software) or a >>>> combination of loop packets and hardware archive records ( >>>> record_generation=hardware). That may explain why no one else sees >>>> this issue. I am certain I have seen another WeeWX user who was using the >>>> WeatherDuino as a Davis emulator and there were some clear issues with the >>>> emulator. Unfortunately I cannot find that thread. >>>> >>>> You can obtain extra debug info in the log by setting debug=1 in >>>> weewx.conf, though for the vantage driver I doubt this will give you >>>> too much more information. The vantage driver does not have the debug >>>> capabilities that some other driver have so that you can see individual >>>> commands and responses. >>>> >>>> Gary >>>> >>>> On Wednesday, 20 March 2019 08:26:33 UTC+10, engolling wrote: >>>>> >>>>> Hi Gary, >>>>> >>>>> I disabled my data service and the errors still were there. See the >>>>> logfile in attachment. >>>>> >>>>> So I assumed that there is a problem with the vantage driver in >>>>> combination with my hardware. >>>>> So I changed in weewx.conf to >>>>> # If possible, new archive records are downloaded from the station >>>>> # hardware. If the hardware does not support this, then new >>>>> archive >>>>> # records will be generated in software. >>>>> # Set the following to "software" to force software record >>>>> generation. >>>>> record_generation = software >>>>> this solves the problem with the logging errors. >>>>> >>>>> With the "unable to add records" problem solved I finished my data >>>>> service - also in attachment - which seems to be working now :). >>>>> >>>>> Back to the issue with the archive records - I read in the >>>>> customization guide, that in case of the record generation is set >>>>> hardware, >>>>> the archive records are directly genereted by the hardware. So I assume, >>>>> that my hardware emitts an archive record to often. >>>>> As I have seen in the vantage protocol descriptin the archive data can >>>>> be requested via the DMPAFT command and sending the timestamp of the >>>>> last received archive record. >>>>> I think somewhere is a slight different behaviour between my >>>>> WeatherDuino and an original Davis device - because the developper of the >>>>> WeatherDuino says the emulation is compatible with the original Davis >>>>> software and nearly each other weather software. I can confirm this. >>>>> Maybe it is necessary to use some kind of port sniffer to check the >>>>> communication or do you know if there are additional debug outputs of the >>>>> driver available. >>>>> >>>>> Best regards, >>>>> engolling >>>>> >>>>> Am Dienstag, 19. März 2019 00:57:16 UTC+1 schrieb gjr80: >>>>>> >>>>>> On Tuesday, 19 March 2019 07:57:06 UTC+10, engolling wrote: >>>>>>> >>>>>>> Hi Gary, >>>>>>> thanks for your patience. >>>>>>> >>>>>>> This is not normal operation. Because something it triggering an >>>>>>>> archive record to be saved every few seconds the NEW_ARCHIVE_RECORD >>>>>>>> event >>>>>>>> is also triggered which causes your service to fire as well. The issue >>>>>>>> here >>>>>>>> is not your service but whatever is causing this archive record to be >>>>>>>> written every few seconds. I suggest we step back a bit and get a >>>>>>>> clear >>>>>>>> picture of how your system is configured and what it is running. >>>>>>>> >>>>>>> >>>>>>> I was wondering where these errors where coming from and I already >>>>>>> tried to get more information about it. I found somewhere a statement >>>>>>> that >>>>>>> is not too bad and might happen if a very narrow saving interval is >>>>>>> used. >>>>>>> But now I increasingly understand... >>>>>>> >>>>>> >>>>>> Errors involving 'Unable to add record' are not good and are a sign >>>>>> something is wrong, at the very least you are wasting processor time, >>>>>> something you probably should avoid with such a short archive interval. >>>>>> A >>>>>> short archive interval should not be the cause in itself >>>>>> >>>>>> On Tuesday, 19 March 2019 08:21:12 UTC+10, engolling wrote: >>>>>> >>>>>>> One more thing - I do not own a original davis station - I'm using a >>>>>>> WeatherDuino, emulating a davis station for import of "standard" data. >>>>>>> https://www.meteocercal.info/forum/index.php >>>>>>> >>>>>> >>>>>> OK, not saying that is the issue but it is another complicating >>>>>> factor. Your weewx.conf looks fine and I don't see anything in your >>>>>> service that would be causing the archive record problem. Let's step >>>>>> right >>>>>> back to basics and disable your service so that you have WeeWX running >>>>>> just >>>>>> the vantage driver to talk the WeatherDuino. To do this: >>>>>> >>>>>> 1. edit weewx.conf, and change: >>>>>> >>>>>> [Engine] >>>>>> >>>>>> [[Services]] >>>>>> # This section specifies the services that should be run. >>>>>> They are >>>>>> # grouped by type, and the order of services within each >>>>>> group >>>>>> # determines the order in which the services will be run. >>>>>> prep_services = weewx.engine.StdTimeSynch >>>>>> data_services = user.WeeWx_WeatherDuino_Logger_plugin. >>>>>> WeeWxService, >>>>>> process_services = weewx.engine.StdConvert, weewx.engine. >>>>>> StdCalibrate, weewx.engine.StdQC, weewx.wxservices.StdWXCalculate >>>>>> >>>>>> to >>>>>> >>>>>> [Engine] >>>>>> >>>>>> [[Services]] >>>>>> # This section specifies the services that should be run. >>>>>> They are >>>>>> # grouped by type, and the order of services within each >>>>>> group >>>>>> # determines the order in which the services will be run. >>>>>> prep_services = weewx.engine.StdTimeSynch >>>>>> data_services = , # >>>>>> user.WeeWx_WeatherDuino_Logger_plugin.WeeWxService, >>>>>> process_services = weewx.engine.StdConvert, weewx.engine. >>>>>> StdCalibrate, weewx.engine.StdQC, weewx.wxservices.StdWXCalculate >>>>>> >>>>>> 2. save weewx.conf >>>>>> >>>>>> 3. restart WeeWX and let it run for 10 minutes or so then take a copy >>>>>> of the log from when WeeWX was restarted and post it here, make sure you >>>>>> capture the full WeeWX startup >>>>>> >>>>>> This should show us whether the WeatherDuino/vantage driver is >>>>>> causing the problem or if it is your service. >>>>>> >>>>>> Gary >>>>>> >>>>>> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
