Hello Andrew, indeed it actually is only augmenting. But in case the archive data is gained from a DMPAFT packet it seems to be converted complete after augmenting. If the archive data is gained from a LOOP packet it converted before.
So the following things would probably solve the issue: - Only using record generation hardare - Database in US units My intention is that my plugin works with any configuration, so I would like to work out an solution for this. So my idea would be to find an indicator of the archive data is gained from an LOOP packet or a DMPAFT packet - so conversion could be done if necessary. Regards, engolling Am Sonntag, 26. Mai 2019 13:53:46 UTC+2 schrieb Andrew Milner: > > The script you posted SHOULD only be augmenting REC packets, and not be > affected by any LOOP packets. > > Maybe you should set record generation to either use hardware or use > software and not prefer hardware or prefer software. Your augmented data > should be use software I would have thought > > > > > > On Sunday, 26 May 2019 13:37:27 UTC+3, engolling wrote: >> >> Hallo, >> think I tracked the issue down. >> First it starts with record generation software. >> >> So each time a archive dataset is retreivedn form the hardware logger my >> data is augmented and then it runs apparently through the unit converter, >> converting it to metric units (in case of my augmented data coming in >> metric this is bad). >> This happens whenn WeeWx was off for some time. >> Each time an archive paket is gained from the LOOP data it was seemingly >> already converted my data is augmented and it is saved correctly to the >> database. >> >> So my question is simplified. Is it possible to determine if a archive >> record is archived of hardware logger data or LOOP data or can I influence >> the order of augmenting and converting? >> >> Best regards, >> engolling >> >> Am Mittwoch, 22. Mai 2019 23:20:08 UTC+2 schrieb engolling: >>> >>> Hello, >>> >>> of course I can give more details :-) >>> >>> So as you know I'm augmenting some extra data to my archive records >>> which I get from emulated davis loop packaged. >>> My file whichs contains always the latest data looks like this: >>> >>> https://github.com/menachers/WeatherDuino/blob/master/WeeWx_Plugin/WeeWx_Exp.txt >>> So there is written a signal name, a unit group and the actual data as >>> csv. >>> >>> The data is then augmented with the following script: >>> >>> https://github.com/menachers/WeatherDuino/blob/master/WeeWx_Plugin/WeeWx_WeatherDuino_Logger_plugin.py >>> called as data service in the weewx.conf >>> >>> record_generation is set to software, target_unit is set to metric. >>> >>> The augmented data is in °C and most of the time it is saved to the >>> database without being converted but sometimes a few datasets are double >>> converted. >>> >>> When record_generation is set to hardware the data is always converted, >>> so I'm exporting the data in US units and weewx converts it back. This is >>> no problem either. >>> But it is hard to handle when weewx jumps between no conversion and >>> conversion. >>> >>> Hopefully it is more clear now what I mean. >>> >>> Regards, >>> engolling >>> >>> >>> Am Montag, 20. Mai 2019 00:49:56 UTC+2 schrieb gjr80: >>>> >>>> Hi, >>>> >>>> Perhaps you could give us some more details as to how the extra >>>> temperature data is being obtained/provided/added to WeeWX, happy to see >>>> code/data formats etc - code usually gives an unambiguous picture, >>>> descriptions are often open to interpretation. I am having a little >>>> difficulty understanding how the numeric data is always being received but >>>> not the units, but perhaps that will become clear once you provide some >>>> more detail. >>>> >>>> Gary >>>> >>>> On Monday, 20 May 2019 05:55:18 UTC+10, engolling wrote: >>>>> >>>>> Hello, >>>>> >>>>> me again :). >>>>> Thank you for your last answer Thomas. Works fine. >>>>> >>>>> Now I have a new "thing" I do not understand. >>>>> As discussed above I'm augmenting different data - also rain data to >>>>> the archive records as described in the customization guide. >>>>> >>>>> I also augment temperature signals and sometimes they are converted to >>>>> metric variables and sometimes not. The problem is the signal is already >>>>> in >>>>> °C and should be in °C, but if it is converted it looks like this: >>>>> >>>>> >>>>> <http://www.google.com/url?q=http%3A%2F%2Fwetter2.kuntn-forum.de%2Fdaytempdew.png&sa=D&sntz=1&usg=AFQjCNGQMmsS1cPqrlD8F5tydTWEueDaEg> >>>>> The database is mertric. If I get for example a temperature of 22.61°C >>>>> which should be imported into the database directly and it is most time. >>>>> But sometimes I get -5.21°C (which is the result of the Farenheit Celsius >>>>> conversion of 22.61°C). >>>>> >>>>> My Raspberry Pi has a bad WIFI internet connection and FTP often fails >>>>> and it seems to be related to this. >>>>> >>>>> Could this be a reason and do you know how to handle the conversion? >>>>> >>>>> Regards, >>>>> engolling >>>>> >>>>> Am Dienstag, 7. Mai 2019 23:30:27 UTC+2 schrieb Thomas Keffer: >>>>>> >>>>>> Yes. That's certainly possible. >>>>>> >>>>>> -tk >>>>>> >>>>>> On Tue, May 7, 2019 at 1:35 PM engolling <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hello together, >>>>>>> >>>>>>> it's me again with hopefully a last question... >>>>>>> As you know I'm augmenting some extra data from my service to the >>>>>>> WeeWx archive records. But if archive records are loaded of the >>>>>>> stations >>>>>>> logger this data (which is in the future) is also augmented. >>>>>>> Is it possible to read the timestamp of the archive record beeing >>>>>>> saved at the moment and then decide if data should be augmented or not? >>>>>>> Im thinking of a code like this: >>>>>>> if dateTime_AugmentedData - event.record[dateTime] < 300: >>>>>>> --> augment_data >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Am Mittwoch, 3. April 2019 00:54:17 UTC+2 schrieb gjr80: >>>>>>>> >>>>>>>> Good that you have tracked down the issue. Running hardware record >>>>>>>> generation on a Davis station has it advantages, though maybe not so >>>>>>>> advantageous on emulated hardware. >>>>>>>> >>>>>>>> Regards plotting rain data from two sensors. The current WeeWX plot >>>>>>>> engine will certainly allow you to plot two obs in a bar graph plot, >>>>>>>> though >>>>>>>> I expect the outcome will be one bar plotted over the top of the >>>>>>>> other. Not >>>>>>>> really what you are seeking. There is no option to plot the bars >>>>>>>> adjacent >>>>>>>> to each other. It may be possible to modify the plot engine though I >>>>>>>> do not >>>>>>>> expect it would be a simple modification, the plot engine is fairly >>>>>>>> rudimentary. You may need to use an external charting package which >>>>>>>> should >>>>>>>> easily do what you want, though this too will not be a turn key >>>>>>>> solution as >>>>>>>> you will need to generate the necessary data for the package as well >>>>>>>> as >>>>>>>> integrate it into your web page(s). >>>>>>>> >>>>>>>> Gary >>>>>>>> >>>>>>>> On Wednesday, 3 April 2019 03:44:09 UTC+10, engolling wrote: >>>>>>>>> >>>>>>>>> Hi Gary, >>>>>>>>> >>>>>>>>> again thanks for your extended reply. >>>>>>>>> Meanwhile I added a lot of debugging code in the vantage driver >>>>>>>>> and the WeatherDuino code and was able to find out the reason what >>>>>>>>> happened. >>>>>>>>> Their is a buffer on the flash chip which is written before >>>>>>>>> storing the data in the flash page. And this buffer always had the >>>>>>>>> outdated >>>>>>>>> data of the page before in the last entries. >>>>>>>>> This lead to this behaviour that the actual written page contained >>>>>>>>> wrong data, but the page before and the page after was pretty fine... >>>>>>>>> >>>>>>>>> It took me quite some time to find the system behind this error. >>>>>>>>> I'm in touch with the developper of the WeatherDuino and this >>>>>>>>> issue will be fixed in the next firmware release so there will be >>>>>>>>> full >>>>>>>>> compatibility with WeeWx. >>>>>>>>> >>>>>>>>> Now that erverything is working I have a last question - do you, >>>>>>>>> or anybody else, think it is possible to draw a bar graph having the >>>>>>>>> rain >>>>>>>>> bars of two sensors right beside? >>>>>>>>> I did not find anything in the manual and if I define the diagramm >>>>>>>>> like in the line / dot graphs the two bars overlay, so that the bar >>>>>>>>> of the >>>>>>>>> second rain sensor is not visible, if it is a bit smaller. >>>>>>>>> >>>>>>>>> Thank you for all your replies. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> engolling >>>>>>>>> >>>>>>>>> Am Dienstag, 2. April 2019 16:01:11 UTC+2 schrieb gjr80: >>>>>>>>>> >>>>>>>>>> Sorry for being a little tardy in replying but I wanted to sit >>>>>>>>>> down and work my way through the Davis protocol. It seems everything >>>>>>>>>> is >>>>>>>>>> being followed, I do not know if the lack of a final acknowledgement >>>>>>>>>> is >>>>>>>>>> critical or not but what I would say is that the Davis driver works >>>>>>>>>> with >>>>>>>>>> real Davis hardware without issue. >>>>>>>>>> >>>>>>>>>> One thing I did wonder, the DMPAFT command downloads all archive >>>>>>>>>> records after a given timestamp. Archive records are downloaded a >>>>>>>>>> page at a >>>>>>>>>> time and a page contains 5 archive records. When we look at the >>>>>>>>>> earlier log >>>>>>>>>> extracts you have provided there seems to be groups of 5 archive >>>>>>>>>> records >>>>>>>>>> being processed (1 page?) but of course all that are already in the >>>>>>>>>> archive >>>>>>>>>> are rejected, for example: >>>>>>>>>> >>>>>>>>>> Mar 18 22:45:18 WeatherDuinoPi weewx[12220]: manager: Unable to >>>>>>>>>> add record 2019-03-18 22:43:00 CET (1552945380) to database >>>>>>>>>> 'weewx.sdb': UNIQUE constraint failed: archive.dateTime >>>>>>>>>> Mar 18 22:45:20 WeatherDuinoPi weewx[12220]: manager: Added >>>>>>>>>> record 2019-03-18 22:44:00 CET (1552945440) to database >>>>>>>>>> 'weewx.sdb' >>>>>>>>>> Mar 18 22:45:20 WeatherDuinoPi weewx[12220]: manager: Added >>>>>>>>>> record 2019-03-18 22:44:00 CET (1552945440) to daily summary in >>>>>>>>>> 'weewx.sdb' >>>>>>>>>> Mar 18 22:45:22 WeatherDuinoPi weewx[12220]: manager: Unable to >>>>>>>>>> add record 2019-03-18 22:40:00 CET (1552945200) to database >>>>>>>>>> 'weewx.sdb': UNIQUE constraint failed: archive.dateTime >>>>>>>>>> Mar 18 22:45:24 WeatherDuinoPi weewx[12220]: manager: Unable to >>>>>>>>>> add record 2019-03-18 22:41:00 CET (1552945260) to database >>>>>>>>>> 'weewx.sdb': UNIQUE constraint failed: archive.dateTime >>>>>>>>>> Mar 18 22:45:25 WeatherDuinoPi weewx[12220]: manager: Unable to >>>>>>>>>> add record 2019-03-18 22:42:00 CET (1552945320) to database >>>>>>>>>> 'weewx.sdb': UNIQUE constraint failed: archive.dateTime >>>>>>>>>> >>>>>>>>>> I did not see anywhere where you ran WeeWX directly >>>>>>>>>> <http://weewx.com/docs/usersguide.htm#Running_directly> and >>>>>>>>>> provided a few minutes of console output. Maybe that may shed more >>>>>>>>>> light on >>>>>>>>>> the problem. Failing that you may need to put some logging in the >>>>>>>>>> vantage >>>>>>>>>> driver to see exactly what the driver is reading and then sending to >>>>>>>>>> WeeWX. >>>>>>>>>> >>>>>>>>>> Gary >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wednesday, 27 March 2019 07:44:59 UTC+10, engolling wrote: >>>>>>>>>>> >>>>>>>>>>> Hi Gary, >>>>>>>>>>> >>>>>>>>>>> I tried to debug the system and therefore I built a "serial >>>>>>>>>>> sniffer" in form of an Arduino Mega passing through the data >>>>>>>>>>> between >>>>>>>>>>> serial0 and serial1 and printing all messages out on serial2. >>>>>>>>>>> I used HTERM to display it and I will provide some screenshots >>>>>>>>>>> for a better visibility. >>>>>>>>>>> Here we see a typical archive request: >>>>>>>>>>> >>>>>>>>>>> 1. DMPAFT\n - request of WeeWx >>>>>>>>>>> 2. WeatherDuino acknowledge with 0x06 >>>>>>>>>>> 3. WeeWx sends Date, Time and Checksum >>>>>>>>>>> 4. WeatherDuino acknowledge with 0x06 >>>>>>>>>>> 5. WeatherDuino sends number of pages and checksum >>>>>>>>>>> 6. WeeWx acknowledge >>>>>>>>>>> 7. WeatherDuino sends page and overhead - in sum 267 bytes as >>>>>>>>>>> expected. >>>>>>>>>>> *8. WeeWx sends LineFeed, but should acknowledge, send a CRC >>>>>>>>>>> failure or skip - that is strange...* >>>>>>>>>>> >>>>>>>>>>> See documentation line 35 and 36 >>>>>>>>>>> >>>>>>>>>>> https://www.davisinstruments.com/support/weather/download/VantageSerialProtocolDocs_v261.pdf >>>>>>>>>>> >>>>>>>>>>> 1 Minute and 360ms later the next DMPAFT command is sent. >>>>>>>>>>> >>>>>>>>>>> This example is starting today at 19:05:16 local time - this is >>>>>>>>>>> written into the logfile. >>>>>>>>>>> Mar 26 19:05:17 WeatherDuinoPi weewx[17760]: manager: Added >>>>>>>>>>> record 2019-03-26 19:04:00 CET (1553623440) to database >>>>>>>>>>> 'weewx.sdb' >>>>>>>>>>> Mar 26 19:05:17 WeatherDuinoPi weewx[17760]: manager: Added >>>>>>>>>>> record 2019-03-26 19:04:00 CET (1553623440) to daily summary in >>>>>>>>>>> 'weewx.sdb' >>>>>>>>>>> Mar 26 19:05:19 WeatherDuinoPi weewx[17760]: manager: Unable to >>>>>>>>>>> add record 2019-03-26 19:00:00 CET (1553623200) to database >>>>>>>>>>> 'weewx.sdb': UNIQUE constraint failed: archive.dateTime >>>>>>>>>>> Mar 26 19:05:21 WeatherDuinoPi weewx[17760]: manager: Unable to >>>>>>>>>>> add record 2019-03-26 19:01:00 CET (1553623260) to database >>>>>>>>>>> 'weewx.sdb': UNIQUE constraint failed: archive.dateTime >>>>>>>>>>> Mar 26 19:05:31 WeatherDuinoPi weewx[17760]: cheetahgenerator: >>>>>>>>>>> Generated 9 files for report StandardReport in 10.25 seconds >>>>>>>>>>> Mar 26 19:06:19 WeatherDuinoPi weewx[17760]: manager: Added >>>>>>>>>>> record 2019-03-26 19:05:00 CET (1553623500) to database >>>>>>>>>>> 'weewx.sdb' >>>>>>>>>>> Mar 26 19:06:20 WeatherDuinoPi weewx[17760]: manager: Added >>>>>>>>>>> record 2019-03-26 19:05:00 CET (1553623500) to daily summary in >>>>>>>>>>> 'weewx.sdb' >>>>>>>>>>> Mar<span style="color:rgb(0, >>>>>>>>>>> >>>>>>>>>> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/dfc8c592-4ed5-4e2f-aeab-715b2dee6e89%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
