The number one rule when augmenting archive records or loop packets with
data is that you must check the unit system used in the record or packet
(ie check the usUnits field) you are adding data to and make sure that any
data you do add is in the same units as used by that unit system. If you
fail to do this you may be fortunate to have something that works, but it
will be good luck rather than good planning and the resulting system will
be fragile with any future changes likely to result in unit conversion
inconsistencies. For example, say you have a service that is adding a rain
value to an archive record and your service produces a variable rainfall
that contains the rain data to be added in mm. The archive record field you
wish to create is 'our_new_field'. You might use some code in your service
like the following to actually augment the archive record (untested):
import weewx
if event.record['usUnits'] == weewx.US:
# archive record uses US unit system so we need rainfall in inches
event.record['our_new_field'] = rainfall/25.4 if rainfall is not None
else None
elif event.record['usUnits'] == weewx.METRIC:
# archive record uses the Metric unit system so we need rainfall in cm
event.record['our_new_field'] = rainfall/10.0 if rainfall is not None
else None
elif event.record['usUnits'] == weewx.METRICWX:
# archive record uses the MetricWX unit system so we need rainfall in
mm - leave as is
event.record['our_new_field'] = rainfall if rainfall is not None else
None
or if you want to be a bit more sophisticated you can use the internal
WeeWX unit conversion routines to do the unit conversion for you (much more
compact code and useful for more complex/less well known conversions)
(again untested):
import weewx.units
# express our rainfall value as a ValueTuple
rainfall_vt = weewx.units.ValueTuple(rainfall, 'mm', 'group_rain')
# now augment the archive record with the appropriately converted value
event.record['our_new_field'] = weewx.units.convertStd(rainfall_vt, event.
record['usUnits'])
The above examples augment an archive record but they apply equally to loop
packets. In the cases above we knew the incoming data was always in mm, if
that is not the case and you need to handle the incoming data being in any
one of a number of units then you will need some extra code to identify the
incoming units and ensure the corect conversions are applied.
Gary
On Thursday, 23 May 2019 07:20:08 UTC+10, engolling wrote:
>
> Hello,
>
> of course I can give more details :-)
>
> So as you know I'm augmenting some extra data to my archive records which
> I get from emulated davis loop packaged.
> My file whichs contains always the latest data looks like this:
>
> https://github.com/menachers/WeatherDuino/blob/master/WeeWx_Plugin/WeeWx_Exp.txt
> So there is written a signal name, a unit group and the actual data as csv.
>
> The data is then augmented with the following script:
>
> https://github.com/menachers/WeatherDuino/blob/master/WeeWx_Plugin/WeeWx_WeatherDuino_Logger_plugin.py
> called as data service in the weewx.conf
>
> record_generation is set to software, target_unit is set to metric.
>
> The augmented data is in °C and most of the time it is saved to the
> database without being converted but sometimes a few datasets are double
> converted.
>
> When record_generation is set to hardware the data is always converted, so
> I'm exporting the data in US units and weewx converts it back. This is no
> problem either.
> But it is hard to handle when weewx jumps between no conversion and
> conversion.
>
> Hopefully it is more clear now what I mean.
>
> Regards,
> engolling
>
>
> Am Montag, 20. Mai 2019 00:49:56 UTC+2 schrieb gjr80:
>>
>> Hi,
>>
>> Perhaps you could give us some more details as to how the extra
>> temperature data is being obtained/provided/added to WeeWX, happy to see
>> code/data formats etc - code usually gives an unambiguous picture,
>> descriptions are often open to interpretation. I am having a little
>> difficulty understanding how the numeric data is always being received but
>> not the units, but perhaps that will become clear once you provide some
>> more detail.
>>
>> Gary
>>
>> On Monday, 20 May 2019 05:55:18 UTC+10, engolling wrote:
>>>
>>> Hello,
>>>
>>> me again :).
>>> Thank you for your last answer Thomas. Works fine.
>>>
>>> Now I have a new "thing" I do not understand.
>>> As discussed above I'm augmenting different data - also rain data to the
>>> archive records as described in the customization guide.
>>>
>>> I also augment temperature signals and sometimes they are converted to
>>> metric variables and sometimes not. The problem is the signal is already in
>>> °C and should be in °C, but if it is converted it looks like this:
>>>
>>>
>>> <http://www.google.com/url?q=http%3A%2F%2Fwetter2.kuntn-forum.de%2Fdaytempdew.png&sa=D&sntz=1&usg=AFQjCNGQMmsS1cPqrlD8F5tydTWEueDaEg>
>>> The database is mertric. If I get for example a temperature of 22.61°C
>>> which should be imported into the database directly and it is most time.
>>> But sometimes I get -5.21°C (which is the result of the Farenheit Celsius
>>> conversion of 22.61°C).
>>>
>>> My Raspberry Pi has a bad WIFI internet connection and FTP often fails
>>> and it seems to be related to this.
>>>
>>> Could this be a reason and do you know how to handle the conversion?
>>>
>>> Regards,
>>> engolling
>>>
>>> Am Dienstag, 7. Mai 2019 23:30:27 UTC+2 schrieb Thomas Keffer:
>>>>
>>>> Yes. That's certainly possible.
>>>>
>>>> -tk
>>>>
>>>> On Tue, May 7, 2019 at 1:35 PM engolling <[email protected]> wrote:
>>>>
>>>>> Hello together,
>>>>>
>>>>> it's me again with hopefully a last question...
>>>>> As you know I'm augmenting some extra data from my service to the
>>>>> WeeWx archive records. But if archive records are loaded of the stations
>>>>> logger this data (which is in the future) is also augmented.
>>>>> Is it possible to read the timestamp of the archive record beeing
>>>>> saved at the moment and then decide if data should be augmented or not?
>>>>> Im thinking of a code like this:
>>>>> if dateTime_AugmentedData - event.record[dateTime] < 300:
>>>>> --> augment_data
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Am Mittwoch, 3. April 2019 00:54:17 UTC+2 schrieb gjr80:
>>>>>>
>>>>>> Good that you have tracked down the issue. Running hardware record
>>>>>> generation on a Davis station has it advantages, though maybe not so
>>>>>> advantageous on emulated hardware.
>>>>>>
>>>>>> Regards plotting rain data from two sensors. The current WeeWX plot
>>>>>> engine will certainly allow you to plot two obs in a bar graph plot,
>>>>>> though
>>>>>> I expect the outcome will be one bar plotted over the top of the other.
>>>>>> Not
>>>>>> really what you are seeking. There is no option to plot the bars
>>>>>> adjacent
>>>>>> to each other. It may be possible to modify the plot engine though I do
>>>>>> not
>>>>>> expect it would be a simple modification, the plot engine is fairly
>>>>>> rudimentary. You may need to use an external charting package which
>>>>>> should
>>>>>> easily do what you want, though this too will not be a turn key solution
>>>>>> as
>>>>>> you will need to generate the necessary data for the package as well as
>>>>>> integrate it into your web page(s).
>>>>>>
>>>>>> Gary
>>>>>>
>>>>>> On Wednesday, 3 April 2019 03:44:09 UTC+10, engolling wrote:
>>>>>>>
>>>>>>> Hi Gary,
>>>>>>>
>>>>>>> again thanks for your extended reply.
>>>>>>> Meanwhile I added a lot of debugging code in the vantage driver and
>>>>>>> the WeatherDuino code and was able to find out the reason what happened.
>>>>>>> Their is a buffer on the flash chip which is written before storing
>>>>>>> the data in the flash page. And this buffer always had the outdated
>>>>>>> data of
>>>>>>> the page before in the last entries.
>>>>>>> This lead to this behaviour that the actual written page contained
>>>>>>> wrong data, but the page before and the page after was pretty fine...
>>>>>>>
>>>>>>> It took me quite some time to find the system behind this error.
>>>>>>> I'm in touch with the developper of the WeatherDuino and this issue
>>>>>>> will be fixed in the next firmware release so there will be full
>>>>>>> compatibility with WeeWx.
>>>>>>>
>>>>>>> Now that erverything is working I have a last question - do you, or
>>>>>>> anybody else, think it is possible to draw a bar graph having the rain
>>>>>>> bars
>>>>>>> of two sensors right beside?
>>>>>>> I did not find anything in the manual and if I define the diagramm
>>>>>>> like in the line / dot graphs the two bars overlay, so that the bar of
>>>>>>> the
>>>>>>> second rain sensor is not visible, if it is a bit smaller.
>>>>>>>
>>>>>>> Thank you for all your replies.
>>>>>>>
>>>>>>> Regards,
>>>>>>> engolling
>>>>>>>
>>>>>>> Am Dienstag, 2. April 2019 16:01:11 UTC+2 schrieb gjr80:
>>>>>>>>
>>>>>>>> Sorry for being a little tardy in replying but I wanted to sit down
>>>>>>>> and work my way through the Davis protocol. It seems everything is
>>>>>>>> being
>>>>>>>> followed, I do not know if the lack of a final acknowledgement is
>>>>>>>> critical
>>>>>>>> or not but what I would say is that the Davis driver works with real
>>>>>>>> Davis
>>>>>>>> hardware without issue.
>>>>>>>>
>>>>>>>> One thing I did wonder, the DMPAFT command downloads all archive
>>>>>>>> records after a given timestamp. Archive records are downloaded a page
>>>>>>>> at a
>>>>>>>> time and a page contains 5 archive records. When we look at the
>>>>>>>> earlier log
>>>>>>>> extracts you have provided there seems to be groups of 5 archive
>>>>>>>> records
>>>>>>>> being processed (1 page?) but of course all that are already in the
>>>>>>>> archive
>>>>>>>> are rejected, for example:
>>>>>>>>
>>>>>>>> Mar 18 22:45:18 WeatherDuinoPi weewx[12220]: manager: Unable to
>>>>>>>> add record 2019-03-18 22:43:00 CET (1552945380) to database
>>>>>>>> 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
>>>>>>>> Mar 18 22:45:20 WeatherDuinoPi weewx[12220]: manager: Added record
>>>>>>>> 2019-03-18 22:44:00 CET (1552945440) to database 'weewx.sdb'
>>>>>>>> Mar 18 22:45:20 WeatherDuinoPi weewx[12220]: manager: Added record
>>>>>>>> 2019-03-18 22:44:00 CET (1552945440) to daily summary in
>>>>>>>> 'weewx.sdb'
>>>>>>>> Mar 18 22:45:22 WeatherDuinoPi weewx[12220]: manager: Unable to
>>>>>>>> add record 2019-03-18 22:40:00 CET (1552945200) to database
>>>>>>>> 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
>>>>>>>> Mar 18 22:45:24 WeatherDuinoPi weewx[12220]: manager: Unable to
>>>>>>>> add record 2019-03-18 22:41:00 CET (1552945260) to database
>>>>>>>> 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
>>>>>>>> Mar 18 22:45:25 WeatherDuinoPi weewx[12220]: manager: Unable to
>>>>>>>> add record 2019-03-18 22:42:00 CET (1552945320) to database
>>>>>>>> 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
>>>>>>>>
>>>>>>>> I did not see anywhere where you ran WeeWX directly
>>>>>>>> <http://weewx.com/docs/usersguide.htm#Running_directly> and
>>>>>>>> provided a few minutes of console output. Maybe that may shed more
>>>>>>>> light on
>>>>>>>> the problem. Failing that you may need to put some logging in the
>>>>>>>> vantage
>>>>>>>> driver to see exactly what the driver is reading and then sending to
>>>>>>>> WeeWX.
>>>>>>>>
>>>>>>>> Gary
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wednesday, 27 March 2019 07:44:59 UTC+10, engolling wrote:
>>>>>>>>>
>>>>>>>>> Hi Gary,
>>>>>>>>>
>>>>>>>>> I tried to debug the system and therefore I built a "serial
>>>>>>>>> sniffer" in form of an Arduino Mega passing through the data between
>>>>>>>>> serial0 and serial1 and printing all messages out on serial2.
>>>>>>>>> I used HTERM to display it and I will provide some screenshots for
>>>>>>>>> a better visibility.
>>>>>>>>> Here we see a typical archive request:
>>>>>>>>>
>>>>>>>>> 1. DMPAFT\n - request of WeeWx
>>>>>>>>> 2. WeatherDuino acknowledge with 0x06
>>>>>>>>> 3. WeeWx sends Date, Time and Checksum
>>>>>>>>> 4. WeatherDuino acknowledge with 0x06
>>>>>>>>> 5. WeatherDuino sends number of pages and checksum
>>>>>>>>> 6. WeeWx acknowledge
>>>>>>>>> 7. WeatherDuino sends page and overhead - in sum 267 bytes as
>>>>>>>>> expected.
>>>>>>>>> *8. WeeWx sends LineFeed, but should acknowledge, send a CRC
>>>>>>>>> failure or skip - that is strange...*
>>>>>>>>>
>>>>>>>>> See documentation line 35 and 36
>>>>>>>>>
>>>>>>>>> https://www.davisinstruments.com/support/weather/download/VantageSerialProtocolDocs_v261.pdf
>>>>>>>>>
>>>>>>>>> 1 Minute and 360ms later the next DMPAFT command is sent.
>>>>>>>>>
>>>>>>>>> This example is starting today at 19:05:16 local time - this is
>>>>>>>>> written into the logfile.
>>>>>>>>> Mar 26 19:05:17 WeatherDuinoPi weewx[17760]: manager: Added
>>>>>>>>> record 2019-03-26 19:04:00 CET (1553623440) to database
>>>>>>>>> 'weewx.sdb'
>>>>>>>>> Mar 26 19:05:17 WeatherDuinoPi weewx[17760]: manager: Added
>>>>>>>>> record 2019-03-26 19:04:00 CET (1553623440) to daily summary in
>>>>>>>>> 'weewx.sdb'
>>>>>>>>> Mar 26 19:05:19 WeatherDuinoPi weewx[17760]: manager: Unable to
>>>>>>>>> add record 2019-03-26 19:00:00 CET (1553623200) to database
>>>>>>>>> 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
>>>>>>>>> Mar 26 19:05:21 WeatherDuinoPi weewx[17760]: manager: Unable to
>>>>>>>>> add record 2019-03-26 19:01:00 CET (1553623260) to database
>>>>>>>>> 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
>>>>>>>>> Mar 26 19:05:31 WeatherDuinoPi weewx[17760]: cheetahgenerator:
>>>>>>>>> Generated 9 files for report StandardReport in 10.25 seconds
>>>>>>>>> Mar 26 19:06:19 WeatherDuinoPi weewx[17760]: manager: Added
>>>>>>>>> record 2019-03-26 19:05:00 CET (1553623500) to database
>>>>>>>>> 'weewx.sdb'
>>>>>>>>> Mar 26 19:06:20 WeatherDuinoPi weewx[17760]: manager: Added
>>>>>>>>> record 2019-03-26 19:05:00 CET (1553623500) to daily summary in
>>>>>>>>> 'weewx.sdb'
>>>>>>>>> Mar 26 19:06:23 WeatherDuinoPi weewx[17760]: manager: Unable to
>>>>>>>>> add record 2019-03-26 19:01:00 CET (1553623260) to database
>>>>>>>>> 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
>>>>>>>>>
>>>>>>>>> In my opinion the WeatherDuino is acting as described in the
>>>>>>>>> protocol description. It is also a bit strange that WeeWx tries to
>>>>>>>>> add the
>>>>>>>>> "older" records now to the database which are also saved in the same
>>>>>>>>> page
>>>>>>>>> of the flash. But as it does not acknowledge the received page
>>>>>>>>> something
>>>>>>>>> strange is going on at this point.
>>>>>>>>>
>>>>>>>>> Maybe you or somebody else owning an original Davis station has an
>>>>>>>>> idea.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> engolling
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Am Montag, 25. März 2019 14:47:08 UTC+1 schrieb gjr80:
>>>>>>>>>>
>>>>>>>>>> The vantage driver is somewhat complex. How the vantage driver
>>>>>>>>>> operates depends on whether hardware or software record generation
>>>>>>>>>> is in
>>>>>>>>>> use. If software record generation is in use then the vantage driver
>>>>>>>>>> obtains loop packets from the console (using the LOOP command) and
>>>>>>>>>> the loop
>>>>>>>>>> packets are then passed to WeeWX. This goes on
>>>>>>>>>>
>>>>>>>>>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/f299e0a4-8423-463a-8628-546a0c942dfa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
