Hi Gary,
maybe I have found my problem after looking at my code and your advices
again.
Because it seems that I have bound my additional signals to the archive
record and not to the LOOP packet
class WeeWxService(StdService):
def __init__(self, engine, config_dict):
super(WeeWxService, self).__init__(engine, config_dict)
d = config_dict.get('WeatherDuino_logger_service', {})
self.filename = d.get('filename',
'/home/pi/WeatherDuino/WeeWx_Exp.txt')
syslog.syslog(syslog.LOG_INFO, "WeatherDuino: using %s" %
self.filename)
self.bind(weewx.NEW_ARCHIVE_RECORD, self.read_file)
self.last_rain = []
But the logfile tells me my plugin is executed in a frequency like the loop
packets:
Mar 17 23:22:19 WeatherDuinoPi weewx[7972]: WeatherDuino: Valid values found
Mar 17 23:22:23 WeatherDuinoPi weewx[7972]: manager: Added record
2019-03-17 23:21:00 CET (1552861260) to database 'weewx.sdb'
Mar 17 23:22:23 WeatherDuinoPi weewx[7972]: manager: Added record
2019-03-17 23:21:00 CET (1552861260) to daily summary in 'weewx.sdb'
Mar 17 23:22:24 WeatherDuinoPi weewx[7972]: WeatherDuino: Valid values found
Mar 17 23:22:27 WeatherDuinoPi weewx[7972]: manager: Unable to add record
2019-03-17 23:17:00 CET (1552861020) to database 'weewx.sdb': UNIQUE
constraint failed: archive.dateTime
Mar 17 23:22:27 WeatherDuinoPi weewx[7972]: WeatherDuino: Valid values found
Mar 17 23:22:30 WeatherDuinoPi weewx[7972]: manager: Unable to add record
2019-03-17 23:18:00 CET (1552861080) to database 'weewx.sdb': UNIQUE
constraint failed: archive.dateTime
Mar 17 23:22:30 WeatherDuinoPi weewx[7972]: WeatherDuino: Valid values found
Mar 17 23:22:33 WeatherDuinoPi weewx[7972]: manager: Unable to add record
2019-03-17 23:19:00 CET (1552861140) to database 'weewx.sdb': UNIQUE
constraint failed: archive.dateTime
Mar 17 23:22:33 WeatherDuinoPi weewx[7972]: WeatherDuino: Valid values found
According to your answers this should be the problem because in this case I
have to take care that the rain is only augmented once each minute, right?
Should I change the binding to
self.bind(weewx.NEW_LOOP_PACKET, self.read_file)
?
Moreover I try to simplify my code as you have proposed, so I added the
last line in the init function (is it a function ?) of the class.
My idea was to make a empty list, which I can expand as I want because
there can be up to 4 rain values - is this a reasonable solution or do I
have to make a list with 4 entries at the beginning?
Thanks for all your help.
Best regards,
engolling
Am Sonntag, 17. März 2019 02:32:13 UTC+1 schrieb gjr80:
>
> OK, your archive schema appears fine, that rules out one variable. WeeWX
> services are loaded at WeeWX startup and are retained as long as WeeWX
> continues to run so any properties you add are retained. Have a go at
> simplifying your service then I suggest you run WeeWX directly
> <http://weewx.com/docs/usersguide.htm#Running_directly> so you can see
> the loop packets and archive records that are generated. It should be clear
> then what is/is not going on.
>
> Gary
>
> On Sunday, 17 March 2019 09:40:03 UTC+10, engolling wrote:
>>
>> Hello Gary,
>>
>>
>>
>> thanks fot your efforts. You are right with your description how my code
>> should work.
>>
>> Let's say my main problem is that I know how I construct software in
>> principle, but my python knowledgde is not very good, so in order to safe
>> the last cumulative rain value my only solution was to write it into a
>> file. Otherwise I tought the value might get lost between executing the
>> data service
>>
>>
>>
>> I will try to 'buffer' my last rain value in a variable as you describe
>> it and give you feedback.
>>
>>
>> But I still think WeeWx does not accumulate correctly in my case, because
>> im nearly 100% sure, that I have checked that my (indeed strange) code
>> emits the values correctly.
>>
>>
>> My schema looks like this - I think it is fine.
>>
>>> sqlite> .schema archive
>>> CREATE TABLE archive (`dateTime` INTEGER NOT NULL UNIQUE PRIMARY KEY,
>>> `usUnits` INTEGER NOT NULL, `interval` INTEGER NOT NULL, `barometer` REAL,
>>> `pressure` REAL, `altimeter` REAL, `inTemp` REAL, `outTemp` REAL,
>>> `inHumidity` REAL, `outHumidity` REAL, `windSpeed` REAL, `windDir` REAL,
>>> `windGust` REAL, `windGustDir` REAL, `rainRate` REAL, `rain` REAL,
>>> `dewpoint` REAL, `windchill` REAL, `heatindex` REAL, `ET` REAL, `radiation`
>>> REAL, `UV` REAL, `extraTemp1` REAL, `extraTemp2` REAL, `extraTemp3` REAL,
>>> `soilTemp1` REAL, `soilTemp2` REAL, `soilTemp3` REAL, `soilTemp4` REAL,
>>> `leafTemp1` REAL, `leafTemp2` REAL, `extraHumid1` REAL, `extraHumid2` REAL,
>>> `soilMoist1` REAL, `soilMoist2` REAL, `soilMoist3` REAL, `soilMoist4` REAL,
>>> `leafWet1` REAL, `leafWet2` REAL, `rxCheckPercent` REAL, `txBatteryStatus`
>>> REAL, `consBatteryVoltage` REAL, `hail` REAL, `hailRate` REAL,
>>> `heatingTemp` REAL, `heatingVoltage` REAL, `supplyVoltage` REAL,
>>> `referenceVoltage` REAL, `windBatteryStatus` REAL, `rainBatteryStatus`
>>> REAL, `outTempBatteryStatus` REAL, `inTempBatteryStatus` REAL,
>>> `AVG_Rcv_RX0` REAL, `AVG_Rcv_RX1` REAL, `AVG_Rcv_RX2` REAL, `BatVolt_0`
>>> REAL, `SysTemp_0` REAL, `Rsfan_0` REAL, `PacketsSentPerHour_0` REAL,
>>> `Snow_Height` REAL, `BatVolt_1` REAL, `SysTemp_1` REAL, `Rsfan_1` REAL,
>>> `PacketsSentPerHour_1` REAL, `BatVolt_2` REAL, `SysTemp_2` REAL, `Rsfan_2`
>>> REAL, `PacketsSentPerHour_2` REAL, `Soil_Temp_Full1` REAL,
>>> `Soil_Temp_Full2` REAL, `Soil_Temp_Full3` REAL, `Soil_Temp_Full4` REAL,
>>> `AQI_PM1_0` REAL, `AQI_PM2_5` REAL, `AQI_PM10_0` REAL, `AQI_Index` REAL,
>>> `AQI_Temp` REAL, `AQI_Hum` REAL, `CO2` REAL, `GAS_2` REAL, `WiFi_T0` REAL,
>>> `WiFi_H0` REAL, `Signal_Quality_TX0` REAL, `Signal_Quality_TX1` REAL,
>>> `Signal_Quality_TX2` REAL, `Rain_RG11` REAL, `Rain_Rate_RG11` REAL,
>>> `Rain_TX2` REAL, `Rain_Rate_TX2` REAL);
>>>
>>
>> Thank you,
>> engolling
>>
>> Am Samstag, 16. März 2019 07:44:10 UTC+1 schrieb gjr80:
>>>
>>> Hi,
>>>
>>> I had a look through your code and I think you have made things overly
>>> complex for yourself. As I understand your setup you have a file,
>>> /home/pi/WeatherDuino/WeeWx_Exp.txt, that contains your WeatherDuino
>>> data, there are three rows of semicolon separated data, the rows being
>>> labels/names, units and values. One of those fields is a cumulative rain
>>> value. You also use the file /home/pi/WeatherDuino/Rain_tmp.txt that
>>> you use in an elaborate arrangement to read and save the last cumulative
>>> rain value and you obtain the delta rain value by taking the difference
>>> between the current cumulative rain value and the value read from
>>> /home/pi/WeatherDuino/Rain_tmp.txt.
>>>
>>> The logic appears to be sound but you are going about it in a very
>>> complex manner and I suspect that is tripping you up. The need to determine
>>> delta rain from a stream of cumulative rain values is common in many WeeWX
>>> drivers. the approach taken in theses drivers is to create a property,
>>> let's call it last_rain that is initialised to None in the drivers
>>> __init__(). Then when the driver obtains a cumulative rain value, let's
>>> call it cumulative_rain, there is a simple check whether last_rain is
>>> None, if it is then delta_rain is set to None, if last_rain is not None
>>> the delta_rain is set to cumulative_rain - last_rain. Finally last_rain
>>> is set to cumulative_rain. In code it might look like this:
>>>
>>>
>>> class SomeDriver():
>>>
>>> def __init__():
>>> ....
>>> self.last_rain = None
>>> ....
>>>
>>> def get_delta_rain(cumulative_rain):
>>>
>>> if self.last_rain is not None:
>>> delta_rain = cumulative_rain - self.last_rain
>>> else:
>>> delta_rain =None
>>> self.last_rain = cumulative_rain
>>> return delta_rain
>>>
>>> I have left a lot of things out there but the point was to illustrate
>>> the overall logic and structure. By using the last_rain property there
>>> is no need to write temporary values to file; your code is a lot neater,
>>> simpler to understand and faster. The example I have used is for a driver,
>>> your service, whilst not a driver, operates in the same manner in many
>>> respects so you should be able to apply a similar construct in your
>>> service. You don' need a separate get_delta_rain method if you don't
>>> want, you can put it all in your read_file method.
>>>
>>> As for data not appearing in your database there are two conditions that
>>> need to be met (1) your data needs ot appear in the archive record
>>> generated by WeeWX and (2) the archive field name containing your data
>>> needs to be included in the archive table schema. In your case your service
>>> should take care of (1). At the moment I am not convinced that it is
>>> operating as intended, but if you simplify/restructure your code as
>>> suggested we will get there. As for (2) I am not convinced that your code
>>> for setting the modified schema is doing as you think. Have you actually
>>> looked in your database to see what schema has been implemented? You can
>>> easily check the schema of a SQLite database table using the sqlite3
>>> utility (note you may need to install it on your machine using $ sudo
>>> apt-get install sqlite3). To check the schema (assuming your database
>>> is in /home/weewx/archive - it may be in /var/lib/weewx):
>>>
>>> $ sqlite3 /home/weewx/archive/weewx.sdb
>>> sqlite> .schema archive
>>> <schema will be listed here>
>>> sqlite> .q
>>>
>>> What does the above show you, is field Rain_RG11 in the schema?
>>>
>>> Gary
>>>
>>> On Saturday, 16 March 2019 09:09:32 UTC+10, engolling wrote:
>>>>
>>>> Hi Gary,
>>>>
>>>> thanks for your very interesting reply. I tried to augment my extra
>>>> data to the NEW_LOOP_PACKET, whichs works in priciple for all my data
>>>> except the rain delta.
>>>> As I mentioned it seems that the rain delta values are getting sorted
>>>> out somehow.
>>>>
>>>> You can find the code here:
>>>> https://github.com/menachers/WeatherDuino/tree/master/WeeWx_Plugin
>>>> and an excerpt of my weewx.conf in an earlier post.
>>>>
>>>> So if I hardcode a deltarain of 0.1[in] it sums up correctly with each
>>>> archive record - so after 5 minutes I get a total rain of 0.5[in]. If it
>>>> is
>>>> augmented to one loop packet (beeing emitted aproximatly every 3 seconds)
>>>> i
>>>> can also see it in the live display of the loop packet, but it can not be
>>>> found in the archive record. So it seems that is is not accumulated
>>>> internaly.
>>>>
>>>> Best regards,
>>>> engolling
>>>>
>>>> Am Freitag, 15. März 2019 06:09:09 UTC+1 schrieb gjr80:
>>>>>
>>>>> Hi,
>>>>>
>>>>> When using a data service to add data to WeeWX you have two choices,
>>>>> you either add your data to the archive records or you add your data to
>>>>> the
>>>>> loop packets. Either way will work. If you decide that your data service
>>>>> is
>>>>> to augment archive records with your rainfall data then your data service
>>>>> must (1) bind to the event NEW_ARCHIVE_RECORD and (2) add the total
>>>>> rainfall seen by your sensor during that archive period (in your case
>>>>> that
>>>>> is the last 1 minute) to the archive record. If you decide your data
>>>>> service is to augment the loop packets then your service must (1) bind to
>>>>> the NEW_LOOP_PACKET event and (2) add the total rainfall seen since the
>>>>> last loop packet you augmented to the loop packet concerned. Note that
>>>>> you
>>>>> do not need to add your data to every loop packet, you can add it to
>>>>> whichever loop packets you wish, but the value you add must be the total
>>>>> rainfall since the last loop packet you augmented. The WeeWX internal
>>>>> machinery will then take care of accumulating the loop data and your
>>>>> accumulated data will appear in the archive record generated by WeeWX.
>>>>>
>>>>> Which way you decide to go is up to you. If you decide to augment the
>>>>> archive record then your service would need to (in your case) obtain the
>>>>> total rainfall for each 1 minute archive period, this may or may not be
>>>>> easy depending on your hardware. The key here is you need to stick to the
>>>>> archive period. If you decide to augment loop packets you can augment
>>>>> whenever you want, you can skip loop packets if you want as long as you
>>>>> stick to the rule of providing total rainfall since you last augmented a
>>>>> loop packet. The loop packet approach is probably simpler, you just add
>>>>> data when ever you can/want; you are not forced to rigidly stick to a
>>>>> fixed
>>>>> interval as you are with the archive approach.
>>>>>
>>>>> In terms of saving data to the archive there are two conditions that
>>>>> must be met. Firstly, the field/data you wish to save must appear in the
>>>>> WeeWX generated archive record (you can see the WeeWX generated loop
>>>>> packets and archive records by running weeWX directly
>>>>> <http://weewx.com/docs/usersguide.htm#Running_directly>). Secondly,
>>>>> your archive table schema must contain a field with the same name as the
>>>>> field of interest in the archive record. So if your service added a field
>>>>> named 'rain2' to the archive records (or loop packets) and the archive
>>>>> table your WeeWX database had a field named 'rain2', then WeeWX would
>>>>> automatically save your rain2 data to the archive. The steps involved in
>>>>> changing your schema are covered in the Customization Guide
>>>>> <http://weewx.com/docs/customizing.htm> under Adding a new type to
>>>>> the database <http://weewx.com/docs/customizing.htm#add_archive_type>.
>>>>> Note there are other approaches to saving new data to archive (eg
>>>>> creating
>>>>> a second database) but these approaches are usually more complex or more
>>>>> involved so I have ignored them in this case.
>>>>>
>>>>> You might also find that posting your code may help as that way we can
>>>>> see exactly what you are/are not doing.
>>>>>
>>>>> Gary
>>>>>
>>>>> On Friday, 15 March 2019 08:45:03 UTC+10, engolling wrote:
>>>>>>
>>>>>> Short update:
>>>>>> I have found out that my plugin is executed via data_services each
>>>>>> time a loop package is generated and this is approximatly every 3
>>>>>> seconds.
>>>>>> An archive dataset is written every 60 seconds, leading to chance of
>>>>>> 1/20 that the raindelta is saved correctly - thats too less :-D
>>>>>>
>>>>>> So I see two possible solutions:
>>>>>> 1. The data service is only executed once before the archiv dataset
>>>>>> is written. Has anybody a idea how this could be done?
>>>>>>
>>>>>> 2. I have access to some kind of software flag saying that a archive
>>>>>> record has been written.
>>>>>>
>>>>>> I had also a look into the vantage driver and I think this piece of
>>>>>> code does the magic of calculating the data:
>>>>>> # Because the Davis stations do not offer bucket tips in
>>>>>> LOOP data, we
>>>>>> # must calculate it by looking for changes in rain totals.
>>>>>> This won't
>>>>>> # work for the very first rain packet.
>>>>>> if self.save_monthRain is None:
>>>>>> delta = None
>>>>>> else:
>>>>>> delta = loop_packet['monthRain'] - self.save_monthRain
>>>>>> # If the difference is negative, we're at the beginning
>>>>>> of a month.
>>>>>> if delta < 0: delta = None
>>>>>> loop_packet['rain'] = delta
>>>>>> self.save_monthRain = loop_packet['monthRain']
>>>>>>
>>>>>> return loop_packet
>>>>>>
>>>>>> But I do not understand how overwriting the delta is prevented here.
>>>>>>
>>>>>> Hoping for some replys.
>>>>>>
>>>>>> Best wishes,
>>>>>> engolling
>>>>>>
>>>>>>
>>>>>> Am Sonntag, 3. März 2019 13:16:49 UTC+1 schrieb engolling:
>>>>>>>
>>>>>>> Hello,
>>>>>>> I tried to implement a driver providing the rainfall in intervall
>>>>>>> but until now it does not work as expected.
>>>>>>>
>>>>>>> I'am able to handle the correct data to WeeWx with this command:
>>>>>>> syslog.syslog(syslog.LOG_DEBUG, "WeatherDuino:
>>>>>>> " + str(names[n+1]) + ": " + str(deltarain))
>>>>>>> event.record[str(names[n+1])] = float(
>>>>>>> deltarain)
>>>>>>> The debug log output says:
>>>>>>>
>>>>>>>> Mar 3 11:56:16 WeatherDuinoPi weewx[1366]: WeatherDuino:
>>>>>>>> Rain_RG11: 0.17716535433
>>>>>>>>
>>>>>>>
>>>>>>> But in the database there is a "0" logged.
>>>>>>>
>>>>>>> If i change the code hardcoding the rain intervall to 10 it works
>>>>>>> fine.
>>>>>>> syslog.syslog(syslog.LOG_DEBUG, "WeatherDuino:
>>>>>>> " + str(names[n+1]) + ": " + str(10))
>>>>>>> event.record[str(names[n+1])] = 10
>>>>>>>
>>>>>>> So I think my driver is executed more often then my one minute
>>>>>>> logging intervall and so the value of the event.record is overwritten
>>>>>>> with
>>>>>>> a zero again - because the driver thinks the value is already in the
>>>>>>> WeeWx
>>>>>>> database.
>>>>>>>
>>>>>>> You can find my code here:
>>>>>>> https://github.com/menachers/WeatherDuino/tree/master/WeeWx_Plugin
>>>>>>>
>>>>>>> It is embedded in the weewx.conf:
>>>>>>> # This section configures the internal weewx engine.
>>>>>>>
>>>>>>> [Engine]
>>>>>>>
>>>>>>> [[Services]]
>>>>>>> # This section specifies the services that should be run.
>>>>>>> They are
>>>>>>> # grouped by type, and the order of services within each
>>>>>>> group
>>>>>>> # determines the order in which the services will be run.
>>>>>>> prep_services = weewx.engine.StdTimeSynch
>>>>>>> data_services = user.WeeWx_WeatherDuino_Logger_plugin.
>>>>>>> WeeWxService,
>>>>>>> process_services = weewx.engine.StdConvert, weewx.engine.
>>>>>>> StdCalibrate, weewx.engine.StdQC, weewx.wxservices.StdWXCalculate
>>>>>>> archive_services = weewx.engine.StdArchive
>>>>>>> restful_services = weewx.restx.StdStationRegistry, weewx.
>>>>>>> restx.StdWunderground, weewx.restx.StdPWSweather, weewx.restx.
>>>>>>> StdCWOP, weewx.restx.StdWOW, weewx.restx.StdAWEKAS
>>>>>>> report_services = weewx.engine.StdPrint, weewx.engine.
>>>>>>> StdReport
>>>>>>>
>>>>>>> Regards,
>>>>>>> engolling
>>>>>>>
>>>>>>>
>>>>>>> Am Samstag, 23. Februar 2019 14:10:36 UTC+1 schrieb Andrew Milner:
>>>>>>>>
>>>>>>>> weewx requires rain during archive interval for storing in the
>>>>>>>> database archive records. A driver may have to convert whatever it
>>>>>>>> receives (eg running total) so as to obtain the value for the
>>>>>>>> interval.
>>>>>>>> Daily is accumulated by weewx from the archive interval values and
>>>>>>>> weekly
>>>>>>>> and monthly are derived form the daily totals. This is possibly an
>>>>>>>> over
>>>>>>>> simplification - but should give the idea.
>>>>>>>>
>>>>>>>> so if the second source gives a value for rainfall in interval it
>>>>>>>> should be enough for weewx to derive the remainder.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Saturday, 23 February 2019 12:30:38 UTC+2, engolling wrote:
>>>>>>>>>
>>>>>>>>> Hello community,
>>>>>>>>>
>>>>>>>>> I would like to add an additional rain gauge as additional source
>>>>>>>>> described in the customization guide.
>>>>>>>>> Can you give me any hints how to do this the easiest way, to get
>>>>>>>>> the daily, weekly and monthly percipation.
>>>>>>>>> Can I use any modules or calculations that are already done inside
>>>>>>>>> the system or is this normally done by the corresponding driver, so I
>>>>>>>>> have
>>>>>>>>> to handle all the signals precalculated.
>>>>>>>>>
>>>>>>>>> I hope you understand what I mean.
>>>>>>>>>
>>>>>>>>> Best Regards,
>>>>>>>>> engolling
>>>>>>>>>
>>>>>>>>
>>>>>
>>>>>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
