I see the problem. It's a subtle bug. Try this version of accum.py. -tk
On Sun, Dec 13, 2020 at 9:43 AM Vetti52 <[email protected]> wrote: > ok, this time, I think, I got several loop in. Sorry... > > > > [email protected] schrieb am Sonntag, 13. Dezember 2020 um 17:14:26 UTC+1: > >> OK, looks like the gust direction is not getting added to >> the accumulators. Same drill: replace your copy of accum.py with this one. >> >> Make sure you let it run through the first reporting cycle. In your first >> run, you stopped it a bit too early. >> >> -tk >> >> On Sun, Dec 13, 2020 at 6:42 AM Vetti52 <[email protected]> wrote: >> >>> Just to complete the log file, I restarted again using GW1000 driver, >>> which I would like to use, as sson as the missing . Looks different. >>> >>> HTH >>> >>> --ph >>> >>> Vetti52 schrieb am Sonntag, 13. Dezember 2020 um 15:27:27 UTC+1: >>> >>>> Here is the result >>>> >>>> Thanks >>>> -ph >>>> >>>> BTW, I had a look at the most recent thread, presenting a greek version >>>> (https://groups.google.com/g/weewx-user/c/h7nEzNRpNhM/m/tCXM0sgYAgAJ). >>>> There seems to be the same problem with N/A for windGustDir. >>>> [email protected] schrieb am Samstag, 12. Dezember 2020 um 23:33:28 >>>> UTC+1: >>>> >>>>> That tells us that the driver is not emitting windGustDir, but it's >>>>> not telling us why it cannot be extracted out of the accumulators by >>>>> WeeWX. >>>>> >>>>> Attached is an instrumented version of accum.py. Swap the version you >>>>> have with it. It should be in /usr/share/weewx/weewx/accum.py. >>>>> >>>>> Set debug=1, restart weewx, then let it run through the first >>>>> reporting cycle. Post the log. >>>>> >>>>> -tk >>>>> >>>>> On Sat, Dec 12, 2020 at 9:33 AM Vetti52 <[email protected]> wrote: >>>>> >>>>>> Here a comparison of three outputs, first as GW1000.service , then as >>>>>> gw1000 driver and finally the current working ecowitt-client by the >>>>>> interceptor driver: >>>>>> >>>>>> # PYTHONPATH=/usr/share/weewx python3 -m user.gw1000 --test-driver >>>>>> Using configuration file /etc/weewx/weewx.conf >>>>>> Interrogating GW1000 at 192.168.100.150:45000 >>>>>> 2020-12-12 17:55:00 CET (1607792100): UV: 0, dateTime: 1607792100, >>>>>> dayRain: 0.0, daymaxwind: 7.7, inHumidity: 59, inTemp: 17.7, luminosity: >>>>>> 0.0, monthRain: 5.7, outHumidity: 93, outTemp: 2.7, pressure: 999.0, >>>>>> rain: >>>>>> None, rainRate: 0.0, relbarometer: 1003.1, stormRain: 0.0, usUnits: 17, >>>>>> uvradiation: 0.1, weekRain: 3.1, wh65_batt: 0, windDir: 70, windGust: >>>>>> 2.6, >>>>>> windSpeed: 1.0, yearRain: 729.9 >>>>>> >>>>>> # PYTHONPATH=/usr/share/weewx python3 -m user.gw1000 --test-service >>>>>> Using configuration file /etc/weewx/weewx.conf >>>>>> Interrogating GW1000 at 192.168.100.150:45000 >>>>>> LOOP: 2020-12-12 17:56:02 CET (1607792162) UV: 0, dateTime: >>>>>> 1607792162, dayRain: 0.0, daymaxwind: 7.7, dummyTemp: 96.3, inHumidity: >>>>>> 59, >>>>>> inTemp: 63.86, luminosity: 0.0, monthRain: 0.22440944881889766, >>>>>> outHumidity: 94, outTemp: 36.86, pressure: 29.5004575125, rain: None, >>>>>> rainRate: 0.0, relbarometer: 1003.1, stormRain: 0.0, usUnits: 1, >>>>>> uvradiation: 0.1, weekRain: 3.1, wh65_batt: 0, windDir: 77, windGust: >>>>>> 4.473883703878609, windSpeed: 2.6843302223271652, yearRain: >>>>>> 28.736220472440944 >>>>>> LOOP: 2020-12-12 17:56:12 CET (1607792172) dateTime: 1607792172, >>>>>> dummyTemp: 96.3, usUnits: 1 >>>>>> >>>>>> # PYTHONPATH=/usr/share/weewx python3 user/interceptor.py >>>>>> --device=ecowitt-client --mode=listen --port=9000 >>>>>> mapped packet: {'dateTime': 1607792207, 'usUnits': 1, 'pressure': >>>>>> 29.492, 'barometer': 29.613, 'outHumidity': 94.0, 'inHumidity': 58.0, >>>>>> 'outTemp': 36.9, 'inTemp': 63.9, 'windSpeed': 2.46, 'windGust': 5.82, >>>>>> 'windDir': 108.0, 'radiation': 0.0, 'rain': None, 'rainRate': 0.0, >>>>>> 'rainEvent': 0.0, 'UV': 0.0, 'txBatteryStatus': 0.0} >>>>>> >>>>>> The windGustDir does not seem to be emitted. There are two other >>>>>> issues: "rainEvent" and "txBatteryStatus" are missing in the GW1000 >>>>>> data or are not mapped correctly. Thus, Weewx shows no results for Rain >>>>>> Event, and indicates a low transmitter battery status. I would like to >>>>>> move >>>>>> to the GW1000 driver (or better use GW1000.service?), if it would work >>>>>> flawlessly. But it would be nice, to know, how to proceed now. >>>>>> >>>>>> I had GW1000.service activated in parallel to the interceptor driver, >>>>>> although this is somewhat nonsense, as both data come from the same >>>>>> sensors, either "translated" to Ecowitt style by the GW1000, or pulled by >>>>>> the GW1000.service directly. This I did just for a short time to see, if >>>>>> there were any errors occuring. Although the raw data seem not to be >>>>>> comparable, I could not see any effect. Actually I went back to the >>>>>> interceptor driver, so at least rainEvent and the battery status are ok >>>>>> again. >>>>>> [email protected] schrieb am Samstag, 12. Dezember 2020 um 15:20:29 >>>>>> UTC+1: >>>>>> >>>>>>> I do not think your problem is related to the "weighting" problem. >>>>>>> It affected only the daily summaries and, even then, only fields that >>>>>>> are >>>>>>> weighted by the archive length. Wind direction is not one of them. >>>>>>> >>>>>>> Unfortunately, I am not very familiar with any of the drivers and >>>>>>> services you are using. If I understand correctly, the data in question >>>>>>> was >>>>>>> collected by interceptor.py. I would assume that's where the problem is. >>>>>>> Does the driver emit windGustDir on every LOOP packet? Or, does it rely >>>>>>> on >>>>>>> software record generation to provide it at the end of an archive >>>>>>> period? >>>>>>> >>>>>>> Sorry I cannot be of more help. >>>>>>> >>>>>>> -tk >>>>>>> >>>>>>> On Thu, Dec 10, 2020 at 11:29 AM Vetti52 <[email protected]> wrote: >>>>>>> >>>>>>>> So, I am still not sure, if wee_database --update or --reweight should >>>>>>>> solve the problem. Or do I have to wait for a special fix? >>>>>>>> >>>>>>>> Vetti52 schrieb am Dienstag, 8. Dezember 2020 um 21:01:56 UTC+1: >>>>>>>> >>>>>>>>> BTW, there are some values not null in the database after update, >>>>>>>>> as seen, when filtering the archive for windGustDir <> null: >>>>>>>>> dateTime windGustDir >>>>>>>>> 2020-12-08 10:50:00 89.0 >>>>>>>>> 2020-12-08 05:35:00 95.0 >>>>>>>>> 2020-12-08 01:25:00 148.0 >>>>>>>>> 2020-12-07 22:20:00 168.0 >>>>>>>>> 2020-12-05 01:35:00 120.0 >>>>>>>>> 2020-12-04 22:00:00 101.0 >>>>>>>>> >>>>>>>>> Vetti52 schrieb am Dienstag, 8. Dezember 2020 um 19:33:13 UTC+1: >>>>>>>>> >>>>>>>>>> tha update was on 2020-12-04, right. >>>>>>>>>> >>>>>>>>>> at 1. >>>>>>>>>> I am running Weewx on a Raspi4 under buster, installed with >>>>>>>>>> apt-get install weewx. My station is a EFWS2500, which is a clone of >>>>>>>>>> Ecowitt 2500. The data are collected by a Froggit DS2500, which is a >>>>>>>>>> clone >>>>>>>>>> of GW1000. Weewx still collects the data from interceptor.py >>>>>>>>>> ecowitt-client. Although user.gw1000 is already implemented as a >>>>>>>>>> service, >>>>>>>>>> I still have not yet replaced interceptor.py. >>>>>>>>>> >>>>>>>>>> at 2. >>>>>>>>>> didn't detect ignore_zero_wind in weewx.conf. >>>>>>>>>> >>>>>>>>>> at 3. >>>>>>>>>> record_generation = software >>>>>>>>>> >>>>>>>>>> HTH >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> -ph >>>>>>>>>> >>>>>>>>>> [email protected] schrieb am Dienstag, 8. Dezember 2020 um >>>>>>>>>> 18:42:04 UTC+1: >>>>>>>>>> >>>>>>>>>>> Thanks. >>>>>>>>>>> >>>>>>>>>>> I assume you updated on 2020-12-04? There is no value for >>>>>>>>>>> "max_dir" (which is where "gust_dir" comes from) in your daily >>>>>>>>>>> summaries >>>>>>>>>>> since that date. >>>>>>>>>>> >>>>>>>>>>> Looking through your daily summaries, you are suffering from >>>>>>>>>>> issue #623 <https://github.com/weewx/weewx/issues/623> (as are >>>>>>>>>>> most V4.2 users). I am working on a fix for this. However, as far >>>>>>>>>>> as I >>>>>>>>>>> know, this problem should not be affecting max_dir, but I may be >>>>>>>>>>> missing >>>>>>>>>>> something. >>>>>>>>>>> >>>>>>>>>>> A few questions: >>>>>>>>>>> 1. What kind of hardware? >>>>>>>>>>> 2. What is your setting for option ignore_zero_wind, if any? >>>>>>>>>>> 3. What is your setting for option record_generation? >>>>>>>>>>> >>>>>>>>>>> -tk >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Dec 8, 2020 at 8:44 AM Vetti52 <[email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> It says "N/A". >>>>>>>>>>>> >>>>>>>>>>>> The output of sqlite: >>>>>>>>>>>> >>>>>>>>>>>> 2020-12-08 >>>>>>>>>>>> 00:00:00|1607382000|0.0|1607382032|10.29|1607442892|148.553777777778|211|148.553777777778|211||141.25403836784|8.54431223906981|211|307.861736888889|307.861736888889 >>>>>>>>>>>> 2020-12-07 >>>>>>>>>>>> 00:00:00|1607295600|0.0|1607295628|18.34|1607348466|969.075666666667|288|969.075666666667|288||563.975677916051|-355.104682689556|288|4027.90666114815|4027.90666114815 >>>>>>>>>>>> 2020-12-06 >>>>>>>>>>>> 00:00:00|1607209200|0.0|1607209233|9.17|1607219761|77.5104166666667|286|77.5104166666667|286||69.480956552117|2.39263945299658|286|132.33436038966|132.33436038966 >>>>>>>>>>>> 2020-12-05 >>>>>>>>>>>> 00:00:00|1607122800|0.0|1607123329|12.53|1607140258|550.433333333334|288|550.433333333334|288||357.29020463216|-182.82480166047|288|1638.89117222222|1638.89117222222 >>>>>>>>>>>> 2020-12-04 >>>>>>>>>>>> 00:00:00|1607036400|0.0|1607039337|20.58|1607050956|806.906472222223|288|806.906472222223|288||609.110049598142|-419.170693241216|288|2757.25666137114|2757.25666137114 >>>>>>>>>>>> 2020-12-03 >>>>>>>>>>>> 00:00:00|1606950000|0.0|1606950381|14.76|1606982936|938.132555555555|286|161542.062333333|52611|136.0|63684.8752720733|-143160.707316072|52611|3367.86934245679|550406.753590012 >>>>>>>>>>>> 2020-12-02 >>>>>>>>>>>> 00:00:00|1606863600|0.0|1606863628|10.29|1606933960|298.527702380952|288|89558.3107142857|86400|97.0|42277.4937890549|-72643.1316507023|61200|647.727487337254|194318.246201176 >>>>>>>>>>>> 2020-12-01 >>>>>>>>>>>> 00:00:00|1606777200|0.0|1606778615|9.17|1606778252|265.736888888889|288|79721.0666666667|86400|194.0|42931.7123928354|-43844.9965062139|69900|434.78745508642|130436.236525926 >>>>>>>>>>>> 2020-11-30 >>>>>>>>>>>> 00:00:00|1606690800|0.0|1606690818|12.53|1606753726|458.85391053391|287|137656.173160173|86100|138.0|-43283.5066028223|-120674.014788181|77100|1142.32670378797|342698.01113639 >>>>>>>>>>>> 2020-11-29 >>>>>>>>>>>> 00:00:00|1606604400|0.0|1606604418|5.82|1606650983|50.8892222222223|288|15266.7666666667|86400|309.0|-11596.0986434007|7995.68851878374|24600|54.2518490740741|16275.5547222222 >>>>>>>>>>>> 2020-11-28 >>>>>>>>>>>> 00:00:00|1606518000|0.0|1606518015|11.41|1606565426|165.474444444445|287|49642.3333333333|86100|63.0|47302.3988525243|1887.80479760903|28200|423.739858888889|127121.957666667 >>>>>>>>>>>> 2020-11-27 >>>>>>>>>>>> 00:00:00|1606431600|0.0|1606431616|4.47|1606483066|9.17855555555556|288|2753.56666666667|86400|131.0|2261.0873478704|-539.506810163124|4200|10.6575301851852|3197.25905555556 >>>>>>>>>>>> 2020-11-26 >>>>>>>>>>>> 00:00:00|1606345200|0.0|1606345204|8.05|1606394153|107.788222222222|287|32336.4666666667|86100|259.0|-25803.5675497035|-15488.3625339286|42000|134.601425382716|40380.4276148148 >>>>>>>>>>>> 2020-11-25 >>>>>>>>>>>> 00:00:00|1606258800|0.0|1606258804|9.17|1606302300|337.383111111111|288|101214.933333333|86400|189.0|-19895.5174799498|-97556.2211722166|76500|636.16824345679|190850.473037037 >>>>>>>>>>>> 2020-11-24 >>>>>>>>>>>> 00:00:00|1606172400|0.0|1606172999|10.29|1606187816|577.891555555556|288|173367.466666667|86400|213.0|-90246.0852000086|-146738.507905923|86400|1319.29265822222|395787.797466666 >>>>>>>>>>>> 2020-11-23 >>>>>>>>>>>> 00:00:00|1606086000|0.0|1606086005|10.29|1606137950|368.728555555556|288|110618.566666667|86400|261.0|-97191.5602413715|-50554.3362971601|72300|718.446958555555|215534.087566667 >>>>>>>>>>>> 2020-11-22 >>>>>>>>>>>> 00:00:00|1605999600|0.0|1606008451|14.76|1606041552|645.880888888889|288|193764.266666666|86400|245.0|-171272.270757009|-84245.3229881982|73500|2109.85465123457|632956.39537037 >>>>>>>>>>>> 2020-11-21 >>>>>>>>>>>> 00:00:00|1605913200|0.0|1605913439|15.88|1605942546|1114.47377777778|288|334342.133333333|86400|191.0|-175428.251363656|-273150.682819638|86400|4553.4892182716|1366046.76548148 >>>>>>>>>>>> 2020-11-20 >>>>>>>>>>>> 00:00:00|1605826800|0.0|1605826872|9.17|1605833902|169.842222222222|288|50952.6666666667|86400|238.0|-42329.2825046772|-24698.2615292903|59400|247.763276197531|74328.9828592593 >>>>>>>>>>>> 2020-11-19 >>>>>>>>>>>> 00:00:00|1605740400|0.0|1605798095|22.82|1605788229|919.519666666666|288|275855.9|86400|255.0|-208559.830853793|-91789.2604135498|82800|3926.4786627037|1177943.59881111 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [email protected] schrieb am Dienstag, 8. Dezember 2020 um >>>>>>>>>>>> 17:30:54 UTC+1: >>>>>>>>>>>> >>>>>>>>>>>>> What do you mean by, "there are no longer windDir data in the >>>>>>>>>>>>> hi/lo section"? Is there a placeholder? Or, white space? Or, N/A? >>>>>>>>>>>>> >>>>>>>>>>>>> It might be worth seeing what is in your database. This >>>>>>>>>>>>> assumes your database is located at /var/lib/weewx/weewx.sdb. If >>>>>>>>>>>>> you used >>>>>>>>>>>>> the setup.py install method, it would be at >>>>>>>>>>>>> /home/weewx/archive/weewx.sdb. >>>>>>>>>>>>> >>>>>>>>>>>>> *sqlite3 /var/lib/weewx/weewx.sdb* >>>>>>>>>>>>> sqlite> *select datetime(dateTime, 'unixepoch', 'localtime'), >>>>>>>>>>>>> * from archive_day_wind order by dateTime desc limit 20;* >>>>>>>>>>>>> sqlite> *.quit* >>>>>>>>>>>>> >>>>>>>>>>>>> -tk >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Dec 8, 2020 at 8:15 AM Vetti52 <[email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Since update to version 4.2.0, I have a curious result in >>>>>>>>>>>>>> season skin. Although $current.windDir is properly displayed and >>>>>>>>>>>>>> continuously collected, there are no longer windDir data in the >>>>>>>>>>>>>> hi/lo >>>>>>>>>>>>>> section. Wind Max shows speed correctly, but no direction. I >>>>>>>>>>>>>> checked >>>>>>>>>>>>>> hilo.inc. There are no changes, compared to the original version >>>>>>>>>>>>>> (hilo.inc.dpkg-dist dated from 2020-04-30). >>>>>>>>>>>>>> >>>>>>>>>>>>>> Where should I start to trace the bug? >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>>> Google Groups "weewx-user" group. >>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>>>> https://groups.google.com/d/msgid/weewx-user/bb0a2343-45cf-4885-a7a8-4212c4426592n%40googlegroups.com >>>>>>>>>>>>>> <https://groups.google.com/d/msgid/weewx-user/bb0a2343-45cf-4885-a7a8-4212c4426592n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>>>> . >>>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>> Google Groups "weewx-user" group. >>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>>> >>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>> https://groups.google.com/d/msgid/weewx-user/38c3e7ba-b73d-4291-a17f-0620c9f80ac7n%40googlegroups.com >>>>>>>>>>>> <https://groups.google.com/d/msgid/weewx-user/38c3e7ba-b73d-4291-a17f-0620c9f80ac7n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>> . >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "weewx-user" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> >>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/weewx-user/625e77f0-c4f3-4640-aecf-1128cb2af4abn%40googlegroups.com >>>>>>>> <https://groups.google.com/d/msgid/weewx-user/625e77f0-c4f3-4640-aecf-1128cb2af4abn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "weewx-user" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> >>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/weewx-user/220a3041-b2a2-4b37-a6c2-3c158b60e20bn%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/weewx-user/220a3041-b2a2-4b37-a6c2-3c158b60e20bn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "weewx-user" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/weewx-user/83d99b6d-863c-4424-afd6-74c119fd25f7n%40googlegroups.com >>> <https://groups.google.com/d/msgid/weewx-user/83d99b6d-863c-4424-afd6-74c119fd25f7n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/weewx-user/b260aeab-db22-4a65-90a0-20cfa61bf408n%40googlegroups.com > <https://groups.google.com/d/msgid/weewx-user/b260aeab-db22-4a65-90a0-20cfa61bf408n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zEAkqFhvhGj2wzZJQLkiy-WCrn02UYORg_G8aT5M01yzdQ%40mail.gmail.com.
# # Copyright (c) 2009-2020 Tom Keffer <[email protected]> # # See the file LICENSE.txt for your full rights. # """Statistical accumulators. They accumulate the highs, lows, averages, etc., of a sequence of records.""" # # General strategy. # # Most observation types are scalars, so they can be treated simply. Values are added to a scalar # accumulator, which keeps track of highs, lows, and a sum. When it comes time for extraction, the # average over the archive period is typically produced. # # However, wind is a special case. It is a vector, which has been flatted over at least two # scalars, windSpeed and windDir. Some stations, notably the Davis Vantage, add windGust and # windGustDir. The accumulators cannot simply treat them individually as if they were just another # scalar. Instead they must be grouped together. This is done by treating windSpeed as a 'special' # scalar. When it appears, it is coupled with windDir and, if available, windGust and windGustDir, # and added to a vector accumulator. When the other types ( windDir, windGust, and windGustDir) # appear, they are ignored, having already been handled during the processing of type windSpeed. # # When it comes time to extract wind, vector averages are calculated, then the results are # flattened again. # from __future__ import absolute_import import logging import math import configobj import six from six.moves import StringIO import weewx from weeutil.weeutil import ListOfDicts, to_float log = logging.getLogger(__name__) # # Default mappings from observation types to accumulator classes and functions # DEFAULTS_INI = """ [Accumulator] [[dateTime]] adder = noop [[dayET]] extractor = last [[dayRain]] extractor = last [[ET]] extractor = sum [[hourRain]] extractor = last [[rain]] extractor = sum [[rain24]] extractor = last [[monthET]] extractor = last [[monthRain]] extractor = last [[stormRain]] extractor = last [[totalRain]] extractor = last [[usUnits]] adder = check_units [[wind]] accumulator = vector extractor = wind [[windDir]] extractor = noop [[windGust]] extractor = noop [[windGustDir]] extractor = noop [[windGust10]] extractor = last [[windGustDir10]] extractor = last [[windrun]] extractor = sum [[windSpeed]] adder = add_wind merger = avg extractor = noop [[windSpeed2]] extractor = last [[windSpeed10]] extractor = last [[yearET]] extractor = last [[yearRain]] extractor = last [[lightning_strike_count]] extractor = sum """ defaults_dict = configobj.ConfigObj(StringIO(DEFAULTS_INI), encoding='utf-8') accum_dict = ListOfDicts(defaults_dict['Accumulator'].dict()) class OutOfSpan(ValueError): """Raised when attempting to add a record outside of the timespan held by an accumulator""" # =============================================================================== # ScalarStats # =============================================================================== class ScalarStats(object): """Accumulates statistics (min, max, average, etc.) for a scalar value. Property 'last' is the last non-None value seen. Property 'lasttime' is the time it was seen. """ default_init = (None, None, None, None, 0.0, 0, 0.0, 0) def __init__(self, stats_tuple=None): self.setStats(stats_tuple) self.last = None self.lasttime = None def setStats(self, stats_tuple=None): (self.min, self.mintime, self.max, self.maxtime, self.sum, self.count, self.wsum, self.sumtime) = stats_tuple if stats_tuple else ScalarStats.default_init def getStatsTuple(self): """Return a stats-tuple. That is, a tuple containing the gathered statistics. This tuple can be used to update the stats database""" return (self.min, self.mintime, self.max, self.maxtime, self.sum, self.count, self.wsum, self.sumtime) def mergeHiLo(self, x_stats): """Merge the highs and lows of another accumulator into myself.""" if x_stats.min is not None: if self.min is None or x_stats.min < self.min: self.min = x_stats.min self.mintime = x_stats.mintime if x_stats.max is not None: if self.max is None or x_stats.max > self.max: self.max = x_stats.max self.maxtime = x_stats.maxtime if x_stats.lasttime is not None: if self.lasttime is None or x_stats.lasttime >= self.lasttime: self.lasttime = x_stats.lasttime self.last = x_stats.last def mergeSum(self, x_stats): """Merge the sum and count of another accumulator into myself.""" self.sum += x_stats.sum self.count += x_stats.count self.wsum += x_stats.wsum self.sumtime += x_stats.sumtime def addHiLo(self, val, ts): """Include a scalar value in my highs and lows. val: A scalar value ts: The timestamp. """ # If this is a string, try to convert it to a float. if isinstance(val, (six.string_types, six.text_type)): # Fail hard if unable to do the conversion: val = to_float(val) # Check for None and NaN: if val is not None and val == val: if self.min is None or val < self.min: self.min = val self.mintime = ts if self.max is None or val > self.max: self.max = val self.maxtime = ts if self.lasttime is None or ts >= self.lasttime: self.last = val self.lasttime = ts def addSum(self, val, weight=1): """Add a scalar value to my running sum and count.""" # If this is a string, try to convert it to a float. if isinstance(val, (six.string_types, six.text_type)): # Fail hard if unable to do the conversion: val = to_float(val) # Check for None and NaN: if val is not None and val == val: self.sum += val self.count += 1 self.wsum += val * weight self.sumtime += weight @property def avg(self): return self.wsum / self.sumtime if self.count else None class VecStats(object): """Accumulates statistics for a vector value. Property 'last' is the last non-None value seen. It is a two-way tuple (mag, dir). Property 'lasttime' is the time it was seen. """ default_init = (None, None, None, None, 0.0, 0, 0.0, 0, None, 0.0, 0.0, 0, 0.0, 0.0) def __init__(self, stats_tuple=None): self.setStats(stats_tuple) self.last = (None, None) self.lasttime = None def setStats(self, stats_tuple=None): (self.min, self.mintime, self.max, self.maxtime, self.sum, self.count, self.wsum, self.sumtime, self.max_dir, self.xsum, self.ysum, self.dirsumtime, self.squaresum, self.wsquaresum) = stats_tuple if stats_tuple else VecStats.default_init def getStatsTuple(self): """Return a stats-tuple. That is, a tuple containing the gathered statistics.""" return (self.min, self.mintime, self.max, self.maxtime, self.sum, self.count, self.wsum, self.sumtime, self.max_dir, self.xsum, self.ysum, self.dirsumtime, self.squaresum, self.wsquaresum) def mergeHiLo(self, x_stats): """Merge the highs and lows of another accumulator into myself.""" if x_stats.min is not None: if self.min is None or x_stats.min < self.min: self.min = x_stats.min self.mintime = x_stats.mintime if x_stats.max is not None: if self.max is None or x_stats.max > self.max: self.max = x_stats.max self.maxtime = x_stats.maxtime self.max_dir = x_stats.max_dir if x_stats.lasttime is not None: if self.lasttime is None or x_stats.lasttime >= self.lasttime: self.lasttime = x_stats.lasttime self.last = x_stats.last def mergeSum(self, x_stats): """Merge the sum and count of another accumulator into myself.""" self.sum += x_stats.sum self.count += x_stats.count self.wsum += x_stats.wsum self.sumtime += x_stats.sumtime self.xsum += x_stats.xsum self.ysum += x_stats.ysum self.dirsumtime += x_stats.dirsumtime self.squaresum += x_stats.squaresum self.wsquaresum += x_stats.wsquaresum def addHiLo(self, val, ts): """Include a vector value in my highs and lows. val: A vector value. It is a 2-way tuple (mag, dir). ts: The timestamp. """ speed, dirN = val # If this is a string, try to convert it to a float. if isinstance(speed, (six.string_types, six.text_type)): # Fail hard if unable to do the conversion: speed = to_float(speed) if isinstance(dirN, (six.string_types, six.text_type)): # Fail hard if unable to do the conversion: dirN = to_float(dirN) # Check for None and NaN: if speed is not None and speed == speed: if self.min is None or speed < self.min: self.min = speed self.mintime = ts if self.max is None or speed > self.max: self.max = speed self.maxtime = ts self.max_dir = dirN if self.lasttime is None or ts >= self.lasttime: self.last = (speed, dirN) self.lasttime = ts def addSum(self, val, weight=1): """Add a vector value to my sum and squaresum. val: A vector value. It is a 2-way tuple (mag, dir) """ speed, dirN = val # If this is a string, try to convert it to a float. if isinstance(speed, (six.string_types, six.text_type)): # Fail hard if unable to do the conversion: speed = to_float(speed) if isinstance(dirN, (six.string_types, six.text_type)): # Fail hard if unable to do the conversion: dirN = to_float(dirN) # Check for None and NaN: if speed is not None and speed == speed: self.sum += speed self.count += 1 self.wsum += weight * speed self.sumtime += weight self.squaresum += speed ** 2 self.wsquaresum += weight * speed ** 2 if dirN is not None: self.xsum += weight * speed * math.cos(math.radians(90.0 - dirN)) self.ysum += weight * speed * math.sin(math.radians(90.0 - dirN)) # It's OK for direction to be None, provided speed is zero: if dirN is not None or speed == 0: self.dirsumtime += weight @property def avg(self): return self.wsum / self.sumtime if self.count else None @property def rms(self): return math.sqrt(self.wsquaresum / self.sumtime) if self.count else None @property def vec_avg(self): if self.count: return math.sqrt((self.xsum ** 2 + self.ysum ** 2) / self.sumtime ** 2) @property def vec_dir(self): if self.dirsumtime and (self.ysum or self.xsum): _result = 90.0 - math.degrees(math.atan2(self.ysum, self.xsum)) if _result < 0.0: _result += 360.0 return _result # Return the last known direction when our vector sum is 0 return self.last[1] # =============================================================================== # FirstLastAccum # =============================================================================== class FirstLastAccum(object): """Minimal accumulator, suitable for strings. It can only return the first and last strings it has seen, along with their timestamps. """ default_init = (None, None, None, None) def __init__(self, stats_tuple=None): self.first = None self.firsttime = None self.last = None self.lasttime = None def setStats(self, stats_tuple=None): self.first, self.firsttime, self.last, self.lasttime = stats_tuple \ if stats_tuple else FirstLastAccum.default_init def getStatsTuple(self): """Return a stats-tuple. That is, a tuple containing the gathered statistics. This tuple can be used to update the stats database""" return self.first, self.firsttime, self.last, self.lasttime def mergeHiLo(self, x_stats): """Merge the highs and lows of another accumulator into myself.""" if x_stats.firsttime is not None: if self.firsttime is None or x_stats.firsttime < self.firsttime: self.firsttime = x_stats.firsttime self.first = x_stats.first if x_stats.lasttime is not None: if self.lasttime is None or x_stats.lasttime >= self.lasttime: self.lasttime = x_stats.lasttime self.last = x_stats.last def mergeSum(self, x_stats): """Merge the count of another accumulator into myself.""" pass def addHiLo(self, val, ts): """Include a value in my stats. val: A value of almost any type. It will be converted to a string before being accumulated. ts: The timestamp. """ if val is not None: string_val = str(val) if self.firsttime is None or ts < self.firsttime: self.first = string_val self.firsttime = ts if self.lasttime is None or ts >= self.lasttime: self.last = string_val self.lasttime = ts def addSum(self, val, weight=1): """Add a scalar value to my running count.""" pass # =============================================================================== # Class Accum # =============================================================================== class Accum(dict): """Accumulates statistics for a set of observation types.""" def __init__(self, timespan, unit_system=None): """Initialize a Accum. timespan: The time period over which stats will be accumulated. unit_system: The unit system used by the accumulator""" self.timespan = timespan # Set the accumulator's unit system. Usually left unspecified until the # first observation comes in for normal operation or pre-set if # obtaining a historical accumulator. self.unit_system = unit_system def addRecord(self, record, add_hilo=True, weight=1): """Add a record to my running statistics. The record must have keys 'dateTime' and 'usUnits'.""" # Check to see if the record is within my observation timespan if not self.timespan.includesArchiveTime(record['dateTime']): raise OutOfSpan("Attempt to add out-of-interval record (%s) to timespan (%s)" % (record['dateTime'], self.timespan)) for obs_type in record: # Get the proper function ... func = get_add_function(obs_type) # ... then call it. func(self, record, obs_type, add_hilo, weight) def updateHiLo(self, accumulator): """Merge the high/low stats of another accumulator into me.""" if accumulator.timespan.start < self.timespan.start \ or accumulator.timespan.stop > self.timespan.stop: raise OutOfSpan("Attempt to merge an accumulator whose timespan is not a subset") self._check_units(accumulator.unit_system) for obs_type in accumulator: # Initialize the type if we have not seen it before self._init_type(obs_type) # Get the proper function ... func = get_merge_function(obs_type) # ... then call it func(self, accumulator, obs_type) def getRecord(self): """Extract a record out of the results in the accumulator.""" # All records have a timestamp and unit type record = {'dateTime': self.timespan.stop, 'usUnits': self.unit_system} return self.augmentRecord(record) def augmentRecord(self, record): # Go through all observation types. for obs_type in self: # If the type does not appear in the record, then add it: if obs_type not in record: # Get the proper extraction function... func = get_extract_function(obs_type) # ... then call it func(self, record, obs_type) return record def set_stats(self, obs_type, stats_tuple): self._init_type(obs_type) self[obs_type].setStats(stats_tuple) # # Begin add functions. These add a record to the accumulator. # def add_value(self, record, obs_type, add_hilo, weight): """Add a single observation to myself.""" val = record[obs_type] # If the type has not been seen before, initialize it self._init_type(obs_type) # Then add to highs/lows, and to the running sum: if add_hilo: self[obs_type].addHiLo(val, record['dateTime']) self[obs_type].addSum(val, weight=weight) def add_wind_value(self, record, obs_type, add_hilo, weight): """Add a single observation of type wind to myself.""" if obs_type in ['windDir', 'windGust', 'windGustDir']: return if weewx.debug: assert (obs_type == 'windSpeed') # First add it to regular old 'windSpeed', then # treat it like a vector. self.add_value(record, obs_type, add_hilo, weight) # If the type has not been seen before, initialize it. self._init_type('wind') # Then add to highs/lows. if add_hilo: # If the station does not provide windGustDir, then substitute windDir. # See issue #320, https://bit.ly/2HSo0ju wind_gust_dir = record.get('windGustDir') if wind_gust_dir is None: wind_gust_dir = record.get('windDir') # Do windGust first, so that the last packet entered is windSpeed, not windGust # See Slack discussion https://bit.ly/3qV1nBV self['wind'].addHiLo((record.get('windGust'), wind_gust_dir), record['dateTime']) self['wind'].addHiLo((record.get('windSpeed'), record.get('windDir')), record['dateTime']) # Add to the running sum. self['wind'].addSum((record['windSpeed'], record.get('windDir')), weight=weight) def check_units(self, record, obs_type, add_hilo, weight): if weewx.debug: assert (obs_type == 'usUnits') self._check_units(record['usUnits']) def noop(self, record, obs_type, add_hilo=True, weight=1): pass # # Begin hi/lo merge functions. These are called when merging two accumulators # def merge_minmax(self, x_accumulator, obs_type): """Merge value in another accumulator, using min/max""" self[obs_type].mergeHiLo(x_accumulator[obs_type]) def merge_avg(self, x_accumulator, obs_type): """Merge value in another accumulator, using avg for max""" x_stats = x_accumulator[obs_type] if x_stats.min is not None: if self[obs_type].min is None or x_stats.min < self[obs_type].min: self[obs_type].min = x_stats.min self[obs_type].mintime = x_stats.mintime if x_stats.avg is not None: if self[obs_type].max is None or x_stats.avg > self[obs_type].max: self[obs_type].max = x_stats.avg self[obs_type].maxtime = x_accumulator.timespan.stop if x_stats.lasttime is not None: if self[obs_type].lasttime is None or x_stats.lasttime >= self[obs_type].lasttime: self[obs_type].lasttime = x_stats.lasttime self[obs_type].last = x_stats.last # # Begin extraction functions. These extract a record out of the accumulator. # def extract_wind(self, record, obs_type): """Extract wind values from myself, and put in a record.""" # Wind records must be flattened into the separate categories: if 'windSpeed' not in record: record['windSpeed'] = self[obs_type].avg if 'windDir' not in record: record['windDir'] = self[obs_type].vec_dir if 'windGust' not in record: record['windGust'] = self[obs_type].max if 'windGustDir' not in record: record['windGustDir'] = self[obs_type].max_dir def extract_sum(self, record, obs_type): record[obs_type] = self[obs_type].sum def extract_last(self, record, obs_type): record[obs_type] = self[obs_type].last def extract_avg(self, record, obs_type): record[obs_type] = self[obs_type].avg def extract_min(self, record, obs_type): record[obs_type] = self[obs_type].min def extract_max(self, record, obs_type): record[obs_type] = self[obs_type].max def extract_count(self, record, obs_type): record[obs_type] = self[obs_type].count # # Miscellaneous, utility functions # def _init_type(self, obs_type): """Add a given observation type to my dictionary.""" # Do nothing if this type has already been initialized: if obs_type in self: return # Get a new accumulator of the proper type self[obs_type] = new_accumulator(obs_type) def _check_units(self, new_unit_system): # If no unit system has been specified for me yet, adopt the incoming # system if self.unit_system is None: self.unit_system = new_unit_system else: # Otherwise, make sure they match if self.unit_system != new_unit_system: raise ValueError("Unit system mismatch %d v. %d" % (self.unit_system, new_unit_system)) @property def isEmpty(self): return self.unit_system is None # =============================================================================== # Configuration dictionaries # =============================================================================== # # Mappings from convenient string nicknames, which can be used in a config file, # to actual functions and classes # ACCUM_TYPES = { 'scalar': ScalarStats, 'vector': VecStats, 'firstlast': FirstLastAccum } ADD_FUNCTIONS = { 'add': Accum.add_value, 'add_wind': Accum.add_wind_value, 'check_units': Accum.check_units, 'noop': Accum.noop } MERGE_FUNCTIONS = { 'minmax': Accum.merge_minmax, 'avg': Accum.merge_avg } EXTRACT_FUNCTIONS = { 'avg': Accum.extract_avg, 'count': Accum.extract_count, 'last': Accum.extract_last, 'max': Accum.extract_max, 'min': Accum.extract_min, 'noop': Accum.noop, 'sum': Accum.extract_sum, 'wind': Accum.extract_wind, } # The default actions for an individual observation type OBS_DEFAULTS = { 'accumulator': 'scalar', 'adder': 'add', 'merger': 'minmax', 'extractor': 'avg' } def initialize(config_dict): # Add the configuration dictionary to the beginning of the list of maps. # This will cause it to override the defaults global accum_dict accum_dict.maps.insert(0, config_dict.get('Accumulator', {})) def new_accumulator(obs_type): """Instantiate an accumulator, appropriate for type 'obs_type'.""" global accum_dict # Get the options for this type. Substitute the defaults if they have not been specified obs_options = accum_dict.get(obs_type, OBS_DEFAULTS) # Get the nickname of the accumulator. Default is 'scalar' accum_nickname = obs_options.get('accumulator', 'scalar') # Instantiate and return the accumulator. # If we don't know this nickname, then fail hard with a KeyError return ACCUM_TYPES[accum_nickname]() def get_add_function(obs_type): """Get an adder function appropriate for type 'obs_type'.""" global accum_dict # Get the options for this type. Substitute the defaults if they have not been specified obs_options = accum_dict.get(obs_type, OBS_DEFAULTS) # Get the nickname of the adder. Default is 'add' add_nickname = obs_options.get('adder', 'add') # If we don't know this nickname, then fail hard with a KeyError return ADD_FUNCTIONS[add_nickname] def get_merge_function(obs_type): """Get a merge function appropriate for type 'obs_type'.""" global accum_dict # Get the options for this type. Substitute the defaults if they have not been specified obs_options = accum_dict.get(obs_type, OBS_DEFAULTS) # Get the nickname of the merger. Default is 'minmax' add_nickname = obs_options.get('merger', 'minmax') # If we don't know this nickname, then fail hard with a KeyError return MERGE_FUNCTIONS[add_nickname] def get_extract_function(obs_type): """Get an extraction function appropriate for type 'obs_type'.""" global accum_dict # Get the options for this type. Substitute the defaults if they have not been specified obs_options = accum_dict.get(obs_type, OBS_DEFAULTS) # Get the nickname of the extractor. Default is 'avg' add_nickname = obs_options.get('extractor', 'avg') # If we don't know this nickname, then fail hard with a KeyError return EXTRACT_FUNCTIONS[add_nickname]
