OK, looks like the gust direction is not getting added to the accumulators. Same drill: replace your copy of accum.py with this one.
Make sure you let it run through the first reporting cycle. In your first run, you stopped it a bit too early. -tk On Sun, Dec 13, 2020 at 6:42 AM Vetti52 <[email protected]> wrote: > Just to complete the log file, I restarted again using GW1000 driver, > which I would like to use, as sson as the missing . Looks different. > > HTH > > --ph > > Vetti52 schrieb am Sonntag, 13. Dezember 2020 um 15:27:27 UTC+1: > >> Here is the result >> >> Thanks >> -ph >> >> BTW, I had a look at the most recent thread, presenting a greek version ( >> https://groups.google.com/g/weewx-user/c/h7nEzNRpNhM/m/tCXM0sgYAgAJ). >> There seems to be the same problem with N/A for windGustDir. >> [email protected] schrieb am Samstag, 12. Dezember 2020 um 23:33:28 UTC+1: >> >>> That tells us that the driver is not emitting windGustDir, but it's not >>> telling us why it cannot be extracted out of the accumulators by WeeWX. >>> >>> Attached is an instrumented version of accum.py. Swap the version you >>> have with it. It should be in /usr/share/weewx/weewx/accum.py. >>> >>> Set debug=1, restart weewx, then let it run through the first reporting >>> cycle. Post the log. >>> >>> -tk >>> >>> On Sat, Dec 12, 2020 at 9:33 AM Vetti52 <[email protected]> wrote: >>> >>>> Here a comparison of three outputs, first as GW1000.service , then as >>>> gw1000 driver and finally the current working ecowitt-client by the >>>> interceptor driver: >>>> >>>> # PYTHONPATH=/usr/share/weewx python3 -m user.gw1000 --test-driver >>>> Using configuration file /etc/weewx/weewx.conf >>>> Interrogating GW1000 at 192.168.100.150:45000 >>>> 2020-12-12 17:55:00 CET (1607792100): UV: 0, dateTime: 1607792100, >>>> dayRain: 0.0, daymaxwind: 7.7, inHumidity: 59, inTemp: 17.7, luminosity: >>>> 0.0, monthRain: 5.7, outHumidity: 93, outTemp: 2.7, pressure: 999.0, rain: >>>> None, rainRate: 0.0, relbarometer: 1003.1, stormRain: 0.0, usUnits: 17, >>>> uvradiation: 0.1, weekRain: 3.1, wh65_batt: 0, windDir: 70, windGust: 2.6, >>>> windSpeed: 1.0, yearRain: 729.9 >>>> >>>> # PYTHONPATH=/usr/share/weewx python3 -m user.gw1000 --test-service >>>> Using configuration file /etc/weewx/weewx.conf >>>> Interrogating GW1000 at 192.168.100.150:45000 >>>> LOOP: 2020-12-12 17:56:02 CET (1607792162) UV: 0, dateTime: >>>> 1607792162, dayRain: 0.0, daymaxwind: 7.7, dummyTemp: 96.3, inHumidity: 59, >>>> inTemp: 63.86, luminosity: 0.0, monthRain: 0.22440944881889766, >>>> outHumidity: 94, outTemp: 36.86, pressure: 29.5004575125, rain: None, >>>> rainRate: 0.0, relbarometer: 1003.1, stormRain: 0.0, usUnits: 1, >>>> uvradiation: 0.1, weekRain: 3.1, wh65_batt: 0, windDir: 77, windGust: >>>> 4.473883703878609, windSpeed: 2.6843302223271652, yearRain: >>>> 28.736220472440944 >>>> LOOP: 2020-12-12 17:56:12 CET (1607792172) dateTime: 1607792172, >>>> dummyTemp: 96.3, usUnits: 1 >>>> >>>> # PYTHONPATH=/usr/share/weewx python3 user/interceptor.py >>>> --device=ecowitt-client --mode=listen --port=9000 >>>> mapped packet: {'dateTime': 1607792207, 'usUnits': 1, 'pressure': >>>> 29.492, 'barometer': 29.613, 'outHumidity': 94.0, 'inHumidity': 58.0, >>>> 'outTemp': 36.9, 'inTemp': 63.9, 'windSpeed': 2.46, 'windGust': 5.82, >>>> 'windDir': 108.0, 'radiation': 0.0, 'rain': None, 'rainRate': 0.0, >>>> 'rainEvent': 0.0, 'UV': 0.0, 'txBatteryStatus': 0.0} >>>> >>>> The windGustDir does not seem to be emitted. There are two other >>>> issues: "rainEvent" and "txBatteryStatus" are missing in the GW1000 >>>> data or are not mapped correctly. Thus, Weewx shows no results for Rain >>>> Event, and indicates a low transmitter battery status. I would like to move >>>> to the GW1000 driver (or better use GW1000.service?), if it would work >>>> flawlessly. But it would be nice, to know, how to proceed now. >>>> >>>> I had GW1000.service activated in parallel to the interceptor driver, >>>> although this is somewhat nonsense, as both data come from the same >>>> sensors, either "translated" to Ecowitt style by the GW1000, or pulled by >>>> the GW1000.service directly. This I did just for a short time to see, if >>>> there were any errors occuring. Although the raw data seem not to be >>>> comparable, I could not see any effect. Actually I went back to the >>>> interceptor driver, so at least rainEvent and the battery status are ok >>>> again. >>>> [email protected] schrieb am Samstag, 12. Dezember 2020 um 15:20:29 >>>> UTC+1: >>>> >>>>> I do not think your problem is related to the "weighting" problem. It >>>>> affected only the daily summaries and, even then, only fields that are >>>>> weighted by the archive length. Wind direction is not one of them. >>>>> >>>>> Unfortunately, I am not very familiar with any of the drivers and >>>>> services you are using. If I understand correctly, the data in question >>>>> was >>>>> collected by interceptor.py. I would assume that's where the problem is. >>>>> Does the driver emit windGustDir on every LOOP packet? Or, does it rely on >>>>> software record generation to provide it at the end of an archive period? >>>>> >>>>> Sorry I cannot be of more help. >>>>> >>>>> -tk >>>>> >>>>> On Thu, Dec 10, 2020 at 11:29 AM Vetti52 <[email protected]> wrote: >>>>> >>>>>> So, I am still not sure, if wee_database --update or --reweight should >>>>>> solve the problem. Or do I have to wait for a special fix? >>>>>> >>>>>> Vetti52 schrieb am Dienstag, 8. Dezember 2020 um 21:01:56 UTC+1: >>>>>> >>>>>>> BTW, there are some values not null in the database after update, as >>>>>>> seen, when filtering the archive for windGustDir <> null: >>>>>>> dateTime windGustDir >>>>>>> 2020-12-08 10:50:00 89.0 >>>>>>> 2020-12-08 05:35:00 95.0 >>>>>>> 2020-12-08 01:25:00 148.0 >>>>>>> 2020-12-07 22:20:00 168.0 >>>>>>> 2020-12-05 01:35:00 120.0 >>>>>>> 2020-12-04 22:00:00 101.0 >>>>>>> >>>>>>> Vetti52 schrieb am Dienstag, 8. Dezember 2020 um 19:33:13 UTC+1: >>>>>>> >>>>>>>> tha update was on 2020-12-04, right. >>>>>>>> >>>>>>>> at 1. >>>>>>>> I am running Weewx on a Raspi4 under buster, installed with apt-get >>>>>>>> install weewx. My station is a EFWS2500, which is a clone of Ecowitt >>>>>>>> 2500. >>>>>>>> The data are collected by a Froggit DS2500, which is a clone of GW1000. >>>>>>>> Weewx still collects the data from interceptor.py ecowitt-client. >>>>>>>> Although >>>>>>>> user.gw1000 is already implemented as a service, I still have not yet >>>>>>>> replaced interceptor.py. >>>>>>>> >>>>>>>> at 2. >>>>>>>> didn't detect ignore_zero_wind in weewx.conf. >>>>>>>> >>>>>>>> at 3. >>>>>>>> record_generation = software >>>>>>>> >>>>>>>> HTH >>>>>>>> >>>>>>>> Thanks >>>>>>>> -ph >>>>>>>> >>>>>>>> [email protected] schrieb am Dienstag, 8. Dezember 2020 um 18:42:04 >>>>>>>> UTC+1: >>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> >>>>>>>>> I assume you updated on 2020-12-04? There is no value for >>>>>>>>> "max_dir" (which is where "gust_dir" comes from) in your daily >>>>>>>>> summaries >>>>>>>>> since that date. >>>>>>>>> >>>>>>>>> Looking through your daily summaries, you are suffering from issue >>>>>>>>> #623 <https://github.com/weewx/weewx/issues/623> (as are most >>>>>>>>> V4.2 users). I am working on a fix for this. However, as far as I >>>>>>>>> know, >>>>>>>>> this problem should not be affecting max_dir, but I may be missing >>>>>>>>> something. >>>>>>>>> >>>>>>>>> A few questions: >>>>>>>>> 1. What kind of hardware? >>>>>>>>> 2. What is your setting for option ignore_zero_wind, if any? >>>>>>>>> 3. What is your setting for option record_generation? >>>>>>>>> >>>>>>>>> -tk >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Dec 8, 2020 at 8:44 AM Vetti52 <[email protected]> wrote: >>>>>>>>> >>>>>>>>>> It says "N/A". >>>>>>>>>> >>>>>>>>>> The output of sqlite: >>>>>>>>>> >>>>>>>>>> 2020-12-08 >>>>>>>>>> 00:00:00|1607382000|0.0|1607382032|10.29|1607442892|148.553777777778|211|148.553777777778|211||141.25403836784|8.54431223906981|211|307.861736888889|307.861736888889 >>>>>>>>>> 2020-12-07 >>>>>>>>>> 00:00:00|1607295600|0.0|1607295628|18.34|1607348466|969.075666666667|288|969.075666666667|288||563.975677916051|-355.104682689556|288|4027.90666114815|4027.90666114815 >>>>>>>>>> 2020-12-06 >>>>>>>>>> 00:00:00|1607209200|0.0|1607209233|9.17|1607219761|77.5104166666667|286|77.5104166666667|286||69.480956552117|2.39263945299658|286|132.33436038966|132.33436038966 >>>>>>>>>> 2020-12-05 >>>>>>>>>> 00:00:00|1607122800|0.0|1607123329|12.53|1607140258|550.433333333334|288|550.433333333334|288||357.29020463216|-182.82480166047|288|1638.89117222222|1638.89117222222 >>>>>>>>>> 2020-12-04 >>>>>>>>>> 00:00:00|1607036400|0.0|1607039337|20.58|1607050956|806.906472222223|288|806.906472222223|288||609.110049598142|-419.170693241216|288|2757.25666137114|2757.25666137114 >>>>>>>>>> 2020-12-03 >>>>>>>>>> 00:00:00|1606950000|0.0|1606950381|14.76|1606982936|938.132555555555|286|161542.062333333|52611|136.0|63684.8752720733|-143160.707316072|52611|3367.86934245679|550406.753590012 >>>>>>>>>> 2020-12-02 >>>>>>>>>> 00:00:00|1606863600|0.0|1606863628|10.29|1606933960|298.527702380952|288|89558.3107142857|86400|97.0|42277.4937890549|-72643.1316507023|61200|647.727487337254|194318.246201176 >>>>>>>>>> 2020-12-01 >>>>>>>>>> 00:00:00|1606777200|0.0|1606778615|9.17|1606778252|265.736888888889|288|79721.0666666667|86400|194.0|42931.7123928354|-43844.9965062139|69900|434.78745508642|130436.236525926 >>>>>>>>>> 2020-11-30 >>>>>>>>>> 00:00:00|1606690800|0.0|1606690818|12.53|1606753726|458.85391053391|287|137656.173160173|86100|138.0|-43283.5066028223|-120674.014788181|77100|1142.32670378797|342698.01113639 >>>>>>>>>> 2020-11-29 >>>>>>>>>> 00:00:00|1606604400|0.0|1606604418|5.82|1606650983|50.8892222222223|288|15266.7666666667|86400|309.0|-11596.0986434007|7995.68851878374|24600|54.2518490740741|16275.5547222222 >>>>>>>>>> 2020-11-28 >>>>>>>>>> 00:00:00|1606518000|0.0|1606518015|11.41|1606565426|165.474444444445|287|49642.3333333333|86100|63.0|47302.3988525243|1887.80479760903|28200|423.739858888889|127121.957666667 >>>>>>>>>> 2020-11-27 >>>>>>>>>> 00:00:00|1606431600|0.0|1606431616|4.47|1606483066|9.17855555555556|288|2753.56666666667|86400|131.0|2261.0873478704|-539.506810163124|4200|10.6575301851852|3197.25905555556 >>>>>>>>>> 2020-11-26 >>>>>>>>>> 00:00:00|1606345200|0.0|1606345204|8.05|1606394153|107.788222222222|287|32336.4666666667|86100|259.0|-25803.5675497035|-15488.3625339286|42000|134.601425382716|40380.4276148148 >>>>>>>>>> 2020-11-25 >>>>>>>>>> 00:00:00|1606258800|0.0|1606258804|9.17|1606302300|337.383111111111|288|101214.933333333|86400|189.0|-19895.5174799498|-97556.2211722166|76500|636.16824345679|190850.473037037 >>>>>>>>>> 2020-11-24 >>>>>>>>>> 00:00:00|1606172400|0.0|1606172999|10.29|1606187816|577.891555555556|288|173367.466666667|86400|213.0|-90246.0852000086|-146738.507905923|86400|1319.29265822222|395787.797466666 >>>>>>>>>> 2020-11-23 >>>>>>>>>> 00:00:00|1606086000|0.0|1606086005|10.29|1606137950|368.728555555556|288|110618.566666667|86400|261.0|-97191.5602413715|-50554.3362971601|72300|718.446958555555|215534.087566667 >>>>>>>>>> 2020-11-22 >>>>>>>>>> 00:00:00|1605999600|0.0|1606008451|14.76|1606041552|645.880888888889|288|193764.266666666|86400|245.0|-171272.270757009|-84245.3229881982|73500|2109.85465123457|632956.39537037 >>>>>>>>>> 2020-11-21 >>>>>>>>>> 00:00:00|1605913200|0.0|1605913439|15.88|1605942546|1114.47377777778|288|334342.133333333|86400|191.0|-175428.251363656|-273150.682819638|86400|4553.4892182716|1366046.76548148 >>>>>>>>>> 2020-11-20 >>>>>>>>>> 00:00:00|1605826800|0.0|1605826872|9.17|1605833902|169.842222222222|288|50952.6666666667|86400|238.0|-42329.2825046772|-24698.2615292903|59400|247.763276197531|74328.9828592593 >>>>>>>>>> 2020-11-19 >>>>>>>>>> 00:00:00|1605740400|0.0|1605798095|22.82|1605788229|919.519666666666|288|275855.9|86400|255.0|-208559.830853793|-91789.2604135498|82800|3926.4786627037|1177943.59881111 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [email protected] schrieb am Dienstag, 8. Dezember 2020 um >>>>>>>>>> 17:30:54 UTC+1: >>>>>>>>>> >>>>>>>>>>> What do you mean by, "there are no longer windDir data in the >>>>>>>>>>> hi/lo section"? Is there a placeholder? Or, white space? Or, N/A? >>>>>>>>>>> >>>>>>>>>>> It might be worth seeing what is in your database. This assumes >>>>>>>>>>> your database is located at /var/lib/weewx/weewx.sdb. If you used >>>>>>>>>>> the >>>>>>>>>>> setup.py install method, it would be at >>>>>>>>>>> /home/weewx/archive/weewx.sdb. >>>>>>>>>>> >>>>>>>>>>> *sqlite3 /var/lib/weewx/weewx.sdb* >>>>>>>>>>> sqlite> *select datetime(dateTime, 'unixepoch', 'localtime'), * >>>>>>>>>>> from archive_day_wind order by dateTime desc limit 20;* >>>>>>>>>>> sqlite> *.quit* >>>>>>>>>>> >>>>>>>>>>> -tk >>>>>>>>>>> >>>>>>>>>>> On Tue, Dec 8, 2020 at 8:15 AM Vetti52 <[email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Since update to version 4.2.0, I have a curious result in >>>>>>>>>>>> season skin. Although $current.windDir is properly displayed and >>>>>>>>>>>> continuously collected, there are no longer windDir data in the >>>>>>>>>>>> hi/lo >>>>>>>>>>>> section. Wind Max shows speed correctly, but no direction. I >>>>>>>>>>>> checked >>>>>>>>>>>> hilo.inc. There are no changes, compared to the original version >>>>>>>>>>>> (hilo.inc.dpkg-dist dated from 2020-04-30). >>>>>>>>>>>> >>>>>>>>>>>> Where should I start to trace the bug? >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>> Google Groups "weewx-user" group. >>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>> https://groups.google.com/d/msgid/weewx-user/bb0a2343-45cf-4885-a7a8-4212c4426592n%40googlegroups.com >>>>>>>>>>>> <https://groups.google.com/d/msgid/weewx-user/bb0a2343-45cf-4885-a7a8-4212c4426592n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>> . >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "weewx-user" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to [email protected]. >>>>>>>>>> >>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/d/msgid/weewx-user/38c3e7ba-b73d-4291-a17f-0620c9f80ac7n%40googlegroups.com >>>>>>>>>> <https://groups.google.com/d/msgid/weewx-user/38c3e7ba-b73d-4291-a17f-0620c9f80ac7n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>> . >>>>>>>>>> >>>>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "weewx-user" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> >>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/weewx-user/625e77f0-c4f3-4640-aecf-1128cb2af4abn%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/weewx-user/625e77f0-c4f3-4640-aecf-1128cb2af4abn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "weewx-user" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> >>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/weewx-user/220a3041-b2a2-4b37-a6c2-3c158b60e20bn%40googlegroups.com >>>> <https://groups.google.com/d/msgid/weewx-user/220a3041-b2a2-4b37-a6c2-3c158b60e20bn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/weewx-user/83d99b6d-863c-4424-afd6-74c119fd25f7n%40googlegroups.com > <https://groups.google.com/d/msgid/weewx-user/83d99b6d-863c-4424-afd6-74c119fd25f7n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zEDCKU5UhT7o6xjihAwCMa%3D7Dnj4W9TMaKpW0t1kHkZ1xQ%40mail.gmail.com.
# # Copyright (c) 2009-2019 Tom Keffer <[email protected]> # # See the file LICENSE.txt for your full rights. # """Statistical accumulators. They accumulate the highs, lows, averages, etc., of a sequence of records.""" # # General strategy. # # Most observation types are scalars, so they can be treated simply. Values are added to a scalar accumulator, # which keeps track of highs, lows, and a sum. When it comes time for extraction, the average over the archive period # is typically produced. # # However, wind is a special case. It is a vector, which has been flatted over at least two scalars, windSpeed and # windDir. Some stations, notably the Davis Vantage, add windGust and windGustDir. The accumulators cannot simply treat # them individually as if they were just another scalar. Instead they must be grouped together. This is done by treating # windSpeed as a 'special' scalar. When it appears, it is coupled with windDir and, if available, windGust and # windGustDir, and added to a vector accumulator. When the other types ( windDir, windGust, and windGustDir) appear, # they are ignored, having already been handled during the processing of type windSpeed. # # When it comes time to extract wind, vector averages are calculated, then the results are flattened again. # from __future__ import absolute_import import logging import math import configobj import six from six.moves import StringIO import weewx from weeutil.weeutil import ListOfDicts, to_float log = logging.getLogger(__name__) # # Default mappings from observation types to accumulator classes and functions # DEFAULTS_INI = """ [Accumulator] [[dateTime]] adder = noop [[dayET]] extractor = last [[dayRain]] extractor = last [[ET]] extractor = sum [[hourRain]] extractor = last [[rain]] extractor = sum [[rain24]] extractor = last [[monthET]] extractor = last [[monthRain]] extractor = last [[stormRain]] extractor = last [[totalRain]] extractor = last [[usUnits]] adder = check_units [[wind]] accumulator = vector extractor = wind [[windDir]] extractor = noop [[windGust]] extractor = noop [[windGustDir]] extractor = noop [[windGust10]] extractor = last [[windGustDir10]] extractor = last [[windrun]] extractor = sum [[windSpeed]] adder = add_wind merger = avg extractor = noop [[windSpeed2]] extractor = last [[windSpeed10]] extractor = last [[yearET]] extractor = last [[yearRain]] extractor = last [[lightning_strike_count]] extractor = sum """ defaults_dict = configobj.ConfigObj(StringIO(DEFAULTS_INI), encoding='utf-8') accum_dict = ListOfDicts(defaults_dict['Accumulator'].dict()) class OutOfSpan(ValueError): """Raised when attempting to add a record outside of the timespan held by an accumulator""" # =============================================================================== # ScalarStats # =============================================================================== class ScalarStats(object): """Accumulates statistics (min, max, average, etc.) for a scalar value. Property 'last' is the last non-None value seen. Property 'lasttime' is the time it was seen. """ default_init = (None, None, None, None, 0.0, 0, 0.0, 0) def __init__(self, stats_tuple=None): self.setStats(stats_tuple) self.last = None self.lasttime = None def setStats(self, stats_tuple=None): (self.min, self.mintime, self.max, self.maxtime, self.sum, self.count, self.wsum, self.sumtime) = stats_tuple if stats_tuple else ScalarStats.default_init def getStatsTuple(self): """Return a stats-tuple. That is, a tuple containing the gathered statistics. This tuple can be used to update the stats database""" return (self.min, self.mintime, self.max, self.maxtime, self.sum, self.count, self.wsum, self.sumtime) def mergeHiLo(self, x_stats): """Merge the highs and lows of another accumulator into myself.""" if x_stats.min is not None: if self.min is None or x_stats.min < self.min: self.min = x_stats.min self.mintime = x_stats.mintime if x_stats.max is not None: if self.max is None or x_stats.max > self.max: self.max = x_stats.max self.maxtime = x_stats.maxtime if x_stats.lasttime is not None: if self.lasttime is None or x_stats.lasttime >= self.lasttime: self.lasttime = x_stats.lasttime self.last = x_stats.last def mergeSum(self, x_stats): """Merge the sum and count of another accumulator into myself.""" self.sum += x_stats.sum self.count += x_stats.count self.wsum += x_stats.wsum self.sumtime += x_stats.sumtime def addHiLo(self, val, ts): """Include a scalar value in my highs and lows. val: A scalar value ts: The timestamp. """ # If this is a string, try to convert it to a float. if isinstance(val, (six.string_types, six.text_type)): # Fail hard if unable to do the conversion: val = to_float(val) # Check for None and NaN: if val is not None and val == val: if self.min is None or val < self.min: self.min = val self.mintime = ts if self.max is None or val > self.max: self.max = val self.maxtime = ts if self.lasttime is None or ts >= self.lasttime: self.last = val self.lasttime = ts def addSum(self, val, weight=1): """Add a scalar value to my running sum and count.""" # If this is a string, try to convert it to a float. if isinstance(val, (six.string_types, six.text_type)): # Fail hard if unable to do the conversion: val = to_float(val) # Check for None and NaN: if val is not None and val == val: self.sum += val self.count += 1 self.wsum += val * weight self.sumtime += weight @property def avg(self): return self.wsum / self.sumtime if self.count else None class VecStats(object): """Accumulates statistics for a vector value. Property 'last' is the last non-None value seen. It is a two-way tuple (mag, dir). Property 'lasttime' is the time it was seen. """ default_init = (None, None, None, None, 0.0, 0, 0.0, 0, None, 0.0, 0.0, 0, 0.0, 0.0) def __init__(self, stats_tuple=None): self.setStats(stats_tuple) self.last = (None, None) self.lasttime = None def setStats(self, stats_tuple=None): (self.min, self.mintime, self.max, self.maxtime, self.sum, self.count, self.wsum, self.sumtime, self.max_dir, self.xsum, self.ysum, self.dirsumtime, self.squaresum, self.wsquaresum) = stats_tuple if stats_tuple else VecStats.default_init def getStatsTuple(self): """Return a stats-tuple. That is, a tuple containing the gathered statistics.""" return (self.min, self.mintime, self.max, self.maxtime, self.sum, self.count, self.wsum, self.sumtime, self.max_dir, self.xsum, self.ysum, self.dirsumtime, self.squaresum, self.wsquaresum) def mergeHiLo(self, x_stats): """Merge the highs and lows of another accumulator into myself.""" log.debug("mergeHiLo: self.max=%s, x_stats.max=%s", self.max, x_stats.max) if x_stats.min is not None: if self.min is None or x_stats.min < self.min: self.min = x_stats.min self.mintime = x_stats.mintime if x_stats.max is not None: if self.max is None or x_stats.max > self.max: self.max = x_stats.max self.maxtime = x_stats.maxtime self.max_dir = x_stats.max_dir if x_stats.lasttime is not None: if self.lasttime is None or x_stats.lasttime >= self.lasttime: self.lasttime = x_stats.lasttime self.last = x_stats.last def mergeSum(self, x_stats): """Merge the sum and count of another accumulator into myself.""" self.sum += x_stats.sum self.count += x_stats.count self.wsum += x_stats.wsum self.sumtime += x_stats.sumtime self.xsum += x_stats.xsum self.ysum += x_stats.ysum self.dirsumtime += x_stats.dirsumtime self.squaresum += x_stats.squaresum self.wsquaresum += x_stats.wsquaresum def addHiLo(self, val, ts): """Include a vector value in my highs and lows. val: A vector value. It is a 2-way tuple (mag, dir). ts: The timestamp. """ speed, dirN = val # If this is a string, try to convert it to a float. if isinstance(speed, (six.string_types, six.text_type)): # Fail hard if unable to do the conversion: speed = to_float(speed) if isinstance(dirN, (six.string_types, six.text_type)): # Fail hard if unable to do the conversion: dirN = to_float(dirN) log.debug("in addHiLo: speed=%s, dirN=%s, self.max=%s", speed, dirN, self.max) # Check for None and NaN: if speed is not None and speed == speed: if self.min is None or speed < self.min: self.min = speed self.mintime = ts if self.max is None or speed > self.max: self.max = speed self.maxtime = ts self.max_dir = dirN if self.lasttime is None or ts >= self.lasttime: self.last = (speed, dirN) self.lasttime = ts def addSum(self, val, weight=1): """Add a vector value to my sum and squaresum. val: A vector value. It is a 2-way tuple (mag, dir) """ speed, dirN = val # If this is a string, try to convert it to a float. if isinstance(speed, (six.string_types, six.text_type)): # Fail hard if unable to do the conversion: speed = to_float(speed) if isinstance(dirN, (six.string_types, six.text_type)): # Fail hard if unable to do the conversion: dirN = to_float(dirN) # Check for None and NaN: if speed is not None and speed == speed: self.sum += speed self.count += 1 self.wsum += weight * speed self.sumtime += weight self.squaresum += speed ** 2 self.wsquaresum += weight * speed ** 2 if dirN is not None: self.xsum += weight * speed * math.cos(math.radians(90.0 - dirN)) self.ysum += weight * speed * math.sin(math.radians(90.0 - dirN)) # It's OK for direction to be None, provided speed is zero: if dirN is not None or speed == 0: self.dirsumtime += weight @property def avg(self): return self.wsum / self.sumtime if self.count else None @property def rms(self): return math.sqrt(self.wsquaresum / self.sumtime) if self.count else None @property def vec_avg(self): if self.count: return math.sqrt((self.xsum ** 2 + self.ysum ** 2) / self.sumtime ** 2) @property def vec_dir(self): if self.dirsumtime and (self.ysum or self.xsum): _result = 90.0 - math.degrees(math.atan2(self.ysum, self.xsum)) if _result < 0.0: _result += 360.0 return _result # Return the last known direction when our vector sum is 0 return self.last[1] # =============================================================================== # FirstLastAccum # =============================================================================== class FirstLastAccum(object): """Minimal accumulator, suitable for strings. It can only return the first and last strings it has seen, along with their timestamps. """ default_init = (None, None, None, None) def __init__(self, stats_tuple=None): self.first = None self.firsttime = None self.last = None self.lasttime = None def setStats(self, stats_tuple=None): self.first, self.firsttime, self.last, self.lasttime = stats_tuple \ if stats_tuple else FirstLastAccum.default_init def getStatsTuple(self): """Return a stats-tuple. That is, a tuple containing the gathered statistics. This tuple can be used to update the stats database""" return self.first, self.firsttime, self.last, self.lasttime def mergeHiLo(self, x_stats): """Merge the highs and lows of another accumulator into myself.""" if x_stats.firsttime is not None: if self.firsttime is None or x_stats.firsttime < self.firsttime: self.firsttime = x_stats.firsttime self.first = x_stats.first if x_stats.lasttime is not None: if self.lasttime is None or x_stats.lasttime >= self.lasttime: self.lasttime = x_stats.lasttime self.last = x_stats.last def mergeSum(self, x_stats): """Merge the count of another accumulator into myself.""" pass def addHiLo(self, val, ts): """Include a value in my stats. val: A value of almost any type. It will be converted to a string before being accumulated. ts: The timestamp. """ if val is not None: string_val = str(val) if self.firsttime is None or ts < self.firsttime: self.first = string_val self.firsttime = ts if self.lasttime is None or ts >= self.lasttime: self.last = string_val self.lasttime = ts def addSum(self, val, weight=1): """Add a scalar value to my running count.""" pass # =============================================================================== # Class Accum # =============================================================================== class Accum(dict): """Accumulates statistics for a set of observation types.""" def __init__(self, timespan, unit_system=None): """Initialize a Accum. timespan: The time period over which stats will be accumulated. unit_system: The unit system used by the accumulator""" self.timespan = timespan # Set the accumulator's unit system. Usually left unspecified until the # first observation comes in for normal operation or pre-set if # obtaining a historical accumulator. self.unit_system = unit_system def addRecord(self, record, add_hilo=True, weight=1): """Add a record to my running statistics. The record must have keys 'dateTime' and 'usUnits'.""" # Check to see if the record is within my observation timespan if not self.timespan.includesArchiveTime(record['dateTime']): raise OutOfSpan("Attempt to add out-of-interval record (%s) to timespan (%s)" % (record['dateTime'], self.timespan)) for obs_type in record: # Get the proper function ... func = get_add_function(obs_type) # ... then call it. func(self, record, obs_type, add_hilo, weight) def updateHiLo(self, accumulator): """Merge the high/low stats of another accumulator into me.""" if accumulator.timespan.start < self.timespan.start or accumulator.timespan.stop > self.timespan.stop: raise OutOfSpan("Attempt to merge an accumulator whose timespan is not a subset") self._check_units(accumulator.unit_system) for obs_type in accumulator: # Initialize the type if we have not seen it before self._init_type(obs_type) # Get the proper function ... func = get_merge_function(obs_type) # ... then call it func(self, accumulator, obs_type) def getRecord(self): """Extract a record out of the results in the accumulator.""" # All records have a timestamp and unit type record = {'dateTime': self.timespan.stop, 'usUnits': self.unit_system} return self.augmentRecord(record) def augmentRecord(self, record): # Go through all observation types. for obs_type in self: # If the type does not appear in the record, then add it: if obs_type not in record: # Get the proper extraction function... func = get_extract_function(obs_type) # ... then call it func(self, record, obs_type) return record def set_stats(self, obs_type, stats_tuple): self._init_type(obs_type) self[obs_type].setStats(stats_tuple) # # Begin add functions. These add a record to the accumulator. # def add_value(self, record, obs_type, add_hilo, weight): """Add a single observation to myself.""" val = record[obs_type] # If the type has not been seen before, initialize it self._init_type(obs_type) # Then add to highs/lows, and to the running sum: if add_hilo: self[obs_type].addHiLo(val, record['dateTime']) self[obs_type].addSum(val, weight=weight) def add_wind_value(self, record, obs_type, add_hilo, weight): """Add a single observation of type wind to myself.""" if obs_type in ['windDir', 'windGust', 'windGustDir']: return if weewx.debug: assert (obs_type == 'windSpeed') # First add it to regular old 'windSpeed', then # treat it like a vector. self.add_value(record, obs_type, add_hilo, weight) # If the type has not been seen before, initialize it. self._init_type('wind') # Then add to highs/lows. if add_hilo: speed, dirN = record.get('windSpeed'), record.get('windDir') gust_speed, gust_dirN = record.get('windGust'), record.get('windGustDir', record.get('windDir')) log.debug("about to addHiLo windSpeed=%s, windDir=%s", speed, dirN) log.debug("about to addHiLo windGust=%s, windGustDir=%s", gust_speed, gust_dirN) self['wind'].addHiLo((record.get('windSpeed'), record.get('windDir')), record['dateTime']) # If the station does not provide windGustDir, then substitute windDir. # See issue #320, https://bit.ly/2HSo0ju self['wind'].addHiLo((record.get('windGust'), record.get('windGustDir', record.get('windDir'))), record['dateTime']) # Add to the running sum. self['wind'].addSum((record['windSpeed'], record.get('windDir')), weight=weight) def check_units(self, record, obs_type, add_hilo, weight): if weewx.debug: assert (obs_type == 'usUnits') self._check_units(record['usUnits']) def noop(self, record, obs_type, add_hilo=True, weight=1): pass # # Begin hi/lo merge functions. These are called when merging two accumulators # def merge_minmax(self, x_accumulator, obs_type): """Merge value in another accumulator, using min/max""" self[obs_type].mergeHiLo(x_accumulator[obs_type]) def merge_avg(self, x_accumulator, obs_type): """Merge value in another accumulator, using avg for max""" x_stats = x_accumulator[obs_type] if x_stats.min is not None: if self[obs_type].min is None or x_stats.min < self[obs_type].min: self[obs_type].min = x_stats.min self[obs_type].mintime = x_stats.mintime if x_stats.avg is not None: if self[obs_type].max is None or x_stats.avg > self[obs_type].max: self[obs_type].max = x_stats.avg self[obs_type].maxtime = x_accumulator.timespan.stop if x_stats.lasttime is not None: if self[obs_type].lasttime is None or x_stats.lasttime >= self[obs_type].lasttime: self[obs_type].lasttime = x_stats.lasttime self[obs_type].last = x_stats.last # # Begin extraction functions. These extract a record out of the accumulator. # def extract_wind(self, record, obs_type): """Extract wind values from myself, and put in a record.""" # Wind records must be flattened into the separate categories: if 'windSpeed' not in record: record['windSpeed'] = self[obs_type].avg if 'windDir' not in record: record['windDir'] = self[obs_type].vec_dir if 'windGust' not in record: record['windGust'] = self[obs_type].max if 'windGustDir' not in record: log.info("Extracting wind; windGustDir not in record. Using %s", self[obs_type].max_dir) record['windGustDir'] = self[obs_type].max_dir else: log.info("Extracting wind; record['windGustDir']=%s", record['windGustDir']) def extract_sum(self, record, obs_type): record[obs_type] = self[obs_type].sum def extract_last(self, record, obs_type): record[obs_type] = self[obs_type].last def extract_avg(self, record, obs_type): record[obs_type] = self[obs_type].avg def extract_min(self, record, obs_type): record[obs_type] = self[obs_type].min def extract_max(self, record, obs_type): record[obs_type] = self[obs_type].max def extract_count(self, record, obs_type): record[obs_type] = self[obs_type].count # # Miscellaneous, utility functions # def _init_type(self, obs_type): """Add a given observation type to my dictionary.""" # Do nothing if this type has already been initialized: if obs_type in self: return # Get a new accumulator of the proper type self[obs_type] = new_accumulator(obs_type) def _check_units(self, new_unit_system): # If no unit system has been specified for me yet, adopt the incoming # system if self.unit_system is None: self.unit_system = new_unit_system else: # Otherwise, make sure they match if self.unit_system != new_unit_system: raise ValueError("Unit system mismatch %d v. %d" % (self.unit_system, new_unit_system)) @property def isEmpty(self): return self.unit_system is None # =============================================================================== # Configuration dictionaries # =============================================================================== # # Mappings from convenient string nicknames, which can be used in a config file, # to actual functions and classes # ACCUM_TYPES = { 'scalar': ScalarStats, 'vector': VecStats, 'firstlast': FirstLastAccum } ADD_FUNCTIONS = { 'add': Accum.add_value, 'add_wind': Accum.add_wind_value, 'check_units': Accum.check_units, 'noop': Accum.noop } MERGE_FUNCTIONS = { 'minmax': Accum.merge_minmax, 'avg': Accum.merge_avg } EXTRACT_FUNCTIONS = { 'avg': Accum.extract_avg, 'count': Accum.extract_count, 'last': Accum.extract_last, 'max': Accum.extract_max, 'min': Accum.extract_min, 'noop': Accum.noop, 'sum': Accum.extract_sum, 'wind': Accum.extract_wind, } # The default actions for an individual observation type OBS_DEFAULTS = { 'accumulator': 'scalar', 'adder': 'add', 'merger': 'minmax', 'extractor': 'avg' } def initialize(config_dict): # Add the configuration dictionary to the beginning of the list of maps. # This will cause it to override the defaults global accum_dict accum_dict.maps.insert(0, config_dict.get('Accumulator', {})) def new_accumulator(obs_type): """Instantiate an accumulator, appropriate for type 'obs_type'.""" global accum_dict # Get the options for this type. Substitute the defaults if they have not been specified obs_options = accum_dict.get(obs_type, OBS_DEFAULTS) # Get the nickname of the accumulator. Default is 'scalar' accum_nickname = obs_options.get('accumulator', 'scalar') # Instantiate and return the accumulator. # If we don't know this nickname, then fail hard with a KeyError return ACCUM_TYPES[accum_nickname]() def get_add_function(obs_type): """Get an adder function appropriate for type 'obs_type'.""" global accum_dict # Get the options for this type. Substitute the defaults if they have not been specified obs_options = accum_dict.get(obs_type, OBS_DEFAULTS) # Get the nickname of the adder. Default is 'add' add_nickname = obs_options.get('adder', 'add') # If we don't know this nickname, then fail hard with a KeyError return ADD_FUNCTIONS[add_nickname] def get_merge_function(obs_type): """Get a merge function appropriate for type 'obs_type'.""" global accum_dict # Get the options for this type. Substitute the defaults if they have not been specified obs_options = accum_dict.get(obs_type, OBS_DEFAULTS) # Get the nickname of the merger. Default is 'minmax' add_nickname = obs_options.get('merger', 'minmax') # If we don't know this nickname, then fail hard with a KeyError return MERGE_FUNCTIONS[add_nickname] def get_extract_function(obs_type): """Get an extraction function appropriate for type 'obs_type'.""" global accum_dict # Get the options for this type. Substitute the defaults if they have not been specified obs_options = accum_dict.get(obs_type, OBS_DEFAULTS) # Get the nickname of the extractor. Default is 'avg' add_nickname = obs_options.get('extractor', 'avg') # If we don't know this nickname, then fail hard with a KeyError return EXTRACT_FUNCTIONS[add_nickname]
