That tells us that the driver is not emitting windGustDir, but it's not telling us why it cannot be extracted out of the accumulators by WeeWX.
Attached is an instrumented version of accum.py. Swap the version you have with it. It should be in /usr/share/weewx/weewx/accum.py. Set debug=1, restart weewx, then let it run through the first reporting cycle. Post the log. -tk On Sat, Dec 12, 2020 at 9:33 AM Vetti52 <[email protected]> wrote: > Here a comparison of three outputs, first as GW1000.service , then as > gw1000 driver and finally the current working ecowitt-client by the > interceptor driver: > > # PYTHONPATH=/usr/share/weewx python3 -m user.gw1000 --test-driver > Using configuration file /etc/weewx/weewx.conf > Interrogating GW1000 at 192.168.100.150:45000 > 2020-12-12 17:55:00 CET (1607792100): UV: 0, dateTime: 1607792100, > dayRain: 0.0, daymaxwind: 7.7, inHumidity: 59, inTemp: 17.7, luminosity: > 0.0, monthRain: 5.7, outHumidity: 93, outTemp: 2.7, pressure: 999.0, rain: > None, rainRate: 0.0, relbarometer: 1003.1, stormRain: 0.0, usUnits: 17, > uvradiation: 0.1, weekRain: 3.1, wh65_batt: 0, windDir: 70, windGust: 2.6, > windSpeed: 1.0, yearRain: 729.9 > > # PYTHONPATH=/usr/share/weewx python3 -m user.gw1000 --test-service > Using configuration file /etc/weewx/weewx.conf > Interrogating GW1000 at 192.168.100.150:45000 > LOOP: 2020-12-12 17:56:02 CET (1607792162) UV: 0, dateTime: 1607792162, > dayRain: 0.0, daymaxwind: 7.7, dummyTemp: 96.3, inHumidity: 59, inTemp: > 63.86, luminosity: 0.0, monthRain: 0.22440944881889766, outHumidity: 94, > outTemp: 36.86, pressure: 29.5004575125, rain: None, rainRate: 0.0, > relbarometer: 1003.1, stormRain: 0.0, usUnits: 1, uvradiation: 0.1, > weekRain: 3.1, wh65_batt: 0, windDir: 77, windGust: 4.473883703878609, > windSpeed: 2.6843302223271652, yearRain: 28.736220472440944 > LOOP: 2020-12-12 17:56:12 CET (1607792172) dateTime: 1607792172, > dummyTemp: 96.3, usUnits: 1 > > # PYTHONPATH=/usr/share/weewx python3 user/interceptor.py > --device=ecowitt-client --mode=listen --port=9000 > mapped packet: {'dateTime': 1607792207, 'usUnits': 1, 'pressure': 29.492, > 'barometer': 29.613, 'outHumidity': 94.0, 'inHumidity': 58.0, 'outTemp': > 36.9, 'inTemp': 63.9, 'windSpeed': 2.46, 'windGust': 5.82, 'windDir': > 108.0, 'radiation': 0.0, 'rain': None, 'rainRate': 0.0, 'rainEvent': 0.0, > 'UV': 0.0, 'txBatteryStatus': 0.0} > > The windGustDir does not seem to be emitted. There are two other issues: " > rainEvent" and "txBatteryStatus" are missing in the GW1000 data or are > not mapped correctly. Thus, Weewx shows no results for Rain Event, and > indicates a low transmitter battery status. I would like to move to the > GW1000 driver (or better use GW1000.service?), if it would work > flawlessly. But it would be nice, to know, how to proceed now. > > I had GW1000.service activated in parallel to the interceptor driver, > although this is somewhat nonsense, as both data come from the same > sensors, either "translated" to Ecowitt style by the GW1000, or pulled by > the GW1000.service directly. This I did just for a short time to see, if > there were any errors occuring. Although the raw data seem not to be > comparable, I could not see any effect. Actually I went back to the > interceptor driver, so at least rainEvent and the battery status are ok > again. > [email protected] schrieb am Samstag, 12. Dezember 2020 um 15:20:29 UTC+1: > >> I do not think your problem is related to the "weighting" problem. It >> affected only the daily summaries and, even then, only fields that are >> weighted by the archive length. Wind direction is not one of them. >> >> Unfortunately, I am not very familiar with any of the drivers and >> services you are using. If I understand correctly, the data in question was >> collected by interceptor.py. I would assume that's where the problem is. >> Does the driver emit windGustDir on every LOOP packet? Or, does it rely on >> software record generation to provide it at the end of an archive period? >> >> Sorry I cannot be of more help. >> >> -tk >> >> On Thu, Dec 10, 2020 at 11:29 AM Vetti52 <[email protected]> wrote: >> >>> So, I am still not sure, if wee_database --update or --reweight should >>> solve the problem. Or do I have to wait for a special fix? >>> >>> Vetti52 schrieb am Dienstag, 8. Dezember 2020 um 21:01:56 UTC+1: >>> >>>> BTW, there are some values not null in the database after update, as >>>> seen, when filtering the archive for windGustDir <> null: >>>> dateTime windGustDir >>>> 2020-12-08 10:50:00 89.0 >>>> 2020-12-08 05:35:00 95.0 >>>> 2020-12-08 01:25:00 148.0 >>>> 2020-12-07 22:20:00 168.0 >>>> 2020-12-05 01:35:00 120.0 >>>> 2020-12-04 22:00:00 101.0 >>>> >>>> Vetti52 schrieb am Dienstag, 8. Dezember 2020 um 19:33:13 UTC+1: >>>> >>>>> tha update was on 2020-12-04, right. >>>>> >>>>> at 1. >>>>> I am running Weewx on a Raspi4 under buster, installed with apt-get >>>>> install weewx. My station is a EFWS2500, which is a clone of Ecowitt 2500. >>>>> The data are collected by a Froggit DS2500, which is a clone of GW1000. >>>>> Weewx still collects the data from interceptor.py ecowitt-client. Although >>>>> user.gw1000 is already implemented as a service, I still have not yet >>>>> replaced interceptor.py. >>>>> >>>>> at 2. >>>>> didn't detect ignore_zero_wind in weewx.conf. >>>>> >>>>> at 3. >>>>> record_generation = software >>>>> >>>>> HTH >>>>> >>>>> Thanks >>>>> -ph >>>>> >>>>> [email protected] schrieb am Dienstag, 8. Dezember 2020 um 18:42:04 >>>>> UTC+1: >>>>> >>>>>> Thanks. >>>>>> >>>>>> I assume you updated on 2020-12-04? There is no value for "max_dir" >>>>>> (which is where "gust_dir" comes from) in your daily summaries since that >>>>>> date. >>>>>> >>>>>> Looking through your daily summaries, you are suffering from issue >>>>>> #623 <https://github.com/weewx/weewx/issues/623> (as are most V4.2 >>>>>> users). I am working on a fix for this. However, as far as I know, this >>>>>> problem should not be affecting max_dir, but I may be missing something. >>>>>> >>>>>> A few questions: >>>>>> 1. What kind of hardware? >>>>>> 2. What is your setting for option ignore_zero_wind, if any? >>>>>> 3. What is your setting for option record_generation? >>>>>> >>>>>> -tk >>>>>> >>>>>> >>>>>> On Tue, Dec 8, 2020 at 8:44 AM Vetti52 <[email protected]> wrote: >>>>>> >>>>>>> It says "N/A". >>>>>>> >>>>>>> The output of sqlite: >>>>>>> >>>>>>> 2020-12-08 >>>>>>> 00:00:00|1607382000|0.0|1607382032|10.29|1607442892|148.553777777778|211|148.553777777778|211||141.25403836784|8.54431223906981|211|307.861736888889|307.861736888889 >>>>>>> 2020-12-07 >>>>>>> 00:00:00|1607295600|0.0|1607295628|18.34|1607348466|969.075666666667|288|969.075666666667|288||563.975677916051|-355.104682689556|288|4027.90666114815|4027.90666114815 >>>>>>> 2020-12-06 >>>>>>> 00:00:00|1607209200|0.0|1607209233|9.17|1607219761|77.5104166666667|286|77.5104166666667|286||69.480956552117|2.39263945299658|286|132.33436038966|132.33436038966 >>>>>>> 2020-12-05 >>>>>>> 00:00:00|1607122800|0.0|1607123329|12.53|1607140258|550.433333333334|288|550.433333333334|288||357.29020463216|-182.82480166047|288|1638.89117222222|1638.89117222222 >>>>>>> 2020-12-04 >>>>>>> 00:00:00|1607036400|0.0|1607039337|20.58|1607050956|806.906472222223|288|806.906472222223|288||609.110049598142|-419.170693241216|288|2757.25666137114|2757.25666137114 >>>>>>> 2020-12-03 >>>>>>> 00:00:00|1606950000|0.0|1606950381|14.76|1606982936|938.132555555555|286|161542.062333333|52611|136.0|63684.8752720733|-143160.707316072|52611|3367.86934245679|550406.753590012 >>>>>>> 2020-12-02 >>>>>>> 00:00:00|1606863600|0.0|1606863628|10.29|1606933960|298.527702380952|288|89558.3107142857|86400|97.0|42277.4937890549|-72643.1316507023|61200|647.727487337254|194318.246201176 >>>>>>> 2020-12-01 >>>>>>> 00:00:00|1606777200|0.0|1606778615|9.17|1606778252|265.736888888889|288|79721.0666666667|86400|194.0|42931.7123928354|-43844.9965062139|69900|434.78745508642|130436.236525926 >>>>>>> 2020-11-30 >>>>>>> 00:00:00|1606690800|0.0|1606690818|12.53|1606753726|458.85391053391|287|137656.173160173|86100|138.0|-43283.5066028223|-120674.014788181|77100|1142.32670378797|342698.01113639 >>>>>>> 2020-11-29 >>>>>>> 00:00:00|1606604400|0.0|1606604418|5.82|1606650983|50.8892222222223|288|15266.7666666667|86400|309.0|-11596.0986434007|7995.68851878374|24600|54.2518490740741|16275.5547222222 >>>>>>> 2020-11-28 >>>>>>> 00:00:00|1606518000|0.0|1606518015|11.41|1606565426|165.474444444445|287|49642.3333333333|86100|63.0|47302.3988525243|1887.80479760903|28200|423.739858888889|127121.957666667 >>>>>>> 2020-11-27 >>>>>>> 00:00:00|1606431600|0.0|1606431616|4.47|1606483066|9.17855555555556|288|2753.56666666667|86400|131.0|2261.0873478704|-539.506810163124|4200|10.6575301851852|3197.25905555556 >>>>>>> 2020-11-26 >>>>>>> 00:00:00|1606345200|0.0|1606345204|8.05|1606394153|107.788222222222|287|32336.4666666667|86100|259.0|-25803.5675497035|-15488.3625339286|42000|134.601425382716|40380.4276148148 >>>>>>> 2020-11-25 >>>>>>> 00:00:00|1606258800|0.0|1606258804|9.17|1606302300|337.383111111111|288|101214.933333333|86400|189.0|-19895.5174799498|-97556.2211722166|76500|636.16824345679|190850.473037037 >>>>>>> 2020-11-24 >>>>>>> 00:00:00|1606172400|0.0|1606172999|10.29|1606187816|577.891555555556|288|173367.466666667|86400|213.0|-90246.0852000086|-146738.507905923|86400|1319.29265822222|395787.797466666 >>>>>>> 2020-11-23 >>>>>>> 00:00:00|1606086000|0.0|1606086005|10.29|1606137950|368.728555555556|288|110618.566666667|86400|261.0|-97191.5602413715|-50554.3362971601|72300|718.446958555555|215534.087566667 >>>>>>> 2020-11-22 >>>>>>> 00:00:00|1605999600|0.0|1606008451|14.76|1606041552|645.880888888889|288|193764.266666666|86400|245.0|-171272.270757009|-84245.3229881982|73500|2109.85465123457|632956.39537037 >>>>>>> 2020-11-21 >>>>>>> 00:00:00|1605913200|0.0|1605913439|15.88|1605942546|1114.47377777778|288|334342.133333333|86400|191.0|-175428.251363656|-273150.682819638|86400|4553.4892182716|1366046.76548148 >>>>>>> 2020-11-20 >>>>>>> 00:00:00|1605826800|0.0|1605826872|9.17|1605833902|169.842222222222|288|50952.6666666667|86400|238.0|-42329.2825046772|-24698.2615292903|59400|247.763276197531|74328.9828592593 >>>>>>> 2020-11-19 >>>>>>> 00:00:00|1605740400|0.0|1605798095|22.82|1605788229|919.519666666666|288|275855.9|86400|255.0|-208559.830853793|-91789.2604135498|82800|3926.4786627037|1177943.59881111 >>>>>>> >>>>>>> >>>>>>> [email protected] schrieb am Dienstag, 8. Dezember 2020 um 17:30:54 >>>>>>> UTC+1: >>>>>>> >>>>>>>> What do you mean by, "there are no longer windDir data in the hi/lo >>>>>>>> section"? Is there a placeholder? Or, white space? Or, N/A? >>>>>>>> >>>>>>>> It might be worth seeing what is in your database. This assumes >>>>>>>> your database is located at /var/lib/weewx/weewx.sdb. If you used the >>>>>>>> setup.py install method, it would be at /home/weewx/archive/weewx.sdb. >>>>>>>> >>>>>>>> *sqlite3 /var/lib/weewx/weewx.sdb* >>>>>>>> sqlite> *select datetime(dateTime, 'unixepoch', 'localtime'), * >>>>>>>> from archive_day_wind order by dateTime desc limit 20;* >>>>>>>> sqlite> *.quit* >>>>>>>> >>>>>>>> -tk >>>>>>>> >>>>>>>> On Tue, Dec 8, 2020 at 8:15 AM Vetti52 <[email protected]> wrote: >>>>>>>> >>>>>>>>> Since update to version 4.2.0, I have a curious result in season >>>>>>>>> skin. Although $current.windDir is properly displayed and continuously >>>>>>>>> collected, there are no longer windDir data in the hi/lo section. >>>>>>>>> Wind Max >>>>>>>>> shows speed correctly, but no direction. I checked hilo.inc. There >>>>>>>>> are no >>>>>>>>> changes, compared to the original version (hilo.inc.dpkg-dist dated >>>>>>>>> from >>>>>>>>> 2020-04-30). >>>>>>>>> >>>>>>>>> Where should I start to trace the bug? >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "weewx-user" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/weewx-user/bb0a2343-45cf-4885-a7a8-4212c4426592n%40googlegroups.com >>>>>>>>> <https://groups.google.com/d/msgid/weewx-user/bb0a2343-45cf-4885-a7a8-4212c4426592n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "weewx-user" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> >>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/weewx-user/38c3e7ba-b73d-4291-a17f-0620c9f80ac7n%40googlegroups.com >>>>>>> <https://groups.google.com/d/msgid/weewx-user/38c3e7ba-b73d-4291-a17f-0620c9f80ac7n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "weewx-user" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/weewx-user/625e77f0-c4f3-4640-aecf-1128cb2af4abn%40googlegroups.com >>> <https://groups.google.com/d/msgid/weewx-user/625e77f0-c4f3-4640-aecf-1128cb2af4abn%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/weewx-user/220a3041-b2a2-4b37-a6c2-3c158b60e20bn%40googlegroups.com > <https://groups.google.com/d/msgid/weewx-user/220a3041-b2a2-4b37-a6c2-3c158b60e20bn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zEDJoR34ZJU9mhXq1TdYUcV_2JxhYOGN-qVtnH1ZdKbfuA%40mail.gmail.com.
# # Copyright (c) 2009-2020 Tom Keffer <[email protected]> # # See the file LICENSE.txt for your full rights. # """Statistical accumulators. They accumulate the highs, lows, averages, etc., of a sequence of records.""" # # General strategy. # # Most observation types are scalars, so they can be treated simply. Values are added to a scalar # accumulator, which keeps track of highs, lows, and a sum. When it comes time for extraction, the # average over the archive period is typically produced. # # However, wind is a special case. It is a vector, which has been flatted over at least two # scalars, windSpeed and windDir. Some stations, notably the Davis Vantage, add windGust and # windGustDir. The accumulators cannot simply treat them individually as if they were just another # scalar. Instead they must be grouped together. This is done by treating windSpeed as a 'special' # scalar. When it appears, it is coupled with windDir and, if available, windGust and windGustDir, # and added to a vector accumulator. When the other types ( windDir, windGust, and windGustDir) # appear, they are ignored, having already been handled during the processing of type windSpeed. # # When it comes time to extract wind, vector averages are calculated, then the results are # flattened again. # from __future__ import absolute_import import logging import math import configobj import six from six.moves import StringIO import weewx from weeutil.weeutil import ListOfDicts, to_float log = logging.getLogger(__name__) # # Default mappings from observation types to accumulator classes and functions # DEFAULTS_INI = """ [Accumulator] [[dateTime]] adder = noop [[dayET]] extractor = last [[dayRain]] extractor = last [[ET]] extractor = sum [[hourRain]] extractor = last [[rain]] extractor = sum [[rain24]] extractor = last [[monthET]] extractor = last [[monthRain]] extractor = last [[stormRain]] extractor = last [[totalRain]] extractor = last [[usUnits]] adder = check_units [[wind]] accumulator = vector extractor = wind [[windDir]] extractor = noop [[windGust]] extractor = noop [[windGustDir]] extractor = noop [[windGust10]] extractor = last [[windGustDir10]] extractor = last [[windrun]] extractor = sum [[windSpeed]] adder = add_wind merger = avg extractor = noop [[windSpeed2]] extractor = last [[windSpeed10]] extractor = last [[yearET]] extractor = last [[yearRain]] extractor = last [[lightning_strike_count]] extractor = sum """ defaults_dict = configobj.ConfigObj(StringIO(DEFAULTS_INI), encoding='utf-8') accum_dict = ListOfDicts(defaults_dict['Accumulator'].dict()) class OutOfSpan(ValueError): """Raised when attempting to add a record outside of the timespan held by an accumulator""" # =============================================================================== # ScalarStats # =============================================================================== class ScalarStats(object): """Accumulates statistics (min, max, average, etc.) for a scalar value. Property 'last' is the last non-None value seen. Property 'lasttime' is the time it was seen. """ default_init = (None, None, None, None, 0.0, 0, 0.0, 0) def __init__(self, stats_tuple=None): self.setStats(stats_tuple) self.last = None self.lasttime = None def setStats(self, stats_tuple=None): (self.min, self.mintime, self.max, self.maxtime, self.sum, self.count, self.wsum, self.sumtime) = stats_tuple if stats_tuple else ScalarStats.default_init def getStatsTuple(self): """Return a stats-tuple. That is, a tuple containing the gathered statistics. This tuple can be used to update the stats database""" return (self.min, self.mintime, self.max, self.maxtime, self.sum, self.count, self.wsum, self.sumtime) def mergeHiLo(self, x_stats): """Merge the highs and lows of another accumulator into myself.""" if x_stats.min is not None: if self.min is None or x_stats.min < self.min: self.min = x_stats.min self.mintime = x_stats.mintime if x_stats.max is not None: if self.max is None or x_stats.max > self.max: self.max = x_stats.max self.maxtime = x_stats.maxtime if x_stats.lasttime is not None: if self.lasttime is None or x_stats.lasttime >= self.lasttime: self.lasttime = x_stats.lasttime self.last = x_stats.last def mergeSum(self, x_stats): """Merge the sum and count of another accumulator into myself.""" self.sum += x_stats.sum self.count += x_stats.count self.wsum += x_stats.wsum self.sumtime += x_stats.sumtime def addHiLo(self, val, ts): """Include a scalar value in my highs and lows. val: A scalar value ts: The timestamp. """ # If this is a string, try to convert it to a float. if isinstance(val, (six.string_types, six.text_type)): # Fail hard if unable to do the conversion: val = to_float(val) # Check for None and NaN: if val is not None and val == val: if self.min is None or val < self.min: self.min = val self.mintime = ts if self.max is None or val > self.max: self.max = val self.maxtime = ts if self.lasttime is None or ts >= self.lasttime: self.last = val self.lasttime = ts def addSum(self, val, weight=1): """Add a scalar value to my running sum and count.""" # If this is a string, try to convert it to a float. if isinstance(val, (six.string_types, six.text_type)): # Fail hard if unable to do the conversion: val = to_float(val) # Check for None and NaN: if val is not None and val == val: self.sum += val self.count += 1 self.wsum += val * weight self.sumtime += weight @property def avg(self): return self.wsum / self.sumtime if self.count else None class VecStats(object): """Accumulates statistics for a vector value. Property 'last' is the last non-None value seen. It is a two-way tuple (mag, dir). Property 'lasttime' is the time it was seen. """ default_init = (None, None, None, None, 0.0, 0, 0.0, 0, None, 0.0, 0.0, 0, 0.0, 0.0) def __init__(self, stats_tuple=None): self.setStats(stats_tuple) self.last = (None, None) self.lasttime = None def setStats(self, stats_tuple=None): (self.min, self.mintime, self.max, self.maxtime, self.sum, self.count, self.wsum, self.sumtime, self.max_dir, self.xsum, self.ysum, self.dirsumtime, self.squaresum, self.wsquaresum) = stats_tuple if stats_tuple else VecStats.default_init def getStatsTuple(self): """Return a stats-tuple. That is, a tuple containing the gathered statistics.""" return (self.min, self.mintime, self.max, self.maxtime, self.sum, self.count, self.wsum, self.sumtime, self.max_dir, self.xsum, self.ysum, self.dirsumtime, self.squaresum, self.wsquaresum) def mergeHiLo(self, x_stats): """Merge the highs and lows of another accumulator into myself.""" if x_stats.min is not None: if self.min is None or x_stats.min < self.min: self.min = x_stats.min self.mintime = x_stats.mintime if x_stats.max is not None: if self.max is None or x_stats.max > self.max: self.max = x_stats.max self.maxtime = x_stats.maxtime self.max_dir = x_stats.max_dir if x_stats.lasttime is not None: if self.lasttime is None or x_stats.lasttime >= self.lasttime: self.lasttime = x_stats.lasttime self.last = x_stats.last def mergeSum(self, x_stats): """Merge the sum and count of another accumulator into myself.""" self.sum += x_stats.sum self.count += x_stats.count self.wsum += x_stats.wsum self.sumtime += x_stats.sumtime self.xsum += x_stats.xsum self.ysum += x_stats.ysum self.dirsumtime += x_stats.dirsumtime self.squaresum += x_stats.squaresum self.wsquaresum += x_stats.wsquaresum def addHiLo(self, val, ts): """Include a vector value in my highs and lows. val: A vector value. It is a 2-way tuple (mag, dir). ts: The timestamp. """ speed, dirN = val # If this is a string, try to convert it to a float. if isinstance(speed, (six.string_types, six.text_type)): # Fail hard if unable to do the conversion: speed = to_float(speed) if isinstance(dirN, (six.string_types, six.text_type)): # Fail hard if unable to do the conversion: dirN = to_float(dirN) # Check for None and NaN: if speed is not None and speed == speed: if self.min is None or speed < self.min: self.min = speed self.mintime = ts if self.max is None or speed > self.max: self.max = speed self.maxtime = ts self.max_dir = dirN if self.lasttime is None or ts >= self.lasttime: self.last = (speed, dirN) self.lasttime = ts def addSum(self, val, weight=1): """Add a vector value to my sum and squaresum. val: A vector value. It is a 2-way tuple (mag, dir) """ speed, dirN = val # If this is a string, try to convert it to a float. if isinstance(speed, (six.string_types, six.text_type)): # Fail hard if unable to do the conversion: speed = to_float(speed) if isinstance(dirN, (six.string_types, six.text_type)): # Fail hard if unable to do the conversion: dirN = to_float(dirN) # Check for None and NaN: if speed is not None and speed == speed: self.sum += speed self.count += 1 self.wsum += weight * speed self.sumtime += weight self.squaresum += speed ** 2 self.wsquaresum += weight * speed ** 2 if dirN is not None: self.xsum += weight * speed * math.cos(math.radians(90.0 - dirN)) self.ysum += weight * speed * math.sin(math.radians(90.0 - dirN)) # It's OK for direction to be None, provided speed is zero: if dirN is not None or speed == 0: self.dirsumtime += weight @property def avg(self): return self.wsum / self.sumtime if self.count else None @property def rms(self): return math.sqrt(self.wsquaresum / self.sumtime) if self.count else None @property def vec_avg(self): if self.count: return math.sqrt((self.xsum ** 2 + self.ysum ** 2) / self.sumtime ** 2) @property def vec_dir(self): if self.dirsumtime and (self.ysum or self.xsum): _result = 90.0 - math.degrees(math.atan2(self.ysum, self.xsum)) if _result < 0.0: _result += 360.0 return _result # Return the last known direction when our vector sum is 0 return self.last[1] # =============================================================================== # FirstLastAccum # =============================================================================== class FirstLastAccum(object): """Minimal accumulator, suitable for strings. It can only return the first and last strings it has seen, along with their timestamps. """ default_init = (None, None, None, None) def __init__(self, stats_tuple=None): self.first = None self.firsttime = None self.last = None self.lasttime = None def setStats(self, stats_tuple=None): self.first, self.firsttime, self.last, self.lasttime = stats_tuple \ if stats_tuple else FirstLastAccum.default_init def getStatsTuple(self): """Return a stats-tuple. That is, a tuple containing the gathered statistics. This tuple can be used to update the stats database""" return self.first, self.firsttime, self.last, self.lasttime def mergeHiLo(self, x_stats): """Merge the highs and lows of another accumulator into myself.""" if x_stats.firsttime is not None: if self.firsttime is None or x_stats.firsttime < self.firsttime: self.firsttime = x_stats.firsttime self.first = x_stats.first if x_stats.lasttime is not None: if self.lasttime is None or x_stats.lasttime >= self.lasttime: self.lasttime = x_stats.lasttime self.last = x_stats.last def mergeSum(self, x_stats): """Merge the count of another accumulator into myself.""" pass def addHiLo(self, val, ts): """Include a value in my stats. val: A value of almost any type. It will be converted to a string before being accumulated. ts: The timestamp. """ if val is not None: string_val = str(val) if self.firsttime is None or ts < self.firsttime: self.first = string_val self.firsttime = ts if self.lasttime is None or ts >= self.lasttime: self.last = string_val self.lasttime = ts def addSum(self, val, weight=1): """Add a scalar value to my running count.""" pass # =============================================================================== # Class Accum # =============================================================================== class Accum(dict): """Accumulates statistics for a set of observation types.""" def __init__(self, timespan, unit_system=None): """Initialize a Accum. timespan: The time period over which stats will be accumulated. unit_system: The unit system used by the accumulator""" self.timespan = timespan # Set the accumulator's unit system. Usually left unspecified until the # first observation comes in for normal operation or pre-set if # obtaining a historical accumulator. self.unit_system = unit_system def addRecord(self, record, add_hilo=True, weight=1): """Add a record to my running statistics. The record must have keys 'dateTime' and 'usUnits'.""" # Check to see if the record is within my observation timespan if not self.timespan.includesArchiveTime(record['dateTime']): raise OutOfSpan("Attempt to add out-of-interval record (%s) to timespan (%s)" % (record['dateTime'], self.timespan)) for obs_type in record: # Get the proper function ... func = get_add_function(obs_type) # ... then call it. func(self, record, obs_type, add_hilo, weight) def updateHiLo(self, accumulator): """Merge the high/low stats of another accumulator into me.""" if accumulator.timespan.start < self.timespan.start \ or accumulator.timespan.stop > self.timespan.stop: raise OutOfSpan("Attempt to merge an accumulator whose timespan is not a subset") self._check_units(accumulator.unit_system) for obs_type in accumulator: # Initialize the type if we have not seen it before self._init_type(obs_type) # Get the proper function ... func = get_merge_function(obs_type) # ... then call it func(self, accumulator, obs_type) def getRecord(self): """Extract a record out of the results in the accumulator.""" # All records have a timestamp and unit type record = {'dateTime': self.timespan.stop, 'usUnits': self.unit_system} return self.augmentRecord(record) def augmentRecord(self, record): # Go through all observation types. for obs_type in self: # If the type does not appear in the record, then add it: if obs_type not in record: # Get the proper extraction function... func = get_extract_function(obs_type) # ... then call it func(self, record, obs_type) return record def set_stats(self, obs_type, stats_tuple): self._init_type(obs_type) self[obs_type].setStats(stats_tuple) # # Begin add functions. These add a record to the accumulator. # def add_value(self, record, obs_type, add_hilo, weight): """Add a single observation to myself.""" val = record[obs_type] # If the type has not been seen before, initialize it self._init_type(obs_type) # Then add to highs/lows, and to the running sum: if add_hilo: self[obs_type].addHiLo(val, record['dateTime']) self[obs_type].addSum(val, weight=weight) def add_wind_value(self, record, obs_type, add_hilo, weight): """Add a single observation of type wind to myself.""" if obs_type in ['windDir', 'windGust', 'windGustDir']: return if weewx.debug: assert (obs_type == 'windSpeed') # First add it to regular old 'windSpeed', then # treat it like a vector. self.add_value(record, obs_type, add_hilo, weight) # If the type has not been seen before, initialize it. self._init_type('wind') # Then add to highs/lows. if add_hilo: # If the station does not provide windGustDir, then substitute windDir. # See issue #320, https://bit.ly/2HSo0ju # Do windGust first, so the last packet in is windSpeed, not windGust. self['wind'].addHiLo((record.get('windGust'), record.get('windGustDir', record.get('windDir'))), record['dateTime']) self['wind'].addHiLo((record.get('windSpeed'), record.get('windDir')), record['dateTime']) # Add to the running sum. self['wind'].addSum((record['windSpeed'], record.get('windDir')), weight=weight) def check_units(self, record, obs_type, add_hilo, weight): if weewx.debug: assert (obs_type == 'usUnits') self._check_units(record['usUnits']) def noop(self, record, obs_type, add_hilo=True, weight=1): pass # # Begin hi/lo merge functions. These are called when merging two accumulators # def merge_minmax(self, x_accumulator, obs_type): """Merge value in another accumulator, using min/max""" self[obs_type].mergeHiLo(x_accumulator[obs_type]) def merge_avg(self, x_accumulator, obs_type): """Merge value in another accumulator, using avg for max""" x_stats = x_accumulator[obs_type] if x_stats.min is not None: if self[obs_type].min is None or x_stats.min < self[obs_type].min: self[obs_type].min = x_stats.min self[obs_type].mintime = x_stats.mintime if x_stats.avg is not None: if self[obs_type].max is None or x_stats.avg > self[obs_type].max: self[obs_type].max = x_stats.avg self[obs_type].maxtime = x_accumulator.timespan.stop if x_stats.lasttime is not None: if self[obs_type].lasttime is None or x_stats.lasttime >= self[obs_type].lasttime: self[obs_type].lasttime = x_stats.lasttime self[obs_type].last = x_stats.last # # Begin extraction functions. These extract a record out of the accumulator. # def extract_wind(self, record, obs_type): """Extract wind values from myself, and put in a record.""" # Wind records must be flattened into the separate categories: if 'windSpeed' not in record: record['windSpeed'] = self[obs_type].avg if 'windDir' not in record: record['windDir'] = self[obs_type].vec_dir if 'windGust' not in record: record['windGust'] = self[obs_type].max if 'windGustDir' not in record: log.info("Extracting wind; windGustDir not in record. Using %s", self[obs_type].max_dir) record['windGustDir'] = self[obs_type].max_dir else: log.info("Extracting wind; record['windGustDir']=%s", record['windGustDir']) def extract_sum(self, record, obs_type): record[obs_type] = self[obs_type].sum def extract_last(self, record, obs_type): record[obs_type] = self[obs_type].last def extract_avg(self, record, obs_type): record[obs_type] = self[obs_type].avg def extract_min(self, record, obs_type): record[obs_type] = self[obs_type].min def extract_max(self, record, obs_type): record[obs_type] = self[obs_type].max def extract_count(self, record, obs_type): record[obs_type] = self[obs_type].count # # Miscellaneous, utility functions # def _init_type(self, obs_type): """Add a given observation type to my dictionary.""" # Do nothing if this type has already been initialized: if obs_type in self: return # Get a new accumulator of the proper type self[obs_type] = new_accumulator(obs_type) def _check_units(self, new_unit_system): # If no unit system has been specified for me yet, adopt the incoming # system if self.unit_system is None: self.unit_system = new_unit_system else: # Otherwise, make sure they match if self.unit_system != new_unit_system: raise ValueError("Unit system mismatch %d v. %d" % (self.unit_system, new_unit_system)) @property def isEmpty(self): return self.unit_system is None # =============================================================================== # Configuration dictionaries # =============================================================================== # # Mappings from convenient string nicknames, which can be used in a config file, # to actual functions and classes # ACCUM_TYPES = { 'scalar': ScalarStats, 'vector': VecStats, 'firstlast': FirstLastAccum } ADD_FUNCTIONS = { 'add': Accum.add_value, 'add_wind': Accum.add_wind_value, 'check_units': Accum.check_units, 'noop': Accum.noop } MERGE_FUNCTIONS = { 'minmax': Accum.merge_minmax, 'avg': Accum.merge_avg } EXTRACT_FUNCTIONS = { 'avg': Accum.extract_avg, 'count': Accum.extract_count, 'last': Accum.extract_last, 'max': Accum.extract_max, 'min': Accum.extract_min, 'noop': Accum.noop, 'sum': Accum.extract_sum, 'wind': Accum.extract_wind, } # The default actions for an individual observation type OBS_DEFAULTS = { 'accumulator': 'scalar', 'adder': 'add', 'merger': 'minmax', 'extractor': 'avg' } def initialize(config_dict): # Add the configuration dictionary to the beginning of the list of maps. # This will cause it to override the defaults global accum_dict accum_dict.maps.insert(0, config_dict.get('Accumulator', {})) def new_accumulator(obs_type): """Instantiate an accumulator, appropriate for type 'obs_type'.""" global accum_dict # Get the options for this type. Substitute the defaults if they have not been specified obs_options = accum_dict.get(obs_type, OBS_DEFAULTS) # Get the nickname of the accumulator. Default is 'scalar' accum_nickname = obs_options.get('accumulator', 'scalar') # Instantiate and return the accumulator. # If we don't know this nickname, then fail hard with a KeyError return ACCUM_TYPES[accum_nickname]() def get_add_function(obs_type): """Get an adder function appropriate for type 'obs_type'.""" global accum_dict # Get the options for this type. Substitute the defaults if they have not been specified obs_options = accum_dict.get(obs_type, OBS_DEFAULTS) # Get the nickname of the adder. Default is 'add' add_nickname = obs_options.get('adder', 'add') # If we don't know this nickname, then fail hard with a KeyError return ADD_FUNCTIONS[add_nickname] def get_merge_function(obs_type): """Get a merge function appropriate for type 'obs_type'.""" global accum_dict # Get the options for this type. Substitute the defaults if they have not been specified obs_options = accum_dict.get(obs_type, OBS_DEFAULTS) # Get the nickname of the merger. Default is 'minmax' add_nickname = obs_options.get('merger', 'minmax') # If we don't know this nickname, then fail hard with a KeyError return MERGE_FUNCTIONS[add_nickname] def get_extract_function(obs_type): """Get an extraction function appropriate for type 'obs_type'.""" global accum_dict # Get the options for this type. Substitute the defaults if they have not been specified obs_options = accum_dict.get(obs_type, OBS_DEFAULTS) # Get the nickname of the extractor. Default is 'avg' add_nickname = obs_options.get('extractor', 'avg') # If we don't know this nickname, then fail hard with a KeyError return EXTRACT_FUNCTIONS[add_nickname]
