Thank you for the suggestion, I'll take a look into it. I might have to refactor a lot more than I planned. Some parts of "fuzzy archer" didn't age very well, as WeeWX in the mean time, provides functionality that was implemented there in a similar way. But the less code in the extension, the better.
Tom Keffer schrieb am Sonntag, 15. September 2024 um 22:36:47 UTC+2: > If the plot is for a long time period and a daily aggregation is used, > then one can plot "max" values. For example, > > [[[yearhilow]]] > [[[[hi]]]] > data_type = outTemp > aggregate_type = max > label = High > [[[[low]]]] > data_type = outTemp > aggregate_type = min > label = Low Temperature > > With this example, the plot should exactly measure the NOAA data: they are > using the same data, that is, the daily summaries. > > I didn't quite follow your augmentation plan. I can see getting the > regular series of outTemp from the archive table, then squeezing in a false > extra point with timestamp and value of the max seen temperature for the > day. That should look pretty good. If this is what you are trying to do, I > would implement it as a new xtype "get_series()" function. Use the regular > xtype function get_series() to get the series, then augment. > > See the XType documentation <https://github.com/weewx/weewx/wiki/xtypes> > and examples. > > -tk > > > > On Sun, Sep 15, 2024 at 10:53 AM '[email protected]' via > weewx-development <[email protected]> wrote: > >> >> The red line is just like you expect: the "outTemp" for each archive >> record, no aggregation. The chart library I use to draw the plot, has a >> feature to show things like the max and min value of the plot. The user (in >> that case: myself) wants to have the min/max of each day shown in the plot. >> If one doesn't know, that the archive doesn't contain the daily min/max >> values, but the daily summaries do, he might be confused that the plot >> shows a different maximum, than the NOAA data show, even if it is just a >> little off. There is a perfectly "legal" technical explanation, why this is >> the case. Anyway, if the max of the plot matches the max of a particular >> day, no questions ever will be asked in that regard. >> >> What I will do, is augment the archive data with the max/min values and >> their timestamps, and combine archive with archive_day, if the min/max >> values from the daily summary is within the timestamp. >> import weecfg >> import weewx.manager >> import weewx.xtypes >> from weeutil.weeutil import TimeSpan >> from datetime import datetime, time, timedelta >> >> config_path, config_dict = weecfg.read_config( >> '/home/michi/weewx-data/weewx.conf') >> with weewx.manager.open_manager_with_config(config_dict, 'wx_binding') as >> dbmanager: >> start_time = 1726347600 >> end_time = datetime.now() >> start_of_day = datetime.combine(datetime.fromtimestamp(start_time), >> time.min) >> >> min_list = [] >> >> while start_of_day < end_time: >> min_tupel = [] >> start_of_next_day = start_of_day + timedelta(days=1) >> >> min_value = weewx.xtypes.get_aggregate("outTemp", TimeSpan( >> start_of_day.timestamp(), start_of_next_day.timestamp()), "min", >> dbmanager)[0] >> min_time = weewx.xtypes.get_aggregate("outTemp", TimeSpan( >> start_of_day.timestamp(), start_of_next_day.timestamp()), "mintime", >> dbmanager)[0] >> min_tupel.append(min_time * 1000) >> min_tupel.append(min_value) >> if min_time >= start_time and min_time <= end_time.timestamp(): >> min_list.append(min_tupel) >> >> start_of_day = start_of_next_day >> >> print(min_list) >> >> The result (min_list) will be a list of timestamp/values, that will be >> inserted in the data series for the plot. >> Tom Keffer schrieb am Sonntag, 15. September 2024 um 17:23:01 UTC+2: >> >>> I don't know what aggregation type, if any, was used for the red line. >>> Most likely no aggregation type was called for, in which case the curve >>> will represent "outTemp" for each archive record, which by default (it >>> depends on the accumulator extractor type) will be the average value of all >>> LOOP packets seen in the archive interval. >>> >>> By contrast, the max value in the table is the max value seen in any >>> LOOP packet. This can, and most likely will be, higher than these averages. >>> >>> Not sure what the user expected. WeeWX does not save every LOOP packet, >>> so the curve can only represent the values in the archive records. >>> >>> In any case, for a plot of the day's temperatures, the daily archives >>> don't come into play at all. They are only used if the aggregation interval >>> is a multiple of a whole day. >>> >>> Hope that helps. >>> >>> -tk >>> >>> >>> >>> >>> On Sun, Sep 15, 2024 at 12:19 AM '[email protected]' via >>> weewx-development <[email protected]> wrote: >>> >>>> It's connected to this issue I am to resolve currently: >>>> https://github.com/brewster76/fuzzy-archer/issues/156 >>>> >>>> The plot is showing a number lower/higher than the real max/min from >>>> the stats/NOAA data, which obtain their min/max values from the daily >>>> summaries, while the plot is only showing the average of the values for >>>> every archive_interval. My approach will be the following: >>>> >>>> - determine every day boundary for the given timespan >>>> - get the min/max for every calendar day in that timespan >>>> - if the datetime of a single min/max is within the timespan, insert it >>>> to the values of the plot (most likely leading to an extra data point >>>> between two existing data point, but that's desired behavior) >>>> - if the datetime of a single daily min/max is not within the >>>> timespan, ignore >>>> >>>> This way we have the real min/max values, with their exact datetime of >>>> arrival, for every day, in the plot, if they are within the given >>>> timespan. >>>> If they are not in the timespan, we simply don't have a more precise >>>> information about the min/max value for an archive_interval in the plot. >>>> If >>>> they are within the timespan, but not the min/max value of the plot, we >>>> know it is a min/max value of that certain calendar day that value lies >>>> within. The user probably won't recognize these values, if not zooming in >>>> on the chart, but that's perfectly OK. >>>> >>>> >>>> Tom Keffer schrieb am Samstag, 14. September 2024 um 22:06:39 UTC+2: >>>> >>>>> Yes, it is. In the interests of clarity, I did not give the complete >>>>> set of conditions for the daily summaries to be used. The start time must >>>>> either be the first record in the database, or a midnight boundary. The >>>>> stop time must either be the last record in the database, or a midnight >>>>> boundary. Otherwise, the archive database is used. >>>>> >>>>> So, if the ending timestamp of your first example is the last record >>>>> in the database, then the daily summary would have been used. >>>>> >>>>> I'm glad you're showing an interest in this! It's deep in the guts of >>>>> WeeWX, but important. >>>>> >>>>> -tk >>>>> >>>>> On Sat, Sep 14, 2024 at 1:01 PM '[email protected]' via >>>>> weewx-development <[email protected]> wrote: >>>>> >>>>>> You are right, none of those examples meets the condition (starting >>>>>> and ending on day boundaries), but the first example starts at the start >>>>>> of >>>>>> today, ending with the latest timestamp in the database. Which is still >>>>>> today, but not a day's boundary. So my best guess is, that the shortcuts >>>>>> using the daily summaries, is also taken in that particular case. >>>>>> >>>>>> And yes, your explanation helps. >>>>>> Tom Keffer schrieb am Samstag, 14. September 2024 um 21:27:01 UTC+2: >>>>>> >>>>>>> I'm not following your question --- too many numbers flying around >>>>>>> --- but let me try to explain the calculation in the hopes that it >>>>>>> helps. >>>>>>> >>>>>>> When calculating an aggregate, there's always a well-defined answer. >>>>>>> In this case, it's the minimum temperature over the aggregate interval, >>>>>>> so >>>>>>> you scan all the relevant records and look for the minimum and the >>>>>>> timestamp of the record that contains it. >>>>>>> >>>>>>> However, if the interval starts and ends on day boundaries (that is, >>>>>>> at midnight), then a shortcut can be taken. The algorithm can use the >>>>>>> daily >>>>>>> summaries to speed the calculation. Otherwise, the main archive table >>>>>>> is >>>>>>> used. >>>>>>> >>>>>>> I didn't look too closely, but I don't think either of your examples >>>>>>> satisfies this condition. The aggregation interval is something like 20 >>>>>>> hours or 27 hours. So, the main archive table should have been used. >>>>>>> >>>>>>> Either way, you should get the same answer, although the daily >>>>>>> summaries can be slightly more precise for the timestamp. The reason is >>>>>>> that while the archive table has the minimum value seen in the >>>>>>> interval, it >>>>>>> doesn't have the precise time, only the timestamp of the record with >>>>>>> the >>>>>>> lowest value. By contrast, the daily summaries have the timestamp of >>>>>>> the >>>>>>> LOOP packet that provided the minimum value, so the precision can be >>>>>>> down >>>>>>> to a few seconds. >>>>>>> >>>>>>> Otherwise, there should be no difference. >>>>>>> >>>>>>> Does that help? >>>>>>> >>>>>>> -tk >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Sat, Sep 14, 2024 at 10:56 AM '[email protected]' via >>>>>>> weewx-development <[email protected]> wrote: >>>>>>> >>>>>>>> Is it possible, that this function returns different values, if the >>>>>>>> timespan is across more than a single calendar day? >>>>>>>> >>>>>>>> results = weewx.xtypes.get_aggregate("outTemp", TimeSpan(1726264800, >>>>>>>> 1726335600), "min", dbmanager) >>>>>>>> print(results) >>>>>>>> results = weewx.xtypes.get_aggregate("outTemp", TimeSpan( >>>>>>>> 1726264800, 1726335600), "mintime", dbmanager) >>>>>>>> print(results) >>>>>>>> yields >>>>>>>> (4.5, 'degree_C', 'group_temperature') >>>>>>>> (1726303300, 'unix_epoch', 'group_time') >>>>>>>> Where 1726264800 is the start of today in my timezone. 1726303300 >>>>>>>> is the min outtemp of toady as in archive_day_outtemp. >>>>>>>> >>>>>>>> results = weewx.xtypes.get_aggregate("outTemp", TimeSpan(1726238400, >>>>>>>> 1726335600), "min", dbmanager) >>>>>>>> print(results) >>>>>>>> results = weewx.xtypes.get_aggregate("outTemp", TimeSpan( >>>>>>>> 1726238400, 1726335600), "mintime", dbmanager) >>>>>>>> print(results) >>>>>>>> yields >>>>>>>> (4.555172413793103, 'degree_C', 'group_temperature') >>>>>>>> (1726303500, 'unix_epoch', 'group_time') >>>>>>>> Where 1726238400 is some random time yesterday in my timezone. >>>>>>>> 1726303500 is the first timestamp of the record in archive, where >>>>>>>> outtemp >>>>>>>> has the minimum value. >>>>>>>> >>>>>>>> Obviously, when doing the second "query", I don't get the exact >>>>>>>> minimum and it's timestamp. Can I only achieve my desired result, if I >>>>>>>> limit the timespan to not overlap a calendar day change? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> [email protected] schrieb am Samstag, 14. September 2024 um >>>>>>>> 16:15:43 UTC+2: >>>>>>>> >>>>>>>>> Thanks! Although obviously there, I didn't get it :) >>>>>>>>> >>>>>>>>> Tom Keffer schrieb am Samstag, 14. September 2024 um 15:35:04 >>>>>>>>> UTC+2: >>>>>>>>> >>>>>>>>>> Just substitute "mintime" / "maxtime" for "min" / "max". >>>>>>>>>> >>>>>>>>>> See the table of Aggregation Types >>>>>>>>>> <https://www.weewx.com/docs/5.1/reference/aggtypes/>. >>>>>>>>>> >>>>>>>>>> On Sat, Sep 14, 2024 at 2:02 AM '[email protected]' via >>>>>>>>>> weewx-development <[email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Thank you Tom, this works for the max/min value as I need it. >>>>>>>>>>> But the dateTime for that value isn't in the result and I don't >>>>>>>>>>> have a >>>>>>>>>>> clue, how to get it for that particular value. >>>>>>>>>>> >>>>>>>>>>> Tom Keffer schrieb am Freitag, 13. September 2024 um 22:27:31 >>>>>>>>>>> UTC+2: >>>>>>>>>>> >>>>>>>>>>>> The xtypes system can do this. See the section, *XTypes API >>>>>>>>>>>> <https://github.com/weewx/weewx/wiki/xtypes#xtypes-api>*. >>>>>>>>>>>> >>>>>>>>>>>> Your example would be: >>>>>>>>>>>> >>>>>>>>>>>> results = weewx.xtypes.get_aggregate("outTemp", >>>>>>>>>>>> TimeSpan(1726092000, 1726178400), "max", dbmanager) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> where "dbmanager" is an open database manager. The results >>>>>>>>>>>> will be a ValueTuple >>>>>>>>>>>> <https://www.weewx.com/docs/5.1/reference/valuetuple/>. >>>>>>>>>>>> >>>>>>>>>>>> Here's a more complete example: >>>>>>>>>>>> >>>>>>>>>>>> import weecfg >>>>>>>>>>>> import weewx.manager >>>>>>>>>>>> import weewx.xtypes >>>>>>>>>>>> from weeutil.weeutil import TimeSpan >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> config_path, config_dict = >>>>>>>>>>>> weecfg.read_config('/home/weewx/weewx.conf') >>>>>>>>>>>> >>>>>>>>>>>> with weewx.manager.open_manager_with_config(config_dict, >>>>>>>>>>>> 'wx_binding') as dbmanager: >>>>>>>>>>>> >>>>>>>>>>>> results = weewx.xtypes.get_aggregate("outTemp", >>>>>>>>>>>> TimeSpan(1726092000, 1726178400), "max", dbmanager) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> print(results) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> (68.4, 'degree_F', 'group_temperature') >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Sep 13, 2024 at 10:27 AM '[email protected]' via >>>>>>>>>>>> weewx-development <[email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> When I write an extension in python, can I get the >>>>>>>>>>>>> minimum/maximum value for an obs_type and a given timespan and >>>>>>>>>>>>> their >>>>>>>>>>>>> related timestamp? >>>>>>>>>>>>> >>>>>>>>>>>>> Something like: >>>>>>>>>>>>> >>>>>>>>>>>>> maxValue, datetime = getMax("outTemp", 1726092000, >>>>>>>>>>>>> 1726178400) >>>>>>>>>>>>> >>>>>>>>>>>>> The min/max has to be obtained from the archive_day table, the >>>>>>>>>>>>> archive table might not contain the exact max/min value, and also >>>>>>>>>>>>> not the >>>>>>>>>>>>> exact timestamp the value arrived. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>> Google Groups "weewx-development" group. >>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>>> https://groups.google.com/d/msgid/weewx-development/6d13cde1-43c4-4ca3-a4a4-d220fc5e2040n%40googlegroups.com >>>>>>>>>>>>> >>>>>>>>>>>>> <https://groups.google.com/d/msgid/weewx-development/6d13cde1-43c4-4ca3-a4a4-d220fc5e2040n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>>> . >>>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>> Google Groups "weewx-development" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>> >>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>> https://groups.google.com/d/msgid/weewx-development/2aaf12b7-25ad-4013-8185-b379168b4bc5n%40googlegroups.com >>>>>>>>>>> >>>>>>>>>>> <https://groups.google.com/d/msgid/weewx-development/2aaf12b7-25ad-4013-8185-b379168b4bc5n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>>> . >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "weewx-development" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> >>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/weewx-development/c183248b-1a04-47bd-8359-51ec5f215741n%40googlegroups.com >>>>>>>> >>>>>>>> <https://groups.google.com/d/msgid/weewx-development/c183248b-1a04-47bd-8359-51ec5f215741n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "weewx-development" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> >>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/weewx-development/8138a283-82d2-418b-8a06-d919daee47b0n%40googlegroups.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/weewx-development/8138a283-82d2-418b-8a06-d919daee47b0n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "weewx-development" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> >>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/weewx-development/1c0b6e38-6d6f-4ce9-ae27-107f430cc310n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/weewx-development/1c0b6e38-6d6f-4ce9-ae27-107f430cc310n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "weewx-development" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/weewx-development/66060704-cfb5-4a5a-8aaa-1c3db3589ba0n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/weewx-development/66060704-cfb5-4a5a-8aaa-1c3db3589ba0n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "weewx-development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/3eb8ecfc-043a-4ff3-ba48-be9c4d02d961n%40googlegroups.com.
