The focus of the last couple of development cycles has been on making it easy for extension authors. Glad it shows!
On Sun, Sep 15, 2024 at 8:52 PM '[email protected]' via weewx-development <[email protected]> wrote: > Thank you for the suggestion, I'll take a look into it. I might have to > refactor a lot more than I planned. Some parts of "fuzzy archer" didn't age > very well, as WeeWX in the mean time, provides functionality that was > implemented there in a similar way. But the less code in the extension, the > better. > > Tom Keffer schrieb am Sonntag, 15. September 2024 um 22:36:47 UTC+2: > >> If the plot is for a long time period and a daily aggregation is used, >> then one can plot "max" values. For example, >> >> [[[yearhilow]]] >> [[[[hi]]]] >> data_type = outTemp >> aggregate_type = max >> label = High >> [[[[low]]]] >> data_type = outTemp >> aggregate_type = min >> label = Low Temperature >> >> With this example, the plot should exactly measure the NOAA data: they >> are using the same data, that is, the daily summaries. >> >> I didn't quite follow your augmentation plan. I can see getting the >> regular series of outTemp from the archive table, then squeezing in a false >> extra point with timestamp and value of the max seen temperature for the >> day. That should look pretty good. If this is what you are trying to do, I >> would implement it as a new xtype "get_series()" function. Use the regular >> xtype function get_series() to get the series, then augment. >> >> See the XType documentation <https://github.com/weewx/weewx/wiki/xtypes> >> and examples. >> >> -tk >> >> >> >> On Sun, Sep 15, 2024 at 10:53 AM '[email protected]' via >> weewx-development <[email protected]> wrote: >> >>> >>> The red line is just like you expect: the "outTemp" for each archive >>> record, no aggregation. The chart library I use to draw the plot, has a >>> feature to show things like the max and min value of the plot. The user (in >>> that case: myself) wants to have the min/max of each day shown in the plot. >>> If one doesn't know, that the archive doesn't contain the daily min/max >>> values, but the daily summaries do, he might be confused that the plot >>> shows a different maximum, than the NOAA data show, even if it is just a >>> little off. There is a perfectly "legal" technical explanation, why this is >>> the case. Anyway, if the max of the plot matches the max of a particular >>> day, no questions ever will be asked in that regard. >>> >>> What I will do, is augment the archive data with the max/min values and >>> their timestamps, and combine archive with archive_day, if the min/max >>> values from the daily summary is within the timestamp. >>> import weecfg >>> import weewx.manager >>> import weewx.xtypes >>> from weeutil.weeutil import TimeSpan >>> from datetime import datetime, time, timedelta >>> >>> config_path, config_dict = weecfg.read_config( >>> '/home/michi/weewx-data/weewx.conf') >>> with weewx.manager.open_manager_with_config(config_dict, 'wx_binding') >>> as dbmanager: >>> start_time = 1726347600 >>> end_time = datetime.now() >>> start_of_day = datetime.combine(datetime.fromtimestamp(start_time), >>> time.min) >>> >>> min_list = [] >>> >>> while start_of_day < end_time: >>> min_tupel = [] >>> start_of_next_day = start_of_day + timedelta(days=1) >>> >>> min_value = weewx.xtypes.get_aggregate("outTemp", TimeSpan( >>> start_of_day.timestamp(), start_of_next_day.timestamp()), "min", >>> dbmanager)[0] >>> min_time = weewx.xtypes.get_aggregate("outTemp", TimeSpan( >>> start_of_day.timestamp(), start_of_next_day.timestamp()), "mintime", >>> dbmanager)[0] >>> min_tupel.append(min_time * 1000) >>> min_tupel.append(min_value) >>> if min_time >= start_time and min_time <= end_time.timestamp(): >>> min_list.append(min_tupel) >>> >>> start_of_day = start_of_next_day >>> >>> print(min_list) >>> >>> The result (min_list) will be a list of timestamp/values, that will be >>> inserted in the data series for the plot. >>> Tom Keffer schrieb am Sonntag, 15. September 2024 um 17:23:01 UTC+2: >>> >>>> I don't know what aggregation type, if any, was used for the red line. >>>> Most likely no aggregation type was called for, in which case the curve >>>> will represent "outTemp" for each archive record, which by default (it >>>> depends on the accumulator extractor type) will be the average value of all >>>> LOOP packets seen in the archive interval. >>>> >>>> By contrast, the max value in the table is the max value seen in any >>>> LOOP packet. This can, and most likely will be, higher than these averages. >>>> >>>> Not sure what the user expected. WeeWX does not save every LOOP packet, >>>> so the curve can only represent the values in the archive records. >>>> >>>> In any case, for a plot of the day's temperatures, the daily archives >>>> don't come into play at all. They are only used if the aggregation interval >>>> is a multiple of a whole day. >>>> >>>> Hope that helps. >>>> >>>> -tk >>>> >>>> >>>> >>>> >>>> On Sun, Sep 15, 2024 at 12:19 AM '[email protected]' via >>>> weewx-development <[email protected]> wrote: >>>> >>>>> It's connected to this issue I am to resolve currently: >>>>> https://github.com/brewster76/fuzzy-archer/issues/156 >>>>> >>>>> The plot is showing a number lower/higher than the real max/min from >>>>> the stats/NOAA data, which obtain their min/max values from the daily >>>>> summaries, while the plot is only showing the average of the values for >>>>> every archive_interval. My approach will be the following: >>>>> >>>>> - determine every day boundary for the given timespan >>>>> - get the min/max for every calendar day in that timespan >>>>> - if the datetime of a single min/max is within the timespan, insert >>>>> it to the values of the plot (most likely leading to an extra data point >>>>> between two existing data point, but that's desired behavior) >>>>> - if the datetime of a single daily min/max is not within the >>>>> timespan, ignore >>>>> >>>>> This way we have the real min/max values, with their exact datetime of >>>>> arrival, for every day, in the plot, if they are within the given >>>>> timespan. >>>>> If they are not in the timespan, we simply don't have a more precise >>>>> information about the min/max value for an archive_interval in the plot. >>>>> If >>>>> they are within the timespan, but not the min/max value of the plot, we >>>>> know it is a min/max value of that certain calendar day that value lies >>>>> within. The user probably won't recognize these values, if not zooming in >>>>> on the chart, but that's perfectly OK. >>>>> >>>>> >>>>> Tom Keffer schrieb am Samstag, 14. September 2024 um 22:06:39 UTC+2: >>>>> >>>>>> Yes, it is. In the interests of clarity, I did not give the complete >>>>>> set of conditions for the daily summaries to be used. The start time must >>>>>> either be the first record in the database, or a midnight boundary. The >>>>>> stop time must either be the last record in the database, or a midnight >>>>>> boundary. Otherwise, the archive database is used. >>>>>> >>>>>> So, if the ending timestamp of your first example is the last record >>>>>> in the database, then the daily summary would have been used. >>>>>> >>>>>> I'm glad you're showing an interest in this! It's deep in the guts of >>>>>> WeeWX, but important. >>>>>> >>>>>> -tk >>>>>> >>>>>> On Sat, Sep 14, 2024 at 1:01 PM '[email protected]' via >>>>>> weewx-development <[email protected]> wrote: >>>>>> >>>>>>> You are right, none of those examples meets the condition (starting >>>>>>> and ending on day boundaries), but the first example starts at the >>>>>>> start of >>>>>>> today, ending with the latest timestamp in the database. Which is still >>>>>>> today, but not a day's boundary. So my best guess is, that the shortcuts >>>>>>> using the daily summaries, is also taken in that particular case. >>>>>>> >>>>>>> And yes, your explanation helps. >>>>>>> Tom Keffer schrieb am Samstag, 14. September 2024 um 21:27:01 UTC+2: >>>>>>> >>>>>>>> I'm not following your question --- too many numbers flying around >>>>>>>> --- but let me try to explain the calculation in the hopes that it >>>>>>>> helps. >>>>>>>> >>>>>>>> When calculating an aggregate, there's always a well-defined >>>>>>>> answer. In this case, it's the minimum temperature over the aggregate >>>>>>>> interval, so you scan all the relevant records and look for the >>>>>>>> minimum and >>>>>>>> the timestamp of the record that contains it. >>>>>>>> >>>>>>>> However, if the interval starts and ends on day boundaries (that >>>>>>>> is, at midnight), then a shortcut can be taken. The algorithm can use >>>>>>>> the >>>>>>>> daily summaries to speed the calculation. Otherwise, the main archive >>>>>>>> table >>>>>>>> is used. >>>>>>>> >>>>>>>> I didn't look too closely, but I don't think either of your >>>>>>>> examples satisfies this condition. The aggregation interval is >>>>>>>> something >>>>>>>> like 20 hours or 27 hours. So, the main archive table should have been >>>>>>>> used. >>>>>>>> >>>>>>>> Either way, you should get the same answer, although the daily >>>>>>>> summaries can be slightly more precise for the timestamp. The reason is >>>>>>>> that while the archive table has the minimum value seen in the >>>>>>>> interval, it >>>>>>>> doesn't have the precise time, only the timestamp of the record with >>>>>>>> the >>>>>>>> lowest value. By contrast, the daily summaries have the timestamp of >>>>>>>> the >>>>>>>> LOOP packet that provided the minimum value, so the precision can be >>>>>>>> down >>>>>>>> to a few seconds. >>>>>>>> >>>>>>>> Otherwise, there should be no difference. >>>>>>>> >>>>>>>> Does that help? >>>>>>>> >>>>>>>> -tk >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Sat, Sep 14, 2024 at 10:56 AM '[email protected]' via >>>>>>>> weewx-development <[email protected]> wrote: >>>>>>>> >>>>>>>>> Is it possible, that this function returns different values, if >>>>>>>>> the timespan is across more than a single calendar day? >>>>>>>>> >>>>>>>>> results = weewx.xtypes.get_aggregate("outTemp", TimeSpan( >>>>>>>>> 1726264800, 1726335600), "min", dbmanager) >>>>>>>>> print(results) >>>>>>>>> results = weewx.xtypes.get_aggregate("outTemp", TimeSpan( >>>>>>>>> 1726264800, 1726335600), "mintime", dbmanager) >>>>>>>>> print(results) >>>>>>>>> yields >>>>>>>>> (4.5, 'degree_C', 'group_temperature') >>>>>>>>> (1726303300, 'unix_epoch', 'group_time') >>>>>>>>> Where 1726264800 is the start of today in my timezone. 1726303300 >>>>>>>>> is the min outtemp of toady as in archive_day_outtemp. >>>>>>>>> >>>>>>>>> results = weewx.xtypes.get_aggregate("outTemp", TimeSpan( >>>>>>>>> 1726238400, 1726335600), "min", dbmanager) >>>>>>>>> print(results) >>>>>>>>> results = weewx.xtypes.get_aggregate("outTemp", TimeSpan( >>>>>>>>> 1726238400, 1726335600), "mintime", dbmanager) >>>>>>>>> print(results) >>>>>>>>> yields >>>>>>>>> (4.555172413793103, 'degree_C', 'group_temperature') >>>>>>>>> (1726303500, 'unix_epoch', 'group_time') >>>>>>>>> Where 1726238400 is some random time yesterday in my timezone. >>>>>>>>> 1726303500 is the first timestamp of the record in archive, where >>>>>>>>> outtemp >>>>>>>>> has the minimum value. >>>>>>>>> >>>>>>>>> Obviously, when doing the second "query", I don't get the exact >>>>>>>>> minimum and it's timestamp. Can I only achieve my desired result, if I >>>>>>>>> limit the timespan to not overlap a calendar day change? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> [email protected] schrieb am Samstag, 14. September 2024 um >>>>>>>>> 16:15:43 UTC+2: >>>>>>>>> >>>>>>>>>> Thanks! Although obviously there, I didn't get it :) >>>>>>>>>> >>>>>>>>>> Tom Keffer schrieb am Samstag, 14. September 2024 um 15:35:04 >>>>>>>>>> UTC+2: >>>>>>>>>> >>>>>>>>>>> Just substitute "mintime" / "maxtime" for "min" / "max". >>>>>>>>>>> >>>>>>>>>>> See the table of Aggregation Types >>>>>>>>>>> <https://www.weewx.com/docs/5.1/reference/aggtypes/>. >>>>>>>>>>> >>>>>>>>>>> On Sat, Sep 14, 2024 at 2:02 AM '[email protected]' via >>>>>>>>>>> weewx-development <[email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Thank you Tom, this works for the max/min value as I need it. >>>>>>>>>>>> But the dateTime for that value isn't in the result and I don't >>>>>>>>>>>> have a >>>>>>>>>>>> clue, how to get it for that particular value. >>>>>>>>>>>> >>>>>>>>>>>> Tom Keffer schrieb am Freitag, 13. September 2024 um 22:27:31 >>>>>>>>>>>> UTC+2: >>>>>>>>>>>> >>>>>>>>>>>>> The xtypes system can do this. See the section, *XTypes API >>>>>>>>>>>>> <https://github.com/weewx/weewx/wiki/xtypes#xtypes-api>*. >>>>>>>>>>>>> >>>>>>>>>>>>> Your example would be: >>>>>>>>>>>>> >>>>>>>>>>>>> results = weewx.xtypes.get_aggregate("outTemp", >>>>>>>>>>>>> TimeSpan(1726092000, 1726178400), "max", dbmanager) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> where "dbmanager" is an open database manager. The results >>>>>>>>>>>>> will be a ValueTuple >>>>>>>>>>>>> <https://www.weewx.com/docs/5.1/reference/valuetuple/>. >>>>>>>>>>>>> >>>>>>>>>>>>> Here's a more complete example: >>>>>>>>>>>>> >>>>>>>>>>>>> import weecfg >>>>>>>>>>>>> import weewx.manager >>>>>>>>>>>>> import weewx.xtypes >>>>>>>>>>>>> from weeutil.weeutil import TimeSpan >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> config_path, config_dict = >>>>>>>>>>>>> weecfg.read_config('/home/weewx/weewx.conf') >>>>>>>>>>>>> >>>>>>>>>>>>> with weewx.manager.open_manager_with_config(config_dict, >>>>>>>>>>>>> 'wx_binding') as dbmanager: >>>>>>>>>>>>> >>>>>>>>>>>>> results = weewx.xtypes.get_aggregate("outTemp", >>>>>>>>>>>>> TimeSpan(1726092000, 1726178400), "max", dbmanager) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> print(results) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> (68.4, 'degree_F', 'group_temperature') >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Sep 13, 2024 at 10:27 AM '[email protected]' via >>>>>>>>>>>>> weewx-development <[email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> When I write an extension in python, can I get the >>>>>>>>>>>>>> minimum/maximum value for an obs_type and a given timespan and >>>>>>>>>>>>>> their >>>>>>>>>>>>>> related timestamp? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Something like: >>>>>>>>>>>>>> >>>>>>>>>>>>>> maxValue, datetime = getMax("outTemp", 1726092000, >>>>>>>>>>>>>> 1726178400) >>>>>>>>>>>>>> >>>>>>>>>>>>>> The min/max has to be obtained from the archive_day table, >>>>>>>>>>>>>> the archive table might not contain the exact max/min value, and >>>>>>>>>>>>>> also not >>>>>>>>>>>>>> the exact timestamp the value arrived. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>>> Google Groups "weewx-development" group. >>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>>>> https://groups.google.com/d/msgid/weewx-development/6d13cde1-43c4-4ca3-a4a4-d220fc5e2040n%40googlegroups.com >>>>>>>>>>>>>> <https://groups.google.com/d/msgid/weewx-development/6d13cde1-43c4-4ca3-a4a4-d220fc5e2040n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>>>> . >>>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>> Google Groups "weewx-development" group. >>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>>> >>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>> https://groups.google.com/d/msgid/weewx-development/2aaf12b7-25ad-4013-8185-b379168b4bc5n%40googlegroups.com >>>>>>>>>>>> <https://groups.google.com/d/msgid/weewx-development/2aaf12b7-25ad-4013-8185-b379168b4bc5n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>> . >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "weewx-development" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> >>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/weewx-development/c183248b-1a04-47bd-8359-51ec5f215741n%40googlegroups.com >>>>>>>>> <https://groups.google.com/d/msgid/weewx-development/c183248b-1a04-47bd-8359-51ec5f215741n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "weewx-development" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> >>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/weewx-development/8138a283-82d2-418b-8a06-d919daee47b0n%40googlegroups.com >>>>>>> <https://groups.google.com/d/msgid/weewx-development/8138a283-82d2-418b-8a06-d919daee47b0n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "weewx-development" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> >>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/weewx-development/1c0b6e38-6d6f-4ce9-ae27-107f430cc310n%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/weewx-development/1c0b6e38-6d6f-4ce9-ae27-107f430cc310n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "weewx-development" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/weewx-development/66060704-cfb5-4a5a-8aaa-1c3db3589ba0n%40googlegroups.com >>> <https://groups.google.com/d/msgid/weewx-development/66060704-cfb5-4a5a-8aaa-1c3db3589ba0n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "weewx-development" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/weewx-development/3eb8ecfc-043a-4ff3-ba48-be9c4d02d961n%40googlegroups.com > <https://groups.google.com/d/msgid/weewx-development/3eb8ecfc-043a-4ff3-ba48-be9c4d02d961n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "weewx-development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/CAPq0zEASh4m3myxLuhwRUOV-GOFKvf7meKQNHdmnfcw66KdL-g%40mail.gmail.com.
