File attached On Wednesday 9 October 2024 at 19:11:24 UTC-4 [email protected] wrote:
> I am guessing you mean my original code. I’ll recreate and attach shortly. > So, to recap. I had a service MyService that was augmenting loop packets > with data. The standard processing of WeeWX accumulators was used to pull > out the min/max/first/last values (and times of these) and update the > archive record. > At first I was augmenting every loop packet with the min/max. But, I > noticed that the min/max (and times) values were ‘bleeding’ into the next > archive record. So, to mitigate that, I stopped putting the min/max (and > times) in every loop packet. > But there is still a slight chance that the last packet could ‘bleed’ into > the next archive record (In my case, if there is a lightning strike in the > last loop packet that has minimum distance smaller than any in the next > archive interval). I admit it is slight, but it was enough to bother me…. > rich > On Wednesday 9 October 2024 at 18:32:47 UTC-4 Tom Keffer wrote: > >> Perhaps if you posted your code? I'm betting you could use the existing >> accumulators to extract what you need from the collection of LOOP packets. >> >> On Wed, Oct 9, 2024 at 1:08 PM [email protected] <[email protected]> >> wrote: >> >>> Tom, >>> I get life, no apologies necessary. I’m probably not following your >>> thought, but here is what I think you are suggesting. >>> When MyService handles the NEW_LOOP_PACKET event, it checks if the data >>> belongs to the current accumulator (or has a new archive interval started). >>> If it does belong to the current accumulator (current archive interval), >>> it updates the current accumulator with the loop data. >>> If it does not belong (a new archive intervals has started), it creates >>> a new accumulator and puts the data in that (keeping the old one around, >>> similar to StdArchive (I think)). >>> When MyService handles the NEW_ARCHIVE_RECORD, it uses the ‘old’ >>> accumulator to retrieve the desired values and updates the archive record. >>> The challenge I was facing: I could not figure out a good way for >>> MyService to know anything about archive intervals in the NEW_LOOP_PACKET >>> event handler. >>> rich >>> >>> On Wednesday 9 October 2024 at 11:51:22 UTC-4 Tom Keffer wrote: >>> >>>> Sorry, Rich, for not getting back to you sooner. Just got swamped with >>>> another project. >>>> >>>> I can see how this can happen. If a LOOP packet arrives slightly after >>>> the end of an archive interval, it most properly belongs to the previous >>>> archive interval, but you're treating it as if it belongs to the next >>>> interval. >>>> >>>> Is there any reason why you can't use the accumulator machinery? It >>>> already handles min and max values. >>>> >>>> You've probably already seen it but, in case you haven't, take a look >>>> at the wiki article *Accumulators >>>> <https://github.com/weewx/weewx/wiki/Accumulators>*. >>>> >>>> -tk >>>> >>>> >>>> On Sat, Oct 5, 2024 at 4:46 PM [email protected] <[email protected]> >>>> wrote: >>>> >>>>> So to close this out. I updated the service to capture the data via >>>>> the NEW_LOOP_PACKET event. Then in the NEW_ARCHIVE_RECORD I only process >>>>> event data that is in the archive interval. Essentially a ‘poor mans’ >>>>> accumulator. >>>>> This has stopped the ‘bleeding’ of data from a previous archive >>>>> interval into the current interval. I still need to write some more >>>>> tests, >>>>> but I am pretty confident this accomplishes what I desire. >>>>> rich >>>>> >>>>> On Sunday 22 September 2024 at 11:50:07 UTC-4 [email protected] wrote: >>>>> >>>>>> As a followup. When I flipped the order that StdEngine fires the >>>>>> events, such that CHECK_LOOP is before NEW_LOOP this is what I observed. >>>>>> 1. A new loop packet is generated. >>>>>> 2. StdEngine fires the CHECK_LOOP event. >>>>>> 3. StdArchive handles the CHECK_LOOP. >>>>>> a. The dateTime in the loop packet is greater than the end of the >>>>>> archive period. >>>>>> b. StdArchive fires the END_ARCHIVE_PERIOD event. >>>>>> c. The dateTime is not greater than the end of the archive period >>>>>> + the archive delay. >>>>>> 4. MyService handles the END_ARCHIVE_PERIOD. >>>>>> a. Min values are reset to None, etc. >>>>>> 5. StdEngine fires a NEW_LOOP event >>>>>> 6. MyService handles the NEW_LOOP event. >>>>>> a. It sees that no prior value exists >>>>>> b. The current value in the loop packet is added to the loop >>>>>> packet as min_value. >>>>>> 7. StdArchive handles the NEW_LOOP event. >>>>>> a. It tries to add it to the accumulator. >>>>>> b. The accumulator throws OutOfSpan exception >>>>>> c. StdArchive creates a 'new' accumulator for the next archive >>>>>> period. >>>>>> d. StdArchive adds the loop packet to the accumulator, >>>>>> >>>>>> 8. Eventually a packet with dateTime >= the end of archive period + >>>>>> archive delay is generated. >>>>>> a. This causes StdArchive (in CHECK_LOOP) to raise the BREAKLOOP >>>>>> exception >>>>>> b. StdEngine handles the exception, raising POST_LOOP; allowing >>>>>> things like reports to run etc. >>>>>> >>>>>> Again, not saying that changing the order that events are fired >>>>>> should be taken lightly. >>>>>> rich >>>>>> >>>>>> On Sunday 22 September 2024 at 11:28:28 UTC-4 [email protected] >>>>>> wrote: >>>>>> >>>>>>> Tom, >>>>>>> Thanks for the pointer. That is what I am attempting to do. Here is >>>>>>> what I THINK is my problem. See 4.d below. >>>>>>> 1. A new loop packet is generated. >>>>>>> 2. StdEngine fires a NEW_LOOP event >>>>>>> 3. MyService handles the NEW_LOOP event. >>>>>>> a. It sees that a prior value is less then the current value in >>>>>>> the loop packet. >>>>>>> b. The prior value is added to the loop packet as min_value. >>>>>>> 4. StdArchive handles the NEW_LOOP event. >>>>>>> a. It tries to add it to the accumulator. >>>>>>> b. The accumulator throws OutOfSpan exception >>>>>>> c. StdArchive creates a 'new' accumulator for the next archive >>>>>>> period. >>>>>>> d. StdArchive adds the loop packet (which has been updated with >>>>>>> prior loop packet data) to the accumulator, >>>>>>> 5. StdEngine fires the CHECK_LOOP event. >>>>>>> 6. StdArchive handles the CHECK_LOOP. >>>>>>> a. The dateTime in the loop packet is greater than the end of the >>>>>>> archive period. >>>>>>> b. StdArchive fires the END_ARCHIVE_PERIOD event. >>>>>>> c. The dateTime is not greater than the end of the archive period >>>>>>> + the archive delay. >>>>>>> 7. MyService handles the END_ARCHIVE_PERIOD. >>>>>>> a. Min values are reset to None, etc. >>>>>>> >>>>>>> >>>>>>> 8. Eventually a packet with dateTime >= the end of archive period + >>>>>>> archive delay is generated. >>>>>>> a. This causes StdArchive (in CHECK_LOOP) to raise the BREAKLOOP >>>>>>> exception >>>>>>> b. StdEngine handles the exception, raising POST_LOOP; allowing >>>>>>> things like reports to run etc. >>>>>>> >>>>>>> rich >>>>>>> >>>>>>> On Sunday 22 September 2024 at 11:00:00 UTC-4 Tom Keffer wrote: >>>>>>> >>>>>>>> Take a look at the Vantage device driver. In addition to a standard >>>>>>>> device driver, it includes a service, VantageService, that >>>>>>>> participates in >>>>>>>> the weewxd event loop. It binds to STARTUP, NEW_LOOP_PACKET, and >>>>>>>> END_ARCHIVE_PERIOD. The first two are used to reset wind gust values, >>>>>>>> the >>>>>>>> last to update the archive record with the max wind seen. >>>>>>>> >>>>>>>> I think this is pretty similar to what you're trying to do. >>>>>>>> >>>>>>>> -tk >>>>>>>> >>>>>>>> On Saturday, September 21, 2024 at 5:01:26 PM UTC-7 >>>>>>>> [email protected] wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> Goal: Capture the min value and time of lightning strikes. >>>>>>>>> >>>>>>>>> I started out binding to the PRE_LOOP and NEW_LOOP_PACKET events. >>>>>>>>> When the PRE_LOOP event is handled, the min value and time were >>>>>>>>> reset. In >>>>>>>>> the handling of the NEW_LOOP_PACKET event, the comparison is done and >>>>>>>>> the >>>>>>>>> loop packet is updated as necessary. I 'quickly' realized that >>>>>>>>> PRE_LOOP and >>>>>>>>> POST_LOOP events have nothing to do with the archive period. >>>>>>>>> >>>>>>>>> The END_ARCHIVE_PERIOD event looked promising. Resetting of the >>>>>>>>> value and time could be done here. But when I analyze the order of >>>>>>>>> event >>>>>>>>> firing, this is what I am seeing. >>>>>>>>> 2024-09-21 11:44:59 weewxd[4036220] DEBUG user.observationtime: >>>>>>>>> RMB: In new_loop_packet >>>>>>>>> 2024-09-21 11:44:59 weewxd[4036220] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726933499 2024-09-21 11:44:59 >>>>>>>>> 2024-09-21 11:44:59 weewxd[4036220] DEBUG user.observationtime: >>>>>>>>> RMB: In check_loop >>>>>>>>> 2024-09-21 11:44:59 weewxd[4036220] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726933499 2024-09-21 11:44:59 >>>>>>>>> 2024-09-21 11:45:01 weewxd[4036220] DEBUG user.observationtime: >>>>>>>>> RMB: In new_loop_packet >>>>>>>>> 2024-09-21 11:45:01 weewxd[4036220] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726933502 2024-09-21 11:45:02 >>>>>>>>> 2024-09-21 11:45:01 weewxd[4036220] DEBUG user.observationtime: >>>>>>>>> RMB: In check_loop >>>>>>>>> 2024-09-21 11:45:01 weewxd[4036220] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726933502 2024-09-21 11:45:02 >>>>>>>>> 2024-09-21 11:45:01 weewxd[4036220] DEBUG user.observationtime: >>>>>>>>> RMBx: In end_archive_period >>>>>>>>> 2024-09-21 11:45:04 weewxd[4036220] DEBUG user.observationtime: >>>>>>>>> RMB: In new_loop_packet >>>>>>>>> 2024-09-21 11:45:04 weewxd[4036220] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726933504 2024-09-21 11:45:04 >>>>>>>>> This will result in the first packet (1726933502) of the archive >>>>>>>>> period using the data from the previous archive period. >>>>>>>>> >>>>>>>>> Flipping the order in engine.py to the following: >>>>>>>>> for packet in self.console.genLoopPackets(): >>>>>>>>> # Allow services to break the loop by throwing >>>>>>>>> # an exception: >>>>>>>>> self.dispatchEvent(weewx.Event(weewx.CHECK_LOOP, >>>>>>>>> packet=packet)) >>>>>>>>> >>>>>>>>> # Package the packet as an event, then dispatch it. >>>>>>>>> self.dispatchEvent(weewx.Event(weewx.NEW_LOOP_PACKET, >>>>>>>>> packet=packet)) >>>>>>>>> >>>>>>>>> I would have expected to see the last packet in an archive period >>>>>>>>> not being processed. But, I was not seeing that. So I added a bit >>>>>>>>> more >>>>>>>>> logging to try to understand what was going on. This is what I saw. >>>>>>>>> Sorry, >>>>>>>>> it is a bit long, I made a couple of comments with <---. >>>>>>>>> >>>>>>>>> 2024-09-21 19:39:58 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In check_loop >>>>>>>>> 2024-09-21 19:39:58 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726961999 2024-09-21 19:39:59 >>>>>>>>> 2024-09-21 19:39:58 weewxd[235236] DEBUG weewx.engine: RMBx: In >>>>>>>>> check_loop (StdArchive) packet dateTime: 1726961999 >>>>>>>>> end_archive_delay_ts: >>>>>>>>> 1726962035 1726962000 35 >>>>>>>>> 2024-09-21 19:39:58 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In new_loop_packet >>>>>>>>> 2024-09-21 19:39:58 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726961999 2024-09-21 19:39:59 >>>>>>>>> 2024-09-21 19:39:58 weewxd[235236] DEBUG weewx.engine: RMBx: in >>>>>>>>> new_loop_packet (StdArchive) >>>>>>>>> 2024-09-21 19:40:01 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In check_loop >>>>>>>>> 2024-09-21 19:40:01 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962001 2024-09-21 19:40:01 >>>>>>>>> 2024-09-21 19:40:01 weewxd[235236] DEBUG weewx.engine: RMBx: In >>>>>>>>> check_loop (StdArchive) packet dateTime: 1726962001 >>>>>>>>> end_archive_delay_ts: >>>>>>>>> 1726962035 1726962000 35 >>>>>>>>> 2024-09-21 19:40:01 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMBx: In end_archive_period <--- END_ARCHIVE_PERIOD event must have >>>>>>>>> fired >>>>>>>>> 2024-09-21 19:40:01 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In new_loop_packet >>>>>>>>> 2024-09-21 19:40:01 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962001 2024-09-21 19:40:01 <--- And the >>>>>>>>> NEW_LOOP_PACKET has fired afterward >>>>>>>>> 2024-09-21 19:40:01 weewxd[235236] DEBUG weewx.engine: RMBx: in >>>>>>>>> new_loop_packet (StdArchive) >>>>>>>>> 2024-09-21 19:40:03 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In check_loop <--- processing of even packets continues >>>>>>>>> 2024-09-21 19:40:03 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962004 2024-09-21 19:40:04 >>>>>>>>> 2024-09-21 19:40:03 weewxd[235236] DEBUG weewx.engine: RMBx: In >>>>>>>>> check_loop (StdArchive) packet dateTime: 1726962004 >>>>>>>>> end_archive_delay_ts: >>>>>>>>> 1726962035 1726962300 35 >>>>>>>>> 2024-09-21 19:40:03 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In new_loop_packet >>>>>>>>> 2024-09-21 19:40:03 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962004 2024-09-21 19:40:04 >>>>>>>>> 2024-09-21 19:40:03 weewxd[235236] DEBUG weewx.engine: RMBx: in >>>>>>>>> new_loop_packet (StdArchive) >>>>>>>>> 2024-09-21 19:40:06 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In check_loop >>>>>>>>> 2024-09-21 19:40:06 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962006 2024-09-21 19:40:06 >>>>>>>>> 2024-09-21 19:40:06 weewxd[235236] DEBUG weewx.engine: RMBx: In >>>>>>>>> check_loop (StdArchive) packet dateTime: 1726962006 >>>>>>>>> end_archive_delay_ts: >>>>>>>>> 1726962035 1726962300 35 >>>>>>>>> 2024-09-21 19:40:06 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In new_loop_packet >>>>>>>>> 2024-09-21 19:40:06 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962006 2024-09-21 19:40:06 >>>>>>>>> 2024-09-21 19:40:06 weewxd[235236] DEBUG weewx.engine: RMBx: in >>>>>>>>> new_loop_packet (StdArchive) >>>>>>>>> 2024-09-21 19:40:08 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In check_loop >>>>>>>>> 2024-09-21 19:40:08 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962009 2024-09-21 19:40:09 >>>>>>>>> 2024-09-21 19:40:08 weewxd[235236] DEBUG weewx.engine: RMBx: In >>>>>>>>> check_loop (StdArchive) packet dateTime: 1726962009 >>>>>>>>> end_archive_delay_ts: >>>>>>>>> 1726962035 1726962300 35 >>>>>>>>> 2024-09-21 19:40:08 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In new_loop_packet >>>>>>>>> 2024-09-21 19:40:08 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962009 2024-09-21 19:40:09 >>>>>>>>> 2024-09-21 19:40:08 weewxd[235236] DEBUG weewx.engine: RMBx: in >>>>>>>>> new_loop_packet (StdArchive) >>>>>>>>> 2024-09-21 19:40:11 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In check_loop >>>>>>>>> 2024-09-21 19:40:11 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962011 2024-09-21 19:40:11 >>>>>>>>> 2024-09-21 19:40:11 weewxd[235236] DEBUG weewx.engine: RMBx: In >>>>>>>>> check_loop (StdArchive) packet dateTime: 1726962011 >>>>>>>>> end_archive_delay_ts: >>>>>>>>> 1726962035 1726962300 35 >>>>>>>>> 2024-09-21 19:40:11 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In new_loop_packet >>>>>>>>> 2024-09-21 19:40:11 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962011 2024-09-21 19:40:11 >>>>>>>>> 2024-09-21 19:40:11 weewxd[235236] DEBUG weewx.engine: RMBx: in >>>>>>>>> new_loop_packet (StdArchive) >>>>>>>>> 2024-09-21 19:40:13 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In check_loop >>>>>>>>> 2024-09-21 19:40:13 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962014 2024-09-21 19:40:14 >>>>>>>>> 2024-09-21 19:40:13 weewxd[235236] DEBUG weewx.engine: RMBx: In >>>>>>>>> check_loop (StdArchive) packet dateTime: 1726962014 >>>>>>>>> end_archive_delay_ts: >>>>>>>>> 1726962035 1726962300 35 >>>>>>>>> 2024-09-21 19:40:13 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In new_loop_packet >>>>>>>>> 2024-09-21 19:40:13 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962014 2024-09-21 19:40:14 >>>>>>>>> 2024-09-21 19:40:13 weewxd[235236] DEBUG weewx.engine: RMBx: in >>>>>>>>> new_loop_packet (StdArchive) >>>>>>>>> 2024-09-21 19:40:16 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In check_loop >>>>>>>>> 2024-09-21 19:40:16 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962016 2024-09-21 19:40:16 >>>>>>>>> 2024-09-21 19:40:16 weewxd[235236] DEBUG weewx.engine: RMBx: In >>>>>>>>> check_loop (StdArchive) packet dateTime: 1726962016 >>>>>>>>> end_archive_delay_ts: >>>>>>>>> 1726962035 1726962300 35 >>>>>>>>> 2024-09-21 19:40:16 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In new_loop_packet >>>>>>>>> 2024-09-21 19:40:16 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962016 2024-09-21 19:40:16 >>>>>>>>> 2024-09-21 19:40:16 weewxd[235236] DEBUG weewx.engine: RMBx: in >>>>>>>>> new_loop_packet (StdArchive) >>>>>>>>> 2024-09-21 19:40:18 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In check_loop >>>>>>>>> 2024-09-21 19:40:18 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962019 2024-09-21 19:40:19 >>>>>>>>> 2024-09-21 19:40:18 weewxd[235236] DEBUG weewx.engine: RMBx: In >>>>>>>>> check_loop (StdArchive) packet dateTime: 1726962019 >>>>>>>>> end_archive_delay_ts: >>>>>>>>> 1726962035 1726962300 35 >>>>>>>>> 2024-09-21 19:40:18 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In new_loop_packet >>>>>>>>> 2024-09-21 19:40:18 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962019 2024-09-21 19:40:19 >>>>>>>>> 2024-09-21 19:40:18 weewxd[235236] DEBUG weewx.engine: RMBx: in >>>>>>>>> new_loop_packet (StdArchive) >>>>>>>>> 2024-09-21 19:40:21 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In check_loop >>>>>>>>> 2024-09-21 19:40:21 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962021 2024-09-21 19:40:21 >>>>>>>>> 2024-09-21 19:40:21 weewxd[235236] DEBUG weewx.engine: RMBx: In >>>>>>>>> check_loop (StdArchive) packet dateTime: 1726962021 >>>>>>>>> end_archive_delay_ts: >>>>>>>>> 1726962035 1726962300 35 >>>>>>>>> 2024-09-21 19:40:21 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In new_loop_packet >>>>>>>>> 2024-09-21 19:40:21 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962021 2024-09-21 19:40:21 >>>>>>>>> 2024-09-21 19:40:21 weewxd[235236] DEBUG weewx.engine: RMBx: in >>>>>>>>> new_loop_packet (StdArchive) >>>>>>>>> 2024-09-21 19:40:23 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In check_loop >>>>>>>>> 2024-09-21 19:40:23 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962024 2024-09-21 19:40:24 >>>>>>>>> 2024-09-21 19:40:23 weewxd[235236] DEBUG weewx.engine: RMBx: In >>>>>>>>> check_loop (StdArchive) packet dateTime: 1726962024 >>>>>>>>> end_archive_delay_ts: >>>>>>>>> 1726962035 1726962300 35 >>>>>>>>> 2024-09-21 19:40:23 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In new_loop_packet >>>>>>>>> 2024-09-21 19:40:23 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962024 2024-09-21 19:40:24 >>>>>>>>> 2024-09-21 19:40:23 weewxd[235236] DEBUG weewx.engine: RMBx: in >>>>>>>>> new_loop_packet (StdArchive) >>>>>>>>> 2024-09-21 19:40:26 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In check_loop >>>>>>>>> 2024-09-21 19:40:26 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962026 2024-09-21 19:40:26 >>>>>>>>> 2024-09-21 19:40:26 weewxd[235236] DEBUG weewx.engine: RMBx: In >>>>>>>>> check_loop (StdArchive) packet dateTime: 1726962026 >>>>>>>>> end_archive_delay_ts: >>>>>>>>> 1726962035 1726962300 35 >>>>>>>>> 2024-09-21 19:40:26 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In new_loop_packet >>>>>>>>> 2024-09-21 19:40:26 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962026 2024-09-21 19:40:26 >>>>>>>>> 2024-09-21 19:40:26 weewxd[235236] DEBUG weewx.engine: RMBx: in >>>>>>>>> new_loop_packet (StdArchive) >>>>>>>>> 2024-09-21 19:40:28 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In check_loop >>>>>>>>> 2024-09-21 19:40:28 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962029 2024-09-21 19:40:29 >>>>>>>>> 2024-09-21 19:40:28 weewxd[235236] DEBUG weewx.engine: RMBx: In >>>>>>>>> check_loop (StdArchive) packet dateTime: 1726962029 >>>>>>>>> end_archive_delay_ts: >>>>>>>>> 1726962035 1726962300 35 >>>>>>>>> 2024-09-21 19:40:28 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In new_loop_packet >>>>>>>>> 2024-09-21 19:40:28 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962029 2024-09-21 19:40:29 >>>>>>>>> 2024-09-21 19:40:28 weewxd[235236] DEBUG weewx.engine: RMBx: in >>>>>>>>> new_loop_packet (StdArchive) >>>>>>>>> 2024-09-21 19:40:31 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In check_loop >>>>>>>>> 2024-09-21 19:40:31 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962031 2024-09-21 19:40:31 >>>>>>>>> 2024-09-21 19:40:31 weewxd[235236] DEBUG weewx.engine: RMBx: In >>>>>>>>> check_loop (StdArchive) packet dateTime: 1726962031 >>>>>>>>> end_archive_delay_ts: >>>>>>>>> 1726962035 1726962300 35 >>>>>>>>> 2024-09-21 19:40:31 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In new_loop_packet >>>>>>>>> 2024-09-21 19:40:31 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962031 2024-09-21 19:40:31 >>>>>>>>> 2024-09-21 19:40:31 weewxd[235236] DEBUG weewx.engine: RMBx: in >>>>>>>>> new_loop_packet (StdArchive) >>>>>>>>> 2024-09-21 19:40:33 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In check_loop >>>>>>>>> 2024-09-21 19:40:33 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962034 2024-09-21 19:40:34 >>>>>>>>> 2024-09-21 19:40:33 weewxd[235236] DEBUG weewx.engine: RMBx: In >>>>>>>>> check_loop (StdArchive) packet dateTime: 1726962034 >>>>>>>>> end_archive_delay_ts: >>>>>>>>> 1726962035 1726962300 35 >>>>>>>>> 2024-09-21 19:40:33 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In new_loop_packet >>>>>>>>> 2024-09-21 19:40:33 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962034 2024-09-21 19:40:34 >>>>>>>>> 2024-09-21 19:40:33 weewxd[235236] DEBUG weewx.engine: RMBx: in >>>>>>>>> new_loop_packet (StdArchive) >>>>>>>>> 2024-09-21 19:40:36 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In check_loop >>>>>>>>> 2024-09-21 19:40:36 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962036 2024-09-21 19:40:36 >>>>>>>>> 2024-09-21 19:40:36 weewxd[235236] DEBUG weewx.engine: RMBx: In >>>>>>>>> check_loop (StdArchive) packet dateTime: 1726962036 >>>>>>>>> end_archive_delay_ts: >>>>>>>>> 1726962035 1726962300 35 >>>>>>>>> 2024-09-21 19:40:36 weewxd[235236] DEBUG weewx.engine: RMBx: >>>>>>>>> Raising BreakLoop 1726962036 1726962035 <--- due to archive_delay, >>>>>>>>> the loop >>>>>>>>> is 'finally' broken and the archive record can be processed. >>>>>>>>> 2024-09-21 19:40:36 weewxd[235236] DEBUG weewx.engine: RMBx: >>>>>>>>> handling BreakLoop exception >>>>>>>>> 2024-09-21 19:40:36 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMBx: In post_loop >>>>>>>>> 2024-09-21 19:40:36 weewxd[235236] DEBUG weewx.engine: RMBx: In >>>>>>>>> post_loop (StdArchive) >>>>>>>>> 2024-09-21 19:40:36 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMBx: In new_archive_record <--- processing the new archive record >>>>>>>>> 2024-09-21 19:40:56 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMBx: In pre_loop >>>>>>>>> 2024-09-21 19:40:56 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: In check_loop >>>>>>>>> 2024-09-21 19:40:56 weewxd[235236] DEBUG user.observationtime: >>>>>>>>> RMB: packet dateTime: 1726962039 2024-09-21 19:40:39 >>>>>>>>> >>>>>>>>> Because of the archive_delay, the last packet goes into the >>>>>>>>> correct archive record and the END_ARCHIVE_PERIOD event fires before >>>>>>>>> the >>>>>>>>> NEW_LOOP_PACKET event is fired. >>>>>>>>> >>>>>>>>> I realize that changing the order of events firing can have big >>>>>>>>> impact to existing code, but I am curious about possibly making this >>>>>>>>> change? It seems to me that this should only impact anyone that wants >>>>>>>>> the >>>>>>>>> END_ARCHIVE_PERIOD event to fire before the next packet's >>>>>>>>> NEW_LOOP_PACKET >>>>>>>>> is fired. Not sure if that is important to anyone... Or is there a >>>>>>>>> better/more WeeWX way of accomplishing what I desire? >>>>>>>>> If no changes are made to WeeWX, my current thought is to capture >>>>>>>>> the data via the NEW_LOOP_PACKET and perform the processing via the >>>>>>>>> NEW_ARCHIVE_RECORD event and only update the archive record with the >>>>>>>>> min >>>>>>>>> value and time. This would certainly work for me. >>>>>>>>> >>>>>>>>> Thanks. rich >>>>>>>>> >>>>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "weewx-development" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/weewx-development/d5388801-335e-482e-acc0-2fff84cd3633n%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/weewx-development/d5388801-335e-482e-acc0-2fff84cd3633n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "weewx-development" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/weewx-development/cdd39a81-25de-4faf-b383-8d5420041d83n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/weewx-development/cdd39a81-25de-4faf-b383-8d5420041d83n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "weewx-development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/926cd97a-0ea8-4da8-be95-1800776003e6n%40googlegroups.com.
# # Copyright (c) 2024 Rich Bell <[email protected]> # # See the file LICENSE.txt for your full rights. # ''' WeeWX service to capture times and values for the first, last, min, and max of an observation in an archive period. Prerequistes: Python 3.7+ WeeWX 5.x+ Installation: 1. Put this file in the bin/user directory. 2. Update weewx.conf [ObservationTime] as needed to configure the service. 3. Add the service, user.observationtime.ObservationTime to the appropriate 'service group'. This would typically be the 'data_services' group. But it might be the 'process_services' group if it needs to run after any of the WeeWX services in this group. Overview: For a configured WeeWX observation, capture any of the last, first, min, or max values in an archive period. In addition to capturing the value, the time that the value was observed is also captured. These values and times are added to the loop packet. The WeeWX accumulator function is used to populate the archive record with these values. Important: - The daily summaries cannot be used to get the time (observation_time_name) of the value. This is because the daily summary will have the min/max of the time, not the time of the min/max value. Configuration: [ObservationTime] [[observations]] # The first observation whose 'event' value and time will be captured. # For example: lightning_distance, windGust, etc. [[[REPLACE_ME]]] [[[[last]]]] # The name of the WeeWX to store the last observation's value. # For example: lightning_last_distance, windGust_last, etc. observation_name = # The name of the WeeWX to store the last observation's time. # For example: lightning_last_det_time, windGust_last_time, etc. observation_time_name = [[[[first]]]] # The name of the WeeWX to store the first observation's value. # For example: lightning_first_distance, windGust_first, etc. observation_name = # The name of the WeeWX to store the first observation's time. # For example: lightning_first_det_time, windGust_first_time, etc. observation_time_name = [[[[min]]]] # The name of the WeeWX to store the min observation's value. # For example: lightning_min_distance, windGust_min, etc. observation_name # The name of the WeeWX to store the min observation's time. # For example: lightning_min_det_time, windGust_min_time, etc. observation_time_name = [[[[max]]]] # The name of the WeeWX to store the max observation's value. # For example: lightning_max_distance, windGust_max, etc. observation_name = # The name of the WeeWX to store the max observation's time. # For example: lightning_max_det_time, windGust_max_time, etc. observation_time_name = # The next observation whose 'event' value and time will be captured. [[[REPLACE_ME_TOO]]] Below is an example of a 'complete' configuration to capture the last, first, and closest lightning strike in an archive period. This assumes that in the WeeWX loop packet, the 'lightning_distance' field captures the distance to the strike. [Engine] [[Services]] data_services = user.observationtime.ObservationTime [Accumulator] [[lightning_last_distance]] extractor = last [[lightning_last_det_time]] extractor = last [[lightning_first_distance]] extractor = first [[lightning_first_det_time]] extractor = first [[lightning_min_distance]] extractor = last [[lightning_min_det_time]] extractor = last [ObservationTime] [[observations]] [[[lightning_distance]]] [[[[last]]]] observation_name = lightning_last_distance observation_time = lightning_last_det_time [[[[first]]]] observation_name = lightning_first_distance observation_time = lightning_first_det_time [[[[min]]]] observation_name = lightning_min_distance observation_time = lightning_min_det_time Add the additional fields to the database. weectl database add-column lightning_last_distance --type=REAL weectl database add-column lightning_last_det_time --type=INTEGER weectl database add-column lightning_first_distance --type=REAL weectl database add-column lightning_first_det_time --type=INTEGER weectl database add-column lightning_min_distance --type=REAL weectl database add-column lightning_min_det_time --type=INTEGER Update bin/user/extensions.py with the units for the new fields. import weewx.units weewx.units.obs_group_dict['lightning_last_distance'] = 'group_distance' weewx.units.obs_group_dict['lightning_last_det_time'] = 'group_time' weewx.units.obs_group_dict['lightning_first_distance'] = 'group_distance' weewx.units.obs_group_dict['lightning_first_det_time'] = 'group_time' weewx.units.obs_group_dict['lightning_min_distance'] = 'group_distance' weewx.units.obs_group_dict['lightning_min_det_time'] = 'group_time' ''' # ToDo: Decide if the aggregate types should be new/unique, or to continue to override WeeWX's existing types. import logging import weewx import weewx.engine import weedb from weeutil.weeutil import to_bool VERSION = '0.1.0' log = logging.getLogger(__name__) class ObservationTime(weewx.engine.StdService): ''' Save the times and values of the first, last, min, and max of an observation.''' def __init__(self, engine, config_dict): log.info("Version is: %s", VERSION) super(ObservationTime, self).__init__(engine, config_dict) # ToDo: perform a deep copy service_dict = config_dict.get('ObservationTime', {}) self.observations = service_dict.get('observations', {}) log.debug("The configuration is: %s", self.observations) if to_bool(service_dict.get('augment_loop', True)): self.bind(weewx.NEW_LOOP_PACKET, self.new_loop_packet) self.bind(weewx.STARTUP, self.startup) self.bind(weewx.END_ARCHIVE_PERIOD, self.end_archive_period) self.observation_time_xtype = ObservationTimeXtype(self.observations) def _reset_observations(self): for _observation, observation_data in self.observations.items(): for observation_type in observation_data: observation_data[observation_type]['observation'] = None observation_data[observation_type]['observation_time'] = None def startup(self, _event): ''' Handle the WeeWX STARTUP event. ''' self._reset_observations() def end_archive_period(self, _event): ''' Handle the WeeWX END_ARCHIVE_PERIOD event. ''' self._reset_observations() def new_loop_packet(self, event): ''' Handle the WeeWX POST_LOOP event.''' log.debug("Incoming packet is: %s", event.packet) for observation, observation_data in self.observations.items(): if observation not in event.packet: continue log.debug("Processing observation: %s %s", observation, observation_data) observation_value = event.packet[observation] observation_time = event.packet['dateTime'] log.debug("Processing current value: %s and current time: %s", observation_value, observation_time) if observation_value is None: continue for observation_type in observation_data: observation_name = observation_data[observation_type]['observation_name'] observation_time_name = observation_data[observation_type]['observation_time_name'] previous_value = observation_data[observation_type]['observation'] previous_time = observation_data[observation_type]['observation_time'] observation_data[observation_type]['observation'] = previous_value observation_data[observation_type]['observation_time'] = previous_time log.debug("Processing type: %s with observation name: %s and observation time name: %s", observation_type, observation_name, observation_time_name) log.debug("Processing previous value: %s and previous time: %s", previous_value, previous_time) if observation_type == 'last' and observation_value is not None: log.debug("Setting %s of %s value %s and time %s", observation_type, observation, observation_value, observation_time) observation_data[observation_type]['observation'] = observation_value observation_data[observation_type]['observation_time'] = observation_time if observation_type == 'first' and observation_value is not None and previous_value is None: log.debug("Setting %s of %s value %s and time %s", observation_type, observation, observation_value, observation_time) observation_data[observation_type]['observation'] = observation_value observation_data[observation_type]['observation_time'] = observation_time if observation_type == 'min' and observation_value is not None and \ (previous_value is None or observation_value <= previous_value): log.debug("Setting %s of %s value %s and time %s", observation_type, observation, observation_value, observation_time) observation_data[observation_type]['observation'] = observation_value observation_data[observation_type]['observation_time'] = observation_time if observation_type == 'max' and observation_value is not None and \ (previous_value is None or observation_value >= previous_value): log.debug("Setting %s of %s value %s and time %s", observation_type, observation, observation_value, observation_time) observation_data[observation_type]['observation'] = observation_value observation_data[observation_type]['observation_time'] = observation_time event.packet[observation_name] = observation_data[observation_type]['observation'] event.packet[observation_time_name] = observation_data[observation_type]['observation_time'] log.debug("Outgoing packet is: %s", event.packet)
