Thanks Rich - I took a look at mine and my differences were literally a fraction of a second slow in some cases. Pretty amazing.
Since then I have 'one' missed measure in 16 hours after ignoring the start/end times, so much much better so far :-) Just so I can find it via search later on, here's the query I run to search for my external sensors which are mapped to extraTemp1 and the like. The dateTime maps to around 6PM PST yesterday.... echo "select datetime(dateTime, 'unixepoch', 'localtime'),dateTime,extraTemp1,extraTemp2 from archive where (extraTemp1 is null and dateTime > 1639101300);" | sqlite3 /home/weewx/archive/weewx.sdb On Thursday, December 9, 2021 at 5:55:39 PM UTC-8 [email protected] wrote: > Hi, > I think you have gotten bit by some code that needs to be improved. The > code attempts to make sure that data arriving is neither too old nor to > 'new'. Since your data does not have a timestamp, this check is a bit > silly. To turn it off you can use the ignore_start_time = True and > ignore_end_time = True options. See, > https://github.com/bellrichm/WeeWX-MQTTSubscribe/wiki/Date-Time-processing > So, you would end up with something like: > > [MQTTSubscribeService] > [[topics]] > ignore_start_time = True > ignore_end_time = True > [[[mytopic]]] > ignore = true > [[[[CO2_Value_PPM]]]] > name = co2 > ignore = false > expires_after = none > > rich > On Thursday, 9 December 2021 at 15:34:19 UTC-5 [email protected] wrote: > >> I'm sure everyone is tired of me and this topic but I'm trying to >> understand what is happening here. >> >> I have switched to using " $day.co2.last" which should work for all >> hourly data sent except for maybe something very close to midnight. >> >> Looking at the contents of weewx.sdb I still notice gaps in the db. With >> weewx.log, I can see that a couple of missing data were sent correctly but: >> >> Dec 9 12:08:11 n4mrv weewx[448959] INFO user.MQTTSubscribe: (Service) >> TopicManager ignoring record outside of interval 1639069690.000000 >> 1639069692.000000 1639069689.979975 co2: 423.0, dateTime: >> 1639069689.9799752, usUnits: 1 >> Dec 9 13:09:25 n4mrv weewx[448959] INFO user.MQTTSubscribe: (Service) >> TopicManager ignoring record outside of interval 1639073364.000000 >> 1639073366.000000 1639073363.761206 co2: 415.0, dateTime: >> 1639073363.7612064, usUnits: 1 >> >> This happens for a couple of hours and the data is not included in the db. >> >> On the third hour, the data is sent and correctly put into the db >> >> Dec 9 14:10:37 n4mrv weewx[448959] DEBUG user.MQTTSubscribe: (Service) >> TopicManager data-> outgoing accumulated mytopic: co2: 400.0, dateTime: >> 1639077038.0, usUnits: 1 >> >> The attached file shows what is happening. >> >> Thanks in advance for any suggestion where I'm going wrong here. There >> may be something I have incorrect in MQTTSubscribe. I'll be happy to send >> more information if needed. >> Bob >> On Monday, December 6, 2021 at 8:11:50 PM UTC-5 [email protected] wrote: >> >>> No reason why you can't use $day.co2.last. It will give you the last >>> valid value for the day. >>> >>> On Mon, Dec 6, 2021 at 7:38 AM [email protected] <[email protected]> wrote: >>> >>>> Ok, I seem to have jumped the gun :( >>>> >>>> Things are much better but I'm still getting the N/A in place of data >>>> every few archive records (current data). My suspicion here is that the >>>> sensor is not sending on exactly 60 minute intervals. Looking at the >>>> actual >>>> times in the LoRa server, it's 60, 61 or 62 minute intervals--not always >>>> the same. Is it possible, using the basic " $hour.co2.last" syntax, to >>>> change the "hour" to say 65 minutes? This would probably catch the changes >>>> from one hour. I have attached a file of what is in the weewx.sdb showing >>>> the missing data in certain instances. If the sensor has updated just >>>> after >>>> the archive time (15 minutes) then it misses recording new data. >>>> I'm just guessing here at what might be going on. The graph seems to be >>>> picking up the data with no problem. >>>> Thanks. >>>> Bob[image: zbsng7lr.bmp] >>>> >>>> On Sunday, December 5, 2021 at 12:41:03 PM UTC-5 [email protected] wrote: >>>> >>>>> It looks like " $hour.co2.last" did it, at least the last four hours >>>>> are updating correctly with no N/As mixed in. >>>>> >>>>> Many thanks to all who contributed help on this, and much appreciated >>>>> is all the hard work that goes into weewx maintenance and updates by many! >>>>> Bob >>>>> http://grattans.org/wx >>>>> >>>>> On Sunday, December 5, 2021 at 8:10:12 AM UTC-5 [email protected] >>>>> wrote: >>>>> >>>>>> I'd go with Tom's advice, but if you want to get the cache working >>>>>> I'd need a log file with debug set to 2. It needs to have a few archive >>>>>> records before the first CO2 reading arrives via MQTT and a few records >>>>>> after. Warning, the file is going to large, very large. >>>>>> rich >>>>>> >>>>>> On Saturday, 4 December 2021 at 20:08:19 UTC-5 [email protected] >>>>>> wrote: >>>>>> >>>>>>> Take the accumulator stuff out. >>>>>>> >>>>>>> Why not just use $hour.co2.last? >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Sat, Dec 4, 2021 at 4:17 PM gjr80 <[email protected]> wrote: >>>>>>> >>>>>>>> Suggesting use of $hour_delta=1 wasn’t wise on my part as data >>>>>>>> arriving once per 60 minutes will likely lead to some occasions where >>>>>>>> no >>>>>>>> data is displayed depending on exactly when the data arrives. >>>>>>>> Conceivably >>>>>>>> data arriving or being processed a few seconds late could fall outside >>>>>>>> of >>>>>>>> $hour_delta=1. >>>>>>>> >>>>>>>> Actually, thinking some more, you my be able to use the aggregate >>>>>>>> ‘last’ to get the last value in the aggregation period, so try >>>>>>>> $span($hour_delta=2).co2.last. I may be mistaken but I think the >>>>>>>> aggregates >>>>>>>> exclude the None values so this tag may give you the most recent >>>>>>>> non-None >>>>>>>> value in the last two hours. Refer to >>>>>>>> http://weewx.com/docs/customizing.htm#Tag_$span and >>>>>>>> http://weewx.com/docs/customizing.htm#aggregation_types. >>>>>>>> >>>>>>>> Failing that you may like to try $span($time_delta=3900).co2.avg or >>>>>>>> 4200 to display the average over the last 70 or 80 minutes. Be >>>>>>>> aware though that just as $hour_delta=1 missed some records going the >>>>>>>> other >>>>>>>> way to > 1 hour will likely see more than one record be included in >>>>>>>> some >>>>>>>> aggregates, probably not an issue but something to be aware of. >>>>>>>> >>>>>>>> Gary >>>>>>>> On Sunday, 5 December 2021 at 08:35:01 UTC+10 [email protected] >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I've tried a few suggestions. The sensor output time can be >>>>>>>>> changed to a shorter period but, at this point, I prefer to keep it >>>>>>>>> at 60 >>>>>>>>> minutes since the internal battery looks impossible to change easily. >>>>>>>>> At 60 >>>>>>>>> minutes it is supposed to last a little over 3 years. >>>>>>>>> >>>>>>>>> I added the suggestion " $span($hour_delta=1).co2.avg" which seems >>>>>>>>> to bring in data more often but there are still some periods of "N/A." >>>>>>>>> >>>>>>>>> I added " expires_after = none" but for now that doesn't seem to >>>>>>>>> do anything. >>>>>>>>> >>>>>>>>> The suggestion of writing data to a scratch file may be the only >>>>>>>>> way to go if I can't configure it another way. >>>>>>>>> >>>>>>>>> I have produced a daily graph on the main page >>>>>>>>> >>>>>>>>> [image: yafe6kzk.bmp] >>>>>>>>> >>>>>>>>> The sensor is in the house for now and the peaks seem to come from >>>>>>>>> using the oven and burners on the gas stove. >>>>>>>>> >>>>>>>>> Thanks for all the help. I'll give this a day or so to accumulate >>>>>>>>> some data. >>>>>>>>> On Saturday, December 4, 2021 at 4:46:26 PM UTC-5 >>>>>>>>> [email protected] wrote: >>>>>>>>> >>>>>>>>>> Bob, >>>>>>>>>> As noted earlier, now that you have a base MQTTSubscribe >>>>>>>>>> configuration up and running, you probably want to look at its >>>>>>>>>> 'archive >>>>>>>>>> record cache capability'. Details are here, >>>>>>>>>> >>>>>>>>>> https://github.com/bellrichm/WeeWX-MQTTSubscribe/wiki/Configuring-additional-options#expires_after >>>>>>>>>> >>>>>>>>>> I think that setting expires_after to none would be the simplest. >>>>>>>>>> So you would end up with something like this. >>>>>>>>>> [MQTTSubscribeService] >>>>>>>>>> [[topics]] >>>>>>>>>> [[[mytopic]]] >>>>>>>>>> ignore = true >>>>>>>>>> [[[[CO2_Value_PPM]]]] >>>>>>>>>> name = co2 >>>>>>>>>> ignore = false >>>>>>>>>> expires_after = none >>>>>>>>>> >>>>>>>>>> rich >>>>>>>>>> >>>>>>>>>> On Saturday, 4 December 2021 at 15:09:39 UTC-5 gjr80 wrote: >>>>>>>>>> >>>>>>>>>>> The other approach is to give WeeWX at least one co2 value per >>>>>>>>>>> archive period (it can be the same value) but that is a function of >>>>>>>>>>> your >>>>>>>>>>> data source and the driver being used and I don’t know the >>>>>>>>>>> limitations/capabilities of each. >>>>>>>>>>> >>>>>>>>>>> Gary >>>>>>>>>>> >>>>>>>>>>> On Sunday, 5 December 2021 at 05:56:10 UTC+10 gjr80 wrote: >>>>>>>>>>> >>>>>>>>>>>> The issue here is that the ‘co2 field’ receives data only once >>>>>>>>>>>> per hour, so based on a 15 minute archive period only one in four >>>>>>>>>>>> archive >>>>>>>>>>>> records will have co2 data. Changing the accumulator type and >>>>>>>>>>>> extraction >>>>>>>>>>>> policy will not change this. I would leave the co2 accumulator >>>>>>>>>>>> settings out >>>>>>>>>>>> of weewx.conf. I doubt whether $last will be much help either; >>>>>>>>>>>> $last >>>>>>>>>>>> displays data from the last known good record in the database as >>>>>>>>>>>> opposed to >>>>>>>>>>>> the current record used by $current. If there is no co2 data in >>>>>>>>>>>> most >>>>>>>>>>>> records then $last will provide a similar result. >>>>>>>>>>>> >>>>>>>>>>>> You may have better luck with some sort of aggregate over the >>>>>>>>>>>> past hour, say something like (untested) >>>>>>>>>>>> $span($hour_delta=1).co2.avg. >>>>>>>>>>>> The actual aggregate is probably meaningless since you only have >>>>>>>>>>>> one value >>>>>>>>>>>> per hour so min be mx will work just as well. >>>>>>>>>>>> >>>>>>>>>>>> Gary >>>>>>>>>>>> On Sunday, 5 December 2021 at 03:59:46 UTC+10 [email protected] >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Sorry, here is some info. This time, I re-inserted the >>>>>>>>>>>>> [Accumulator] paragraph at a different place in weewx.conf and it >>>>>>>>>>>>> restarted >>>>>>>>>>>>> without an exit. The web data is still showing "$last.co2" >>>>>>>>>>>>> It ran for a while and then stopped. Debug file attached. >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> weewx.conf >>>>>>>>>>>>> >>>>>>>>>>>>> # Options for 'MQTTSubscribeService' >>>>>>>>>>>>> [Accumulator] >>>>>>>>>>>>> [[co2]] >>>>>>>>>>>>> accumulator = firstlast >>>>>>>>>>>>> extractor = last >>>>>>>>>>>>> >>>>>>>>>>>>> [MQTTSubscribeService] >>>>>>>>>>>>> # This section is for the MQTTSubscribe service. >>>>>>>>>>>>> >>>>>>>>>>>>> # Turn the service on and off. >>>>>>>>>>>>> # Default is: true >>>>>>>>>>>>> # Only used by the service. >>>>>>>>>>>>> enable = true # false >>>>>>>>>>>>> ================================================== >>>>>>>>>>>>> index.html.tmpl >>>>>>>>>>>>> <tr class = "even"> >>>>>>>>>>>>> <td class="stats_label">105 Crawl >>>>>>>>>>>>> Temperature</td> >>>>>>>>>>>>> <td class="stats_data">$current.extraTemp1 / >>>>>>>>>>>>> $current.extraTemp1.degree_C</td> >>>>>>>>>>>>> </tr> >>>>>>>>>>>>> <tr class = "even"> >>>>>>>>>>>>> <td class="stats_label">CO2 level</td> >>>>>>>>>>>>> <td class="stats_data">$last.co2</td> >>>>>>>>>>>>> </tr> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> ================================================== >>>>>>>>>>>>> >>>>>>>>>>>>> Standard web page reads: (http://grattans.org/wx) >>>>>>>>>>>>> >>>>>>>>>>>>> 105 Crawl Temperature 72.0°F / 22.2°C >>>>>>>>>>>>> CO2 level $last.co2 >>>>>>>>>>>>> >>>>>>>>>>>>> On Saturday, December 4, 2021 at 12:07:09 PM UTC-5 vince wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> On Saturday, December 4, 2021 at 7:16:00 AM UTC-8 >>>>>>>>>>>>>> [email protected] wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I tried changing $current.co2 to $last.co2 but it only >>>>>>>>>>>>>>> printed "$last.co2" on the web page. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Adding the [Accumulator] section to weewx.conf only caused >>>>>>>>>>>>>>> an error and exit when I restarted weewx. I tried adding this >>>>>>>>>>>>>>> in several >>>>>>>>>>>>>>> places but none seems to let weewx restart without an exit >>>>>>>>>>>>>>> error. Is there >>>>>>>>>>>>>>> a special place to add this? Couldn't find anything in the doc >>>>>>>>>>>>>>> about >>>>>>>>>>>>>>> placement. Thanks >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> Usual answers apply. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Unless you post the error messages verbatim and/or the >>>>>>>>>>>>>> changes you made to the skin, we are not going to be able to >>>>>>>>>>>>>> help you. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "weewx-user" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/weewx-user/60583ac3-73f3-479c-91d0-34fcdf1c0865n%40googlegroups.com >>>>>>>> >>>>>>>> <https://groups.google.com/d/msgid/weewx-user/60583ac3-73f3-479c-91d0-34fcdf1c0865n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "weewx-user" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> >>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/weewx-user/57504a16-555a-4123-a5dd-ae7b39861befn%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/weewx-user/57504a16-555a-4123-a5dd-ae7b39861befn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/08dbeefe-d823-4dc5-83e3-b7a68d22624cn%40googlegroups.com.
