I am guessing that you are getting loop packets with just distance because of the order the data is arriving over MQTT.
For example: message with distance arrives - data is saved another message with distance arrives - packet with saved data is created - current message is saved message with count arrives - count is saved a message with either distance or count arrives - packet with saved data is created - data received is saved A debug level log would confirm this. If a ‘corrections’ expression won’t fix this, I will need a debug level log to get an understanding of the data being received. If you are using rtl_433 to publish to MQTT, I’d also be interested in what is being published to the ‘events’ topic. rich On Thursday, 12 August 2021 at 16:45:35 UTC-4 [email protected] wrote: > Not quite bingo, I am still getting some fixed lightning_distance integers > in the absence of strikes, about 6 per hour. Loops are produced with > lightning_distance = 7.999 and no information on lightning_strike_count > and other loops have with both lightning_distance = 7.999 and > lightning_strike_count = 0 See below output from weewxd for 2 hours. > > I changed [[Corrections]] to > lightning_distance = None if lightning_strike_count == 0 or None else > lightning_distance > > thinking that this correction would set lightning_distance to none in the > absence of a lightning strike count, but the correction had no effect. > Looking at all the loops there is no reason some loops have > lightning_distance: 7.99 while the great majority have the desired > lightning_distance: None. > > I got a great improvement moving to MQTTSubscribeDriver 2.1.0rc but the > problem persists to a small degree. The problem is > happening sporadically and rarely. Any thoughts? I really don't want to > give up MQTTSubscribeDriver since it is so versatile in what data you can > bring into weewx. > > > pi@raspberrypi:~ $ grep "lightning_distance: 7" weewxout1 > LOOP: 2021-08-12 13:58:54 EDT (1628791134) active: 1.0, cloudbase: > 4673.53199141, dateTime: 1628791134.4, dewpoint: 68.5748592378, heatindex: > 84.58478629, humidex: 95.6055479842, lightning_distance: 7.99999999694, > lightning_strike_count: None, maxSolarRad: None, outHumidity: 65.0, > outTemp: 81.5, rainRate: 0, usUnits: 1 > REC: 2021-08-12 14:00:00 EDT (1628791200) active: 0.4, cloudbase: > 4592.04983163, dateTime: 1628791200, dewpoint: 68.8533815408, ET: None, > heatindex: 84.5975618398, humidex: 95.7627385325, interval: 5, > lightning_distance: 7.99999999694, lightning_strike_count: 0.0, > maxSolarRad: None, outHumidity: 65.8, outTemp: 81.4200008, rain: 0.04, > rainRate: 0.0421052631579, usUnits: 1 > LOOP: 2021-08-12 14:42:54 EDT (1628793774) active: 1.0, dateTime: > 1628793774.25, lightning_distance: 7.99999999694, maxSolarRad: None, > rainRate: 0, usUnits: 1 > REC: 2021-08-12 14:45:00 EDT (1628793900) active: 1.0, altimeter: > 30.3586782548, barometer: 30.2545352733, cloudbase: 4768.17055948, > dateTime: 1628793900, dewpoint: 70.2239033928, ET: None, heatindex: > 87.9424503611, humidex: 99.1008122419, interval: 5, lightning_distance: > 7.99999999694, lightning_strike_count: 0.0, maxSolarRad: None, outHumidity: > 64.3243243243, outTemp: 83.5654538545, pressure: 28.5043137197, rain: 0.0, > rainRate: 0.0, usUnits: 1 > LOOP: 2021-08-12 14:58:06 EDT (1628794686) active: 1.0, dateTime: > 1628794686.48, lightning_distance: 7.99999999694, maxSolarRad: None, > rainRate: 0, usUnits: 1 > LOOP: 2021-08-12 14:58:30 EDT (1628794710) active: 1.0, dateTime: > 1628794710.6, lightning_distance: 7.99999999694, maxSolarRad: None, > rainRate: 0, usUnits: 1 > REC: 2021-08-12 15:00:00 EDT (1628794800) active: 0.911764705882, > altimeter: 30.352863321, barometer: 30.248274916, cloudbase: 4781.8918872, > dateTime: 1628794800, dewpoint: 71.158468132, ET: None, heatindex: > 89.7957954486, humidex: 100.932534378, interval: 5, lightning_distance: > 7.99999999694, lightning_strike_count: 0.0, maxSolarRad: None, outHumidity: > 64.3235294118, outTemp: 84.5603924356, pressure: 28.4987883408, rain: 0.0, > rainRate: 0.0, usUnits: 1 > LOOP: 2021-08-12 15:50:46 EDT (1628797846) active: 1.0, dateTime: > 1628797846.38, lightning_distance: 7.99999999694, maxSolarRad: None, > rainRate: 0, usUnits: 1 > REC: 2021-08-12 15:55:00 EDT (1628798100) active: 1.0, altimeter: > 30.3401020979, barometer: 30.2347797117, cloudbase: 4909.75354319, > dateTime: 1628798100, dewpoint: 71.8556696944, ET: None, heatindex: > 91.9474302989, humidex: 102.838512165, interval: 5, lightning_distance: > 7.99999999694, lightning_strike_count: 0.0, maxSolarRad: None, outHumidity: > 63.2432432432, outTemp: 85.8181836364, pressure: 28.4866625997, rain: 0.0, > rainRate: 0.0, usUnits: 1 > > On Wednesday, August 11, 2021 at 3:12:22 PM UTC-4 Gene LeSage wrote: > >> Bingo, fixed by installing v2.1.0-RC MQTTSubscribeDriver. Adding >> collect_observations = True and single_queue = True to [[topics], I get >> desired behavior. Null lightning distance is recorded in absence of >> strikes and with an test simulated lightning strike by battery charger >> shorting, I get lightning strike recording and a lightning distance. Loop >> shows both lightning distance and strikes in same loop: >> >> LOOP: 2021-08-11 14:56:07 EDT (1628708167) active: 1.0, cloudbase: >> 5636.94172822, dateTime: 1628708167.42, dewpoint: 70.6358563958, heatindex: >> 93.4196227576, humidex: 103.698544022, lightning_distance: None, >> lightning_strike_count: 0, maxSolarRad: None, outHumidity: 57.0, outTemp: >> 87.8, rainRate: 0, usUnits: 1 >> LOOP: 2021-08-11 14:56:07 EDT (1628708167) active: 1.0, cloudbase: >> 5636.94172822, dateTime: 1628708167.43, dewpoint: 70.6358563958, heatindex: >> 93.4196227576, humidex: 103.698544022, lightning_distance: None, >> lightning_strike_count: 0, maxSolarRad: None, outHumidity: 57.0, outTemp: >> 87.8, rainRate: 0, usUnits: 1 >> >> Supporting the idea that lightning strikes and distances were being >> interpreted at different times I caught two lightning strikes this morning >> (before installing v2.1.0-RC) which shows the 8 am lightning strike before >> lightning distance change and the 11 am lightning strike after lightning >> distance change. By not being synchronized, the correction >> (lightning_distance = lightning_distance if lightning_strike_count > 0 else >> None) would never work. >> >> Thanks >> >> >> >> >> >> [image: Screenshot (14).png] >> >> On Wednesday, August 11, 2021 at 11:41:31 AM UTC-4 [email protected] >> wrote: >> >>> I fear the root cause is that MQTTSubscribeDriver takes each MQTT >>> message and creates a loop packet. This works ok for 'aggregated' messages >>> like json or keyword (the original implementations). But for 'individual' >>> payloads that are dependent on each other, it is problematic.I've already >>> had to deal with this for wind data. MQTTSubscribe collects speed and >>> direction data into a single loop packet. >>> >>> Since then I've been experimenting with collecting data and creating the >>> loop packet from the collection. Looking at your loop packets, I think it >>> might help you. >>> >>> The code is a pre-release which can be found here, >>> https://github.com/bellrichm/WeeWX-MQTTSubscribe/releases/tag/v2.1.0-rc02 >>> >>> You will need to add two undocumented configuration options. >>> >>> [MQTTSubscribeDriver] >>> . >>> . >>> . >>> >>> [[topics] >>> # With the exception of wind data, by default a packet is >>> created for every MQTT message received. >>> # When this is true, MQTTSubscribe attempts to collect >>> observations across messages into a packet. >>> # Default is False. >>> # This is experimental and may be removed. >>> collect_observations = True >>> >>> # With the exception of wind data, by default a queue is created >>> for every MQTT topic. >>> # When this is true, MQTTSubsribe uses a single queue for all >>> non wind data. >>> # This is useful when 'collect_observations = True'. >>> # Default is False. >>> # This is experimental and may be removed. >>> single_queue = True >>> >>> . >>> . >>> . >>> >>> With this, the incoming data should be collected and a loop packet only >>> created when the arriving observation is already in the collection. Since, >>> hopefully, the loop packet now has both strike and distance data, your >>> [[Corrections]] should work. >>> >>> If this doesn't work. I'll have to go back to the drawing board and look >>> for a different solution. If it comes to this, I'd need a debug level log >>> for at least one archive period so that I can analyze the data. >>> >>> This is beta code, so the usual caveats apply. >>> rich >>> >>> ps. >>> If you are using rtl_433 to publish to MQTT, another option might be to >>> use the 'events' topic that it publishes to. This is a json formatted >>> message so the loop packet created for each MQTT message should have both >>> the strike and distance data. >>> >>> On Wednesday, 11 August 2021 at 09:52:39 UTC-4 [email protected] wrote: >>> >>>> I am using weewx-MQTTSubscribe to bring data into Weewx from sensors, >>>> so i don't use SDR sensor map, see config for mqttsubscribe here >>>> https://github.com/bellrichm/WeeWX-MQTTSubscribe/wiki/Configuring A >>>> number of unsuccessful tries to modify mqttsubscribe are commented out. >>>> [MQTTSubscribeDriver] >>>> [[topics]] >>>> [[[rtl_433/9b13b3f4-rtl433/devices/Acurite-6045M/C/223/strike_count]]] >>>> >>>> [[[[rtl_433/9b13b3f4-rtl433/devices/Acurite-6045M/C/223/strike_count]]]] >>>> name = lightning_strike_count #strikes_total >>>> #strike_count >>>> contains_total = true >>>> # conversion_type = int >>>> # units = count >>>> # expires_after = 0 >>>> >>>> [[[rtl_433/9b13b3f4-rtl433/devices/Acurite-6045M/C/223/storm_dist]]] >>>> >>>> [[[[rtl_433/9b13b3f4-rtl433/devices/Acurite-6045M/C/223/storm_dist]]]] >>>> name = lightning_distance #strike_dist >>>> #LS_1_distance >>>> # units = mile >>>> # conversion_type = int >>>> # expires_after = 0 >>>> >>>> >>>> >>>> [[Corrections]] >>>> # For each type, an arbitrary calibration expression can be >>>> given. >>>> # It should be in the units defined in the StdConvert section. >>>> # Example: >>>> lightning_distance = lightning_distance if >>>> lightning_strike_count > 0 else None >>>> >>>> [Accumulator] >>>> [[lightning_strike_count]] >>>> extractor = sum >>>> [[lightning_distance]] >>>> extractor = min >>>> # merger = minmax >>>> >>>> >>>> On Wednesday, August 11, 2021 at 7:58:38 AM UTC-4 [email protected] >>>> wrote: >>>> >>>>> I think going back to your original issue something seems to be off. >>>>> Can you post you weewx.conf for lightning data? If you're using sdr, >>>>> should >>>>> contain the following in each section: >>>>> >>>>> [SDR] >>>>> [[sensor_map]] >>>>> ...... >>>>> lightning_distance = distance.XXXX.AcuriteLightningPacket >>>>> strikes_total = strikes_total.XXXX.AcuriteLightningPacket >>>>> [[deltas]] >>>>> rain = rain_total >>>>> lightning_strike_count = strikes_total >>>>> >>>>> [StdCalibrate] >>>>> >>>>> [[Corrections]] >>>>> lightning_distance = lightning_distance if >>>>> lightning_strike_count > 0 else None >>>>> >>>>> [Accumulator] >>>>> [[lightning_strike_count]] >>>>> extractor = sum >>>>> [[lightning_distance]] >>>>> extractor = min >>>>> >>>>> On Tuesday, August 10, 2021 at 2:46:44 PM UTC-4 [email protected] >>>>> wrote: >>>>> >>>>>> Initially I thought Rich's solution was going to work, adding his >>>>>> suggested correction <lightning_distance = lightning_distance if >>>>>> 'lightning_strike_count'in locals() and lightning_strike_count> 0 else >>>>>> None> eliminated my fixed lightning_distance data in the absence of >>>>>> lightning_counts, but it created a new problem, now I don't get >>>>>> lightning_distance values with a lightning strike. I triggered the >>>>>> lightning detector with a spark by transiently shorting a battery >>>>>> charger >>>>>> at time 14:22:16 , which produced 2 detected lightnings and 14:22:24 >>>>>> which >>>>>> produced one, but the lightning_distance remains at None. My mqtt >>>>>> explorer >>>>>> reports the lightning detector is sending a constant 12-mile lightning >>>>>> distant >>>>>> >>>>>> LOOP: 2021-08-10 14:22:16 EDT (1628619736) dateTime: 1628619736.82, >>>>>> lightning_distance: None, maxSolarRad: None, outHumidity: 73.0, >>>>>> rainRate: >>>>>> 0.32, usUnits: 1 >>>>>> LOOP: 2021-08-10 14:22:16 EDT (1628619736) dateTime: 1628619736.81, >>>>>> lightning_strike_count: 2.0, maxSolarRad: None, rainRate: 0.32, usUnits: >>>>>> 1 >>>>>> <-------------------------------------------------------- >>>>>> LOOP: 2021-08-10 14:22:16 EDT (1628619736) dateTime: 1628619736.82, >>>>>> lightning_distance: None, lightning_strike_count: 0.0, maxSolarRad: >>>>>> None, >>>>>> rainRate: 0.32, usUnits: 1 >>>>>> LOOP: 2021-08-10 14:22:16 EDT (1628619736) dateTime: 1628619736.83, >>>>>> lightning_distance: None, lightning_strike_count: 0.0, maxSolarRad: >>>>>> None, >>>>>> rainRate: 0.32, usUnits: 1 >>>>>> LOOP: 2021-08-10 14:22:16 EDT (1628619736) dateTime: 1628619736.81, >>>>>> lightning_distance: None, maxSolarRad: None, rainRate: 0.32, usUnits: 1 >>>>>> LOOP: 2021-08-10 14:22:16 EDT (1628619736) dateTime: 1628619736.82, >>>>>> lightning_distance: None, maxSolarRad: None, rainRate: 0.32, usUnits: 1 >>>>>> LOOP: 2021-08-10 14:22:16 EDT (1628619736) dateTime: 1628619736.83, >>>>>> lightning_distance: None, maxSolarRad: None, rainRate: 0.32, usUnits: 1 >>>>>> LOOP: 2021-08-10 14:22:24 EDT (1628619744) dateTime: 1628619744.67, >>>>>> lightning_distance: None, maxSolarRad: None, outTemp: 82.000004, >>>>>> rainRate: >>>>>> 0.32, usUnits: 1 >>>>>> LOOP: 2021-08-10 14:22:24 EDT (1628619744) dateTime: 1628619744.68, >>>>>> lightning_distance: None, maxSolarRad: None, outTemp: 82.000004, >>>>>> rainRate: >>>>>> 0.32, usUnits: 1 >>>>>> LOOP: 2021-08-10 14:22:24 EDT (1628619744) dateTime: 1628619744.69, >>>>>> lightning_distance: None, maxSolarRad: None, outTemp: 82.000004, >>>>>> rainRate: >>>>>> 0.32, usUnits: 1 >>>>>> LOOP: 2021-08-10 14:22:24 EDT (1628619744) dateTime: 1628619744.67, >>>>>> lightning_distance: None, maxSolarRad: None, outHumidity: 73.0, >>>>>> rainRate: >>>>>> 0.32, usUnits: 1 >>>>>> LOOP: 2021-08-10 14:22:24 EDT (1628619744) dateTime: 1628619744.68, >>>>>> lightning_distance: None, maxSolarRad: None, outHumidity: 73.0, >>>>>> rainRate: >>>>>> 0.32, usUnits: 1 >>>>>> LOOP: 2021-08-10 14:22:24 EDT (1628619744) dateTime: 1628619744.69, >>>>>> lightning_distance: None, maxSolarRad: None, outHumidity: 73.0, >>>>>> rainRate: >>>>>> 0.32, usUnits: 1 >>>>>> LOOP: 2021-08-10 14:22:24 EDT (1628619744) dateTime: 1628619744.68, >>>>>> lightning_strike_count: 1.0, maxSolarRad: None, rainRate: 0.32, usUnits: >>>>>> 1 >>>>>> <----------------------------------------------------------- >>>>>> LOOP: 2021-08-10 14:22:24 EDT (1628619744) dateTime: 1628619744.69, >>>>>> lightning_distance: None, lightning_strike_count: 0.0, maxSolarRad: >>>>>> None, >>>>>> rainRate: 0.32, usUnits: 1 >>>>>> LOOP: 2021-08-10 14:22:24 EDT (1628619744) dateTime: 1628619744.69, >>>>>> lightning_distance: None, lightning_strike_count: 0.0, maxSolarRad: >>>>>> None, >>>>>> rainRate: 0.32, usUnits: 1 >>>>>> >>>>>> >>>>>> On Tuesday, August 10, 2021 at 9:20:17 AM UTC-4 [email protected] >>>>>> wrote: >>>>>> >>>>>>> I'm no python expert (probably know just enough to be dangerous), >>>>>>> but something like this might get you want you want. >>>>>>> >>>>>>> # ligtning_strike_count must exist and have a count > 0 for >>>>>>> lightning_distance to have a valid value >>>>>>> lightning_distance = lightning_distance if >>>>>>> 'lightning_strike_count'in locals() and lightning_strike_count> 0 else >>>>>>> None >>>>>>> >>>>>>> rich >>>>>>> On Tuesday, 10 August 2021 at 03:03:01 UTC-4 gjr80 wrote: >>>>>>> >>>>>>>> On Tuesday, 10 August 2021 at 08:37:42 UTC+10 [email protected] >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Here is my output running weewxd directly, I see three lines which >>>>>>>>> show lightning_distance to be None, which is expected from >>>>>>>>> corrections, but >>>>>>>>> the following three lines show lightning_distance to be 11.999... It >>>>>>>>> appears the correction " lightning_distance = lightning_distance if >>>>>>>>> lightning_strike_count > 0 else None" is working but being ignored in >>>>>>>>> final >>>>>>>>> output to graph and database. Any additional thoughts? >>>>>>>>> >>>>>>>> >>>>>>>> The reason you see no change to lightning_distance for some >>>>>>>> packets is that for those packets lightning_strike_count does not >>>>>>>> exist so your calibration expression fails and no change is made, in >>>>>>>> other >>>>>>>> words lightning_distance remains as it was 11.9999999954. Why does >>>>>>>> this happen, in layman's terms when lightning_strike_count is not >>>>>>>> in the loop packet the calibration expression in effect has an unknown >>>>>>>> variable (lightning_strike_count) and the calibration expression >>>>>>>> raises an error. The StdCalibrate service which handles the >>>>>>>> calibration expressions catches that error and discards that >>>>>>>> calibration >>>>>>>> expression. Given the limitations of the StdCalibrate service I am >>>>>>>> not aware of any calibration expression that would do as you want, of >>>>>>>> course a WeeWX and python wizard might come along an prove me wrong! >>>>>>>> >>>>>>>> My thoughts on a solution, unless there is a simple solution in >>>>>>>> your WeeWX driver or elsewhere upstream of WeeWX I would be writing >>>>>>>> your >>>>>>>> own WeeWX service to look at the packet and make the necessary >>>>>>>> correction. >>>>>>>> This way you can use a bit of python code to do exactly what you want >>>>>>>> and >>>>>>>> you won't be limited by the single line expressions as used by >>>>>>>> StdCalibrate. All up it should take no more than 10 lines of code. >>>>>>>> The service could be a data_service or process_service (refer >>>>>>>> weewx.conf [Engine] [[Services]]) but would need to appear before >>>>>>>> StdArchive is called. The advantage of your own service is that >>>>>>>> there is no need to change any one else's code (eg WeeWX driver, >>>>>>>> upstream >>>>>>>> code) so you will not have your changes lost during an upgrade (WeeWX >>>>>>>> or >>>>>>>> otherwise). >>>>>>>> >>>>>>>> Gary >>>>>>>> >>>>>>> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/518ae01c-516b-4be7-9fb5-83ced7864d69n%40googlegroups.com.
