Hi Tom, Here you have my weectl debug output:
https://pastebin.com/bH6fJz5G I've been able to force the simulator to generate plenty of wrong observations for rain and rainRate by simply setting these two lines in lines number 131 of the simulator.py file: 'rain' : Observation(magnitude=110.0, average= 220.0, period=1.0, phase_lag= 0.0, start=start_ts), 'rainRate' : Observation(magnitude=110.0, average= 220.0, period=1.0, phase_lag= 0.0, start=start_ts), But only for LOOP, and they don't go into the database: ene 16 20:35:15 meteopi weewxd[166178]: WARNING weewx.qc: 2025-01-16 20:35:16 CET (1737056116) LOOP value 'rain' 323.92866511135657 outside limits (0.0, 0.3937007874015748) ene 16 20:35:15 meteopi weewxd[166178]: WARNING weewx.qc: 2025-01-16 20:35:16 CET (1737056116) LOOP value 'rainRate' 323.92866511135657 outside limits (0.0, 0.11811023609999999) ene 16 20:35:15 meteopi weewxd[166178]: INFO weewx.manager: Added record 2025-01-16 20:35:00 CET (1737056100) to database 'weewx' ene 16 20:35:16 meteopi weewxd[166178]: INFO weewx.manager: Added record 2025-01-16 20:35:00 CET (1737056100) to daily summary in 'weewx' ene 16 20:35:18 meteopi weewxd[166178]: WARNING weewx.qc: 2025-01-16 20:35:18 CET (1737056118) LOOP value 'rain' 323.7704250868808 outside limits (0.0, 0.3937007874015748) ene 16 20:35:18 meteopi weewxd[166178]: WARNING weewx.qc: 2025-01-16 20:35:18 CET (1737056118) LOOP value 'rainRate' 323.7704250868808 outside limits (0.0, 0.11811023609999999) MariaDB [weewx]> select dateTime, rain, rainRate from archive where dateTime >= 1737056100 order by dateTime limit 10; +------------+------+----------+ | dateTime | rain | rainRate | +------------+------+----------+ | 1737056100 | NULL | 0 | | 1737056400 | 0 | 0 | | 1737056700 | 0 | 0 | | 1737057000 | 0 | 0 | +------------+------+----------+ As you can see the rain value is correctly set to Null. In real working the StdQC errors come not only for LOOP but also for Archive, that is where I think the problem comes, but with the simulator I haven't been able to force those warnings. These are real warnings for both LOOP and Archive: ene 16 19:27:22 meteopi weewxd[146447]: WARNING weewx.qc: 2025-01-16 19:27:22 CET (1737052042) LOOP value 'rain' 0.693 outside limits (0.0, 0.3937007874015748) ene 16 19:27:27 meteopi weewxd[146447]: WARNING weewx.qc: 2025-01-16 19:27:27 CET (1737052047) LOOP value 'rain' 0.42899999999999994 outside limits (0.0, 0.3937007874015748) ene 16 19:30:18 meteopi weewxd[146447]: WARNING weewx.qc: 2025-01-16 19:30:00 CET (1737052200) Archive value 'rain' 6.578 outside limits (0.0, 0.3937007874015748) ene 16 19:30:18 meteopi weewxd[146447]: WARNING weewx.qc: 2025-01-16 19:30:00 CET (1737052200) Archive value 'rainRate' 50.06262295081965 outside limits (0.0, 0.11811023609999999) And those coming from Archive are the ones that put garbage into the DB: MariaDB [weewx]> select dateTime, rain, rainRate from archive where dateTime >= 1737052200 order by dateTime limit 10; +------------+------+--------------------+ | dateTime | rain | rainRate | +------------+------+--------------------+ | 1737052200 | NULL | 52.58000000000002 | | 1737052500 | 0 | 40.392 | | 1737052800 | 0 | 26.311999999999998 | | 1737053100 | 0 | 0 | | 1737053400 | 0 | 0 | | 1737053700 | 0 | 0 | | 1737054000 | 0 | 0 | | 1737054300 | 0 | 0 | | 1737054600 | 0 | 0 | | 1737054900 | 0 | 0 | +------------+------+--------------------+ I hope you can see something I'm not able to get. Thanks really once again. El jueves, 16 de enero de 2025 a las 0:46:43 UTC+1, Tom Keffer escribió: > I don't think you've posted your weewx.conf. It could prove useful. Make > sure you obfuscate any passwords first. Or, use the utility weectl debug. > > Yes, if this is a bug in WeeWX, it should be possible to reproduce it > using the simulator. This is what I used (diff below). It emits a bad value > 5% of the time, zero the rest of the time. > > =================================================================== > diff --git a/src/weewx/drivers/simulator.py > b/src/weewx/drivers/simulator.py > --- a/src/weewx/drivers/simulator.py (revision > 1566a768af0963a43bd1c7b174293c46226a7fb5) > +++ b/src/weewx/drivers/simulator.py (date 1736899735080) > @@ -249,6 +249,11 @@ > else: > self.packet_number = 0 > amt = 0 > + import random > + if random.randint(0, 100) > 95: > + amt = 10 > + else: > + amt = 0 > return amt > > > For StdQC, I used > > [[MinMax]] > rain = 0, 10, mm > rainRate = 0, 3, mm_per_hour > > -tk > > > > On Wed, Jan 15, 2025 at 2:14 PM Francis Perea <[email protected]> > wrote: > >> Hi Tom, >> >> I really appreciate all your efforts. >> >> I don't really know how I could send you that simulator configuration. >> What would you need? My weewx.conf? >> >> I have discovered that it is not really a fault of my sensor but some >> interferences from a neighbour ham radio station. Talking with him he made >> me a comment and later we have confirmed that every time he transmit in a >> concrete frequency my station get wrong observations. At least I know now >> where the faults come from. And more than before I think that a correct >> software QC is the better way to keep wrong values to get into de DB >> because it seems more complicate to me to avoid the interferences. >> >> I would really need to know how to prevent those values from getting into >> the database and still think that a correct StdQC service configuration >> would do the trick but I don't know how. >> >> Would you please suggest any other configuration or tests I could develop >> to try to trim my config? >> >> Thanks once again for all your time and interest. >> >> Regards, >> >> Francis >> >> El miércoles, 15 de enero de 2025 a las 1:52:52 UTC+1, Tom Keffer >> escribió: >> >>> I am sorry, Francis, but I am unable to reproduce this. >>> >>> I modified a copy of the simulator to emit an out-of-range value for >>> "rain" randomly about 5% of the time. After letting it run for 40 minutes, >>> I looked at the database. There was nothing but zeros in both the archive >>> table and in the daily summaries: >>> >>> sqlite> select datetime(dateTime, 'unixepoch', 'localtime'), rain, >>> rainRate from archive; >>> >>> 2025-01-14 16:10:00|0.0|0.0 >>> >>> 2025-01-14 16:15:00|0.0|0.0 >>> >>> 2025-01-14 16:20:00|0.0|0.0 >>> >>> 2025-01-14 16:25:00|0.0|0.0 >>> >>> 2025-01-14 16:30:00|0.0|0.0 >>> >>> 2025-01-14 16:35:00|0.0|0.0 >>> >>> 2025-01-14 16:40:00|0.0|0.0 >>> >>> 2025-01-14 16:45:00|0.0|0.0 >>> >>> sqlite> select datetime(dateTime, 'unixepoch', 'localtime'), sum, max >>> from archive_day_rain; >>> >>> 2025-01-14 00:00:00|0.0|0.0 >>> >>> sqlite> select datetime(dateTime, 'unixepoch', 'localtime'), sum, max >>> from archive_day_rainRate; >>> >>> 2025-01-14 00:00:00|0.0|0.0 >>> >>> There is something peculiar about your setup, but I don't know what it >>> is. If you can send me a configuration that reproduces the behavior by >>> using the simulator, I'm happy to take another look. >>> >>> -tk >>> >>> On Mon, Jan 13, 2025 at 8:17 AM Francis Perea <[email protected]> >>> wrote: >>> >>>> Hi again Tom, >>>> >>>> Definitely not enough with my new StdQC configuration. I still get >>>> wrong observations into my DB. >>>> >>>> Just now I have this in my weewx.conf: >>>> >>>> [StdQC] >>>> >>>> [[MinMax]] >>>> rain = 0, 10, mm >>>> rainRate = 0, 3, mm_per_hour >>>> >>>> And I still get messages like these: >>>> ene 13 16:10:17 meteopi weewxd[193488]: WARNING weewx.qc: 2025-01-13 >>>> 16:10:00 CET (1736781000) Archive value 'rainRate' 4.046533333333339 >>>> outside limits (0.0, 0.11811023609999999) >>>> >>>> ene 13 17:00:17 meteopi weewxd[193488]: WARNING weewx.qc: 2025-01-13 >>>> 17:00:00 CET (1736784000) Archive value 'rain' 0.539 outside limits (0.0, >>>> 0.3937007874015748) >>>> ene 13 17:00:17 meteopi weewxd[193488]: WARNING weewx.qc: Discarding >>>> observation >>>> ene 13 17:00:17 meteopi weewxd[193488]: WARNING weewx.qc: 2025-01-13 >>>> 17:00:00 CET (1736784000) Archive value 'rainRate' 4.399999999999997 >>>> outside limits (0.0, 0.11811023609999999) >>>> ene 13 17:00:17 meteopi weewxd[193488]: WARNING weewx.qc: Discarding >>>> observation >>>> >>>> Which generate these values into my DB: >>>> >>>> dateTime rain >>>> rainRate >>>> 1736781000 0.09899999999999998 3.079999999999999 >>>> 1736784000 NULL 4.3119999999999985 >>>> >>>> It seems that the rainRate filter is not correct or not filtering. >>>> >>>> Any clue? >>>> >>>> Thanks once again for your help. >>>> >>>> >>>> El domingo, 12 de enero de 2025 a las 18:51:15 UTC+1, Francis Perea >>>> escribió: >>>> >>>>> Hi Tom, >>>>> >>>>> I get what you mean. I have been making all kind of tests to avoid >>>>> wrong observations to get into the data base and I even think I began >>>>> setting min-max values for rainRate, but I'll try again. >>>>> >>>>> What I get from your excellent explanation is that I should try to >>>>> filter rainRate, not rain, and see if that keeps "pollution" out of the >>>>> DB, >>>>> isn't it? >>>>> >>>>> I think I'll use this: >>>>> >>>>> rainRate = 0, 3 >>>>> >>>>> Do you think it would be enough? Or would you suggest me any other way >>>>> of filtering? Do you think I should also try to modify the rain_period >>>>> parameter from its default 1800 seconds? >>>>> >>>>> Regarding the [[RainRate]] section you see in my config, I've to be >>>>> honest and say that because of my desperation I even asked ChatGPT how to >>>>> proceed and this is what he propossed. I have already removed it. >>>>> >>>>> Thanks again for your support and patience. >>>>> >>>>> >>>>> El sábado, 11 de enero de 2025 a las 23:56:00 UTC+1, Tom Keffer >>>>> escribió: >>>>> >>>>>> It looks like the field "rain" was nulled out (otherwise, it would be >>>>>> 0.0, not NULL), so no bug. >>>>>> >>>>>> The field rainRate is not null, *however* the fields "rain" and >>>>>> "rainRate" are completely different. The former represents the amount of >>>>>> rain that fell, the latter how fast it fell. The field rainRate may be >>>>>> emitted from your device, in which case, by default, WeeWX uses that >>>>>> value. >>>>>> I suspect your byows hardware does not do this, so WeeWX will calculate >>>>>> it >>>>>> using a *running average* over 15 minutes. So, if you get a bad rain >>>>>> value, it will continue to "pollute" the rainRate value for up to 15 >>>>>> minutes afterwards. You need to filter rainRate as well, or set its >>>>>> running >>>>>> average window, given by option rain_period >>>>>> <https://www.weewx.com/docs/5.1/reference/weewx-options/stdwxcalculate/#rainrater>, >>>>>> >>>>>> to something shorter. >>>>>> >>>>>> [StdQC] >>>>>>> >>>>>> >>>>>>> >>>>>>> [[RainRate]] >>>>>>> min = 0 >>>>>>> max = 3 >>>>>>> min_delta = 0 >>>>>>> max_delta = 3 >>>>>>> reject = True >>>>>>> >>>>>> >>>>>> I don't know what any of this is. Did you write an addition to StdQC? >>>>>> If so, you're going to have to debug it! >>>>>> >>>>>> -tk >>>>>> >>>>> -- >>>> >>> You received this message because you are subscribed to the Google >>>> Groups "weewx-development" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> >>> To view this discussion visit >>>> https://groups.google.com/d/msgid/weewx-development/91ec157e-5f06-406b-8c16-f55fb40cd694n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/weewx-development/91ec157e-5f06-406b-8c16-f55fb40cd694n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "weewx-development" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion visit >> https://groups.google.com/d/msgid/weewx-development/19706d14-3e4d-466a-9455-f438d2d4b7d2n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/weewx-development/19706d14-3e4d-466a-9455-f438d2d4b7d2n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "weewx-development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/weewx-development/05d7e04d-0230-4161-ae77-0ed067188b5cn%40googlegroups.com.
