On the topics of negative numbers, I have also realized that in cases where I shutdown weewx running on Pi, unplug the USB cable running to the WMR200, plug it back and restart the weewx, sometimes I see -ve value for rain getting populated in the database which I have to manually remove. Looks like WMR200 is certainly an interesting WS. Is there a way to guard against such erroneous records ?
Thanks. - Rahul On Sat, Sep 15, 2018 at 6:20 PM Thomas Keffer <[email protected]> wrote: > Implemented in commit ea9fe13 > <https://github.com/weewx/weewx/commit/ea9fe134a242ddd0ea1df9e1c4aa4ddea1542ff6> > > -tk > > On Sat, Sep 15, 2018 at 5:38 AM Thomas Keffer <[email protected]> wrote: > >> Thanks, Chris >> >> I worry that this approach doesn't cover the case where the "previous" >> record has, say, a timestamp far in the future. This patch would cause all >> of the remaining records to get thrown out, but the "future" record to be >> kept. >> >> In any case, it's way better than what we have, which allows negative >> intervals. >> >> -tk >> >> >> >> On Fri, Sep 14, 2018 at 3:53 PM Chris Manton <[email protected]> wrote: >> >>> I suppose if the archived packets are presented out of order there could >>> be a negative interval... >>> >>> I have not seen this but the wmr200 is full of wonderful surprises. >>> >>> One potential drive-by fix is to drop all packets satisfying this >>> failing condition...here is one untested fix: >>> >>> >>> $ git diff >>> diff --git a/bin/weewx/drivers/wmr200.py b/bin/weewx/drivers/wmr200.py >>> index eaab5a14..62b743e0 100644 >>> --- a/bin/weewx/drivers/wmr200.py >>> +++ b/bin/weewx/drivers/wmr200.py >>> @@ -1943,7 +1943,13 @@ class WMR200(weewx.drivers.AbstractDevice): >>> timestamp_packet_interval = timestamp_packet_current \ >>> - timestamp_packet_previous >>> >>> - if pkt.timestamp_record() > (timestamp_packet_previous >>> + if timestamp_packet_interval < 0: >>> + loginf(('genStartup() Discarding received archive' >>> + ' record that presented out-of-order' >>> + ' current_threshold:%s previous_timestamp:%s') >>> + % >>> (weeutil.weeutil.timestamp_to_string(pkt.timestamp_record()) >>> + >>> weeutil.weeutil.timestamp_to_string(timestamp_packet_previous))) >>> + elif pkt.timestamp_record() > (timestamp_packet_previous >>> + self._archive_threshold): >>> loginf(('genStartup() Discarding received archive' >>> ' record exceeding archive interval cnt:%d' >>> >>> On Friday, September 14, 2018 at 6:07:48 AM UTC-7, Thomas Keffer wrote: >>>> >>>> Good. >>>> >>>> Looking through the WMR200 driver code (I didn't write it), it appears >>>> that a positive value for 'interval' is not guaranteed. I'll ask the author >>>> to take a look at it. >>>> >>>> -tk >>>> >>>> On Fri, Sep 14, 2018 at 5:44 AM Rahul Bedi <[email protected]> wrote: >>>> >>>>> Yup found a record with -ve interval, changed that to 5 as thats what >>>>> I am using. >>>>> >>>>> And problem fixed !! >>>>> >>>>> Thanks a lot. >>>>> >>>>> Not sure how this bad data entered into the db. >>>>> >>>>> Will be great if there is utility that finds such errors in the db and >>>>> flags them. >>>>> >>>>> - Rahul >>>>> >>>>> On Fri, Sep 14, 2018 at 5:48 PM Thomas Keffer <[email protected]> >>>>> wrote: >>>>> >>>>>> I think the thing to do is to figure out how a negative value got in >>>>>> there, fix that, then rebuild the daily summaries. >>>>>> >>>>>> One possibility is that a record in your database has a negative >>>>>> value for 'interval' --- the time length of an archive record. Let's >>>>>> check >>>>>> that. >>>>>> >>>>>> *sqlite3 /var/lib/weewx/weewx.sdb* >>>>>> sqlite> *SELECT datetime(dateTime, 'unixepoch','localtime'), >>>>>> interval FROM archive WHERE interval<0;* >>>>>> >>>>>> -tk >>>>>> >>>>>> On Fri, Sep 14, 2018 at 5:06 AM Rahul Bedi <[email protected]> wrote: >>>>>> >>>>>>> Here is what I get: >>>>>>> >>>>>>> 2018-08-02 00:00:00|2851473.0|86400 >>>>>>> 2018-08-03 00:00:00|2139447.0|86400 >>>>>>> 2018-08-04 00:00:00|2867211.0|86400 >>>>>>> 2018-08-05 00:00:00|3306195.0|86400 >>>>>>> 2018-08-06 00:00:00|3167394.0|86400 >>>>>>> 2018-08-07 00:00:00|2924283.0|86400 >>>>>>> 2018-08-08 00:00:00|2255208.0|86400 >>>>>>> 2018-08-09 00:00:00|2797065.0|86400 >>>>>>> 2018-08-10 00:00:00|2693187.0|86400 >>>>>>> 2018-08-11 00:00:00|2981601.0|86400 >>>>>>> 2018-08-12 00:00:00|3869136.0|86400 >>>>>>> 2018-08-13 00:00:00|3632787.0|86400 >>>>>>> 2018-08-14 00:00:00|2651829.0|86400 >>>>>>> 2018-08-15 00:00:00|2104410.0|86400 >>>>>>> 2018-08-16 00:00:00|3682674.0|86400 >>>>>>> 2018-08-17 00:00:00|3499992.0|86400 >>>>>>> 2018-08-18 00:00:00|3855531.0|86400 >>>>>>> 2018-08-19 00:00:00|3875121.0|86400 >>>>>>> 2018-08-20 00:00:00|3915861.0|86400 >>>>>>> 2018-08-21 00:00:00|3523719.0|86400 >>>>>>> 2018-08-22 00:00:00|1637909.95179334|85620 >>>>>>> 2018-08-23 00:00:00|506702.215048778|25380 >>>>>>> 2018-08-24 00:00:00|523285.18735593|101340 >>>>>>> 2018-08-25 00:00:00|2047231.90482286|92280 >>>>>>> 2018-08-26 00:00:00|2207680.59885907|85800 >>>>>>> 2018-08-27 00:00:00|2112653.43729687|78000 >>>>>>> 2018-08-28 00:00:00|1927695.38113441|85200 >>>>>>> 2018-08-29 00:00:00|1696867.80614767|76740 >>>>>>> 2018-08-30 00:00:00|1892572.6564943|85500 >>>>>>> 2018-08-31 00:00:00|1669207.56530998|86100 >>>>>>> 2018-09-01 00:00:00|1882876.90336366|85080 >>>>>>> 2018-09-02 00:00:00|1552040.97205686|86100 >>>>>>> 2018-09-03 00:00:00|1550916.37957731|86100 >>>>>>> 2018-09-04 00:00:00|1093704.34743737|85740 >>>>>>> 2018-09-05 00:00:00|1688380.4274756|86100 >>>>>>> 2018-09-06 00:00:00|2349485.94738889|86100 >>>>>>> 2018-09-07 00:00:00|2676307.36370007|85560 >>>>>>> 2018-09-08 00:00:00|2198034.86178985|86100 >>>>>>> 2018-09-09 00:00:00|-7107216.29856356|-302280 >>>>>>> 2018-09-10 00:00:00|288713.672694377|85560 >>>>>>> 2018-09-11 00:00:00|351094.880509644|85860 >>>>>>> 2018-09-12 00:00:00|441249.222795585|85500 >>>>>>> 2018-09-13 00:00:00|60033.1634844758|85620 >>>>>>> 2018-09-14 00:00:00|21222.2555021576|62820 >>>>>>> >>>>>>> I am guessing that the negative values above may be the culprit. How >>>>>>> are these populated in the archive_day_wind table or should I just edit >>>>>>> these values out ? >>>>>>> >>>>>>> - Rahul >>>>>>> >>>>>>> >>>>>>> On Fri, Sep 14, 2018 at 5:19 PM Thomas Keffer <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> You're on the right track: there must be an bad value in your >>>>>>>> database. Did you, by any chance, use a visual sql editor? They often >>>>>>>> put >>>>>>>> null strings in the database. >>>>>>>> >>>>>>>> In any case, the field 'wsquaresum' and 'sumtime' are used by rms. >>>>>>>> So, try this: >>>>>>>> >>>>>>>> *sqlite3 /var/lib/weewx/weewx.sdb* >>>>>>>> sqlite> *SELECT datetime(dateTime, 'unixepoch','localtime'), >>>>>>>> wsquaresum, sumtime FROM archive_day_wind WHERE dateTime > 1533081600;* >>>>>>>> >>>>>>>> This will show all values from (roughly) 1-August (about 45 >>>>>>>> values). If the bad value is before that, we'll have to use a different >>>>>>>> strategy, rather than go looking for a needle in a haystack. >>>>>>>> >>>>>>>> -tk >>>>>>>> >>>>>>>> On Thu, Sep 13, 2018 at 11:35 PM Rahul Bedi <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I found rms used in the statictics.ico and hilo.ico files in the >>>>>>>>> Seasons directory. removed those segments. Diff as follows: >>>>>>>>> >>>>>>>>> $ diff statistics.14092018.inc statistics.inc >>>>>>>>> >>>>>>>>> 134,140d133 >>>>>>>>> < <tr> >>>>>>>>> < <td class="label">$obs.label.wind_rms</td> >>>>>>>>> < <td class="units">$unit.label.wind</td> >>>>>>>>> < #for $archive in $archive_data >>>>>>>>> < <td class="data >>>>>>>>> new_row">$archive.wind.rms.formatted</td> >>>>>>>>> < #end for >>>>>>>>> < </tr> >>>>>>>>> >>>>>>>>> $ diff hilo.inc hilo.14092018.inc >>>>>>>>> >>>>>>>>> 132a133,140 >>>>>>>>> > <tr> >>>>>>>>> > <td class="label">$obs.label.wind_rms</td> >>>>>>>>> > #for $archive in $archive_data >>>>>>>>> > <td class="data new_row hilo_$archive[0]"> >>>>>>>>> > $archive[1].wind.rms.formatted</td> >>>>>>>>> > #end for >>>>>>>>> > <td class="units">$unit.label.wind</td> >>>>>>>>> > </tr> >>>>>>>>> >>>>>>>>> Apart from this any file referring to rms were exactly the same as >>>>>>>>> the original install. Attaching skin.conf with this message. >>>>>>>>> >>>>>>>>> As expected the problem is gone but rms is not getting reported. >>>>>>>>> Which means the issue still stays. >>>>>>>>> >>>>>>>>> Could you help me with what fields in the archive are referred to >>>>>>>>> by rms ? I can try to manually check for any value discrepancy. >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> >>>>>>>>> - Rahul >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Sep 14, 2018 at 9:14 AM Rahul Bedi <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Thanks Gary. >>>>>>>>>> >>>>>>>>>> Charges were done about a month ago but have seen no errors till >>>>>>>>>> this episode. I will do a diff with the original skin files and >>>>>>>>>> post it >>>>>>>>>> here. >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> >>>>>>>>>> - Rahul >>>>>>>>>> >>>>>>>>>> On Fri 14 Sep, 2018, 7:35 AM gjr80, <[email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> On Friday, 14 September 2018 11:30:42 UTC+10, Rahul Bedi wrote: >>>>>>>>>>>> >>>>>>>>>>>> As I have made no code or config changes since the day before, >>>>>>>>>>>> I believe it doing this for the right data. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> This sounds to me like you have changed some of the seasons skin >>>>>>>>>>> files. The error is coming form the use of a .rms tag in one of >>>>>>>>>>> the seasons .tmpl or .inc files. The seasons skin only uses the >>>>>>>>>>> .rms tag with the WeeWX wind field. As Tom indicated .rms only >>>>>>>>>>> works against the wind field, if your changes have used the .rms >>>>>>>>>>> tag against another field then this could be the cause. I suggest >>>>>>>>>>> you go >>>>>>>>>>> through all of your changes to the seasons skin files and see what >>>>>>>>>>> fields >>>>>>>>>>> are being used with the .rms tag. Until we know for sure, and >>>>>>>>>>> given you seem to have made changes to the skin, we will likely be >>>>>>>>>>> chasing >>>>>>>>>>> our tails. >>>>>>>>>>> >>>>>>>>>>> Gary >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> You received this message because you are subscribed to a topic >>>>>>>>>>> in the Google Groups "weewx-user" group. >>>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>>> https://groups.google.com/d/topic/weewx-user/LWmSpcVfdmo/unsubscribe >>>>>>>>>>> . >>>>>>>>>>> To unsubscribe from this group and all its topics, send an email >>>>>>>>>>> to [email protected]. >>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "weewx-user" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to a topic in >>>>>>>> the Google Groups "weewx-user" group. >>>>>>>> To unsubscribe from this topic, visit >>>>>>>> https://groups.google.com/d/topic/weewx-user/LWmSpcVfdmo/unsubscribe >>>>>>>> . >>>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>>> [email protected]. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "weewx-user" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to a topic in >>>>>> the Google Groups "weewx-user" group. >>>>>> To unsubscribe from this topic, visit >>>>>> https://groups.google.com/d/topic/weewx-user/LWmSpcVfdmo/unsubscribe. >>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>> [email protected]. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "weewx-user" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "weewx-user" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- > You received this message because you are subscribed to a topic in the > Google Groups "weewx-user" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/weewx-user/LWmSpcVfdmo/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
