What's a "-ve" value? -tk
On Sat, Sep 15, 2018 at 6:10 AM Rahul Bedi <[email protected]> wrote: > On the topics of negative numbers, I have also realized that in cases > where I shutdown weewx running on Pi, unplug the USB cable running to the > WMR200, plug it back and restart the weewx, sometimes I see -ve value for > rain getting populated in the database which I have to manually remove. > Looks like WMR200 is certainly an interesting WS. Is there a way to guard > against such erroneous records ? > > Thanks. > > - Rahul > > On Sat, Sep 15, 2018 at 6:20 PM Thomas Keffer <[email protected]> wrote: > >> Implemented in commit ea9fe13 >> <https://github.com/weewx/weewx/commit/ea9fe134a242ddd0ea1df9e1c4aa4ddea1542ff6> >> >> -tk >> >> On Sat, Sep 15, 2018 at 5:38 AM Thomas Keffer <[email protected]> wrote: >> >>> Thanks, Chris >>> >>> I worry that this approach doesn't cover the case where the "previous" >>> record has, say, a timestamp far in the future. This patch would cause all >>> of the remaining records to get thrown out, but the "future" record to be >>> kept. >>> >>> In any case, it's way better than what we have, which allows negative >>> intervals. >>> >>> -tk >>> >>> >>> >>> On Fri, Sep 14, 2018 at 3:53 PM Chris Manton <[email protected]> wrote: >>> >>>> I suppose if the archived packets are presented out of order there >>>> could be a negative interval... >>>> >>>> I have not seen this but the wmr200 is full of wonderful surprises. >>>> >>>> One potential drive-by fix is to drop all packets satisfying this >>>> failing condition...here is one untested fix: >>>> >>>> >>>> $ git diff >>>> diff --git a/bin/weewx/drivers/wmr200.py b/bin/weewx/drivers/wmr200.py >>>> index eaab5a14..62b743e0 100644 >>>> --- a/bin/weewx/drivers/wmr200.py >>>> +++ b/bin/weewx/drivers/wmr200.py >>>> @@ -1943,7 +1943,13 @@ class WMR200(weewx.drivers.AbstractDevice): >>>> timestamp_packet_interval = timestamp_packet_current \ >>>> - timestamp_packet_previous >>>> >>>> - if pkt.timestamp_record() > (timestamp_packet_previous >>>> + if timestamp_packet_interval < 0: >>>> + loginf(('genStartup() Discarding received archive' >>>> + ' record that presented out-of-order' >>>> + ' current_threshold:%s previous_timestamp:%s') >>>> + % >>>> (weeutil.weeutil.timestamp_to_string(pkt.timestamp_record()) >>>> + >>>> weeutil.weeutil.timestamp_to_string(timestamp_packet_previous))) >>>> + elif pkt.timestamp_record() > >>>> (timestamp_packet_previous >>>> + >>>> self._archive_threshold): >>>> loginf(('genStartup() Discarding received archive' >>>> ' record exceeding archive interval cnt:%d' >>>> >>>> On Friday, September 14, 2018 at 6:07:48 AM UTC-7, Thomas Keffer wrote: >>>>> >>>>> Good. >>>>> >>>>> Looking through the WMR200 driver code (I didn't write it), it appears >>>>> that a positive value for 'interval' is not guaranteed. I'll ask the >>>>> author >>>>> to take a look at it. >>>>> >>>>> -tk >>>>> >>>>> On Fri, Sep 14, 2018 at 5:44 AM Rahul Bedi <[email protected]> wrote: >>>>> >>>>>> Yup found a record with -ve interval, changed that to 5 as thats what >>>>>> I am using. >>>>>> >>>>>> And problem fixed !! >>>>>> >>>>>> Thanks a lot. >>>>>> >>>>>> Not sure how this bad data entered into the db. >>>>>> >>>>>> Will be great if there is utility that finds such errors in the db >>>>>> and flags them. >>>>>> >>>>>> - Rahul >>>>>> >>>>>> On Fri, Sep 14, 2018 at 5:48 PM Thomas Keffer <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> I think the thing to do is to figure out how a negative value got in >>>>>>> there, fix that, then rebuild the daily summaries. >>>>>>> >>>>>>> One possibility is that a record in your database has a negative >>>>>>> value for 'interval' --- the time length of an archive record. Let's >>>>>>> check >>>>>>> that. >>>>>>> >>>>>>> *sqlite3 /var/lib/weewx/weewx.sdb* >>>>>>> sqlite> *SELECT datetime(dateTime, 'unixepoch','localtime'), >>>>>>> interval FROM archive WHERE interval<0;* >>>>>>> >>>>>>> -tk >>>>>>> >>>>>>> On Fri, Sep 14, 2018 at 5:06 AM Rahul Bedi <[email protected]> wrote: >>>>>>> >>>>>>>> Here is what I get: >>>>>>>> >>>>>>>> 2018-08-02 00:00:00|2851473.0|86400 >>>>>>>> 2018-08-03 00:00:00|2139447.0|86400 >>>>>>>> 2018-08-04 00:00:00|2867211.0|86400 >>>>>>>> 2018-08-05 00:00:00|3306195.0|86400 >>>>>>>> 2018-08-06 00:00:00|3167394.0|86400 >>>>>>>> 2018-08-07 00:00:00|2924283.0|86400 >>>>>>>> 2018-08-08 00:00:00|2255208.0|86400 >>>>>>>> 2018-08-09 00:00:00|2797065.0|86400 >>>>>>>> 2018-08-10 00:00:00|2693187.0|86400 >>>>>>>> 2018-08-11 00:00:00|2981601.0|86400 >>>>>>>> 2018-08-12 00:00:00|3869136.0|86400 >>>>>>>> 2018-08-13 00:00:00|3632787.0|86400 >>>>>>>> 2018-08-14 00:00:00|2651829.0|86400 >>>>>>>> 2018-08-15 00:00:00|2104410.0|86400 >>>>>>>> 2018-08-16 00:00:00|3682674.0|86400 >>>>>>>> 2018-08-17 00:00:00|3499992.0|86400 >>>>>>>> 2018-08-18 00:00:00|3855531.0|86400 >>>>>>>> 2018-08-19 00:00:00|3875121.0|86400 >>>>>>>> 2018-08-20 00:00:00|3915861.0|86400 >>>>>>>> 2018-08-21 00:00:00|3523719.0|86400 >>>>>>>> 2018-08-22 00:00:00|1637909.95179334|85620 >>>>>>>> 2018-08-23 00:00:00|506702.215048778|25380 >>>>>>>> 2018-08-24 00:00:00|523285.18735593|101340 >>>>>>>> 2018-08-25 00:00:00|2047231.90482286|92280 >>>>>>>> 2018-08-26 00:00:00|2207680.59885907|85800 >>>>>>>> 2018-08-27 00:00:00|2112653.43729687|78000 >>>>>>>> 2018-08-28 00:00:00|1927695.38113441|85200 >>>>>>>> 2018-08-29 00:00:00|1696867.80614767|76740 >>>>>>>> 2018-08-30 00:00:00|1892572.6564943|85500 >>>>>>>> 2018-08-31 00:00:00|1669207.56530998|86100 >>>>>>>> 2018-09-01 00:00:00|1882876.90336366|85080 >>>>>>>> 2018-09-02 00:00:00|1552040.97205686|86100 >>>>>>>> 2018-09-03 00:00:00|1550916.37957731|86100 >>>>>>>> 2018-09-04 00:00:00|1093704.34743737|85740 >>>>>>>> 2018-09-05 00:00:00|1688380.4274756|86100 >>>>>>>> 2018-09-06 00:00:00|2349485.94738889|86100 >>>>>>>> 2018-09-07 00:00:00|2676307.36370007|85560 >>>>>>>> 2018-09-08 00:00:00|2198034.86178985|86100 >>>>>>>> 2018-09-09 00:00:00|-7107216.29856356|-302280 >>>>>>>> 2018-09-10 00:00:00|288713.672694377|85560 >>>>>>>> 2018-09-11 00:00:00|351094.880509644|85860 >>>>>>>> 2018-09-12 00:00:00|441249.222795585|85500 >>>>>>>> 2018-09-13 00:00:00|60033.1634844758|85620 >>>>>>>> 2018-09-14 00:00:00|21222.2555021576|62820 >>>>>>>> >>>>>>>> I am guessing that the negative values above may be the culprit. >>>>>>>> How are these populated in the archive_day_wind table or should I just >>>>>>>> edit >>>>>>>> these values out ? >>>>>>>> >>>>>>>> - Rahul >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Sep 14, 2018 at 5:19 PM Thomas Keffer <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> You're on the right track: there must be an bad value in your >>>>>>>>> database. Did you, by any chance, use a visual sql editor? They often >>>>>>>>> put >>>>>>>>> null strings in the database. >>>>>>>>> >>>>>>>>> In any case, the field 'wsquaresum' and 'sumtime' are used by >>>>>>>>> rms. So, try this: >>>>>>>>> >>>>>>>>> *sqlite3 /var/lib/weewx/weewx.sdb* >>>>>>>>> sqlite> *SELECT datetime(dateTime, 'unixepoch','localtime'), >>>>>>>>> wsquaresum, sumtime FROM archive_day_wind WHERE dateTime > >>>>>>>>> 1533081600;* >>>>>>>>> >>>>>>>>> This will show all values from (roughly) 1-August (about 45 >>>>>>>>> values). If the bad value is before that, we'll have to use a >>>>>>>>> different >>>>>>>>> strategy, rather than go looking for a needle in a haystack. >>>>>>>>> >>>>>>>>> -tk >>>>>>>>> >>>>>>>>> On Thu, Sep 13, 2018 at 11:35 PM Rahul Bedi <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> I found rms used in the statictics.ico and hilo.ico files in the >>>>>>>>>> Seasons directory. removed those segments. Diff as follows: >>>>>>>>>> >>>>>>>>>> $ diff statistics.14092018.inc statistics.inc >>>>>>>>>> >>>>>>>>>> 134,140d133 >>>>>>>>>> < <tr> >>>>>>>>>> < <td class="label">$obs.label.wind_rms</td> >>>>>>>>>> < <td class="units">$unit.label.wind</td> >>>>>>>>>> < #for $archive in $archive_data >>>>>>>>>> < <td class="data >>>>>>>>>> new_row">$archive.wind.rms.formatted</td> >>>>>>>>>> < #end for >>>>>>>>>> < </tr> >>>>>>>>>> >>>>>>>>>> $ diff hilo.inc hilo.14092018.inc >>>>>>>>>> >>>>>>>>>> 132a133,140 >>>>>>>>>> > <tr> >>>>>>>>>> > <td class="label">$obs.label.wind_rms</td> >>>>>>>>>> > #for $archive in $archive_data >>>>>>>>>> > <td class="data new_row hilo_$archive[0]"> >>>>>>>>>> > $archive[1].wind.rms.formatted</td> >>>>>>>>>> > #end for >>>>>>>>>> > <td class="units">$unit.label.wind</td> >>>>>>>>>> > </tr> >>>>>>>>>> >>>>>>>>>> Apart from this any file referring to rms were exactly the same >>>>>>>>>> as the original install. Attaching skin.conf with this message. >>>>>>>>>> >>>>>>>>>> As expected the problem is gone but rms is not getting reported. >>>>>>>>>> Which means the issue still stays. >>>>>>>>>> >>>>>>>>>> Could you help me with what fields in the archive are referred to >>>>>>>>>> by rms ? I can try to manually check for any value discrepancy. >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> >>>>>>>>>> - Rahul >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Sep 14, 2018 at 9:14 AM Rahul Bedi <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Thanks Gary. >>>>>>>>>>> >>>>>>>>>>> Charges were done about a month ago but have seen no errors till >>>>>>>>>>> this episode. I will do a diff with the original skin files and >>>>>>>>>>> post it >>>>>>>>>>> here. >>>>>>>>>>> >>>>>>>>>>> Thanks. >>>>>>>>>>> >>>>>>>>>>> - Rahul >>>>>>>>>>> >>>>>>>>>>> On Fri 14 Sep, 2018, 7:35 AM gjr80, <[email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> On Friday, 14 September 2018 11:30:42 UTC+10, Rahul Bedi wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> As I have made no code or config changes since the day before, >>>>>>>>>>>>> I believe it doing this for the right data. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> This sounds to me like you have changed some of the seasons >>>>>>>>>>>> skin files. The error is coming form the use of a .rms tag in >>>>>>>>>>>> one of the seasons .tmpl or .inc files. The seasons skin only >>>>>>>>>>>> uses the .rms tag with the WeeWX wind field. As Tom indicated >>>>>>>>>>>> .rms only works against the wind field, if your changes have >>>>>>>>>>>> used the .rms tag against another field then this could be the >>>>>>>>>>>> cause. I suggest you go through all of your changes to the seasons >>>>>>>>>>>> skin >>>>>>>>>>>> files and see what fields are being used with the .rms tag. >>>>>>>>>>>> Until we know for sure, and given you seem to have made changes to >>>>>>>>>>>> the >>>>>>>>>>>> skin, we will likely be chasing our tails. >>>>>>>>>>>> >>>>>>>>>>>> Gary >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> You received this message because you are subscribed to a topic >>>>>>>>>>>> in the Google Groups "weewx-user" group. >>>>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>>>> https://groups.google.com/d/topic/weewx-user/LWmSpcVfdmo/unsubscribe >>>>>>>>>>>> . >>>>>>>>>>>> To unsubscribe from this group and all its topics, send an >>>>>>>>>>>> email to [email protected]. >>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "weewx-user" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to [email protected]. >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to a topic in >>>>>>>>> the Google Groups "weewx-user" group. >>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>> https://groups.google.com/d/topic/weewx-user/LWmSpcVfdmo/unsubscribe >>>>>>>>> . >>>>>>>>> To unsubscribe from this group and all its topics, send an email >>>>>>>>> to [email protected]. >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "weewx-user" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to a topic in >>>>>>> the Google Groups "weewx-user" group. >>>>>>> To unsubscribe from this topic, visit >>>>>>> https://groups.google.com/d/topic/weewx-user/LWmSpcVfdmo/unsubscribe >>>>>>> . >>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>> [email protected]. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "weewx-user" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "weewx-user" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "weewx-user" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/weewx-user/LWmSpcVfdmo/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> > -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
