Re: [weewx-user] Re: Amending spurious readings in the database

2017-04-14 Thread Andrew Milner 
>From the noaa reports it is impossible to know if your issues are related
to a single comms issue corrupting many values in one loop record for
example, or if it is different issues with different sensors at different
times and on different occasions.

Not knowing your station type does not help much either!!

Not knowing if the log says anything to match the 'corrupted' data does not
help to identify the problems and causes.

Taking your example of a wind at 128.9 knots - it would be helpful to see
the log and database records for the period of spike - 15 minutes to spike
+ 15 minutes for example.

Supplying us with NOAA data is not a substitute for the actual database
data captured by weewx - the noaa report is, after all, derived from the
database and not the other way around!!

It may also be relevant to know if your station is supplying partial data
from different sensors in different loopr ecords or true complete loop data
and archive records.



On 14 April 2017 at 15:26, Auchtermuchty Weather 
wrote:

> The colour coding makes it easier (for me) to spot the days with rogue
> data although sorting the data by (say) windspeed is another say to find it
> and yes, I realise what I need to do to find the values.  I put the
> spreadsheet up to show people what I was seeing.
>
> My question was if this is a sign of sensors reaching the end of their
> life, given most of the rogue values have occurred since the start of March
> this year.
>
>
>
> On Friday, 14 April 2017 10:02:52 UTC+1, Andrew Milner wrote:
>
>> What you need to do is to use the NOAA data to possibly pinpoint a rogue
>> day, and then check the archive table to see if there is a single rogue
>> record.
>>
>> If there is then you can just correct the rogue record or delete it even,
>> and rebuild the stats tables and noaa files if required.
>>
>> You should check the log at around the time the rogue record or recoprds
>> were inserted into the database and see if there were issues recorded in
>> the log.  Maybe you have a comms problem.
>>
>> I could not tell what we were supposed to deduce from the colour coded
>> noaa data - since there were so many colours in use!!
>>
>>
>>
>> On Friday, 14 April 2017 11:19:15 UTC+3, Auchtermuchty Weather wrote:
>>
>>> Have picked up my NOAA data into excel, and find that nearly all my
>>> 'bad' values are since 1st March this year.  It was December 2016 when I
>>> replaced the Pi with a better one with the then up-to-date version of Weewx.
>>>
>>> XLXS file attached - I have colour coded the values in each appropriate
>>> column.
>>>
>>> Makes me think I do have a sensor problem or problems.  :(
>>>
>>>
>>> On Wednesday, 12 April 2017 09:20:06 UTC+1, Auchtermuchty Weather wrote:
>>>
 I am getting the odd spike in my readings, for example:


TEMPERATURE (C), RAIN (mm), WIND SPEED (knots)

  HEAT   COOL AVG
   MEAN   DEGDEG
 WIND   DOM
 DAY   TEMP   HIGH   TIMELOW   TIME   DAYS   DAYS   RAIN  SPEED
 HIGH   TIMEDIR
 
 ---
 013.5   13.1  11:12   -1.1  04:31   14.80.00.00.4
 6.0  07:19 22
 024.3   13.4  05:56   -0.7  00:02   14.00.0  148.81.8
 44.9  05:56 25
 034.3   11.9  11:31   -2.3  07:56   14.00.05.10.7
 8.6  22:43191
 045.26.7  15:304.0  08:30   13.20.09.90.7
 12.6  09:50207
 055.6   10.3  15:25   -0.2  23:51   12.70.03.60.0
 3.3  07:37108
 065.2   16.7  14:55   -0.3  01:18   13.10.01.20.1
 43.5  15:34130
 075.1   12.4  11:19  -15.1  08:56   13.20.0  807.90.4
 8.0  16:15 77
 087.3   13.3  11:314.8  07:30   11.00.02.41.7
 19.2  11:44360
 098.0   14.3  14:262.7  23:51   10.30.0  271.81.2
 11.9  08:48345
 106.0   10.2  16:021.2  02:30   12.40.0   76.80.1
 26.4  07:35249
 11   10.2   16.3  14:02  -38.0  15:328.10.0  625.50.6
 128.9  06:15 29


 There is no way there has been a wind speed of 128.9 knots!

 I can't remember how to fix these, nor can I remember if there is a
 setup somewhere to reject readings outside the parameters.

 --
> You received this message because you are subscribed to a topic in the
> Google Groups "weewx-user" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/weewx-user/wP9HauD6dNA/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> weewx-user+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the

[weewx-user] Re: Amending spurious readings in the database

2017-04-14 Thread Auchtermuchty Weather 
The colour coding makes it easier (for me) to spot the days with rogue data 
although sorting the data by (say) windspeed is another say to find it and 
yes, I realise what I need to do to find the values.  I put the spreadsheet 
up to show people what I was seeing.

My question was if this is a sign of sensors reaching the end of their 
life, given most of the rogue values have occurred since the start of March 
this year.



On Friday, 14 April 2017 10:02:52 UTC+1, Andrew Milner wrote:

> What you need to do is to use the NOAA data to possibly pinpoint a rogue 
> day, and then check the archive table to see if there is a single rogue 
> record.
>
> If there is then you can just correct the rogue record or delete it even, 
> and rebuild the stats tables and noaa files if required.
>
> You should check the log at around the time the rogue record or recoprds 
> were inserted into the database and see if there were issues recorded in 
> the log.  Maybe you have a comms problem.
>
> I could not tell what we were supposed to deduce from the colour coded 
> noaa data - since there were so many colours in use!!
>
>
>
> On Friday, 14 April 2017 11:19:15 UTC+3, Auchtermuchty Weather wrote:
>
>> Have picked up my NOAA data into excel, and find that nearly all my 'bad' 
>> values are since 1st March this year.  It was December 2016 when I replaced 
>> the Pi with a better one with the then up-to-date version of Weewx.
>>
>> XLXS file attached - I have colour coded the values in each appropriate 
>> column.
>>
>> Makes me think I do have a sensor problem or problems.  :(
>>  
>>
>> On Wednesday, 12 April 2017 09:20:06 UTC+1, Auchtermuchty Weather wrote:
>>
>>> I am getting the odd spike in my readings, for example:
>>>
>>>
>>>TEMPERATURE (C), RAIN (mm), WIND SPEED (knots)
>>>
>>>  HEAT   COOL AVG
>>>   MEAN   DEGDEG  
>>> WIND   DOM
>>> DAY   TEMP   HIGH   TIMELOW   TIME   DAYS   DAYS   RAIN  SPEED   
>>> HIGH   TIMEDIR
>>>
>>> ---
>>> 013.5   13.1  11:12   -1.1  04:31   14.80.00.00.4
>>> 6.0  07:19 22
>>> 024.3   13.4  05:56   -0.7  00:02   14.00.0  148.81.8   
>>> 44.9  05:56 25
>>> 034.3   11.9  11:31   -2.3  07:56   14.00.05.10.7
>>> 8.6  22:43191
>>> 045.26.7  15:304.0  08:30   13.20.09.90.7   
>>> 12.6  09:50207
>>> 055.6   10.3  15:25   -0.2  23:51   12.70.03.60.0
>>> 3.3  07:37108
>>> 065.2   16.7  14:55   -0.3  01:18   13.10.01.20.1   
>>> 43.5  15:34130
>>> 075.1   12.4  11:19  -15.1  08:56   13.20.0  807.90.4
>>> 8.0  16:15 77
>>> 087.3   13.3  11:314.8  07:30   11.00.02.41.7   
>>> 19.2  11:44360
>>> 098.0   14.3  14:262.7  23:51   10.30.0  271.81.2   
>>> 11.9  08:48345
>>> 106.0   10.2  16:021.2  02:30   12.40.0   76.80.1   
>>> 26.4  07:35249
>>> 11   10.2   16.3  14:02  -38.0  15:328.10.0  625.50.6  
>>> 128.9  06:15 29
>>>
>>>
>>> There is no way there has been a wind speed of 128.9 knots!
>>>
>>> I can't remember how to fix these, nor can I remember if there is a 
>>> setup somewhere to reject readings outside the parameters.
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: Amending spurious readings in the database

2017-04-14 Thread Andrew Milner 
What you need to do is to use the NOAA data to possibly pinpoint a rogue 
day, and then check the archive table to see if there is a single rogue 
record.

If there is then you can just correct the rogue record or delete it even, 
and rebuild the stats tables and noaa files if required.

You should check the log at around the time the rogue record or recoprds 
were inserted into the database and see if there were issues recorded in 
the log.  Maybe you have a comms problem.

I could not tell what we were supposed to deduce from the colour coded noaa 
data - since there were so many colours in use!!



On Friday, 14 April 2017 11:19:15 UTC+3, Auchtermuchty Weather wrote:

> Have picked up my NOAA data into excel, and find that nearly all my 'bad' 
> values are since 1st March this year.  It was December 2016 when I replaced 
> the Pi with a better one with the then up-to-date version of Weewx.
>
> XLXS file attached - I have colour coded the values in each appropriate 
> column.
>
> Makes me think I do have a sensor problem or problems.  :(
>  
>
> On Wednesday, 12 April 2017 09:20:06 UTC+1, Auchtermuchty Weather wrote:
>
>> I am getting the odd spike in my readings, for example:
>>
>>
>>TEMPERATURE (C), RAIN (mm), WIND SPEED (knots)
>>
>>  HEAT   COOL AVG
>>   MEAN   DEGDEG  
>> WIND   DOM
>> DAY   TEMP   HIGH   TIMELOW   TIME   DAYS   DAYS   RAIN  SPEED   
>> HIGH   TIMEDIR
>>
>> ---
>> 013.5   13.1  11:12   -1.1  04:31   14.80.00.00.46.0  
>> 07:19 22
>> 024.3   13.4  05:56   -0.7  00:02   14.00.0  148.81.8   44.9  
>> 05:56 25
>> 034.3   11.9  11:31   -2.3  07:56   14.00.05.10.78.6  
>> 22:43191
>> 045.26.7  15:304.0  08:30   13.20.09.90.7   12.6  
>> 09:50207
>> 055.6   10.3  15:25   -0.2  23:51   12.70.03.60.03.3  
>> 07:37108
>> 065.2   16.7  14:55   -0.3  01:18   13.10.01.20.1   43.5  
>> 15:34130
>> 075.1   12.4  11:19  -15.1  08:56   13.20.0  807.90.48.0  
>> 16:15 77
>> 087.3   13.3  11:314.8  07:30   11.00.02.41.7   19.2  
>> 11:44360
>> 098.0   14.3  14:262.7  23:51   10.30.0  271.81.2   11.9  
>> 08:48345
>> 106.0   10.2  16:021.2  02:30   12.40.0   76.80.1   26.4  
>> 07:35249
>> 11   10.2   16.3  14:02  -38.0  15:328.10.0  625.50.6  128.9  
>> 06:15 29
>>
>>
>> There is no way there has been a wind speed of 128.9 knots!
>>
>> I can't remember how to fix these, nor can I remember if there is a setup 
>> somewhere to reject readings outside the parameters.
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: Amending spurious readings in the database

2017-04-13 Thread Glenn McKechnie 
As someone who has problems with the occasional(scoff!) spike value; I
highly recommend Darryn's - beta - qc.py replacement (links below). It may
not fit your comfort level, and I don't think you could use it for rain or
any other instrument (wind?) that can have spikes as a matter of course,
but then again...

As you're aware, if you have failing hardware or similar underlying
problems then it's not necessarily the preferred solution - but when you
consider that an errant value can be caused by many things, and have way
more impact on the historical data,  database, graphs than it should; then
ignoring them is a valid option.
I get spikes in my rain values, but don't use Darryn's qc.py for them -
barometer and outside humidity though? It's been absolutely brilliant for
those sensors. I'm working on replacing them, as something is amiss there,
but the qc spike detection has allowed me to spend some valuable time
finding and isolating the cause and determining my options, the best way to
go for my setup. In the meantime, I still have a working system with the
outliers banished to the bit bucket.

If you're willing and able to replace the stock weewx qc.py with the
qc-spike.py of Darryn's, and spend a bit of time understanding it and
tweaking it to suit your circumstances, it could repay you (in fact:
anyone) with some nice, clean data.

As Gary says, the StdQC service is fairly basic, but I feel this qc-spike
addition enhances that service very nicely, and it's not _too_ complicated
to implement.

Links to Darryn's qc spike posts...
https://groups.google.com/d/topic/weewx-user/p6iGhRUxxa4/discussion
https://groups.google.com/d/topic/weewx-development/vfD9WfHLSaI/discussion


Cheers
 Glenn

rorpi - read only raspberry pi + weewx: now with scripts


On 13 April 2017 at 18:03, Auchtermuchty Weather 
wrote:

> Thanks, the backfill in progress and I have a feeling my weather station
> might have problems as it's clear the erroneous readings have greatly
> increased.
>
> I have a question about the QC.  It's clear what the temperature values
> (for example) should be, but rain is a cumulative thing, so over what
> period of time?  I'm guessing between readings, but to me that doesn't make
> sense with a default QC value of 60 inches!
>
>
> On Wednesday, 12 April 2017 09:20:06 UTC+1, Auchtermuchty Weather wrote:
>
>> I am getting the odd spike in my readings, for example:
>>
>>
>>TEMPERATURE (C), RAIN (mm), WIND SPEED (knots)
>>
>>  HEAT   COOL AVG
>>   MEAN   DEGDEG
>> WIND   DOM
>> DAY   TEMP   HIGH   TIMELOW   TIME   DAYS   DAYS   RAIN  SPEED
>> HIGH   TIMEDIR
>> 
>> ---
>> 013.5   13.1  11:12   -1.1  04:31   14.80.00.00.46.0
>> 07:19 22
>> 024.3   13.4  05:56   -0.7  00:02   14.00.0  148.81.8   44.9
>> 05:56 25
>> 034.3   11.9  11:31   -2.3  07:56   14.00.05.10.78.6
>> 22:43191
>> 045.26.7  15:304.0  08:30   13.20.09.90.7   12.6
>> 09:50207
>> 055.6   10.3  15:25   -0.2  23:51   12.70.03.60.03.3
>> 07:37108
>> 065.2   16.7  14:55   -0.3  01:18   13.10.01.20.1   43.5
>> 15:34130
>> 075.1   12.4  11:19  -15.1  08:56   13.20.0  807.90.48.0
>> 16:15 77
>> 087.3   13.3  11:314.8  07:30   11.00.02.41.7   19.2
>> 11:44360
>> 098.0   14.3  14:262.7  23:51   10.30.0  271.81.2   11.9
>> 08:48345
>> 106.0   10.2  16:021.2  02:30   12.40.0   76.80.1   26.4
>> 07:35249
>> 11   10.2   16.3  14:02  -38.0  15:328.10.0  625.50.6  128.9
>> 06:15 29
>>
>>
>> There is no way there has been a wind speed of 128.9 knots!
>>
>> I can't remember how to fix these, nor can I remember if there is a setup
>> somewhere to reject readings outside the parameters.
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: Amending spurious readings in the database

2017-04-13 Thread gjr80 
The StdQC service is fairly basic and applies a simple max min limit to both 
loop packets and archive records. So the rain max min value is per loop period 
or archive interval. Yes, 60 inches as a default is unrealistic, I thought I 
read something about changing it but I could be wrong. Will make sure it is 
looked at for the next release.

Gary

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Re: Amending spurious readings in the database

2017-04-13 Thread Auchtermuchty Weather 
Thanks, the backfill in progress and I have a feeling my weather station 
might have problems as it's clear the erroneous readings have greatly 
increased.

I have a question about the QC.  It's clear what the temperature values 
(for example) should be, but rain is a cumulative thing, so over what 
period of time?  I'm guessing between readings, but to me that doesn't make 
sense with a default QC value of 60 inches!


On Wednesday, 12 April 2017 09:20:06 UTC+1, Auchtermuchty Weather wrote:

> I am getting the odd spike in my readings, for example:
>
>
>TEMPERATURE (C), RAIN (mm), WIND SPEED (knots)
>
>  HEAT   COOL AVG
>   MEAN   DEGDEG  
> WIND   DOM
> DAY   TEMP   HIGH   TIMELOW   TIME   DAYS   DAYS   RAIN  SPEED   
> HIGH   TIMEDIR
>
> ---
> 013.5   13.1  11:12   -1.1  04:31   14.80.00.00.46.0  
> 07:19 22
> 024.3   13.4  05:56   -0.7  00:02   14.00.0  148.81.8   44.9  
> 05:56 25
> 034.3   11.9  11:31   -2.3  07:56   14.00.05.10.78.6  
> 22:43191
> 045.26.7  15:304.0  08:30   13.20.09.90.7   12.6  
> 09:50207
> 055.6   10.3  15:25   -0.2  23:51   12.70.03.60.03.3  
> 07:37108
> 065.2   16.7  14:55   -0.3  01:18   13.10.01.20.1   43.5  
> 15:34130
> 075.1   12.4  11:19  -15.1  08:56   13.20.0  807.90.48.0  
> 16:15 77
> 087.3   13.3  11:314.8  07:30   11.00.02.41.7   19.2  
> 11:44360
> 098.0   14.3  14:262.7  23:51   10.30.0  271.81.2   11.9  
> 08:48345
> 106.0   10.2  16:021.2  02:30   12.40.0   76.80.1   26.4  
> 07:35249
> 11   10.2   16.3  14:02  -38.0  15:328.10.0  625.50.6  128.9  
> 06:15 29
>
>
> There is no way there has been a wind speed of 128.9 knots!
>
> I can't remember how to fix these, nor can I remember if there is a setup 
> somewhere to reject readings outside the parameters.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.