Gary, Thanks for looking into this. You are correct, that is when I started using weewx and it must have also uploaded archive data to WU. Previously, the Weatherlink software was uploading 3 min after the archive intervals. That explains the interleaving of the data.
As a work around, I spent some time today reformating a csv export from Weatherlink to be compatible with wee_import. I was able to convert the time stamp to epoch time and reformat the wind cardinal directions to degrees. Then I imported the last 9 years of data from the csv file (almost a half million records). It all worked great, I now have all my data going back almost a decade. Best part is, the data is correct. Thanks again for everyone's help. Pete B On Sunday, September 16, 2018 at 2:31:20 AM UTC-4, gjr80 wrote: > > OK, I think I can see what happened here. if you look at the WU data for > 21 August (tried to include an extract in a bit more readable format below > but with limited success) at 17:10:00 you can see a WeeWX 3.8.2 sourced > record and thence every 10 minutes after. The net effect is that there are > WeeWX sourced and 'Wunderground v.1.15' sourced interleaved records. I > suspect the cause is that when Pete first started WeeWX it downloaded the > loggers history and uploaded any archive records to WU, I'll bet at the > time the oldest record in the logger was 21 August 2018 17:10:00. Because > 21 August was a 'partial' day as far as WeeWX was concerned I imagine it > had trouble getting a dayRain value to upload to WU, hence the WeeWX > records break the integrity of the WU rain data (column with first entry > 5.6). Consequently when wee_import was used to import the WU data it > hoovered up everything, and I could see wee_import being easily tricked > with the dayRain values going up and down within a day. Interestingly, if > you look at subsequent days you still see the interleaved records but this > time WeeWX has a full day of data so there is much more consistency in the > dayRain data (still the odd error though) > > 2018-08-21 16:23:20 23.4 22.9 1010.4 East 100 1.6 > 3.2 97 1.3 5.6 56 Wunderground v.1.15 2018-08-21 20:23:20 > 2018-08-21 16:33:12 23.4 22.9 1010.4 East 90 1.6 > 3.2 97 1 5.6 49 Wunderground v.1.15 2018-08-21 20:33:12 > 2018-08-21 16:43:08 23.6 23.1 1010.4 SE 134 1.6 > 3.2 97 0 5.6 72 Wunderground v.1.15 2018-08-21 20:43:08 > 2018-08-21 16:53:04 23.7 23.2 1010.4 SE 132 3.2 > 4.8 97 0 5.6 148 Wunderground v.1.15 2018-08-21 20:53:04 > 2018-08-21 17:03:20 23.8 23.3 1010.4 SE 132 3.2 > 3.2 97 0 5.6 42 Wunderground v.1.15 2018-08-21 21:03:20 > 2018-08-21 17:10:00 23.7 23.1 1010.1 SE 135 1.6 > 9.7 96 0 0 54 weewx-3.8.2 2018-08-21 21:10:00 > 2018-08-21 17:13:12 23.7 23.1 1010 East 98 3.2 3.2 > 96 0 5.6 37 Wunderground v.1.15 2018-08-21 21:13:12 > 2018-08-21 17:20:00 23.7 23.2 1010 East 90 1.6 > 11.3 97 0 0 43 weewx-3.8.2 2018-08-21 21:20:00 > 2018-08-21 17:23:08 23.7 23.2 1010 East 99 1.6 1.6 > 97 0 5.6 37 Wunderground v.1.15 2018-08-21 21:23:08 > 2018-08-21 17:30:00 23.7 23.2 1010 East 90 1.6 6.4 > 97 0 0 43 weewx-3.8.2 2018-08-21 21:30:00 > 2018-08-21 17:33:04 23.7 23.2 1010 East 101 1.6 > 1.6 97 0 5.6 26 Wunderground v.1.15 2018-08-21 21:33:04 > 2018-08-21 17:40:00 23.7 23.2 1009.9 East 90 1.6 8 > 97 0 0 27 weewx-3.8.2 2018-08-21 21:40:00 > 2018-08-21 17:43:20 23.7 23.2 1010 ESE 104 1.6 4.8 > 97 0 5.6 14 Wunderground v.1.15 2018-08-21 21:43:20 > 2018-08-21 17:50:00 23.8 23.3 1009.8 ESE 112 3.2 > 11.3 97 0 0 16 weewx-3.8.2 2018-08-21 21:50:00 > 2018-08-21 17:53:12 23.8 23.3 1009.7 ESE 113 4.8 > 4.8 97 0 5.6 25 Wunderground v.1.15 2018-08-21 21:53:12 > 2018-08-21 18:00:00 23.9 23.4 1009.8 ESE 112 3.2 > 11.3 97 0 0 31 weewx-3.8.2 2018-08-21 22:00:00 > 2018-08-21 18:03:08 23.8 23.2 1009.7 SSE 151 3.2 > 3.2 96 0 5.6 44 Wunderground v.1.15 2018-08-21 22:03:08 > 2018-08-21 18:10:00 23.9 23.2 1009.8 SSE 158 1.6 > 6.4 96 0.8 0.8 36 weewx-3.8.2 2018-08-21 22:10:00 > 2018-08-21 18:13:04 23.8 23.2 1009.7 ESE 116 1.6 > 1.6 96 43.4 7.9 23 Wunderground v.1.15 2018-08-21 22:13:04 > > No point in trying to re-import the WU data. So the available fixes would > be editing the WeeWX archive for 21 August 2018 then rebuilding the daily > summaries for that day or importing the .wlk files via a third party app. > You could delete the 'WeeWX sourced' records from the archive for 21 August > (for that matter through until 8 September when WeeWX is the sole source of > data) or you could delete the 'Wunderground v.1.15' records from 21 August > 2018 17:10:00 to 8 September. I am sure a cunning query could be crafted to > do it. One word of warning, if you use a graphical app to manipulate your > data be careful as they can leave null strings in SQLite databases and that > can causes issues. > > I guess hardening wee_import to these sorts of errors would be good, > going to be pretty complex though and I just see a stack more issues being > thrown up. > > Gary > > On Sunday, 16 September 2018 09:57:00 UTC+10, gjr80 wrote: >> >> Will look closely at this later, got to go out. >> >> Gary >> >> On Sunday, 16 September 2018 09:23:58 UTC+10, Thomas Keffer wrote: >>> >>> I think this is Pete's station: >>> https://www.wunderground.com/personal-weather-station/dashboard?ID=KMDABING1 >>> >>> -tk >>> >>> On Sat, Sep 15, 2018 at 4:17 PM gjr80 <[email protected]> wrote: >>> >>>> On Sunday, 16 September 2018 06:51:59 UTC+10, Thomas Keffer wrote: >>>>> >>>>> On Sat, Sep 15, 2018 at 1:12 PM <[email protected]> wrote: >>>>> >>>>>> I imported the data using the wee_import utility. I edited the >>>>>> import.conf file to download all the data from Weather Underground. >>>>>> >>>>> >>>>> Then we'll have to wait for Gary to chime in. But, it should be >>>>> possible without a big spike like that. >>>>> >>>> >>>> Unlike WU wee_import does not perform any magic incantations on the >>>> data being imported. WU records daily cumulative rain and rain in the last >>>> hour. wee_import uses the daily cumulative value to calculate the >>>> WeeWX rain field (rain per archive period) and ignores the hourly >>>> value. WeeWX field rainRate is calculated as a derived obs as per the >>>> StdCalculate service. >>>> >>>> I guess we are all familiar with the issues associated with WU and the >>>> quality of their data. Anecdotally, I have found that successive queries >>>> on >>>> WU of the type wee_import uses can at times return differing numbers >>>> of records for a given day (try running wunderfixer 10 times in a row >>>> and I guarantee it will find at least three different numbers of missing >>>> records). I can't say that I have seen variations in the data within the >>>> records WU returns, just in the overall number of records returned. >>>> >>>> What is the underlying data like when you display the tabulated >>>> observation data for the day concerned on WU? Is the discrepancy there? If >>>> so I guess all you can do is fix your WeeWX archive data for that day as >>>> Tom said (or import the .wlk files), if WU is showing fine you could >>>> try re-importing just that day. It's easy enough to do just delete the >>>> records for that day (a fairly simple SQL update query will do that) then >>>> run wee_import again but limit it to importing just that day. Of >>>> course you will need to rebuild your daily summaries again but you can >>>> limit that to just that one day. >>>> >>>> As an aside, last year I did look at extending wee_import to import >>>> .wlk files but on my first glance it looked like substantially more >>>> effort than the three sources that wee_import currently handles plus >>>> wview (and Cumulus I think ?) provide a means to indirectly importing data >>>> via a 3rd party utility. I haven't taken it any further since. >>>> >>>> Gary >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "weewx-user" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
