OK, I think I can see what happened here. if you look at the WU data for 21 August (tried to include an extract in a bit more readable format below but with limited success) at 17:10:00 you can see a WeeWX 3.8.2 sourced record and thence every 10 minutes after. The net effect is that there are WeeWX sourced and 'Wunderground v.1.15' sourced interleaved records. I suspect the cause is that when Pete first started WeeWX it downloaded the loggers history and uploaded any archive records to WU, I'll bet at the time the oldest record in the logger was 21 August 2018 17:10:00. Because 21 August was a 'partial' day as far as WeeWX was concerned I imagine it had trouble getting a dayRain value to upload to WU, hence the WeeWX records break the integrity of the WU rain data (column with first entry 5.6). Consequently when wee_import was used to import the WU data it hoovered up everything, and I could see wee_import being easily tricked with the dayRain values going up and down within a day. Interestingly, if you look at subsequent days you still see the interleaved records but this time WeeWX has a full day of data so there is much more consistency in the dayRain data (still the odd error though)
2018-08-21 16:23:20 23.4 22.9 1010.4 East 100 1.6 3.2 97 1.3 5.6 56 Wunderground v.1.15 2018-08-21 20:23:20 2018-08-21 16:33:12 23.4 22.9 1010.4 East 90 1.6 3.2 97 1 5.6 49 Wunderground v.1.15 2018-08-21 20:33:12 2018-08-21 16:43:08 23.6 23.1 1010.4 SE 134 1.6 3.2 97 0 5.6 72 Wunderground v.1.15 2018-08-21 20:43:08 2018-08-21 16:53:04 23.7 23.2 1010.4 SE 132 3.2 4.8 97 0 5.6 148 Wunderground v.1.15 2018-08-21 20:53:04 2018-08-21 17:03:20 23.8 23.3 1010.4 SE 132 3.2 3.2 97 0 5.6 42 Wunderground v.1.15 2018-08-21 21:03:20 2018-08-21 17:10:00 23.7 23.1 1010.1 SE 135 1.6 9.7 96 0 0 54 weewx-3.8.2 2018-08-21 21:10:00 2018-08-21 17:13:12 23.7 23.1 1010 East 98 3.2 3.2 96 0 5.6 37 Wunderground v.1.15 2018-08-21 21:13:12 2018-08-21 17:20:00 23.7 23.2 1010 East 90 1.6 11.3 97 0 0 43 weewx-3.8.2 2018-08-21 21:20:00 2018-08-21 17:23:08 23.7 23.2 1010 East 99 1.6 1.6 97 0 5.6 37 Wunderground v.1.15 2018-08-21 21:23:08 2018-08-21 17:30:00 23.7 23.2 1010 East 90 1.6 6.4 97 0 0 43 weewx-3.8.2 2018-08-21 21:30:00 2018-08-21 17:33:04 23.7 23.2 1010 East 101 1.6 1.6 97 0 5.6 26 Wunderground v.1.15 2018-08-21 21:33:04 2018-08-21 17:40:00 23.7 23.2 1009.9 East 90 1.6 8 97 0 0 27 weewx-3.8.2 2018-08-21 21:40:00 2018-08-21 17:43:20 23.7 23.2 1010 ESE 104 1.6 4.8 97 0 5.6 14 Wunderground v.1.15 2018-08-21 21:43:20 2018-08-21 17:50:00 23.8 23.3 1009.8 ESE 112 3.2 11.3 97 0 0 16 weewx-3.8.2 2018-08-21 21:50:00 2018-08-21 17:53:12 23.8 23.3 1009.7 ESE 113 4.8 4.8 97 0 5.6 25 Wunderground v.1.15 2018-08-21 21:53:12 2018-08-21 18:00:00 23.9 23.4 1009.8 ESE 112 3.2 11.3 97 0 0 31 weewx-3.8.2 2018-08-21 22:00:00 2018-08-21 18:03:08 23.8 23.2 1009.7 SSE 151 3.2 3.2 96 0 5.6 44 Wunderground v.1.15 2018-08-21 22:03:08 2018-08-21 18:10:00 23.9 23.2 1009.8 SSE 158 1.6 6.4 96 0.8 0.8 36 weewx-3.8.2 2018-08-21 22:10:00 2018-08-21 18:13:04 23.8 23.2 1009.7 ESE 116 1.6 1.6 96 43.4 7.9 23 Wunderground v.1.15 2018-08-21 22:13:04 No point in trying to re-import the WU data. So the available fixes would be editing the WeeWX archive for 21 August 2018 then rebuilding the daily summaries for that day or importing the .wlk files via a third party app. You could delete the 'WeeWX sourced' records from the archive for 21 August (for that matter through until 8 September when WeeWX is the sole source of data) or you could delete the 'Wunderground v.1.15' records from 21 August 2018 17:10:00 to 8 September. I am sure a cunning query could be crafted to do it. One word of warning, if you use a graphical app to manipulate your data be careful as they can leave null strings in SQLite databases and that can causes issues. I guess hardening wee_import to these sorts of errors would be good, going to be pretty complex though and I just see a stack more issues being thrown up. Gary On Sunday, 16 September 2018 09:57:00 UTC+10, gjr80 wrote: > > Will look closely at this later, got to go out. > > Gary > > On Sunday, 16 September 2018 09:23:58 UTC+10, Thomas Keffer wrote: >> >> I think this is Pete's station: >> https://www.wunderground.com/personal-weather-station/dashboard?ID=KMDABING1 >> >> -tk >> >> On Sat, Sep 15, 2018 at 4:17 PM gjr80 <[email protected]> wrote: >> >>> On Sunday, 16 September 2018 06:51:59 UTC+10, Thomas Keffer wrote: >>>> >>>> On Sat, Sep 15, 2018 at 1:12 PM <[email protected]> wrote: >>>> >>>>> I imported the data using the wee_import utility. I edited the >>>>> import.conf file to download all the data from Weather Underground. >>>>> >>>> >>>> Then we'll have to wait for Gary to chime in. But, it should be >>>> possible without a big spike like that. >>>> >>> >>> Unlike WU wee_import does not perform any magic incantations on the >>> data being imported. WU records daily cumulative rain and rain in the last >>> hour. wee_import uses the daily cumulative value to calculate the WeeWX >>> rain field (rain per archive period) and ignores the hourly value. >>> WeeWX field rainRate is calculated as a derived obs as per the >>> StdCalculate service. >>> >>> I guess we are all familiar with the issues associated with WU and the >>> quality of their data. Anecdotally, I have found that successive queries on >>> WU of the type wee_import uses can at times return differing numbers of >>> records for a given day (try running wunderfixer 10 times in a row and >>> I guarantee it will find at least three different numbers of missing >>> records). I can't say that I have seen variations in the data within the >>> records WU returns, just in the overall number of records returned. >>> >>> What is the underlying data like when you display the tabulated >>> observation data for the day concerned on WU? Is the discrepancy there? If >>> so I guess all you can do is fix your WeeWX archive data for that day as >>> Tom said (or import the .wlk files), if WU is showing fine you could >>> try re-importing just that day. It's easy enough to do just delete the >>> records for that day (a fairly simple SQL update query will do that) then >>> run wee_import again but limit it to importing just that day. Of course >>> you will need to rebuild your daily summaries again but you can limit that >>> to just that one day. >>> >>> As an aside, last year I did look at extending wee_import to import .wlk >>> files but on my first glance it looked like substantially more effort than >>> the three sources that wee_import currently handles plus wview (and >>> Cumulus I think ?) provide a means to indirectly importing data via a 3rd >>> party utility. I haven't taken it any further since. >>> >>> Gary >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "weewx-user" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
