On Monday, 27 May 2019 13:16:53 UTC+10, Leon Shaner wrote:
>
> Gary,
>
> In practice, WU seems to discard data that is not close to their
> APPARENTly preferred 5-minute "normalization buckets."
>
> I upload via rapidfire *and* regular loop on 1-minute intervals, and
> irregardless of same, the queries only ever show the records most closely
> aligned to their "preferred" 5-minute "buckets."
>
> I would love to see them preserve the same interval that I upload, but
> either by design or omission or bug, they simply don't. :-(
> Could be the layers in-between are dropping data by code-exception.
> Could be it's deliberate.
> I'm not here to debug their code unless they want to start paying me a
> worthy salary. ;-)
>
> Well... MOST of the time the behavior I have described seems to be true.
> :-/
>
> While WU normally SEEMS to only keeps records on 5-minute boundaries, I
> have seen where if wunderfixer is used persistently enough (say more than
> 10 times spread out every 20 minutes across a total span of 200 minutes)
> then it will keep records that are more frequent than every 5-minutes.
>
> Right now / especially lately, things are so sporadic with WU that I don't
> even want to spend cycles chasing what's really going on there. I would
> rather just only have wunderfixer ignore most records except those nearest
> the 5-minute boundaries that WU seems to most care about.
>
> I am not suggesting we/you/anyone should try to solve WU
issues/problems/whatever, merely making the point that WU is a blackbox,
nobody seems to really know what it does with the data that it is fed. If
we are going to have WU recreate/create 'missing' data then the logical
approach would be to feed it with the data it was orignally fed rather than
trying to guess what it wants.
> I have just now solved the "align to 5-minute boundaries" exercise with:
>
> _gen_rows = dbmanager.genSql("""SELECT dateTime FROM archive WHERE
> dateTime>=? AND dateTime<? GROUP BY (dateTime - 60 ) / 300 """,
> (start_ts, end_ts))
>
> Example:
>
> $ /usr/share/weewx4/bin/wunderfixer_403 --epsilon=125 --date=2019-05-25
> --test --verbose | head -30
> Using configuration file /usr/share/weewx4/weewx.conf.
> Using database binding 'wx_binding', which is bound to database
> 'archive_sqlite'
> Weather Underground Station: KMIDEARB5
> Date to check: 2019-05-25
> Number of archive records: 289
> Number of WU records: 260
> Number of missing records: 288
>
> Missing records:
> 2019-05-25 00:05:00 EDT (1558757100); 29.326"; 61.3F; 76%; 0.0 mph; N/A
> deg; 3.6 mph gust; 53.7F; 0.00" rain ... skipped.
> 2019-05-25 00:10:00 EDT (1558757400); 29.326"; 61.2F; 76%; 0.0 mph; N/A
> deg; 2.0 mph gust; 53.5F; 0.00" rain ... skipped.
> 2019-05-25 00:15:00 EDT (1558757700); 29.326"; 61.2F; 76%; 1.3 mph; 93
> deg; 3.1 mph gust; 53.5F; 0.00" rain ... skipped.
> 2019-05-25 00:20:00 EDT (1558758000); 29.317"; 61.1F; 76%; 1.3 mph; 93
> deg; 2.5 mph gust; 53.5F; 0.00" rain ... skipped.
> 2019-05-25 00:25:00 EDT (1558758300); 29.317"; 61.0F; 77%; 1.3 mph; 121
> deg; 4.3 mph gust; 53.7F; 0.00" rain ... skipped.
> 2019-05-25 00:30:00 EDT (1558758600); 29.317"; 61.0F; 76%; 1.3 mph; 121
> deg; 3.6 mph gust; 53.4F; 0.00" rain ... skipped.
> 2019-05-25 00:35:00 EDT (1558758900); 29.309"; 61.0F; 77%; 1.8 mph; 112
> deg; 4.0 mph gust; 53.7F; 0.00" rain ... skipped.
> 2019-05-25 00:40:00 EDT (1558759200); 29.309"; 60.9F; 76%; 1.8 mph; 112
> deg; 3.1 mph gust; 53.3F; 0.00" rain ... skipped.
> 2019-05-25 00:45:00 EDT (1558759500); 29.309"; 60.7F; 77%; 1.8 mph; 116
> deg; 5.8 mph gust; 53.3F; 0.00" rain ... skipped.
> 2019-05-25 00:50:00 EDT (1558759800); 29.306"; 60.6F; 77%; 1.8 mph; 116
> deg; 5.1 mph gust; 53.4F; 0.00" rain ... skipped.
> 2019-05-25 00:55:00 EDT (1558760100); 29.306"; 60.6F; 77%; 2.5 mph; 113
> deg; 2.9 mph gust; 53.4F; 0.00" rain ... skipped.
> 2019-05-25 01:00:00 EDT (1558760400); 29.306"; 60.5F; 77%; 2.5 mph; 113
> deg; 2.5 mph gust; 53.3F; 0.00" rain ... skipped.
> 2019-05-25 01:05:00 EDT (1558760700); 29.303"; 60.5F; 77%; 1.3 mph; 120
> deg; 3.6 mph gust; 53.2F; 0.00" rain ... skipped.
> 2019-05-25 01:10:00 EDT (1558761000); 29.303"; 60.4F; 78%; 1.3 mph; 120
> deg; 1.3 mph gust; 53.5F; 0.00" rain ... skipped.
> 2019-05-25 01:15:00 EDT (1558761300); 29.303"; 60.4F; 78%; 1.3 mph; 110
> deg; 4.7 mph gust; 53.5F; 0.00" rain ... skipped.
> 2019-05-25 01:20:00 EDT (1558761600); 29.285"; 60.4F; 78%; 1.3 mph; 110
> deg; 5.1 mph gust; 53.5F; 0.00" rain ... skipped.
> 2019-05-25 01:25:00 EDT (1558761900); 29.285"; 60.3F; 78%; 2.0 mph; 102
> deg; 3.1 mph gust; 53.4F; 0.00" rain ... skipped.
> 2019-05-25 01:30:00 EDT (1558762200); 29.285"; 60.3F; 78%; 2.0 mph; 102
> deg; 5.4 mph gust; 53.4F; 0.00" rain ... skipped.
> 2019-05-25 01:35:00 EDT (1558762500); 29.282"; 60.2F; 78%; 2.5 mph; 123
> deg; 2.5 mph gust; 53.3F; 0.00" rain ... skipped.
> 2019-05-25 01:40:00 EDT (1558762800); 29.282"; 60.1F; 78%; 2.5 mph; 123
> deg; 3.6 mph gust; 53.2F; 0.00" rain ... skipped.
> 2019-05-25 01:45:00 EDT (1558763100); 29.282"; 60.1F; 78%; 2.0 mph; 111
> deg; 5.4 mph gust; 53.2F; 0.00" rain ... skipped.
>
> But as you can see, I still have to solve the REAL problem that something
> is in disagreement about the localtime vs. UTC-time w.r.t. the dates in
> question.
>
> *I'm now forced 100% on that part, which is the real issue.*
>
> (Being that if I am within X hours of UTC where X = my localtime UTC
> offset, I get 100% records being marked as missing).
>
> The problem is likely in that same block of code I pasted above, but I
> don't yet have my head around it. :-/ Could also be a bug in the IBM API
> w.r.t. date specification in the URL a la 20190525, which I glean is
> supposed to be normalized ON THIER END to the local time of the PWS (since
> they know where it's located)...but could be a bug in their API
> implementation and isn't being handled properly. :-/
>
> I think I recently mentioned, that I am a persistent SOB, so it won't be
> long before I put the nail in this coffin in a way that is in our favor.
> LOL =D
>
> If you can see any shorter paths to a more reliable outcome than I have
> achieved so far, then you know know know I will be very grateful. =D
>
I am not sure what local/UTC issue you refer to. When I do a
api.weather.com/v2/pws/history query on a station to the east of Greenwich
I am returned all records for the date specified (eg 20190525 gives me all
records for 25 May 2019), each record contains an epoch timestamp which is
correct and consistent with 25 May 2019. Everything is as I would expect.
However, when I perform the same query on a station to the west of
Greenwich I am returned records for the day before the date specified (ie
20190525 gives me all records for 24 May 2019 not 25 May 2019), again each
record contains an epoch timestamp but the timestamp is for the previous
day Ie 24 May 2019. I have checked a number of data records in the stations
history table and WU is definitely returning the midnight to midnight data
for the day before. I have confirmed this behaviour with a number of
stations both east and west of Greenwich.
I don't think there is a local/UTC time issue, I think WU is having some
implementation issues and for stations west of Greenwich they are returning
the wrong day of data.
Gary
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/5aa7ea3b-f071-4ecd-bb49-69702cdaac44%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
