Gary,
In practice, WU seems to discard data that is not close to their APPARENTly
preferred 5-minute "normalization buckets."
I upload via rapidfire *and* regular loop on 1-minute intervals, and
irregardless of same, the queries only ever show the records most closely
aligned to their "preferred" 5-minute "buckets."
I would love to see them preserve the same interval that I upload, but either
by design or omission or bug, they simply don't. :-(
Could be the layers in-between are dropping data by code-exception.
Could be it's deliberate.
I'm not here to debug their code unless they want to start paying me a worthy
salary. ;-)
Well... MOST of the time the behavior I have described seems to be true. :-/
While WU normally SEEMS to only keeps records on 5-minute boundaries, I have
seen where if wunderfixer is used persistently enough (say more than 10 times
spread out every 20 minutes across a total span of 200 minutes) then it will
keep records that are more frequent than every 5-minutes.
Right now / especially lately, things are so sporadic with WU that I don't even
want to spend cycles chasing what's really going on there. I would rather just
only have wunderfixer ignore most records except those nearest the 5-minute
boundaries that WU seems to most care about.
I have just now solved the "align to 5-minute boundaries" exercise with:
_gen_rows = dbmanager.genSql("""SELECT dateTime FROM archive WHERE dateTime>=?
AND dateTime<? GROUP BY (dateTime - 60 ) / 300 """,
(start_ts, end_ts))
Example:
$ /usr/share/weewx4/bin/wunderfixer_403 --epsilon=125 --date=2019-05-25 --test
--verbose | head -30
Using configuration file /usr/share/weewx4/weewx.conf.
Using database binding 'wx_binding', which is bound to database 'archive_sqlite'
Weather Underground Station: KMIDEARB5
Date to check: 2019-05-25
Number of archive records: 289
Number of WU records: 260
Number of missing records: 288
Missing records:
2019-05-25 00:05:00 EDT (1558757100); 29.326"; 61.3F; 76%; 0.0 mph; N/A deg;
3.6 mph gust; 53.7F; 0.00" rain ... skipped.
2019-05-25 00:10:00 EDT (1558757400); 29.326"; 61.2F; 76%; 0.0 mph; N/A deg;
2.0 mph gust; 53.5F; 0.00" rain ... skipped.
2019-05-25 00:15:00 EDT (1558757700); 29.326"; 61.2F; 76%; 1.3 mph; 93 deg;
3.1 mph gust; 53.5F; 0.00" rain ... skipped.
2019-05-25 00:20:00 EDT (1558758000); 29.317"; 61.1F; 76%; 1.3 mph; 93 deg;
2.5 mph gust; 53.5F; 0.00" rain ... skipped.
2019-05-25 00:25:00 EDT (1558758300); 29.317"; 61.0F; 77%; 1.3 mph; 121 deg;
4.3 mph gust; 53.7F; 0.00" rain ... skipped.
2019-05-25 00:30:00 EDT (1558758600); 29.317"; 61.0F; 76%; 1.3 mph; 121 deg;
3.6 mph gust; 53.4F; 0.00" rain ... skipped.
2019-05-25 00:35:00 EDT (1558758900); 29.309"; 61.0F; 77%; 1.8 mph; 112 deg;
4.0 mph gust; 53.7F; 0.00" rain ... skipped.
2019-05-25 00:40:00 EDT (1558759200); 29.309"; 60.9F; 76%; 1.8 mph; 112 deg;
3.1 mph gust; 53.3F; 0.00" rain ... skipped.
2019-05-25 00:45:00 EDT (1558759500); 29.309"; 60.7F; 77%; 1.8 mph; 116 deg;
5.8 mph gust; 53.3F; 0.00" rain ... skipped.
2019-05-25 00:50:00 EDT (1558759800); 29.306"; 60.6F; 77%; 1.8 mph; 116 deg;
5.1 mph gust; 53.4F; 0.00" rain ... skipped.
2019-05-25 00:55:00 EDT (1558760100); 29.306"; 60.6F; 77%; 2.5 mph; 113 deg;
2.9 mph gust; 53.4F; 0.00" rain ... skipped.
2019-05-25 01:00:00 EDT (1558760400); 29.306"; 60.5F; 77%; 2.5 mph; 113 deg;
2.5 mph gust; 53.3F; 0.00" rain ... skipped.
2019-05-25 01:05:00 EDT (1558760700); 29.303"; 60.5F; 77%; 1.3 mph; 120 deg;
3.6 mph gust; 53.2F; 0.00" rain ... skipped.
2019-05-25 01:10:00 EDT (1558761000); 29.303"; 60.4F; 78%; 1.3 mph; 120 deg;
1.3 mph gust; 53.5F; 0.00" rain ... skipped.
2019-05-25 01:15:00 EDT (1558761300); 29.303"; 60.4F; 78%; 1.3 mph; 110 deg;
4.7 mph gust; 53.5F; 0.00" rain ... skipped.
2019-05-25 01:20:00 EDT (1558761600); 29.285"; 60.4F; 78%; 1.3 mph; 110 deg;
5.1 mph gust; 53.5F; 0.00" rain ... skipped.
2019-05-25 01:25:00 EDT (1558761900); 29.285"; 60.3F; 78%; 2.0 mph; 102 deg;
3.1 mph gust; 53.4F; 0.00" rain ... skipped.
2019-05-25 01:30:00 EDT (1558762200); 29.285"; 60.3F; 78%; 2.0 mph; 102 deg;
5.4 mph gust; 53.4F; 0.00" rain ... skipped.
2019-05-25 01:35:00 EDT (1558762500); 29.282"; 60.2F; 78%; 2.5 mph; 123 deg;
2.5 mph gust; 53.3F; 0.00" rain ... skipped.
2019-05-25 01:40:00 EDT (1558762800); 29.282"; 60.1F; 78%; 2.5 mph; 123 deg;
3.6 mph gust; 53.2F; 0.00" rain ... skipped.
2019-05-25 01:45:00 EDT (1558763100); 29.282"; 60.1F; 78%; 2.0 mph; 111 deg;
5.4 mph gust; 53.2F; 0.00" rain ... skipped.
But as you can see, I still have to solve the REAL problem that something is in
disagreement about the localtime vs. UTC-time w.r.t. the dates in question.
I'm now forced 100% on that part, which is the real issue.
(Being that if I am within X hours of UTC where X = my localtime UTC offset, I
get 100% records being marked as missing).
The problem is likely in that same block of code I pasted above, but I don't
yet have my head around it. :-/ Could also be a bug in the IBM API w.r.t.
date specification in the URL a la 20190525, which I glean is supposed to be
normalized ON THIER END to the local time of the PWS (since they know where
it's located)...but could be a bug in their API implementation and isn't being
handled properly. :-/
I think I recently mentioned, that I am a persistent SOB, so it won't be long
before I put the nail in this coffin in a way that is in our favor. LOL =D
If you can see any shorter paths to a more reliable outcome than I have
achieved so far, then you know know know I will be very grateful. =D
Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)
> On May 26, 2019, at 10:59 PM, gjr80 <[email protected]> wrote:
>
>> On Saturday, 25 May 2019 22:54:25 UTC+10, Leon Shaner wrote:
>>
>> There is no point re-uploading 00:00:00, then 00:01:00, then 00:02:00,
>> because WU is only going to keep 00:00:00, and then later it will keep
>> whatever is closest to 00:05:00. That's been the behavior of wunderfixer
>> vs. WU for as long as I've been working with it, however. :-/
>
> Maybe, maybe not. The problem is WU acts as a black box that accepts your
> data at (more or less) whatever interval you choose WU then does who knows
> what with the data and (usually) 5 minute interval records appear. You could
> argue that re-uploading of data should be done in the same manner as it was
> originally uploaded.
>
> Gary
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/weewx-user/c3476bc2-674f-4928-bf58-55f2150159b2%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/309313C3-2AD2-43B4-941C-7BBD90DB35C8%40isylum.org.
For more options, visit https://groups.google.com/d/optout.
