Gary, In practice, WU seems to discard data that is not close to their APPARENTly preferred 5-minute "normalization buckets."
I upload via rapidfire *and* regular loop on 1-minute intervals, and irregardless of same, the queries only ever show the records most closely aligned to their "preferred" 5-minute "buckets." I would love to see them preserve the same interval that I upload, but either by design or omission or bug, they simply don't. :-( Could be the layers in-between are dropping data by code-exception. Could be it's deliberate. I'm not here to debug their code unless they want to start paying me a worthy salary. ;-) Well... MOST of the time the behavior I have described seems to be true. :-/ While WU normally SEEMS to only keeps records on 5-minute boundaries, I have seen where if wunderfixer is used persistently enough (say more than 10 times spread out every 20 minutes across a total span of 200 minutes) then it will keep records that are more frequent than every 5-minutes. Right now / especially lately, things are so sporadic with WU that I don't even want to spend cycles chasing what's really going on there. I would rather just only have wunderfixer ignore most records except those nearest the 5-minute boundaries that WU seems to most care about. I have just now solved the "align to 5-minute boundaries" exercise with: _gen_rows = dbmanager.genSql("""SELECT dateTime FROM archive WHERE dateTime>=? AND dateTime<? GROUP BY (dateTime - 60 ) / 300 """, (start_ts, end_ts)) Example: $ /usr/share/weewx4/bin/wunderfixer_403 --epsilon=125 --date=2019-05-25 --test --verbose | head -30 Using configuration file /usr/share/weewx4/weewx.conf. Using database binding 'wx_binding', which is bound to database 'archive_sqlite' Weather Underground Station: KMIDEARB5 Date to check: 2019-05-25 Number of archive records: 289 Number of WU records: 260 Number of missing records: 288 Missing records: 2019-05-25 00:05:00 EDT (1558757100); 29.326"; 61.3F; 76%; 0.0 mph; N/A deg; 3.6 mph gust; 53.7F; 0.00" rain ... skipped. 2019-05-25 00:10:00 EDT (1558757400); 29.326"; 61.2F; 76%; 0.0 mph; N/A deg; 2.0 mph gust; 53.5F; 0.00" rain ... skipped. 2019-05-25 00:15:00 EDT (1558757700); 29.326"; 61.2F; 76%; 1.3 mph; 93 deg; 3.1 mph gust; 53.5F; 0.00" rain ... skipped. 2019-05-25 00:20:00 EDT (1558758000); 29.317"; 61.1F; 76%; 1.3 mph; 93 deg; 2.5 mph gust; 53.5F; 0.00" rain ... skipped. 2019-05-25 00:25:00 EDT (1558758300); 29.317"; 61.0F; 77%; 1.3 mph; 121 deg; 4.3 mph gust; 53.7F; 0.00" rain ... skipped. 2019-05-25 00:30:00 EDT (1558758600); 29.317"; 61.0F; 76%; 1.3 mph; 121 deg; 3.6 mph gust; 53.4F; 0.00" rain ... skipped. 2019-05-25 00:35:00 EDT (1558758900); 29.309"; 61.0F; 77%; 1.8 mph; 112 deg; 4.0 mph gust; 53.7F; 0.00" rain ... skipped. 2019-05-25 00:40:00 EDT (1558759200); 29.309"; 60.9F; 76%; 1.8 mph; 112 deg; 3.1 mph gust; 53.3F; 0.00" rain ... skipped. 2019-05-25 00:45:00 EDT (1558759500); 29.309"; 60.7F; 77%; 1.8 mph; 116 deg; 5.8 mph gust; 53.3F; 0.00" rain ... skipped. 2019-05-25 00:50:00 EDT (1558759800); 29.306"; 60.6F; 77%; 1.8 mph; 116 deg; 5.1 mph gust; 53.4F; 0.00" rain ... skipped. 2019-05-25 00:55:00 EDT (1558760100); 29.306"; 60.6F; 77%; 2.5 mph; 113 deg; 2.9 mph gust; 53.4F; 0.00" rain ... skipped. 2019-05-25 01:00:00 EDT (1558760400); 29.306"; 60.5F; 77%; 2.5 mph; 113 deg; 2.5 mph gust; 53.3F; 0.00" rain ... skipped. 2019-05-25 01:05:00 EDT (1558760700); 29.303"; 60.5F; 77%; 1.3 mph; 120 deg; 3.6 mph gust; 53.2F; 0.00" rain ... skipped. 2019-05-25 01:10:00 EDT (1558761000); 29.303"; 60.4F; 78%; 1.3 mph; 120 deg; 1.3 mph gust; 53.5F; 0.00" rain ... skipped. 2019-05-25 01:15:00 EDT (1558761300); 29.303"; 60.4F; 78%; 1.3 mph; 110 deg; 4.7 mph gust; 53.5F; 0.00" rain ... skipped. 2019-05-25 01:20:00 EDT (1558761600); 29.285"; 60.4F; 78%; 1.3 mph; 110 deg; 5.1 mph gust; 53.5F; 0.00" rain ... skipped. 2019-05-25 01:25:00 EDT (1558761900); 29.285"; 60.3F; 78%; 2.0 mph; 102 deg; 3.1 mph gust; 53.4F; 0.00" rain ... skipped. 2019-05-25 01:30:00 EDT (1558762200); 29.285"; 60.3F; 78%; 2.0 mph; 102 deg; 5.4 mph gust; 53.4F; 0.00" rain ... skipped. 2019-05-25 01:35:00 EDT (1558762500); 29.282"; 60.2F; 78%; 2.5 mph; 123 deg; 2.5 mph gust; 53.3F; 0.00" rain ... skipped. 2019-05-25 01:40:00 EDT (1558762800); 29.282"; 60.1F; 78%; 2.5 mph; 123 deg; 3.6 mph gust; 53.2F; 0.00" rain ... skipped. 2019-05-25 01:45:00 EDT (1558763100); 29.282"; 60.1F; 78%; 2.0 mph; 111 deg; 5.4 mph gust; 53.2F; 0.00" rain ... skipped. But as you can see, I still have to solve the REAL problem that something is in disagreement about the localtime vs. UTC-time w.r.t. the dates in question. I'm now forced 100% on that part, which is the real issue. (Being that if I am within X hours of UTC where X = my localtime UTC offset, I get 100% records being marked as missing). The problem is likely in that same block of code I pasted above, but I don't yet have my head around it. :-/ Could also be a bug in the IBM API w.r.t. date specification in the URL a la 20190525, which I glean is supposed to be normalized ON THIER END to the local time of the PWS (since they know where it's located)...but could be a bug in their API implementation and isn't being handled properly. :-/ I think I recently mentioned, that I am a persistent SOB, so it won't be long before I put the nail in this coffin in a way that is in our favor. LOL =D If you can see any shorter paths to a more reliable outcome than I have achieved so far, then you know know know I will be very grateful. =D Regards, \Leon -- Leon Shaner :: Dearborn, Michigan (iPad Pro) > On May 26, 2019, at 10:59 PM, gjr80 <gjroder...@gmail.com> wrote: > >> On Saturday, 25 May 2019 22:54:25 UTC+10, Leon Shaner wrote: >> >> There is no point re-uploading 00:00:00, then 00:01:00, then 00:02:00, >> because WU is only going to keep 00:00:00, and then later it will keep >> whatever is closest to 00:05:00. That's been the behavior of wunderfixer >> vs. WU for as long as I've been working with it, however. :-/ > > Maybe, maybe not. The problem is WU acts as a black box that accepts your > data at (more or less) whatever interval you choose WU then does who knows > what with the data and (usually) 5 minute interval records appear. You could > argue that re-uploading of data should be done in the same manner as it was > originally uploaded. > > Gary > -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to weewx-user+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/weewx-user/c3476bc2-674f-4928-bf58-55f2150159b2%40googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/309313C3-2AD2-43B4-941C-7BBD90DB35C8%40isylum.org. For more options, visit https://groups.google.com/d/optout.