Hey, WeeWX'ers, I wonder how many folks are experiencing issues with WU timing out / not responding / failing to accept station updates from weewx?
I've resorted to running wunderfixer roughly every 10 minutes against "today" and "yesterday" and it's almost always finding missing records throughout "today" which after enough wunderfixer re-uploads, they do eventually "stick." I can see from my "weewx_watchdog" logs that wunderfixer was timing out almost every attempt to run against the "yesterday" date, but was usually fine when running against the "today" date. I'm guessing that WU may have some optimizations in place to respond quickly to "current" day queries, and maybe they have intentionally allocated less "capacity" to historical queries. That might explain the timeouts I was seeing mainly when querying the prior day. I found that increasing the wunderfixer timeout from 10 seconds to 20 seconds was helpful in working around the WU connection issues. There is an updated wunderfixer in the that accepts a --timeout option, over here. Thanks, Tom! =D https://github.com/weewx/weewx/blob/master/bin/wunderfixer Now, because wunderfixer is pretty much always finding missing records, I am wondering of weewx itself is falling victim to the WU "capacity issues" during the normal uploading of records? As in could the records be missing on the WU side due to similar timeouts that wunderfixer is experiencing? I have debug = 1, but there are no issues being reported, so do I need a higher debug, such as debut = 3? Could it be that the connections are going through but WU is losing the data, anyway? And/or could it be that running wunderfixer every 10 minutes isn't allowing WU enough time to process the data, such that is is available by the next query, 10 minutes later? Here are some logs of what I am seeing re: the trend of missing WU data. You can see the "overlap" in that multiple runs of wunderfixer 10 minutes apart are finding and re-uploading the exact same missing records until eventually they do "stick" in that they are not shown missing / not re-uploaded. In the below, you can see that a record from 1556709420 did get posted properly, because it was not mentioned again 10 minutes later. However, there are three other records in bold, which did not "stick" between those wunderfixer runs, 10 minutes apart. And then 10 minutes later, two more overlapping re-uploads, which didn't "stick." Wed 1 May 10:15:11 EDT 2019 Weather Underground Station: KMIDEARB5 Wed 1 May 10:15:11 EDT 2019 Date to check: 2019-04-30 Wed 1 May 10:15:11 EDT 2019 Number of archive records: 1438 Wed 1 May 10:15:11 EDT 2019 Number of WU records: 381 Wed 1 May 10:15:11 EDT 2019 Number of missing records: 0 Wed 1 May 10:25:01 EDT 2019 Using configuration file /etc/weewx/weewx.conf. Wed 1 May 10:25:01 EDT 2019 Using database binding 'wx_binding', which is bound to database 'archive_sqlite' Wed 1 May 10:25:01 EDT 2019 Weather Underground Station: KMIDEARB5 Wed 1 May 10:25:01 EDT 2019 Date to check: 2019-05-01 Wed 1 May 10:25:01 EDT 2019 Number of archive records: 624 Wed 1 May 10:25:01 EDT 2019 Number of WU records: 144 Wed 1 May 10:25:01 EDT 2019 Number of missing records: 5 Wed 1 May 10:25:01 EDT 2019 Wed 1 May 10:25:01 EDT 2019 Missing records: Wed 1 May 10:25:01 EDT 2019 2019-05-01 07:17:00 EDT (1556709420); 29.297"; 47.7F; 93%; 4.7 mph; 98 deg; 7.6 mph gust; 45.7F; 0.00" rain ...published. Wed 1 May 10:25:01 EDT 2019 2019-05-01 09:57:00 EDT (1556719020); 29.226"; 50.4F; 93%; 2.9 mph; 111 deg; 6.9 mph gust; 48.4F; 0.00" rain ...published. Wed 1 May 10:25:01 EDT 2019 2019-05-01 10:17:00 EDT (1556720220); 29.223"; 52.0F; 92%; 2.0 mph; 136 deg; 3.6 mph gust; 49.9F; 0.00" rain ...published. Wed 1 May 10:25:01 EDT 2019 2019-05-01 10:23:00 EDT (1556720580); 29.214"; 52.4F; 92%; 2.5 mph; 115 deg; 4.3 mph gust; 50.1F; 0.00" rain ...published. Wed 1 May 10:25:01 EDT 2019 2019-05-01 10:24:00 EDT (1556720640); 29.214"; 52.5F; 92%; 2.5 mph; 115 deg; 5.4 mph gust; 50.3F; 0.00" rain ...published. Wed 1 May 10:25:14 EDT 2019 Using configuration file /etc/weewx/weewx.conf. Wed 1 May 10:25:14 EDT 2019 Using database binding 'wx_binding', which is bound to database 'archive_sqlite' Wed 1 May 10:25:14 EDT 2019 Weather Underground Station: KMIDEARB5 Wed 1 May 10:25:14 EDT 2019 Date to check: 2019-04-30 Wed 1 May 10:25:14 EDT 2019 Number of archive records: 1438 Wed 1 May 10:25:14 EDT 2019 Number of WU records: 381 Wed 1 May 10:25:14 EDT 2019 Number of missing records: 0 Wed 1 May 10:35:01 EDT 2019 Using configuration file /etc/weewx/weewx.conf. Wed 1 May 10:35:01 EDT 2019 Using database binding 'wx_binding', which is bound to database 'archive_sqlite' Wed 1 May 10:35:01 EDT 2019 Weather Underground Station: KMIDEARB5 Wed 1 May 10:35:01 EDT 2019 Date to check: 2019-05-01 Wed 1 May 10:35:01 EDT 2019 Number of archive records: 634 Wed 1 May 10:35:01 EDT 2019 Number of WU records: 147 Wed 1 May 10:35:01 EDT 2019 Number of missing records: 6 Wed 1 May 10:35:01 EDT 2019 Wed 1 May 10:35:01 EDT 2019 Missing records: Wed 1 May 10:35:01 EDT 2019 2019-05-01 09:57:00 EDT (1556719020); 29.226"; 50.4F; 93%; 2.9 mph; 111 deg; 6.9 mph gust; 48.4F; 0.00" rain ...published. Wed 1 May 10:35:01 EDT 2019 2019-05-01 10:17:00 EDT (1556720220); 29.223"; 52.0F; 92%; 2.0 mph; 136 deg; 3.6 mph gust; 49.9F; 0.00" rain ...published. Wed 1 May 10:35:01 EDT 2019 2019-05-01 10:27:00 EDT (1556720820); 29.214"; 52.9F; 92%; 2.5 mph; 115 deg; 6.5 mph gust; 50.6F; 0.00" rain ...published. Wed 1 May 10:35:01 EDT 2019 2019-05-01 10:32:00 EDT (1556721120); 29.214"; 53.2F; 91%; 2.5 mph; 113 deg; 2.5 mph gust; 50.7F; 0.00" rain ...published. Wed 1 May 10:35:01 EDT 2019 2019-05-01 10:33:00 EDT (1556721180); 29.214"; 53.2F; 91%; 2.5 mph; 113 deg; 4.3 mph gust; 50.7F; 0.00" rain ...published. Wed 1 May 10:35:01 EDT 2019 2019-05-01 10:34:00 EDT (1556721240); 29.207"; 53.4F; 91%; 2.5 mph; 113 deg; 3.6 mph gust; 50.9F; 0.00" rain ...published. And again 10 minutes later: Wed 1 May 10:45:01 EDT 2019 Weather Underground Station: KMIDEARB5 Wed 1 May 10:45:01 EDT 2019 Date to check: 2019-05-01 Wed 1 May 10:45:01 EDT 2019 Number of archive records: 644 Wed 1 May 10:45:01 EDT 2019 Number of WU records: 149 Wed 1 May 10:45:01 EDT 2019 Number of missing records: 6 Wed 1 May 10:45:01 EDT 2019 Wed 1 May 10:45:01 EDT 2019 Missing records: Wed 1 May 10:45:01 EDT 2019 2019-05-01 09:57:00 EDT (1556719020); 29.226"; 50.4F; 93%; 2.9 mph; 111 deg; 6.9 mph gust; 48.4F; 0.00" rain ...published. Wed 1 May 10:45:01 EDT 2019 2019-05-01 10:17:00 EDT (1556720220); 29.223"; 52.0F; 92%; 2.0 mph; 136 deg; 3.6 mph gust; 49.9F; 0.00" rain ...published. Wed 1 May 10:45:01 EDT 2019 2019-05-01 10:27:00 EDT (1556720820); 29.214"; 52.9F; 92%; 2.5 mph; 115 deg; 6.5 mph gust; 50.6F; 0.00" rain ...published. Wed 1 May 10:45:01 EDT 2019 2019-05-01 10:32:00 EDT (1556721120); 29.214"; 53.2F; 91%; 2.5 mph; 113 deg; 2.5 mph gust; 50.7F; 0.00" rain ...published. Wed 1 May 10:45:01 EDT 2019 2019-05-01 10:43:00 EDT (1556721780); 29.202"; 54.1F; 91%; 2.0 mph; 111 deg; 6.5 mph gust; 51.4F; 0.00" rain ...published. Wed 1 May 10:45:01 EDT 2019 2019-05-01 10:44:00 EDT (1556721840); 29.202"; 54.1F; 90%; 2.0 mph; 111 deg; 2.9 mph gust; 51.3F; 0.00" rain ...published. It's like a "rolling window" of missing records that eventually do catch up, but seemingly that is only because I am being persistent about running wunderfixer. I could try running it only every 20 minutes to see if results change. Let me think about that some more. :-/ Meanwhile, about the connection issues, from when wunderfixer timeout was 10 seconds, vs. after changing it to 20 seconds. The timeouts still occur, but less frequently ### timeouts almost every time I run wunderfixer 10 minutes apart, for prior day Tue 30 Apr 09:55:01 EDT 2019 Could not get Weather Underground data. Exiting. Tue 30 Apr 10:45:02 EDT 2019 Could not get Weather Underground data. Exiting. Tue 30 Apr 10:55:02 EDT 2019 Could not get Weather Underground data. Exiting. Tue 30 Apr 11:05:01 EDT 2019 Could not get Weather Underground data. Exiting. Tue 30 Apr 11:15:02 EDT 2019 Could not get Weather Underground data. Exiting. Tue 30 Apr 12:05:01 EDT 2019 Could not get Weather Underground data. Exiting. Tue 30 Apr 12:35:02 EDT 2019 Could not get Weather Underground data. Exiting. Tue 30 Apr 12:45:01 EDT 2019 Could not get Weather Underground data. Exiting. Tue 30 Apr 12:55:02 EDT 2019 Could not get Weather Underground data. Exiting. Tue 30 Apr 13:05:01 EDT 2019 Could not get Weather Underground data. Exiting. Tue 30 Apr 13:15:15 EDT 2019 Could not get Weather Underground data. Exiting. Tue 30 Apr 13:35:01 EDT 2019 Could not get Weather Underground data. Exiting. Tue 30 Apr 13:55:01 EDT 2019 Could not get Weather Underground data. Exiting. Tue 30 Apr 13:55:15 EDT 2019 Could not get Weather Underground data. Exiting. Tue 30 Apr 14:35:12 EDT 2019 Could not get Weather Underground data. Exiting. Tue 30 Apr 14:45:01 EDT 2019 Could not get Weather Underground data. Exiting. Tue 30 Apr 14:55:02 EDT 2019 Could not get Weather Underground data. Exiting. Tue 30 Apr 15:15:01 EDT 2019 Could not get Weather Underground data. Exiting. Tue 30 Apr 15:55:19 EDT 2019 Could not get Weather Underground data. Exiting. Tue 30 Apr 16:05:13 EDT 2019 Could not get Weather Underground data. Exiting. Tue 30 Apr 16:15:19 EDT 2019 Could not get Weather Underground data. Exiting. Tue 30 Apr 16:25:14 EDT 2019 Could not get Weather Underground data. Exiting. Tue 30 Apr 16:35:19 EDT 2019 Could not get Weather Underground data. Exiting. Tue 30 Apr 16:45:02 EDT 2019 Could not get Weather Underground data. Exiting. Tue 30 Apr 16:45:15 EDT 2019 Could not get Weather Underground data. Exiting. ### timeout changed from 10 seconds to 20 seconds; far fewer timeouts occurring Tue 30 Apr 21:55:23 EDT 2019 Could not get Weather Underground data. Exiting. Tue 30 Apr 23:15:33 EDT 2019 Could not get Weather Underground data. Exiting. Wed 1 May 01:15:20 EDT 2019 Could not get Weather Underground data. Exiting. Wed 1 May 01:25:14 EDT 2019 Could not get Weather Underground data. Exiting. Wed 1 May 01:35:15 EDT 2019 Could not get Weather Underground data. Exiting. ### no timeouts seen since that time, so far, today Regards, \Leon -- Leon Shaner :: Dearborn, Michigan (iPad Pro) -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
