Re: [weewx-user] Re: WU outage

[Leon Shaner]

Jamie,

Thanks for the heads up.  Cheers!

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad)

> On Jun 9, 2020, at 8:57 PM, Jamie Stephens  wrote:
> 
> IBM is having a major cloud outage.  Lots of sites impacted 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/89ddfd4c-e519-48c7-8980-23cb9fbf656fo%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/29B1FBCB-3F98-4757-B3E4-F4F3AFC7E437%40isylum.org.

Re: [weewx-user] WU outage

[Leon Shaner]

Greg, thanks, for the confirmation.

As a result of this I took a look at my API key and found that it expired in 
Nov 2019, even though it's been working all this time.  I just renewed it and 
updated it in my config, but that didn't help.

As a side-note, apparently API keys are only good for 6 months.  New one is 
only good through Dec.
Clearly they don't bother to send a reminder about expiry, so I set one 
manually.   Par for the course re: (lack-there-of) quality from IBM, since 
taking over WU.  :-(

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad)

> On Jun 9, 2020, at 8:48 PM, Leon Shaner  wrote:
> 
> Actually, I was only glancing at the chart.
> According to the table, my last record was posted at 7:15 p.m. US/Eastern.
> Anyone else having the same issue?
> 
> Regards,
> \Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPad)
> 
>> On Jun 9, 2020, at 8:31 PM, Leon Shaner  wrote:
>> 
>> FYI, I just happened to be checking my station on WU and noticed it said it 
>> was offline since a little after 6 US/Eastern.  No problem with WeeWx...
>> 
>> This is going on right now:
>> 
>> Jun  9 20:25:37 nixie weewx[357] DEBUG weewx.restx: Wunderground-RF: Failed 
>> upload attempt 1: 
>> Jun  9 20:25:37 nixie weewx[357] ERROR weewx.restx: Wunderground-RF: Failed 
>> to publish record 2020-06-09 20:24:55 EDT (1591748695): Failed upload after 
>> 1 tries
>> Jun  9 20:25:40 nixie weewx[357] INFO weewx.drivers.wmr300: history buffer 
>> at 0.79% (291)
>> Jun  9 20:25:49 nixie weewx[357] DEBUG weewx.restx: Wunderground-PWS: Failed 
>> upload attempt 1: 
>> Jun  9 20:26:07 nixie weewx[357] DEBUG weewx.restx: Wunderground-RF: Failed 
>> upload attempt 1: 
>> Jun  9 20:26:07 nixie weewx[357] ERROR weewx.restx: Wunderground-RF: Failed 
>> to publish record 2020-06-09 20:25:37 EDT (1591748737): Failed upload after 
>> 1 tries
>> Jun  9 20:26:14 nixie weewx[357] INFO weewx.manager: Added record 2020-06-09 
>> 20:26:00 EDT (1591748760) to database 'weewx.sdb'
>> Jun  9 20:26:14 nixie weewx[357] INFO weewx.manager: Added record 2020-06-09 
>> 20:26:00 EDT (1591748760) to daily summary in 'weewx.sdb'
>> Jun  9 20:26:25 nixie weewx[357] DEBUG weewx.restx: Wunderground-PWS: Failed 
>> upload attempt 2: 
>> Jun  9 20:26:37 nixie weewx[357] DEBUG weewx.restx: Wunderground-RF: Failed 
>> upload attempt 1: 
>> Jun  9 20:26:37 nixie weewx[357] ERROR weewx.restx: Wunderground-RF: Failed 
>> to publish record 2020-06-09 20:26:07 EDT (1591748767): Failed upload after 
>> 1 tries
>> Jun  9 20:27:00 nixie weewx[357] DEBUG weewx.restx: Wunderground-PWS: Failed 
>> upload attempt 3: 
>> Jun  9 20:27:00 nixie weewx[357] ERROR weewx.restx: Wunderground-PWS: Failed 
>> to publish record 2020-06-09 20:25:00 EDT (1591748700): Failed upload after 
>> 3 tries
>> Jun  9 20:27:08 nixie weewx[357] DEBUG weewx.restx: Wunderground-RF: Failed 
>> upload attempt 1: 
>> Jun  9 20:27:08 nixie weewx[357] ERROR weewx.restx: Wunderground-RF: Failed 
>> to publish record 2020-06-09 20:26:38 EDT (1591748798): Failed upload after 
>> 1 tries
>> Jun  9 20:27:14 nixie weewx[357] INFO weewx.manager: Added record 2020-06-09 
>> 20:27:00 EDT (1591748820) to database 'weewx.sdb'
>> Jun  9 20:27:14 nixie weewx[357] INFO weewx.manager: Added record 2020-06-09 
>> 20:27:00 EDT (1591748820) to daily summary in 'weewx.sdb'
>> Jun  9 20:27:30 nixie weewx[357] DEBUG weewx.restx: Wunderground-PWS: Failed 
>> upload attempt 1: 
>> Jun  9 20:27:38 nixie weewx[357] DEBUG weewx.restx: Wunderground-RF: Failed 
>> upload attempt 1: 
>> Jun  9 20:27:38 nixie weewx[357] ERROR weewx.restx: Wunderground-RF: Failed 
>> to publish record 2020-06-09 20:27:08 EDT (1591748828): Failed upload after 
>> 1 tries
>> Jun  9 20:28:05 nixie weewx[357] DEBUG weewx.restx: Wunderground-PWS: Failed 
>> upload attempt 2: 
>> Jun  9 20:28:08 nixie weewx[357] DEBUG weewx.restx: Wunderground-RF: Failed 
>> upload attempt 1: 
>> Jun  9 20:28:08 nixie weewx[357] ERROR weewx.restx: Wunderground-RF: Failed 
>> to publish record 2020-06-09 20:27:37 EDT (1591748857): Failed upload after 
>> 1 tries
>> Jun  9 20:28:15 nixie weewx[357] INFO weewx.manager: Added record 2020-06-09 
>> 20:28:00 EDT (1591748880) to database 'weewx.sdb'
>> Jun  9 20:28:15 nixie weewx[357] INFO weewx.manager: Added record 2020-06-09 
>> 20:28:00 EDT (1591748880) to daily summary in 'weewx.sdb'
>> Jun  9 20:28:38 nixie weewx[357] DEBUG weewx.restx: Wunderground-RF: Failed 
>> upload attempt 1: 
>> Jun  9 20:28:38 nixie weewx[357] ERROR weewx.restx: Wunderground-RF: Failed 
>> 

Re: [weewx-user] WU outage

[Leon Shaner]

Actually, I was only glancing at the chart.
According to the table, my last record was posted at 7:15 p.m. US/Eastern.
Anyone else having the same issue?

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad)

> On Jun 9, 2020, at 8:31 PM, Leon Shaner  wrote:
> 
> FYI, I just happened to be checking my station on WU and noticed it said it 
> was offline since a little after 6 US/Eastern.  No problem with WeeWx...
> 
> This is going on right now:
> 
> Jun  9 20:25:37 nixie weewx[357] DEBUG weewx.restx: Wunderground-RF: Failed 
> upload attempt 1: 
> Jun  9 20:25:37 nixie weewx[357] ERROR weewx.restx: Wunderground-RF: Failed 
> to publish record 2020-06-09 20:24:55 EDT (1591748695): Failed upload after 1 
> tries
> Jun  9 20:25:40 nixie weewx[357] INFO weewx.drivers.wmr300: history buffer at 
> 0.79% (291)
> Jun  9 20:25:49 nixie weewx[357] DEBUG weewx.restx: Wunderground-PWS: Failed 
> upload attempt 1: 
> Jun  9 20:26:07 nixie weewx[357] DEBUG weewx.restx: Wunderground-RF: Failed 
> upload attempt 1: 
> Jun  9 20:26:07 nixie weewx[357] ERROR weewx.restx: Wunderground-RF: Failed 
> to publish record 2020-06-09 20:25:37 EDT (1591748737): Failed upload after 1 
> tries
> Jun  9 20:26:14 nixie weewx[357] INFO weewx.manager: Added record 2020-06-09 
> 20:26:00 EDT (1591748760) to database 'weewx.sdb'
> Jun  9 20:26:14 nixie weewx[357] INFO weewx.manager: Added record 2020-06-09 
> 20:26:00 EDT (1591748760) to daily summary in 'weewx.sdb'
> Jun  9 20:26:25 nixie weewx[357] DEBUG weewx.restx: Wunderground-PWS: Failed 
> upload attempt 2: 
> Jun  9 20:26:37 nixie weewx[357] DEBUG weewx.restx: Wunderground-RF: Failed 
> upload attempt 1: 
> Jun  9 20:26:37 nixie weewx[357] ERROR weewx.restx: Wunderground-RF: Failed 
> to publish record 2020-06-09 20:26:07 EDT (1591748767): Failed upload after 1 
> tries
> Jun  9 20:27:00 nixie weewx[357] DEBUG weewx.restx: Wunderground-PWS: Failed 
> upload attempt 3: 
> Jun  9 20:27:00 nixie weewx[357] ERROR weewx.restx: Wunderground-PWS: Failed 
> to publish record 2020-06-09 20:25:00 EDT (1591748700): Failed upload after 3 
> tries
> Jun  9 20:27:08 nixie weewx[357] DEBUG weewx.restx: Wunderground-RF: Failed 
> upload attempt 1: 
> Jun  9 20:27:08 nixie weewx[357] ERROR weewx.restx: Wunderground-RF: Failed 
> to publish record 2020-06-09 20:26:38 EDT (1591748798): Failed upload after 1 
> tries
> Jun  9 20:27:14 nixie weewx[357] INFO weewx.manager: Added record 2020-06-09 
> 20:27:00 EDT (1591748820) to database 'weewx.sdb'
> Jun  9 20:27:14 nixie weewx[357] INFO weewx.manager: Added record 2020-06-09 
> 20:27:00 EDT (1591748820) to daily summary in 'weewx.sdb'
> Jun  9 20:27:30 nixie weewx[357] DEBUG weewx.restx: Wunderground-PWS: Failed 
> upload attempt 1: 
> Jun  9 20:27:38 nixie weewx[357] DEBUG weewx.restx: Wunderground-RF: Failed 
> upload attempt 1: 
> Jun  9 20:27:38 nixie weewx[357] ERROR weewx.restx: Wunderground-RF: Failed 
> to publish record 2020-06-09 20:27:08 EDT (1591748828): Failed upload after 1 
> tries
> Jun  9 20:28:05 nixie weewx[357] DEBUG weewx.restx: Wunderground-PWS: Failed 
> upload attempt 2: 
> Jun  9 20:28:08 nixie weewx[357] DEBUG weewx.restx: Wunderground-RF: Failed 
> upload attempt 1: 
> Jun  9 20:28:08 nixie weewx[357] ERROR weewx.restx: Wunderground-RF: Failed 
> to publish record 2020-06-09 20:27:37 EDT (1591748857): Failed upload after 1 
> tries
> Jun  9 20:28:15 nixie weewx[357] INFO weewx.manager: Added record 2020-06-09 
> 20:28:00 EDT (1591748880) to database 'weewx.sdb'
> Jun  9 20:28:15 nixie weewx[357] INFO weewx.manager: Added record 2020-06-09 
> 20:28:00 EDT (1591748880) to daily summary in 'weewx.sdb'
> Jun  9 20:28:38 nixie weewx[357] DEBUG weewx.restx: Wunderground-RF: Failed 
> upload attempt 1: 
> Jun  9 20:28:38 nixie weewx[357] ERROR weewx.restx: Wunderground-RF: Failed 
> to publish record 2020-06-09 20:28:07 EDT (1591748887): Failed upload after 1 
> tries
> Jun  9 20:28:40 nixie weewx[357] DEBUG weewx.restx: Wunderground-PWS: Failed 
> upload attempt 3: 
> Jun  9 20:28:40 nixie weewx[357] ERROR weewx.restx: Wunderground-PWS: Failed 
> to publish record 2020-06-09 20:26:00 EDT (1591748760): Failed upload after 3 
> tries
> Jun  9 20:29:08 nixie weewx[357] DEBUG weewx.restx: Wunderground-RF: Failed 
> upload attempt 1: 
> Jun  9 20:29:08 nixie weewx[357] ERROR weewx.restx: Wunderground-RF: Failed 
> to publish record 2020-06-09 20:28:38 EDT (1591748918): Failed upload after 1 
> tries
> Jun  9 20:29:10 nixie weewx[357] DEBUG weewx.restx: Wunderground-PWS: Failed 
> upload attempt 1: 
> 
> 
> Regards,
> \Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPad)
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> &qu

[weewx-user] WU outage

[Leon Shaner]

FYI, I just happened to be checking my station on WU and noticed it said it was 
offline since a little after 6 US/Eastern.  No problem with WeeWx...

This is going on right now:

Jun  9 20:25:37 nixie weewx[357] DEBUG weewx.restx: Wunderground-RF: Failed 
upload attempt 1: 
Jun  9 20:25:37 nixie weewx[357] ERROR weewx.restx: Wunderground-RF: Failed to 
publish record 2020-06-09 20:24:55 EDT (1591748695): Failed upload after 1 tries
Jun  9 20:25:40 nixie weewx[357] INFO weewx.drivers.wmr300: history buffer at 
0.79% (291)
Jun  9 20:25:49 nixie weewx[357] DEBUG weewx.restx: Wunderground-PWS: Failed 
upload attempt 1: 
Jun  9 20:26:07 nixie weewx[357] DEBUG weewx.restx: Wunderground-RF: Failed 
upload attempt 1: 
Jun  9 20:26:07 nixie weewx[357] ERROR weewx.restx: Wunderground-RF: Failed to 
publish record 2020-06-09 20:25:37 EDT (1591748737): Failed upload after 1 tries
Jun  9 20:26:14 nixie weewx[357] INFO weewx.manager: Added record 2020-06-09 
20:26:00 EDT (1591748760) to database 'weewx.sdb'
Jun  9 20:26:14 nixie weewx[357] INFO weewx.manager: Added record 2020-06-09 
20:26:00 EDT (1591748760) to daily summary in 'weewx.sdb'
Jun  9 20:26:25 nixie weewx[357] DEBUG weewx.restx: Wunderground-PWS: Failed 
upload attempt 2: 
Jun  9 20:26:37 nixie weewx[357] DEBUG weewx.restx: Wunderground-RF: Failed 
upload attempt 1: 
Jun  9 20:26:37 nixie weewx[357] ERROR weewx.restx: Wunderground-RF: Failed to 
publish record 2020-06-09 20:26:07 EDT (1591748767): Failed upload after 1 tries
Jun  9 20:27:00 nixie weewx[357] DEBUG weewx.restx: Wunderground-PWS: Failed 
upload attempt 3: 
Jun  9 20:27:00 nixie weewx[357] ERROR weewx.restx: Wunderground-PWS: Failed to 
publish record 2020-06-09 20:25:00 EDT (1591748700): Failed upload after 3 tries
Jun  9 20:27:08 nixie weewx[357] DEBUG weewx.restx: Wunderground-RF: Failed 
upload attempt 1: 
Jun  9 20:27:08 nixie weewx[357] ERROR weewx.restx: Wunderground-RF: Failed to 
publish record 2020-06-09 20:26:38 EDT (1591748798): Failed upload after 1 tries
Jun  9 20:27:14 nixie weewx[357] INFO weewx.manager: Added record 2020-06-09 
20:27:00 EDT (1591748820) to database 'weewx.sdb'
Jun  9 20:27:14 nixie weewx[357] INFO weewx.manager: Added record 2020-06-09 
20:27:00 EDT (1591748820) to daily summary in 'weewx.sdb'
Jun  9 20:27:30 nixie weewx[357] DEBUG weewx.restx: Wunderground-PWS: Failed 
upload attempt 1: 
Jun  9 20:27:38 nixie weewx[357] DEBUG weewx.restx: Wunderground-RF: Failed 
upload attempt 1: 
Jun  9 20:27:38 nixie weewx[357] ERROR weewx.restx: Wunderground-RF: Failed to 
publish record 2020-06-09 20:27:08 EDT (1591748828): Failed upload after 1 tries
Jun  9 20:28:05 nixie weewx[357] DEBUG weewx.restx: Wunderground-PWS: Failed 
upload attempt 2: 
Jun  9 20:28:08 nixie weewx[357] DEBUG weewx.restx: Wunderground-RF: Failed 
upload attempt 1: 
Jun  9 20:28:08 nixie weewx[357] ERROR weewx.restx: Wunderground-RF: Failed to 
publish record 2020-06-09 20:27:37 EDT (1591748857): Failed upload after 1 tries
Jun  9 20:28:15 nixie weewx[357] INFO weewx.manager: Added record 2020-06-09 
20:28:00 EDT (1591748880) to database 'weewx.sdb'
Jun  9 20:28:15 nixie weewx[357] INFO weewx.manager: Added record 2020-06-09 
20:28:00 EDT (1591748880) to daily summary in 'weewx.sdb'
Jun  9 20:28:38 nixie weewx[357] DEBUG weewx.restx: Wunderground-RF: Failed 
upload attempt 1: 
Jun  9 20:28:38 nixie weewx[357] ERROR weewx.restx: Wunderground-RF: Failed to 
publish record 2020-06-09 20:28:07 EDT (1591748887): Failed upload after 1 tries
Jun  9 20:28:40 nixie weewx[357] DEBUG weewx.restx: Wunderground-PWS: Failed 
upload attempt 3: 
Jun  9 20:28:40 nixie weewx[357] ERROR weewx.restx: Wunderground-PWS: Failed to 
publish record 2020-06-09 20:26:00 EDT (1591748760): Failed upload after 3 tries
Jun  9 20:29:08 nixie weewx[357] DEBUG weewx.restx: Wunderground-RF: Failed 
upload attempt 1: 
Jun  9 20:29:08 nixie weewx[357] ERROR weewx.restx: Wunderground-RF: Failed to 
publish record 2020-06-09 20:28:38 EDT (1591748918): Failed upload after 1 tries
Jun  9 20:29:10 nixie weewx[357] DEBUG weewx.restx: Wunderground-PWS: Failed 
upload attempt 1: 


Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad)

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/FA59913F-B1CF-4173-9509-A613FB7C37F5%40isylum.org.

Re: [weewx-user] FTPS error in WeeWX version 4.0.0

[Leon Shaner]

Stephen,

Not looking at the code, but shouldn't there be some prefix to these paths?
Nobody is going to let you write to root (/).

> *put* 'MKD /\r\n'
> *put* 'STOR /yearrain.png\r\n'
> *put* 'MKD /font\r\n'
> *put* 'MKD /font\r\n'
> *put* 'MKD /font\r\n'

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad)

> On May 11, 2020, at 1:30 PM, Stephen  wrote:
> 
> 
> 
> Hi
> 
> My WeeWX installation is failing to upload files to my website.  I'm using 
> FTPS and from the logs it looks like the connection to the server is made OK 
> but things go wrong as soon as the first file transfer is attempted.  Using 
> an ftp client I can connect manually and write files successfully using FTPS. 
>  The debug output and logs below are from running wee_reports but it also 
> fails when running as a daemon.
> 
> There is another error when it tries to make the directory /font on the 
> server but I suspect that is a consequence of the earlier failure. At the 
> start the "MKD /" command behaves as expected. In addition there appears to 
> be a problem with the exception handling.
> 
> This is a fresh install using apt-get on Ubuntu 19.10 but I have had the same 
> problem with WeeWX 3.9.2 on Ubuntu 16.04 LTS. Python is version 3.7.5.
> 
> Can anyone help?
> 
> Thanks
> 
> Stephen
> 
> --
> wee_reports output
> --
> Using configuration file /etc/weewx/weewx.conf
> Generating for all time
> *get* '220-- Welcome to Pure-FTPd [privsep] [TLS] --\n'
> *get* '220-You are user number 1 of 50 allowed.\n'
> *get* '220-Local time is now 22:49. Server port: 21.\n'
> *get* '220-This is a private system - No anonymous login\n'
> *get* '220-IPv6 connections are also welcome on this server.\n'
> *get* '220 You will be disconnected after 15 minutes of inactivity.\n'
> *resp* '220-- Welcome to Pure-FTPd [privsep] [TLS] 
> --\n220-You are user number 1 of 50 allowed.\n220-Local time is now 
> 22:49. Server port: 21.\n220-This is a private system - No anonymous 
> login\n220-IPv6 connections are also welcome on this server.\n220 You will be 
> disconnected after 15 minutes of inactivity.'
> *cmd* 'AUTH TLS'
> *put* 'AUTH TLS\r\n'
> *get* '234 AUTH TLS OK.\n'
> *resp* '234 AUTH TLS OK.'
> *cmd* 'USER u...@mydomain.com'
> *put* 'USER u...@mydomain.com\r\n'
> *get* '331 User u...@mydomain.com OK. Password required\n'
> *resp* '331 User u...@mydomain.com OK. Password required'
> *cmd* 'PASS '
> *put* 'PASS \r\n'
> *get* '230 OK. Current restricted directory is /\n'
> *resp* '230 OK. Current restricted directory is /'
> *cmd* 'PBSZ 0'
> *put* 'PBSZ 0\r\n'
> *get* '200 PBSZ=0\n'
> *resp* '200 PBSZ=0'
> *cmd* 'PROT P'
> *put* 'PROT P\r\n'
> *get* '200 Data protection level set to "private"\n'
> *resp* '200 Data protection level set to "private"'
> *cmd* 'MKD /'
> *put* 'MKD /\r\n'
> *get* "550 Can't create directory: File exists\n"
> *resp* "550 Can't create directory: File exists"
> *cmd* 'TYPE I'
> *put* 'TYPE I\r\n'
> *get* '200 TYPE is now 8-bit binary\n'
> *resp* '200 TYPE is now 8-bit binary'
> *cmd* 'PASV'
> *put* 'PASV\r\n'
> *get* '227 Entering Passive Mode (185,24,98,215,210,73)\n'
> *resp* '227 Entering Passive Mode (185,24,98,215,210,73)'
> *cmd* 'STOR /yearrain.png'
> *put* 'STOR /yearrain.png\r\n'
> *get* '150 Accepted data connection\n'
> *resp* '150 Accepted data connection'
> *cmd* 'TYPE I'
> *put* 'TYPE I\r\n'
> *get* ''
> 
> repeats last three lines > 200 times
> 
> *cmd* 'TYPE I'
> *put* 'TYPE I\r\n'
> *get* ''
> *cmd* 'MKD /font'
> *put* 'MKD /font\r\n'
> *get* ''
> *cmd* 'MKD /font'
> *put* 'MKD /font\r\n'
> *get* ''
> *cmd* 'MKD /font'
> *put* 'MKD /font\r\n'
> *get* ''
> *cmd* 'QUIT'
> *put* 'QUIT\r\n'
> *get* ''
> 
> ---
> Log extract
> ---
> May 11 17:15:13 localhost wee_reports[17945] DEBUG weewx.reportengine: 
> Running report 'FTP'
> May 11 17:15:13 localhost wee_reports[17945] DEBUG weewx.reportengine: Found 
> configuration file /etc/weewx/skins/Ftp/skin.conf for report 'FTP'
> May 11 17:15:13 localhost wee_reports[17945] DEBUG weeutil.ftpupload: 
> Attempting secure connection to server.myhost.net
> May 11 17:15:13 localhost wee_reports[17945] DEBUG weeutil.ftpupload: Secure 
> data connection to server.myhost.net
> May 11 17:15:13 localhost wee_reports[17945] ERROR weeutil.ftpupload: Attempt 
> #1. Failed uploading /yearrain.png to server.myhost.net. Reason: [Errno 0] 
> Error
> May 11 17:15:13 localhost wee_reports[17945] ERROR weeutil.ftpupload: Attempt 
> #2. Failed uploading /y

Re: [weewx-user] how to switch from python2 to python3

[Leon Shaner]

Tamo,

I did it in such a way as to avoid changing the system-wide default.
I'm on Raspberry Pi, and several system-related functions still expect 
Python 2.7.


What I did is certainly not the only way, but it works for me...
On my system, /usr/local/sbin is already in the path ahead of system 
default locations and it is set via /etc/profile (but you could do it in 
your ~/.profile or ~/.bashrc or other dot files):


   PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

Based on that, I did the following:

   pi@nixie:~ $ cd /usr/local/sbin
   pi@nixie:/usr/local/sbin $ sudo ln -s ../../bin/python3 python

That yields this behavior:

   pi@nixie:~ $ which python
   /usr/local/sbin/python
   pi@nixie:~ $ env python --version
   Python 3.7.3

And because the majority of weewx python scripts begin with the 
following, it means they will end up using Python 3:


   #!/usr/bin/env python

HTH,

Regards,
\Leon

Tarmo wrote on 4/30/20 2:54 PM:

Hi!

Upgraded successfully to 4.0. Now I would like to switch to Pyhton3. 
Is the clean install preferred option?


running raspi & buster + few extensions
--
You received this message because you are subscribed to the Google 
Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to weewx-user+unsubscr...@googlegroups.com 
.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/a44ca17c-eb35-494b-bdbf-a8d4799207e3%40googlegroups.com 
.



--
l...@isylum.org - Dearborn, Michigan

--
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/c13d7d8a-5e62-2e9a-4db7-58f41acfa093%40isylum.org.

Re: [weewx-user] Wunderfixer (Development Version)

[Leon Shaner]

Hey, Ernest.

If you check after some time has passed are the records there?
There is always a "lag" between the time the records are uploaded and when they 
appear.
Also, be sure to check via the WU web portal and not some app like 
WunderStation (which they've deprecated).

Anyway, the wunderfixer is doing its job.  What happens on the WU side is 
subject to their many bugs and infrastructure issues.  :-/

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad)

> On Feb 9, 2020, at 9:59 PM, Ernest Jillson  wrote:
> 
> 
> It "looks" like it's working for me, but doesn't.  I get this message:
>  
> "No results returned from Weather Underground (perhaps a bad station name??).
> Publishing anyway."
>  
> It then proceeds to upload all my 5 minute obs.  Only thing is, they never 
> show up on wunderground.
>  
> 
> 
>> On Sunday, February 9, 2020 at 4:02:21 PM UTC-5, Richard G wrote:
>> 
>> Worked for me thanks.
>> 
>> Richard
>>> On Sunday, 9 February 2020 18:19:00 UTC, Leon Shaner wrote:
>>> Richard,
>>> It was a long thread.  Here is my workaround for the 204 response when the 
>>> request is valid, but there are no records to return:
>>> 
>>> $ diff wunderfixer wunderfixer_tk
>>> 407,409c407
>>> < # Valid response, it's just that WU has no records for 
>>> the requested date
>>> < # Return an empty list of TimeStamps (as in WU has no 
>>> records)
>>> < return {}
>>> ---
>>> > raise IOError("Probably a bad station ID or invalid date")
>>> 
>>> Or if the line numbers don't match yours, it looks like this in context:
>>> 
>>> if hasattr(response, 'code') and response.code != 200:
>>> if response.code == 204:
>>> # Valid response, it's just that WU has no records for the 
>>> requested date
>>> # Return an empty list of TimeStamps (as in WU has no 
>>> records)
>>> return {}
>>> else:
>>> raise IOError("Bad response code returned: %d" % 
>>> response.code)
>>> 
>>> Regards,
>>> \Leon
>>> --
>>> Leon Shaner :: Dearborn, Michigan (iPhone)
>>> 
>>>>> On Feb 9, 2020, at 12:40 PM, Richard G  wrote:
>>>>> 
>>>> 
>>>> I had an outage of a week where no data was uploaded to wunderground. I 
>>>> discovered all the posts about the change of API etc. so pulled down the 
>>>> development branch. This version with the api_key allowed me to fix data 
>>>> on partial days i.e. when there was some data but on days where there is 
>>>> no data at all then it fails
>>>> 
>>>> Using database binding 'wx_binding', which is bound to database 
>>>> 'archive_sqlite'
>>>> 
>>>> Weather Underground Station:   INORFOLK**
>>>> 
>>>> Date to check: 2020-02-07
>>>> 
>>>> Number of archive records: 288
>>>> 
>>>> Could not get Weather Underground data.
>>>> 
>>>> Reason: Probably a bad station ID or invalid date
>>>> 
>>>> Exiting.
>>>> 
>>>> 
>>>> The message "could not get Weather Underground data" is correct as there 
>>>> isn't any for that day, but it then needs to upload so there is. Looks 
>>>> like a bug but I thought I would check before reporting.
>>>> 
>>>> Thanks
>>>> 
>>>> 
>>>> Richard
>>>> -- 
>>>> You received this message because you are subscribed to the Google Groups 
>>>> "weewx-user" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>>> email to weewx...@googlegroups.com.
>>>> To view this discussion on the web visit 
>>>> https://groups.google.com/d/msgid/weewx-user/04c6f04e-502a-40a2-ad9d-62a4fad5a8b5%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/30ac8208-34d8-4b47-a8ac-636151c794c2%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/2417BCC3-A907-4DF6-B374-6CF6DD577657%40isylum.org.

Re: [weewx-user] Wunderfixer (Development Version)

[Leon Shaner]

Richard,
It was a long thread.  Here is my workaround for the 204 response when the 
request is valid, but there are no records to return:

$ diff wunderfixer wunderfixer_tk
407,409c407
< # Valid response, it's just that WU has no records for the 
requested date
< # Return an empty list of TimeStamps (as in WU has no records)
< return {}
---
> raise IOError("Probably a bad station ID or invalid date")

Or if the line numbers don't match yours, it looks like this in context:

if hasattr(response, 'code') and response.code != 200:
if response.code == 204:
# Valid response, it's just that WU has no records for the 
requested date
# Return an empty list of TimeStamps (as in WU has no records)
return {}
else:
raise IOError("Bad response code returned: %d" % response.code)

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPhone)

> On Feb 9, 2020, at 12:40 PM, Richard G  wrote:
> 
> 
> I had an outage of a week where no data was uploaded to wunderground. I 
> discovered all the posts about the change of API etc. so pulled down the 
> development branch. This version with the api_key allowed me to fix data on 
> partial days i.e. when there was some data but on days where there is no data 
> at all then it fails
> 
> Using database binding 'wx_binding', which is bound to database 
> 'archive_sqlite'
> 
> Weather Underground Station:   INORFOLK**
> 
> Date to check: 2020-02-07
> 
> Number of archive records: 288
> 
> Could not get Weather Underground data.
> 
> Reason: Probably a bad station ID or invalid date
> 
> Exiting.
> 
> 
> The message "could not get Weather Underground data" is correct as there 
> isn't any for that day, but it then needs to upload so there is. Looks like a 
> bug but I thought I would check before reporting.
> 
> Thanks
> 
> 
> Richard
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/04c6f04e-502a-40a2-ad9d-62a4fad5a8b5%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/01A0DEC2-AFB3-400F-A30B-AE517FAB4BC9%40isylum.org.

Re: [weewx-user] Uploading missing data to Weather Underground

[Leon Shaner]

Ernest, you would need to pull a newer version of 3.x to avoid this 
error and for api_key to have any meaning.


Ernest Jillson wrote on 2/6/20 4:21 PM:
Definitely does cause weewx to fail. Keep in mind that we are adding 
the key to version 3.x, not 4, then running the wunderfixer pointing 
at our old weewx.conf file.



|
Feb  6 15:56:56 LakeFantasia weewx[21278]: restx: WU essentials: {}
Feb  6 15:56:56 LakeFantasia weewx[21278]: engine: Caught 
unrecoverable exception in engine:
Feb  6 15:56:56 LakeFantasia weewx[21278]:       __init__() got an 
unexpected keyword argument 'api_key'
Feb  6 15:56:56 LakeFantasia weewx[21278]:       Traceback (most 
recent call last):
Feb  6 15:56:56 LakeFantasia weewx[21278]:         File 
"/usr/share/weewx/weewx/engine.py", line 888, in main
Feb  6 15:56:56 LakeFantasia weewx[21278]:           engine = 
engine_class(config_dict)
Feb  6 15:56:56 LakeFantasia weewx[21278]:         File 
"/usr/share/weewx/weewx/engine.py", line 78, in __init__
Feb  6 15:56:56 LakeFantasia weewx[21278]:      
self.loadServices(config_dict)
Feb  6 15:56:56 LakeFantasia weewx[21278]:         File 
"/usr/share/weewx/weewx/engine.py", line 142, in loadServices
Feb  6 15:56:56 LakeFantasia weewx[21278]:      
self.service_obj.append(weeutil.weeutil._get_object(svc)(self, 
config_dict))
Feb  6 15:56:56 LakeFantasia weewx[21278]:         File 
"/usr/share/weewx/weewx/restx.py", line 632, in __init__

Feb  6 15:56:56 LakeFantasia weewx[21278]:      **_ambient_dict)
Feb  6 15:56:56 LakeFantasia weewx[21278]:       TypeError: 
__init__() got an unexpected keyword argument 'api_key'

Feb  6 15:56:56 LakeFantasia weewx[21278]:       Exiting.

|



On Thursday, February 6, 2020 at 3:12:29 PM UTC-5, gjr80 wrote:

Hi,

Simply adding a key to weewx.conf should not cause any problems.
Would you please give us some details (console output, log extract
etc) so we can see if indeed there is a problem with WeeWX and if
so fix it.

Gary

--
You received this message because you are subscribed to the Google 
Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to weewx-user+unsubscr...@googlegroups.com 
.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/ab22ae83-46b6-4a4b-a8ea-6d58a4263c38%40googlegroups.com 
.



--
l...@isylum.org - Dearborn, Michigan

--
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/411635df-49d5-ceba-89cd-1690efcdaca3%40isylum.org.

Re: [weewx-user] Re: rtupdate.wunderground.com certificate error

[Leon Shaner]

FWIW, I haven't seen any 404's since around these times from last night:

Feb  4 20:43:52 Wunderground-RF: Failed upload attempt 1: HTTP Error 404: Not 
Found
Feb  4 20:43:52 Wunderground-RF: Failed to publish record 2020-02-04 20:43:53 
EST (1580867033): Failed upload after 1 tries
Feb  4 20:49:10 Wunderground-RF: Failed upload attempt 1: 
Feb  4 20:49:10 Wunderground-RF: Failed to publish record 2020-02-04 20:48:49 
EST (1580867329): Failed upload after 1 tries
Feb  4 20:49:45 Wunderground-RF: Failed upload attempt 1: 
Feb  4 20:49:45 Wunderground-RF: Failed to publish record 2020-02-04 20:49:15 
EST (1580867355): Failed upload after 1 tries
Feb  4 20:50:19 Wunderground-RF: Failed upload attempt 1: 
Feb  4 20:50:19 Wunderground-RF: Failed to publish record 2020-02-04 20:49:49 
EST (1580867389): Failed upload after 1 tries
Feb  4 21:04:41 Wunderground-RF: Failed upload attempt 1: 
Feb  4 21:04:41 Wunderground-RF: Failed to publish record 2020-02-04 21:04:41 
EST (1580868281): Failed upload after 1 tries
Feb  4 21:04:44 Wunderground-RF: Failed upload attempt 1: HTTP Error 404: Not 
Found


Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad)

> On Feb 4, 2020, at 9:15 PM, Thomas Keffer  wrote:
> 
> 
> I agree with your conclusions. I learned a long time ago not to waste too 
> much energy chasing WU flaws. There are many, and they come and go. Good way 
> to frustrate yourself...
> 
>> On Mon, Feb 3, 2020 at 4:32 PM Leon Shaner  wrote:
>> Tom, All,
>> 
>> On that WU forum, an official rep said the rtupdate and weatherstation URL's 
>> go the same place.
>> He said it shouldn't matter which one.  So we can converge on a single URL.
>> 
>> This is down to same old same old "WU is a steaming pile" being the reason 
>> for the 404's.
>> 
>> Regards,
>> \Leon
>> --
>> Leon Shaner :: Dearborn, Michigan (iPad)
>> 
>>>> On Feb 3, 2020, at 3:35 PM, Leon Shaner  wrote:
>>>> 
>>>  Hey, WeeWx'ers.
>>> 
>>> Even my workaround of setting pws_url to the value of the rf_url is still 
>>> getting 404's, even if I explicitly set a Host header.   :-(
>>> 
>>> This is what I have tried...  I guess WU is just a steaming pile as per 
>>> usual.  :-/
>>> 
>>> Here is my attempted fix, which is part hack.
>>> The "fix" is adding the header.   The "hack" is forcing the code to use the 
>>> rtupdate URL (because I couldn't spot why the PWS URL was being used for 
>>> RF).
>>> 
>>> $ diff restx.py restx.py_getting404 
>>> 100d99
>>> < from urllib.parse import urlparse
>>> 454,458c453
>>> < _parseduri = urlparse(url)
>>> < _parsedhost = '{uri.netloc}'.format(uri=_parseduri)
>>> < _request.add_header("Host", "%s" % _parsedhost)
>>> < #log.debug("%s: Added Header: 'Host: %s'"
>>> < #  % (self.protocol_name, _parsedhost))
>>> ---
>>> > _request.add_header("User-Agent", "weewx/%s" % weewx.__version__)
>>> 610,611c605
>>> < #pws_url = 
>>> "https://weatherstation.wunderground.com/weatherstation/updateweatherstation.php;
>>> < pws_url = 
>>> "https://rtupdate.wunderground.com/weatherstation/updateweatherstation.php;
>>> ---
>>> > pws_url = 
>>> > "https://weatherstation.wunderground.com/weatherstation/updateweatherstation.php;
>>> 
>>> Or in case your line numbers don't match mine (likely), here it is in 
>>> context:
>>> ...
>>> # Python 2/3 compatiblity shims
>>> import six
>>> from six.moves import http_client
>>> from six.moves import queue
>>> from six.moves import urllib
>>> from urllib.parse import urlparse
>>> 
>>> ...
>>> 
>>> def get_request(self, url):
>>> """Get a request object. This can be overridden to add any special 
>>> headers."""
>>> _request = urllib.request.Request(url)
>>> _parseduri = urlparse(url)
>>> _parsedhost = '{uri.netloc}'.format(uri=_parseduri)
>>> _request.add_header("Host", "%s" % _parsedhost)
>>> #log.debug("%s: Added Header: 'Host: %s'"
>>> #  % (self.protocol_name, _parsedhost))
>>> _request.add_header("User-Agent", "weewx/%s" % weewx.__version__)
>>> return _request
>>> 
>>> 
>>>

Re: [weewx-user] Re: rtupdate.wunderground.com certificate error

[Leon Shaner]

Tom, All,

On that WU forum, an official rep said the rtupdate and weatherstation URL's go 
the same place.
He said it shouldn't matter which one.  So we can converge on a single URL.

This is down to same old same old "WU is a steaming pile" being the reason for 
the 404's.

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad)

> On Feb 3, 2020, at 3:35 PM, Leon Shaner  wrote:
> 
>  Hey, WeeWx'ers.
> 
> Even my workaround of setting pws_url to the value of the rf_url is still 
> getting 404's, even if I explicitly set a Host header.   :-(
> 
> This is what I have tried...  I guess WU is just a steaming pile as per 
> usual.  :-/
> 
> Here is my attempted fix, which is part hack.
> The "fix" is adding the header.   The "hack" is forcing the code to use the 
> rtupdate URL (because I couldn't spot why the PWS URL was being used for RF).
> 
> $ diff restx.py restx.py_getting404 
> 100d99
> < from urllib.parse import urlparse
> 454,458c453
> < _parseduri = urlparse(url)
> < _parsedhost = '{uri.netloc}'.format(uri=_parseduri)
> < _request.add_header("Host", "%s" % _parsedhost)
> < #log.debug("%s: Added Header: 'Host: %s'"
> < #  % (self.protocol_name, _parsedhost))
> ---
> > _request.add_header("User-Agent", "weewx/%s" % weewx.__version__)
> 610,611c605
> < #pws_url = 
> "https://weatherstation.wunderground.com/weatherstation/updateweatherstation.php;
> < pws_url = 
> "https://rtupdate.wunderground.com/weatherstation/updateweatherstation.php;
> ---
> > pws_url = 
> > "https://weatherstation.wunderground.com/weatherstation/updateweatherstation.php;
> 
> Or in case your line numbers don't match mine (likely), here it is in context:
> ...
> # Python 2/3 compatiblity shims
> import six
> from six.moves import http_client
> from six.moves import queue
> from six.moves import urllib
> from urllib.parse import urlparse
> 
> ...
> 
> def get_request(self, url):
> """Get a request object. This can be overridden to add any special 
> headers."""
> _request = urllib.request.Request(url)
> _parseduri = urlparse(url)
> _parsedhost = '{uri.netloc}'.format(uri=_parseduri)
> _request.add_header("Host", "%s" % _parsedhost)
> #log.debug("%s: Added Header: 'Host: %s'"
> #  % (self.protocol_name, _parsedhost))
> _request.add_header("User-Agent", "weewx/%s" % weewx.__version__)
> return _request
> 
> 
> ...
> 
> # the rapidfire URL:
> rf_url = 
> "https://rtupdate.wunderground.com/weatherstation/updateweatherstation.php;
> # the personal weather station URL:
> #pws_url = 
> "https://weatherstation.wunderground.com/weatherstation/updateweatherstation.php;
> pws_url = 
> "https://rtupdate.wunderground.com/weatherstation/updateweatherstation.php;
> 
> 
> 
> 
> l...@isylum.org - Dearborn, Michigan
>> On 2/3/20 2:43 PM, Leon Shaner wrote:
>> Tom,
>> 
>> Looking at the code, under the do_rapidfire_post, we clearly see it uses 
>> rf_url.  In my debug output I referenced protocol_name and it's showing 
>> Wunderground-RF as the protocol, but the hostname is from pws_url.
>> I have temporarily set pws_url to the rtupdate URL and my 404's have stopped.
>> Looks like the rtupdate URL will take regular posts, but the weatherstation 
>> URL won't take rf posts.  :-/
>> 
>> My debug with pws_url replaced with the rf_url value now shows:
>> Feb  3 14:36:30 nixie weewx[8747] DEBUG weewx.restx: Wunderground-RF: Added 
>> Header: 'Host: rtupdate.wunderground.com'
>> That's what it should have shown if rf_url was getting used for 
>> do_rapidfire_post.
>> I'm stumped.  :-(
>> 
>> Regards,
>> \Leon
>> l...@isylum.org - Dearborn, Michigan
>>> On 2/3/20 2:23 PM, Leon Shaner wrote:
>>> Brice,
>>> 
>>> Thanks!  That's interesting...  I just added an explicit Host header to the 
>>> rtupdate request and put debug output there.
>>> This was unexpected:
>>> Feb  3 14:16:34 nixie weewx[8584] DEBUG weewx.restx: Wunderground-RF: Added 
>>> Header: 'Host: weatherstation.wunderground.com'
>>> That isn't the host that should be used for rapidfire, but it's taken from 
>>> the URL being used.  :-(
>>> We have:
>>> $ grep wunderground.com restx.py
>>> rf_url = 
>>> "https://rtupdate.wunderground.

Re: [weewx-user] Re: rtupdate.wunderground.com certificate error

[Leon Shaner]

Hey, WeeWx'ers.

Even my workaround of setting pws_url to the value of the rf_url is 
still getting 404's, even if I explicitly set a Host header. :-(


This is what I have tried...  I guess WU is just a steaming pile as per 
usual.  :-/


Here is my attempted fix, which is part hack.
The "fix" is adding the header.   The "hack" is forcing the code to use 
the rtupdate URL (because I couldn't spot why the PWS URL was being used 
for RF).


   $ diff restx.py restx.py_getting404
   100d99
   < from urllib.parse import urlparse
   454,458c453
   < _parseduri = urlparse(url)
   < _parsedhost = '{uri.netloc}'.format(uri=_parseduri)
   < _request.add_header("Host", "%s" % _parsedhost)
   < #    log.debug("%s: Added Header: 'Host: %s'"
   < #  % (self.protocol_name, _parsedhost))
   ---
> _request.add_header("User-Agent", "weewx/%s" %
   weewx.__version__)
   610,611c605
   < #pws_url =
   
"https://weatherstation.wunderground.com/weatherstation/updateweatherstation.php;
   < pws_url =
   "https://rtupdate.wunderground.com/weatherstation/updateweatherstation.php;
   ---
> pws_url =
   
"https://weatherstation.wunderground.com/weatherstation/updateweatherstation.php;

Or in case your line numbers don't match mine (likely), here it is in 
context:


   ...
   # Python 2/3 compatiblity shims
   import six
   from six.moves import http_client
   from six.moves import queue
   from six.moves import urllib
   from urllib.parse import urlparse

   ...

    def get_request(self, url):
    """Get a request object. This can be overridden to add any
   special headers."""
    _request = urllib.request.Request(url)
    _parseduri = urlparse(url)
    _parsedhost = '{uri.netloc}'.format(uri=_parseduri)
    _request.add_header("Host", "%s" % _parsedhost)
   #    log.debug("%s: Added Header: 'Host: %s'"
   #  % (self.protocol_name, _parsedhost))
    _request.add_header("User-Agent", "weewx/%s" %
   weewx.__version__)
    return _request


   ...

    # the rapidfire URL:
    rf_url =
   "https://rtupdate.wunderground.com/weatherstation/updateweatherstation.php;
    # the personal weather station URL:
    #pws_url =
   
"https://weatherstation.wunderground.com/weatherstation/updateweatherstation.php;
    pws_url =
   "https://rtupdate.wunderground.com/weatherstation/updateweatherstation.php;




l...@isylum.org - Dearborn, Michigan

On 2/3/20 2:43 PM, Leon Shaner wrote:

Tom,

Looking at the code, under the do_rapidfire_post, we clearly see it 
uses rf_url.  In my debug output I referenced protocol_name and it's 
showing Wunderground-RF as the protocol, but the hostname is from pws_url.
I have temporarily set pws_url to the rtupdate URL and my 404's have 
stopped.
Looks like the rtupdate URL will take regular posts, but the 
weatherstation URL won't take rf posts.  :-/


My debug with pws_url replaced with the rf_url value now shows:

Feb  3 14:36:30 nixie weewx[8747] DEBUG weewx.restx:
Wunderground-RF: Added Header: 'Host: rtupdate.wunderground.com'

That's what it should have shown if rf_url was getting used for 
do_rapidfire_post.

I'm stumped.  :-(

Regards,
\Leon
l...@isylum.org  - Dearborn, Michigan
On 2/3/20 2:23 PM, Leon Shaner wrote:

Brice,

Thanks!  That's interesting...  I just added an explicit Host header 
to the rtupdate request and put debug output there.

This was unexpected:

Feb  3 14:16:34 nixie weewx[8584] DEBUG weewx.restx:
Wunderground-RF: Added Header: 'Host:
/*weatherstation.wunderground.com*/'

That isn't the host that should be used for rapidfire, but it's taken 
from the URL being used.  :-(

We have:

$ grep wunderground.com restx.py
    rf_url =
"https://rtupdate.wunderground.com/weatherstation/updateweatherstation.php;
    pws_url =

"https://weatherstation.wunderground.com/weatherstation/updateweatherstation.php;

It will take me some time to figure out why restx.py is using the 
wrong URL.
Must have been doing this for a long time, but seems that due to 
changes on the WU side, they no longer accept rapidfire updates to 
the weatherstation.wunderground.com host.


Regards,
\Leon
l...@isylum.org  - Dearborn, Michigan
On 2/3/20 12:59 PM, Brice Ruth wrote:
header would be something like `Host: rtupdate.wunderground.com 
<http://rtupdate.wunderground.com>` ... basically, whatever you're 
using as the hostname in the URL, needs to be sent as the `Host` 
header ... that allows load-balancers to route the request to the 
correct backend.


Brice Ruth, FCD
Software Engineer, Madison WI


On Mon, Feb 3, 2020 at 11:44 AM Leon Shaner <mailto:l...@isyl

Re: [weewx-user] Re: rtupdate.wunderground.com certificate error

[Leon Shaner]

Tom,

Looking at the code, under the do_rapidfire_post, we clearly see it uses 
rf_url.  In my debug output I referenced protocol_name and it's showing 
Wunderground-RF as the protocol, but the hostname is from pws_url.
I have temporarily set pws_url to the rtupdate URL and my 404's have 
stopped.
Looks like the rtupdate URL will take regular posts, but the 
weatherstation URL won't take rf posts.  :-/


My debug with pws_url replaced with the rf_url value now shows:

   Feb  3 14:36:30 nixie weewx[8747] DEBUG weewx.restx:
   Wunderground-RF: Added Header: 'Host: rtupdate.wunderground.com'

That's what it should have shown if rf_url was getting used for 
do_rapidfire_post.

I'm stumped.  :-(

Regards,
\Leon

l...@isylum.org - Dearborn, Michigan

On 2/3/20 2:23 PM, Leon Shaner wrote:

Brice,

Thanks!  That's interesting...  I just added an explicit Host header 
to the rtupdate request and put debug output there.

This was unexpected:

Feb  3 14:16:34 nixie weewx[8584] DEBUG weewx.restx:
Wunderground-RF: Added Header: 'Host:
/*weatherstation.wunderground.com*/'

That isn't the host that should be used for rapidfire, but it's taken 
from the URL being used.  :-(

We have:

$ grep wunderground.com restx.py
    rf_url =
"https://rtupdate.wunderground.com/weatherstation/updateweatherstation.php;
    pws_url =

"https://weatherstation.wunderground.com/weatherstation/updateweatherstation.php;

It will take me some time to figure out why restx.py is using the 
wrong URL.
Must have been doing this for a long time, but seems that due to 
changes on the WU side, they no longer accept rapidfire updates to the 
weatherstation.wunderground.com host.


Regards,
\Leon
l...@isylum.org  - Dearborn, Michigan
On 2/3/20 12:59 PM, Brice Ruth wrote:
header would be something like `Host: rtupdate.wunderground.com 
<http://rtupdate.wunderground.com>` ... basically, whatever you're 
using as the hostname in the URL, needs to be sent as the `Host` 
header ... that allows load-balancers to route the request to the 
correct backend.


Brice Ruth, FCD
Software Engineer, Madison WI


On Mon, Feb 3, 2020 at 11:44 AM Leon Shaner <mailto:l...@isylum.org>> wrote:


Hey, WeeWx'ers,

I am still getting 404 errors from the rtupdate URL site.
There is a reply over here about it.
I'm now looking into what the headers need to be.
Does anyone know?


https://apicommunity.wunderground.com/weatherapi/topics/is-the-rtupdate-wunderground-com-address-no-longer-valid


For requests to route properly, the
host header must be set correctly
path must be correct


Regards,
    \Leon
--
Leon Shaner :: Dearborn, Michigan (iPad)


On Feb 1, 2020, at 12:39 PM, Leon Shaner mailto:l...@isylum.org>> wrote:

Hey, WeeWx'ers.

I took a different approach to working around the WU certificate
issue -- keep SSL but don't verify the CERT.  Scroll for that
"solution."

But meanwhile, using Wunderground-RF (rapid fire) I am getting
an HTTP Error 404:  Not Found.
Any ideas on that?   Did they change the URL, or is it just down?

I noticed that even their own WunderStation App on iOS/iPadOS is
subject to the certificate issue and reports all stations as
"not reporting" -- even ones that are, like mine, which I can
verify via the WU website has the data and has reported within
minutes ago.

From the logs, re: RapidFire HTTP 404:

Feb  1 12:31:51 nixie weewx[13184] DEBUG weewx.restx:
Wunderground-RF: Failed upload attempt 1: HTTP Error 404: Not Found
Feb  1 12:31:51 nixie weewx[13184] ERROR weewx.restx:
Wunderground-RF: Failed to publish record 2020-02-01 12:31:51
EST (1580578311): Failed upload after 1 tries
Feb  1 12:31:56 nixie weewx[13184] DEBUG weewx.restx:
Wunderground-RF: Failed upload attempt 1: HTTP Error 404: Not Found
Feb  1 12:31:56 nixie weewx[13184] ERROR weewx.restx:
Wunderground-RF: Failed to publish record 2020-02-01 12:31:56
EST (1580578316): Failed upload after 1 tries
Feb  1 12:32:04 nixie weewx[13184] DEBUG weewx.restx:
Wunderground-RF: Failed upload attempt 1: HTTP Error 404: Not Found
Feb  1 12:32:04 nixie weewx[13184] ERROR weewx.restx:
Wunderground-RF: Failed to publish record 2020-02-01 12:32:04
EST (1580578324): Failed upload after 1 tries
Feb  1 12:32:05 nixie weewx[13184] DEBUG weewx.restx:
Wunderground-RF: Failed upload attempt 1: HTTP Error 404: Not Found
Feb  1 12:32:05 nixie weewx[13184] ERROR weewx.restx:
Wunderground-RF: Failed to publish record 2020-02-01 12:32:05
EST (1580578325): Failed upload after 1 tries
Feb  1 12:32:06 nixie weewx[13184] DEBUG weewx.restx:
Wunderground-RF: Failed upload attempt 1: HTTP Error 404: Not Found
Feb  1 12:32:06 nixie weewx[13184] ERROR weewx.restx:
Wunderground-RF: Failed to publish record 2020-02-01 12:32:05
EST (

Re: [weewx-user] Re: rtupdate.wunderground.com certificate error

[Leon Shaner]

Brice,

Thanks!  That's interesting...  I just added an explicit Host header to 
the rtupdate request and put debug output there.

This was unexpected:

   Feb  3 14:16:34 nixie weewx[8584] DEBUG weewx.restx:
   Wunderground-RF: Added Header: 'Host:
   /*weatherstation.wunderground.com*/'

That isn't the host that should be used for rapidfire, but it's taken 
from the URL being used.  :-(

We have:

   $ grep wunderground.com restx.py
    rf_url =
   "https://rtupdate.wunderground.com/weatherstation/updateweatherstation.php;
    pws_url =
   
"https://weatherstation.wunderground.com/weatherstation/updateweatherstation.php;

It will take me some time to figure out why restx.py is using the wrong URL.
Must have been doing this for a long time, but seems that due to changes 
on the WU side, they no longer accept rapidfire updates to the 
weatherstation.wunderground.com host.


Regards,
\Leon

l...@isylum.org - Dearborn, Michigan

On 2/3/20 12:59 PM, Brice Ruth wrote:
header would be something like `Host: rtupdate.wunderground.com 
<http://rtupdate.wunderground.com>` ... basically, whatever you're 
using as the hostname in the URL, needs to be sent as the `Host` 
header ... that allows load-balancers to route the request to the 
correct backend.


Brice Ruth, FCD
Software Engineer, Madison WI


On Mon, Feb 3, 2020 at 11:44 AM Leon Shaner <mailto:l...@isylum.org>> wrote:


Hey, WeeWx'ers,

I am still getting 404 errors from the rtupdate URL site.
There is a reply over here about it.
I'm now looking into what the headers need to be.
Does anyone know?


https://apicommunity.wunderground.com/weatherapi/topics/is-the-rtupdate-wunderground-com-address-no-longer-valid


For requests to route properly, the
host header must be set correctly
path must be correct


Regards,
    \Leon
--
Leon Shaner :: Dearborn, Michigan (iPad)


On Feb 1, 2020, at 12:39 PM, Leon Shaner mailto:l...@isylum.org>> wrote:

Hey, WeeWx'ers.

I took a different approach to working around the WU certificate
issue -- keep SSL but don't verify the CERT.  Scroll for that
"solution."

But meanwhile, using Wunderground-RF (rapid fire) I am getting an
HTTP Error 404:  Not Found.
Any ideas on that?   Did they change the URL, or is it just down?

I noticed that even their own WunderStation App on iOS/iPadOS is
subject to the certificate issue and reports all stations as "not
reporting" -- even ones that are, like mine, which I can verify
via the WU website has the data and has reported within minutes ago.

From the logs, re: RapidFire HTTP 404:

Feb  1 12:31:51 nixie weewx[13184] DEBUG weewx.restx:
Wunderground-RF: Failed upload attempt 1: HTTP Error 404: Not Found
Feb  1 12:31:51 nixie weewx[13184] ERROR weewx.restx:
Wunderground-RF: Failed to publish record 2020-02-01 12:31:51 EST
(1580578311): Failed upload after 1 tries
Feb  1 12:31:56 nixie weewx[13184] DEBUG weewx.restx:
Wunderground-RF: Failed upload attempt 1: HTTP Error 404: Not Found
Feb  1 12:31:56 nixie weewx[13184] ERROR weewx.restx:
Wunderground-RF: Failed to publish record 2020-02-01 12:31:56 EST
(1580578316): Failed upload after 1 tries
Feb  1 12:32:04 nixie weewx[13184] DEBUG weewx.restx:
Wunderground-RF: Failed upload attempt 1: HTTP Error 404: Not Found
Feb  1 12:32:04 nixie weewx[13184] ERROR weewx.restx:
Wunderground-RF: Failed to publish record 2020-02-01 12:32:04 EST
(1580578324): Failed upload after 1 tries
Feb  1 12:32:05 nixie weewx[13184] DEBUG weewx.restx:
Wunderground-RF: Failed upload attempt 1: HTTP Error 404: Not Found
Feb  1 12:32:05 nixie weewx[13184] ERROR weewx.restx:
Wunderground-RF: Failed to publish record 2020-02-01 12:32:05 EST
(1580578325): Failed upload after 1 tries
Feb  1 12:32:06 nixie weewx[13184] DEBUG weewx.restx:
Wunderground-RF: Failed upload attempt 1: HTTP Error 404: Not Found
Feb  1 12:32:06 nixie weewx[13184] ERROR weewx.restx:
Wunderground-RF: Failed to publish record 2020-02-01 12:32:05 EST
(1580578325): Failed upload after 1 tries


Here is the diff for my SSL CERT workaround:

$ diff restx.py restx.py.20200201.1
110,115d109
< # SSL certificate hack
< global WUssl
< WUssl = ssl.create_default_context();
< WUssl.check_hostname=False
< WUssl.verify_mode=ssl.CERT_NONE
<
454d447
< _request.add_header("User-Agent", "weewx/%s" % weewx.__version__)
543,544c536
< #        _response = urllib.request.urlopen(request,
data=data_bytes, timeout=self.timeout)
<         _response = urllib.request.urlopen(request,
data=data_bytes, timeout=self.timeout, context=WUssl)
---
>         _response = urllib.request.urlopen(request,
data=data_bytes, timeout=self.timeout)
1

Re: [weewx-user] Re: rtupdate.wunderground.com certificate error

[Leon Shaner]

Hey, WeeWx'ers,

I am still getting 404 errors from the rtupdate URL site.
There is a reply over here about it.
I'm now looking into what the headers need to be.
Does anyone know?

https://apicommunity.wunderground.com/weatherapi/topics/is-the-rtupdate-wunderground-com-address-no-longer-valid

> For requests to route properly, the
> host header must be set correctly
> path must be correct

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad)

> On Feb 1, 2020, at 12:39 PM, Leon Shaner  wrote:
> 
> Hey, WeeWx'ers.
> 
> I took a different approach to working around the WU certificate issue -- 
> keep SSL but don't verify the CERT.  Scroll for that "solution."
> 
> But meanwhile, using Wunderground-RF (rapid fire) I am getting an HTTP Error 
> 404:  Not Found.
> Any ideas on that?   Did they change the URL, or is it just down?
> 
> I noticed that even their own WunderStation App on iOS/iPadOS is subject to 
> the certificate issue and reports all stations as "not reporting" -- even 
> ones that are, like mine, which I can verify via the WU website has the data 
> and has reported within minutes ago.
> 
> From the logs, re: RapidFire HTTP 404:
> 
> Feb  1 12:31:51 nixie weewx[13184] DEBUG weewx.restx: Wunderground-RF: Failed 
> upload attempt 1: HTTP Error 404: Not Found
> Feb  1 12:31:51 nixie weewx[13184] ERROR weewx.restx: Wunderground-RF: Failed 
> to publish record 2020-02-01 12:31:51 EST (1580578311): Failed upload after 1 
> tries
> Feb  1 12:31:56 nixie weewx[13184] DEBUG weewx.restx: Wunderground-RF: Failed 
> upload attempt 1: HTTP Error 404: Not Found
> Feb  1 12:31:56 nixie weewx[13184] ERROR weewx.restx: Wunderground-RF: Failed 
> to publish record 2020-02-01 12:31:56 EST (1580578316): Failed upload after 1 
> tries
> Feb  1 12:32:04 nixie weewx[13184] DEBUG weewx.restx: Wunderground-RF: Failed 
> upload attempt 1: HTTP Error 404: Not Found
> Feb  1 12:32:04 nixie weewx[13184] ERROR weewx.restx: Wunderground-RF: Failed 
> to publish record 2020-02-01 12:32:04 EST (1580578324): Failed upload after 1 
> tries
> Feb  1 12:32:05 nixie weewx[13184] DEBUG weewx.restx: Wunderground-RF: Failed 
> upload attempt 1: HTTP Error 404: Not Found
> Feb  1 12:32:05 nixie weewx[13184] ERROR weewx.restx: Wunderground-RF: Failed 
> to publish record 2020-02-01 12:32:05 EST (1580578325): Failed upload after 1 
> tries
> Feb  1 12:32:06 nixie weewx[13184] DEBUG weewx.restx: Wunderground-RF: Failed 
> upload attempt 1: HTTP Error 404: Not Found
> Feb  1 12:32:06 nixie weewx[13184] ERROR weewx.restx: Wunderground-RF: Failed 
> to publish record 2020-02-01 12:32:05 EST (1580578325): Failed upload after 1 
> tries
> 
> 
> Here is the diff for my SSL CERT workaround:
> 
> $ diff restx.py restx.py.20200201.1
> 110,115d109
> < # SSL certificate hack
> < global WUssl
> < WUssl = ssl.create_default_context();
> < WUssl.check_hostname=False
> < WUssl.verify_mode=ssl.CERT_NONE
> <
> 454d447
> < _request.add_header("User-Agent", "weewx/%s" % weewx.__version__)
> 543,544c536
> < #_response = urllib.request.urlopen(request, data=data_bytes, 
> timeout=self.timeout)
> < _response = urllib.request.urlopen(request, data=data_bytes, 
> timeout=self.timeout, context=WUssl)
> ---
> > _response = urllib.request.urlopen(request, data=data_bytes, 
> > timeout=self.timeout)
> 1051,1052c1043
> < #_response = urllib.request.urlopen(request, 
> timeout=self.timeout)
> < _response = urllib.request.urlopen(request, 
> timeout=self.timeout, context=WUssl)
> ---
> > _response = urllib.request.urlopen(request, 
> > timeout=self.timeout)
> 
> 
> Regards,
> \Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPad)
> 
>>> On Jan 31, 2020, at 4:16 PM, Thomas Keffer  wrote:
>>> 
>> 
>> Travis: what is the error? If it's a certificate error, Version 4 will do a 
>> retry after an hour. Unfortunately, Version 3 does not.
>> 
>>> On Fri, Jan 31, 2020 at 1:04 PM Travis Bully  wrote:
>>> Not here yet.  It'll work for awhile and then will get the random cert 
>>> error I sent earlier.  A restart of weewx will get it going again.  I wish 
>>> restx would just try again after xx seconds.
>>> 
>>> 
>>> 
>>>> On Friday, January 31, 2020 at 2:48:50 PM UTC-5, J B wrote:
>>>> Working here too. I had to restart Weewx.
>>>> 
>>>>> On Friday, January 31, 2020 at 11:34:19 AM UTC-8, Brice Ruth wrote:
>>>>> Looking like it's working to me, too.
>>>>> 
>>>>> Brice Ruth, F

Re: [weewx-user] Re: rtupdate.wunderground.com certificate error

[Leon Shaner]

Hey, Tom,

The reason I get 204 back from wunderfixer on date 2020-01-30 is I really did 
have 0 records uploaded to WU (due to a full day of certificate issues).
Under the new API an HTTP 204 is returned for a valid request for which there 
is no data.

Here is my workaround:

$ diff wunderfixer wunderfixer_tk
407,409c407
< # Valid response, it's just that WU has no records for the 
requested date
< # Return an empty list of TimeStamps (as in WU has no records)
< return {}
---
> raise IOError("Probably a bad station ID or invalid date")

Or if the line numbers don't match yours, it looks like this in context:

if hasattr(response, 'code') and response.code != 200:
if response.code == 204:
# Valid response, it's just that WU has no records for the 
requested date
# Return an empty list of TimeStamps (as in WU has no records)
return {}
else:
raise IOError("Bad response code returned: %d" % response.code)


Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad)

> On Feb 1, 2020, at 12:40 PM, Leon Shaner  wrote:
> 
> Hey, WeeWx'ers.
> 
> I took a different approach to working around the WU certificate issue -- 
> keep SSL but don't verify the CERT.  Scroll for that "solution."
> 
> But meanwhile, using Wunderground-RF (rapid fire) I am getting an HTTP Error 
> 404:  Not Found.
> Any ideas on that?   Did they change the URL, or is it just down?
> 
> I noticed that even their own WunderStation App on iOS/iPadOS is subject to 
> the certificate issue and reports all stations as "not reporting" -- even 
> ones that are, like mine, which I can verify via the WU website has the data 
> and has reported within minutes ago.
> 
> From the logs, re: RapidFire HTTP 404:
> 
> Feb  1 12:31:51 nixie weewx[13184] DEBUG weewx.restx: Wunderground-RF: Failed 
> upload attempt 1: HTTP Error 404: Not Found
> Feb  1 12:31:51 nixie weewx[13184] ERROR weewx.restx: Wunderground-RF: Failed 
> to publish record 2020-02-01 12:31:51 EST (1580578311): Failed upload after 1 
> tries
> Feb  1 12:31:56 nixie weewx[13184] DEBUG weewx.restx: Wunderground-RF: Failed 
> upload attempt 1: HTTP Error 404: Not Found
> Feb  1 12:31:56 nixie weewx[13184] ERROR weewx.restx: Wunderground-RF: Failed 
> to publish record 2020-02-01 12:31:56 EST (1580578316): Failed upload after 1 
> tries
> Feb  1 12:32:04 nixie weewx[13184] DEBUG weewx.restx: Wunderground-RF: Failed 
> upload attempt 1: HTTP Error 404: Not Found
> Feb  1 12:32:04 nixie weewx[13184] ERROR weewx.restx: Wunderground-RF: Failed 
> to publish record 2020-02-01 12:32:04 EST (1580578324): Failed upload after 1 
> tries
> Feb  1 12:32:05 nixie weewx[13184] DEBUG weewx.restx: Wunderground-RF: Failed 
> upload attempt 1: HTTP Error 404: Not Found
> Feb  1 12:32:05 nixie weewx[13184] ERROR weewx.restx: Wunderground-RF: Failed 
> to publish record 2020-02-01 12:32:05 EST (1580578325): Failed upload after 1 
> tries
> Feb  1 12:32:06 nixie weewx[13184] DEBUG weewx.restx: Wunderground-RF: Failed 
> upload attempt 1: HTTP Error 404: Not Found
> Feb  1 12:32:06 nixie weewx[13184] ERROR weewx.restx: Wunderground-RF: Failed 
> to publish record 2020-02-01 12:32:05 EST (1580578325): Failed upload after 1 
> tries
> 
> 
> Here is the diff for my SSL CERT workaround:
> 
> $ diff restx.py restx.py.20200201.1
> 110,115d109
> < # SSL certificate hack
> < global WUssl
> < WUssl = ssl.create_default_context();
> < WUssl.check_hostname=False
> < WUssl.verify_mode=ssl.CERT_NONE
> <
> 454d447
> < _request.add_header("User-Agent", "weewx/%s" % weewx.__version__)
> 543,544c536
> < #_response = urllib.request.urlopen(request, data=data_bytes, 
> timeout=self.timeout)
> < _response = urllib.request.urlopen(request, data=data_bytes, 
> timeout=self.timeout, context=WUssl)
> ---
> > _response = urllib.request.urlopen(request, data=data_bytes, 
> > timeout=self.timeout)
> 1051,1052c1043
> < #    _response = urllib.request.urlopen(request, 
> timeout=self.timeout)
> < _response = urllib.request.urlopen(request, 
> timeout=self.timeout, context=WUssl)
> ---
> > _response = urllib.request.urlopen(request, 
> > timeout=self.timeout)
> 
> 
> Regards,
> \Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPad)
> 
>>> On Jan 31, 2020, at 4:16 PM, Thomas Keffer  wrote:
>>> 
>> 
>> Travis: what is the error? If it's a certificate error, Version 4 will do a 
>> retry after an hour. Unfortunately, Version 3 does not

Re: [weewx-user] Re: rtupdate.wunderground.com certificate error

[Leon Shaner]

Hey, WeeWx'ers.

I took a different approach to working around the WU certificate issue -- keep 
SSL but don't verify the CERT.  Scroll for that "solution."

But meanwhile, using Wunderground-RF (rapid fire) I am getting an HTTP Error 
404:  Not Found.
Any ideas on that?   Did they change the URL, or is it just down?

I noticed that even their own WunderStation App on iOS/iPadOS is subject to the 
certificate issue and reports all stations as "not reporting" -- even ones that 
are, like mine, which I can verify via the WU website has the data and has 
reported within minutes ago.

>From the logs, re: RapidFire HTTP 404:

Feb  1 12:31:51 nixie weewx[13184] DEBUG weewx.restx: Wunderground-RF: Failed 
upload attempt 1: HTTP Error 404: Not Found
Feb  1 12:31:51 nixie weewx[13184] ERROR weewx.restx: Wunderground-RF: Failed 
to publish record 2020-02-01 12:31:51 EST (1580578311): Failed upload after 1 
tries
Feb  1 12:31:56 nixie weewx[13184] DEBUG weewx.restx: Wunderground-RF: Failed 
upload attempt 1: HTTP Error 404: Not Found
Feb  1 12:31:56 nixie weewx[13184] ERROR weewx.restx: Wunderground-RF: Failed 
to publish record 2020-02-01 12:31:56 EST (1580578316): Failed upload after 1 
tries
Feb  1 12:32:04 nixie weewx[13184] DEBUG weewx.restx: Wunderground-RF: Failed 
upload attempt 1: HTTP Error 404: Not Found
Feb  1 12:32:04 nixie weewx[13184] ERROR weewx.restx: Wunderground-RF: Failed 
to publish record 2020-02-01 12:32:04 EST (1580578324): Failed upload after 1 
tries
Feb  1 12:32:05 nixie weewx[13184] DEBUG weewx.restx: Wunderground-RF: Failed 
upload attempt 1: HTTP Error 404: Not Found
Feb  1 12:32:05 nixie weewx[13184] ERROR weewx.restx: Wunderground-RF: Failed 
to publish record 2020-02-01 12:32:05 EST (1580578325): Failed upload after 1 
tries
Feb  1 12:32:06 nixie weewx[13184] DEBUG weewx.restx: Wunderground-RF: Failed 
upload attempt 1: HTTP Error 404: Not Found
Feb  1 12:32:06 nixie weewx[13184] ERROR weewx.restx: Wunderground-RF: Failed 
to publish record 2020-02-01 12:32:05 EST (1580578325): Failed upload after 1 
tries


Here is the diff for my SSL CERT workaround:

$ diff restx.py restx.py.20200201.1
110,115d109
< # SSL certificate hack
< global WUssl
< WUssl = ssl.create_default_context();
< WUssl.check_hostname=False
< WUssl.verify_mode=ssl.CERT_NONE
<
454d447
< _request.add_header("User-Agent", "weewx/%s" % weewx.__version__)
543,544c536
< #_response = urllib.request.urlopen(request, data=data_bytes, 
timeout=self.timeout)
< _response = urllib.request.urlopen(request, data=data_bytes, 
timeout=self.timeout, context=WUssl)
---
> _response = urllib.request.urlopen(request, data=data_bytes, 
> timeout=self.timeout)
1051,1052c1043
< #_response = urllib.request.urlopen(request, timeout=self.timeout)
< _response = urllib.request.urlopen(request, timeout=self.timeout, 
context=WUssl)
---
> _response = urllib.request.urlopen(request, timeout=self.timeout)


Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad)

> On Jan 31, 2020, at 4:16 PM, Thomas Keffer  wrote:
> 
> 
> Travis: what is the error? If it's a certificate error, Version 4 will do a 
> retry after an hour. Unfortunately, Version 3 does not.
> 
>> On Fri, Jan 31, 2020 at 1:04 PM Travis Bully  wrote:
>> Not here yet.  It'll work for awhile and then will get the random cert error 
>> I sent earlier.  A restart of weewx will get it going again.  I wish restx 
>> would just try again after xx seconds.
>> 
>> 
>> 
>>> On Friday, January 31, 2020 at 2:48:50 PM UTC-5, J B wrote:
>>> Working here too. I had to restart Weewx.
>>> 
>>>> On Friday, January 31, 2020 at 11:34:19 AM UTC-8, Brice Ruth wrote:
>>>> Looking like it's working to me, too.
>>>> 
>>>> Brice Ruth, FCD
>>>> Software Engineer, Madison WI
>>>> 
>>>> 
>>>>> On Fri, Jan 31, 2020 at 1:31 PM Travis Bully  wrote:
>>>>> Agreed.  Seems like I still get one cert error every now and then.  
>>>>> Likely still propagating changes through their infrastructure?
>>>>> 
>>>>> Jan 31 14:29:25 homeauto03 weewx[3784]: restx: Wunderground-RF: Published 
>>>>> record 2020-01-31 14:29:24 EST (1580498964)
>>>>> Jan 31 14:29:26 homeauto03 weewx[3784]: restx: Wunderground-RF: Failed 
>>>>> upload attempt 1: >>>> certificate verify failed (_ssl.c:727)>
>>>>> Jan 31 14:29:31 homeauto03 weewx[3784]: restx: Wunderground-RF: Failed to 
>>>>> publish record 2020-01-31 14:29:26 EST (1580498966): Failed upload after 
>>>>> 1 tries
>>>>> Jan 31 14:29:31 homeauto03 weewx[3784]: restx: Wundergroun

Re: [weewx-user] Weewx on RPi Zero

[Leon Shaner]

Tom,

Did I also see a reference to refactoring some SQL connection code, also 
having a positive effect on the memory leak?


Regards,
\Leon

Thomas Keffer wrote:


Only issue is the rampant memory leak plaguing many will manifest
more frequently on a RPI Zero W because it only has 512MB RAM. 
Hopefully the leak will be fixed soon.


Just for the record: the leak is in the underlying Debian drivers or, 
possibly, Python 2.7 --- not in WeeWX.


Upgrading operating systems can definitely help. Upgrading my OS from 
Debian 9.8 to 9.11 fixed my problem.


-tk
--
You received this message because you are subscribed to the Google 
Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to weewx-user+unsubscr...@googlegroups.com 
.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/CAPq0zEC1%3DnCK6eefCkG35xa7SiM4jbaZwar-xFMjNQkO8_jMtw%40mail.gmail.com 
.



--
l...@isylum.org - Dearborn, Michigan

--
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/59a30020-5000-ca07-adf1-c44fe162bab1%40isylum.org.

Re: [weewx-user] Weewx on RPi Zero

[Leon Shaner]

Runs great on my RPI Zero W(H).
Only issue is the rampant memory leak plaguing many will manifest more 
frequently on a RPI Zero W because it only has 512MB RAM.  Hopefully the leak 
will be fixed soon.

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPhone)

> On Nov 18, 2019, at 12:09 AM, peter  wrote:
> 
> Does weewx run OK on RPi Zero W? I want to connect a Vantage Envoy to the 
> USB-on-the-go port.
> Thanks!
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/1531b2f1-fcc3-4c34-bfa2-b07a5b0b4283%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/09225575-23D2-42FC-B3FE-042DBD9B4EFC%40isylum.org.

Re: [weewx-user] memory usage

[Leon Shaner]

Rich,

You're not the only one seeing memory leaks, but so far yours is among the more 
complete reports, helping to narrow it down.  Thanks for this!
I know of several of the extended development team who have been trying to 
reproduce and instrument to zero in on it.  I am sure they will be appreciative 
of your efforts here.

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad)

> On Nov 4, 2019, at 8:53 PM, Rich Bell  wrote:
> 
> 
> I'm running on: raspbian stretch, python 2.7.13, and WeeWX 3.9.1.
> 
> I was noticing a steady increase in memory usage. I narrowed it down to one 
> service I had installed. This service binds to the loop event and makes over 
> 10 calls to getSql. I updated the service to open and close the connection on 
> each loop event. This seems to have stopped the memory increase.
> 
> Next, I wrote a small program that does the same steps. Create a connection, 
> get a cursor, execute a select, fetchone, close the cursor. But this isn't 
> exibiting the memory increase.
> 
> Next I wrote a wrapper class for the Connection class and subclassed the 
> Cursor class, like WeeWX does. Running with these, I am again seeing the 
> memory increase.
> 
> I don't see how the 'wrapper' classes should matter... Granted at 10+ calls 
> per loop, this is aproximately 240 calls a minute - so it would add up fast; 
> but I would think others would be seeing something too. I did see this thread 
> about the forecast service, 
> https://groups.google.com/forum/#!topic/weewx-user/H4GVpoI5l70/discussion It 
> looks to make a lot of calls to getSql, but binds to archive, so it would 
> happen a lot slower. Not sure if there was any resolution.
> 
> I've attached my test program. It can call the sqlite classes directly or use 
> the 'wrapper' classes I stole from WeeWx. I may be all wet, but I'm seeing 
> something strange in my environment... I'll keep digging.
> 
> -rich
> 
> 
> 
> 
>  
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/3753360b-d4c7-44b3-82ca-a8140514b310%40googlegroups.com.
> 

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/9C5C610E-29A3-4ADA-B3AD-40545EFD2C51%40isylum.org.

Re: [weewx-user] Potential memory issue

[Leon Shaner]

TK,
If it helps, I don't use any reporting stuff.
I'm strictly using WMR300 with rapid fire and archive simultaneously.
No other features.

Unfortunately, I've restarted weewxd several times today making the attempt to 
debug it, so I can't share the latest memory usage snapshot.  But as a data 
point my watchdog has to reboot the pi roughly every 3 days.  So now that I've 
restarted weewx I'll see if it goes 6 days instead of the usual 3.

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad)

> On Aug 11, 2019, at 7:22 PM, Thomas Keffer  wrote:
> 
> 
> While the tools work well in smaller projects where you can identify and 
> track every object, in a larger, real-time, project like weewx, there is just 
> too much going on to make sense of the huge memory trees the tools produce.
> 
> They also suffer from a Heisenberg uncertainty principle: the very act of 
> measuring consumes memory, which you then chase, looking for your leak.
> 
> So, I have taken to a patient, binary approach. I shut down half the program, 
> say the reporting engine, and see if that makes a difference. If not, I shut 
> down the other half. Keep dividing by half and eventually you'll locate the 
> problem. With most leaks, you can only do one iteration every day or two, so 
> it can take weeks to isolate the problem. With a large leak, like yours, it 
> can be done in a matter of hours.
> 
> Patience is the only thing I've found that works well!
> 
> One thing worth noting: Python uses garbage collection, so it doesn't 
> actually "leak" memory. It may struggle to reclaim memory that has cyclic 
> references, taking a long time to reclaim it, but eventually it will (except 
> if a __del__ method is defined and when using a Python version before 3.4. 
> Weewx never defines __del__, so it is not subject to this limitation.). 
> 
> Most of the time the problem is in an underlying "C" routine, specifically 
> drivers, occasionally graphic utilities.
> 
> -tk
> 
>> On Sun, Aug 11, 2019 at 3:51 PM Leon Shaner  wrote:
>> Hey, TK,
>> 
>> Can you please toss me a bone on how to instrument weewx for memory 
>> profiling?
>> I tried python3-memory-profiler but couldn't make it work.  It seems to 
>> block the normal importing of needed python modules (like weewx.engine).  
>> Maybe it is because weewx uses precompiled modules?
>> 
>> Other than that I found a thread in the weewx google group where you 
>> suggested pympler, but I can't seem to find that on Raspbian, and you had a 
>> specially modified engine.py anyway.  :-/
>> 
>> Regards,
>> \Leon
>> --
>> Leon Shaner :: Dearborn, Michigan (iPad)
>> 
>>>> On Aug 11, 2019, at 4:06 PM, Leon Shaner  wrote:
>>>> 
>>> Hey, TK.  WMR300.  I'll see if I can track it down.
>>> 
>>> Regards,
>>> \Leon
>>> --
>>> Leon Shaner :: Dearborn, Michigan (iPad)
>>> 
>>>>> On Aug 11, 2019, at 12:45 PM, Thomas Keffer  wrote:
>>>>> 
>>>> 
>>>> If memory increases every couple of seconds, which suggests that the leak 
>>>> is in the part of device driver that deals with LOOP packets. But, you're 
>>>> on an WMR300, right? As I recall, their LOOP packets are more like 10-20 
>>>> seconds apart --- too long to correlate with what you're seeing. Or, are 
>>>> you on something like a Vantage with a shorter LOOP packet interval?
>>>> 
>>>> I've noticed memory leaks in other instances, such as my own site 
>>>> http://www.threefools.org/weewx. In my case, I spent a couple of weeks 
>>>> trying to chase it down, and concluded the problem was somewhere in the 
>>>> sqlite database driver, but couldn't put my finger on it conclusively.
>>>> 
>>>> Then, magically, Debian did an update and the problem (nearly) went away. 
>>>> Memory leak dropped from 5 MB/day, to almost nothing (maybe a megabyte or 
>>>> two over a week). Again, it suggests the problem was (is) in a device 
>>>> driver somewhere.
>>>> 
>>>> -tk
>>>> 
>>>>> On Sun, Aug 11, 2019 at 8:55 AM Leon Shaner  wrote:
>>>>> TK, you may want to know...
>>>>> There definitely is a memory leak in weewxd.
>>>>> It grows by 1 KiB about every 2 seconds.
>>>>> 
>>>>> I haven't yet correlated whether weewxd can grow to a point where other 
>>>>> things start dying, and whether that has any impact on the USB stack.
>>>>> 
>>>>> What tipped me off was I just got a warning from sar

Re: [weewx-user] Potential memory issue

[Leon Shaner]

Hey, TK.  WMR300.  I'll see if I can track it down.

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad)

> On Aug 11, 2019, at 12:45 PM, Thomas Keffer  wrote:
> 
> 
> If memory increases every couple of seconds, which suggests that the leak is 
> in the part of device driver that deals with LOOP packets. But, you're on an 
> WMR300, right? As I recall, their LOOP packets are more like 10-20 seconds 
> apart --- too long to correlate with what you're seeing. Or, are you on 
> something like a Vantage with a shorter LOOP packet interval?
> 
> I've noticed memory leaks in other instances, such as my own site 
> http://www.threefools.org/weewx. In my case, I spent a couple of weeks trying 
> to chase it down, and concluded the problem was somewhere in the sqlite 
> database driver, but couldn't put my finger on it conclusively.
> 
> Then, magically, Debian did an update and the problem (nearly) went away. 
> Memory leak dropped from 5 MB/day, to almost nothing (maybe a megabyte or two 
> over a week). Again, it suggests the problem was (is) in a device driver 
> somewhere.
> 
> -tk
> 
>> On Sun, Aug 11, 2019 at 8:55 AM Leon Shaner  wrote:
>> TK, you may want to know...
>> There definitely is a memory leak in weewxd.
>> It grows by 1 KiB about every 2 seconds.
>> 
>> I haven't yet correlated whether weewxd can grow to a point where other 
>> things start dying, and whether that has any impact on the USB stack.
>> 
>> What tipped me off was I just got a warning from sar/sa1 stating it couldn't 
>> do it's job due to "resource unavailable," so I went and had a peek.
>> 
>> This RPI is a Zero W(H), so it has only 1 GB of RAM.
>> It's full enough on memory that it's using some swap.
>> 
>> Here is just one snapshot.  That RES (RZ) value for weewxd goes up by one 
>> digit every 2 seconds.
>> I gleaned that is 1 KiB, but I am not 100% certain. It might only be 1 byte 
>> every 2 seconds -- whatever the actual unit it goes up by one every 2 
>> seconds.  I've never seen it come down.
>> 
>> I may enhance my watchdog to check the weewxd memory size and restart it 
>> after a limit, then see if that has any positive impact on avoiding the USB 
>> stack going south.  Yeah, I guess I'll work on that this week.   I'll put 
>> some logging in so anyone who wants to keep an eye on it can know whether 
>> their setup is memory leaking also.  =D
>> 
>> I have the WMR300 driver, in case the leak could be specific to that driver. 
>>  :-/
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to weewx-user+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/weewx-user/D7820F76-CC3A-4CC2-B359-4AF967A32638%40isylum.org.
>> 
>> 
>> Regards,
>> \Leon
>> --
>> Leon Shaner :: Dearborn, Michigan (iPad)
>> 
>> > On Aug 10, 2019, at 9:01 AM, Leon Shaner  wrote:
>> > 
>> > It isn't the weewx version, it's the pi kernel / usb module version.  The 
>> > usb stack hangs.  I wrote a watchdog to detect it and reboot the pi, 
>> > because so far there has been no fix to the issue.  I am not able to 
>> > correlate this to a memory leak.  What are you seeing re: memory?
>> > 
>> > Regards,
>> > \Leon
>> > --
>> > Leon Shaner :: Dearborn, Michigan (iPhone)
>> > 
>> >> On Aug 10, 2019, at 4:29 AM, Gordie Stirling  wrote:
>> >> 
>> >> Hi all,
>> >> 
>> >> I have been runnin Weewx on a Raspberry Pi (model a) for a couple of 
>> >> years with a Maplin NG96Y, without any issues.
>> >> 
>> >> I recently upgraded to a Pi3, running Stetch, and Lighttpd.  the old Pi 
>> >> was starting to struggle and sat at 100% cpu for most of the time.
>> >> 
>> >> All worked well for a month or so, the the station locked up.  After a 
>> >> reset(batteries out, usb disconnected) it works for an hour or so, then 
>> >> locks the connection again. the base station still continues to function.
>> >> 
>> >> Error in syslog is Aug 10 09:17:28 raspberrypi weewx[10377]: fousb: 
>> >> get_records failed: [Errno 110] Operation timed out.
>> >> 
>> >> so far, other than rebooting, i have used Easy Weather to clear tge 
>> >> memory, but alas had no effect, 85 mins later, timeouts start again

Re: [weewx-user] Potential memory issue

[Leon Shaner]

It isn't the weewx version, it's the pi kernel / usb module version.  The usb 
stack hangs.  I wrote a watchdog to detect it and reboot the pi, because so far 
there has been no fix to the issue.  I am not able to correlate this to a 
memory leak.  What are you seeing re: memory?

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPhone)

> On Aug 10, 2019, at 4:29 AM, Gordie Stirling  wrote:
> 
> Hi all,
> 
> I have been runnin Weewx on a Raspberry Pi (model a) for a couple of years 
> with a Maplin NG96Y, without any issues.
> 
> I recently upgraded to a Pi3, running Stetch, and Lighttpd.  the old Pi was 
> starting to struggle and sat at 100% cpu for most of the time.
> 
> All worked well for a month or so, the the station locked up.  After a 
> reset(batteries out, usb disconnected) it works for an hour or so, then locks 
> the connection again. the base station still continues to function.
> 
> Error in syslog is Aug 10 09:17:28 raspberrypi weewx[10377]: fousb: 
> get_records failed: [Errno 110] Operation timed out.
> 
> so far, other than rebooting, i have used Easy Weather to clear tge memory, 
> but alas had no effect, 85 mins later, timeouts start again.
> 
> I am thunking of downgrading the Pi to Jessie, and using reverting to Weewx 
> 3.9.1, but wondered if anyone has had this problem and managed to fix rather 
> than rebuild.
> 
> TIA
> 
> G
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/afbf7556-425f-4eca-8088-caaf96c06805%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/315802A3-3CD7-4206-B22B-77A377B0F5A7%40isylum.org.

Re: [weewx-user] Cron set up for Wunderfix

[Leon Shaner]

I have an example wunderfixer_wrapper script and corresponding crontab example, 
here:

https://github.com/UberEclectic/weewx/tree/watchdog/examples/watchdog

You can ignore the watchdog if you want, because they're independent (but 
complimentary) utilities.
If you want only the wunderfixer_wrapper then you just need to copy it 
someplace and make the crontab entry match the path.

Important note is that the crontab needs to be tied to the user that can 
actually run wunderfixer.
For many that means "sudo crontab -e" so it's root's crontab.
But if you are running weewx as some user other than root, then become that 
user and then just use "crontab -e" to add the entry.

There are other changes within the wunderfixer_watchdog script which you may 
need to change if you are not running weewx as root, for example where to put 
the supplemental logs that the script writes, which are for debugging purposes. 
 I've left some comments in the script showing alternatives.

Do pay attention to the logrotate example per the readme.txt, so that your 
supplemental log doesn't grow to infinity.  It's that first line in the 
logrotate file that needs to point to wherever you have wunderfixer_wrapper 
writing the supplemental logs.  If running as root, the example works as is, 
since root can write to /var/log and the wrapper will create 
/var/log/weewx.log.  If running as some non-root user you can't keep that 
default log path because /var/log needs super-user privileges.

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad)

> On Aug 7, 2019, at 10:04 AM, CablDeViL  wrote:
> 
> 
> Good Day,
> 
> I have searched the forums but I have not found the proper flags to run with 
> Wunderfix on my Cron job.
> 
> Nor could I locate the best practices for times and tweaks to use on a Pi2.
> 
> If anyone can point me to a github doc or such. I haven't even downloaded it 
> yes as I am doing research for my upgrade from a Pi1 to a Pi2
> 
> I do see great debate with the new and old APIs and Wunderfix so I am not 
> sure if I should just hold off for a quarter to see how it plays out.
> 
> Thank you
> 
> Bill
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/dd63df72-4b79-4ae6-a8b4-eb3d65cb8ccd%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/5ADC6FB5-7EFD-49DD-9C6B-8B4D88328178%40isylum.org.

Re: [weewx-user] Re: Customised logwatch

[Leon Shaner]

The Linux logrotate always works for anything I need.  It's very flexible.  I 
have an example on my weewx clone on GitHub in the watchdog util branch:

https://github.com/UberEclectic/weewx/tree/watchdog/examples/watchdog

Regards,
Leon
--
Leon Shaner :: Dearborn, Michigan (iPhone)

> On Jul 25, 2019, at 2:49 PM, gjr80  wrote:
> 
> The logwatch script needs to be modularised so that users can easily extend 
> the basic logwatch script functionality to cater for extensions and other 
> customisations the user may have installed. The current top to bottom 
> ‘if..elif..else’ structure does not lend itself to being easily extended in a 
> manner that survives upgrades etc.
> 
> We are aware of the issue but it will probably take until we find the 
> roundtoit before we get around to it.
> 
> Gary
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/99093fab-14af-4c1e-adf9-6d79599f460f%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/73E95A25-8757-48AA-8EAC-C044D2485766%40isylum.org.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

 (1563342124)
Wed 17 Jul 01:42:03 EDT 2019 Using configuration file 
/usr/share/weewx4/weewx.conf.
Wed 17 Jul 01:42:03 EDT 2019 Using database binding 'wx_binding', which is 
bound to database 'archive_sqlite'
Wed 17 Jul 01:42:03 EDT 2019 Weather Underground Station:   KMIDEARB5
Wed 17 Jul 01:42:03 EDT 2019 Date to check: 2019-07-17
Wed 17 Jul 01:42:03 EDT 2019 Number of archive records: 22
Wed 17 Jul 01:42:03 EDT 2019 Number of WU records:  21
Wed 17 Jul 01:42:03 EDT 2019 Number of missing records: 1
Wed 17 Jul 01:42:03 EDT 2019
Wed 17 Jul 01:42:03 EDT 2019 Missing records:
Wed 17 Jul 01:42:03 EDT 2019 2019-07-17 00:00:00 EDT (1563336000); 29.167";  
72.0F;  99%; 0.0 mph; N/A deg; 0.0 mph gust;  71.7F; 0.00" rain  ...published.
Wed 17 Jul 01:52:03 EDT 2019 WeeWX: Wunderfixer has been intentionally skipped 
this iteration.
Wed 17 Jul 02:02:04 EDT 2019 WeeWX: Wunderfixer timestamp: (1563343324)
Wed 17 Jul 02:02:04 EDT 2019 Using configuration file 
/usr/share/weewx4/weewx.conf.
Wed 17 Jul 02:02:04 EDT 2019 Using database binding 'wx_binding', which is 
bound to database 'archive_sqlite'
Wed 17 Jul 02:02:04 EDT 2019 Weather Underground Station:   KMIDEARB5
Wed 17 Jul 02:02:04 EDT 2019 Date to check: 2019-07-17
Wed 17 Jul 02:02:04 EDT 2019 Number of archive records: 26
Wed 17 Jul 02:02:04 EDT 2019 Number of WU records:  25
Wed 17 Jul 02:02:04 EDT 2019 Number of missing records: 1
Wed 17 Jul 02:02:04 EDT 2019
Wed 17 Jul 02:02:04 EDT 2019 Missing records:
Wed 17 Jul 02:02:04 EDT 2019 2019-07-17 00:00:00 EDT (1563336000); 29.167";  
72.0F;  99%; 0.0 mph; N/A deg; 0.0 mph gust;  71.7F; 0.00" rain  ...published.
Wed 17 Jul 02:12:03 EDT 2019 WeeWX: Wunderfixer has been intentionally skipped 
this iteration.


Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad)

> On Jul 18, 2019, at 11:11 AM, Thomas Keffer  wrote:
> 
> 
> I've ported wunderfixer to the new WU API. Commit 32c35ce on the development 
> branch.
> 
> 
> 
> Unfortunately, it looks like the WU no longer allows re-posting old records, 
> so the utility may no longer be useful. I'd be interested in other people's 
> experience.
> 
> 
> 
> Also, as I understand it, there are issues for people living east of 
> Greenwich. I live west, so I wasn't able to check that. Other people's 
> experiences would be welcome.
> 
> 
> 
> NB: this is on the development branch. You will have to clone the weewx 
> GitHub repository, then check out the development branch. 
> 
> 
> 
> -tk
> 
> 
>> On Monday, June 3, 2019 at 3:21:54 PM UTC-7, Rod Yager wrote:
>> I’ve already migrated the today logic into my local version of wunderfixer, 
>> which runs as a cron job at 05:58, 11:58, 17:58 and 23:58.
>> 
>> This works as expected here.
>> 
>> The only circumstance I can see in which it would break is if I make a 
>> request for today’s data  at 23:59:59.95 on my machine. The transmission 
>> delay (and possibly differences in the clocks) will mean that WU receives my 
>> request at a time when it thinks it is 00:00:00.01 on the following day (my 
>> time) and so it will return the (empty) data for the day after the date on 
>> which I made the request. We can’t worry about such things.
>> 
>> 
>> Rod
>> 
>> 
>> 
>>> On 4 Jun 2019, at 7:55 am, Leon Shaner  wrote:
>>> 
>>> Thanks, Rod!  =D
>>> 
>>> Those are precisely the same tests I ran and exact same results that noted, 
>>> before  before publishing.  =D
>>> Plus a ton of checks against my own station, KMIDEARB5, west of UTC.
>>> 
>>> As before, even more important than these 'historical' API tests is to 
>>> query against the current day at different times throughout the day, such 
>>> as before and after your midnight and before and after UTC midnight.  That 
>>> logic which uses the 'today' API (rapid) for today is the one we will 
>>> likely keep no matter what IBM does with the well-known bugs in the 
>>> 'historical' API.
>>> 
>>> (I can't test the 'today' logic 100% without changing my localtime on my 
>>> system and I don't want to do that, which is why I appreciate help from 
>>> others to validate the code).  =D
>>> 
>>> Regards,
>>> \Leon
>>> --
>>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>>> 
>>> On Jun 3, 2019, at 5:39 PM, Rod Yager  wrote:
>>> 
>>>> Dear Leon,
>>>> 
>>>> Thanks for all your work on this. This works as well as we are going to be 
>>>> able to manage unless and until WU fixes the bug. 
>>>> 
>>>> The remaining issue affect

Re: [weewx-user] New Install -> Permanetly Error

[Leon Shaner]

WMR300 or WMR200?

https://github.com/weewx/weewx/issues/375

That has a link to the fix for WMR200.
There is a similar fix for WMR300.

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad)

> On Jul 11, 2019, at 10:13 AM, 'Sebastian Pendsa' via weewx-user 
>  wrote:
> 
> 
> Hello
> 
> After 2 years of operation, I had to reinstall weewx.
> Now I constantly get an error message.
> No matter if RPi 1 or 2 or 3 or 3b+
> Always the same. I also tried different Jessy systems.
> Also different weewx versions .
> 
> -..snip-
> 
> root@orangepipc:~# sudo service weewx status
> â weewx.service - LSB: weewx weather system
>Loaded: loaded (/etc/init.d/weewx; generated; vendor preset: enabled)
>Active: active (exited) since Thu 2019-07-11 13:49:32 UTC; 16min ago
>  Docs: man:systemd-sysv-generator(8)
> Tasks: 0 (limit: 4915)
>CGroup: /system.slice/weewx.service
> 
> Jul 11 13:54:05 orangepipc weewx[17631]: File 
> "/usr/share/weewx/weewx/engine.py", line 601, in new_archive_record
> Jul 11 13:54:05 orangepipc weewx[17631]:   
> dbmanager.addRecord(event.record, accumulator=self.old_accumulator)
> Jul 11 13:54:05 orangepipc weewx[17631]: File 
> "/usr/share/weewx/weewx/manager.py", line 246, in addRecord
> Jul 11 13:54:05 orangepipc weewx[17631]:   
> self._addSingleRecord(record, cursor, log_level)
> Jul 11 13:54:05 orangepipc weewx[17631]: File 
> "/usr/share/weewx/weewx/manager.py", line 1212, in _addSingleRecord
> Jul 11 13:54:05 orangepipc weewx[17631]:   _weight = 
> self._calc_weight(record)
> Jul 11 13:54:05 orangepipc weewx[17631]: File 
> "/usr/share/weewx/weewx/manager.py", line 1582, in _calc_weight
> Jul 11 13:54:05 orangepipc weewx[17631]:   raise 
> ValueError("Non-positive value for record field 'interval': %s" % 
> (record['interval'], ))
> Jul 11 13:54:05 orangepipc weewx[17631]:   ValueError: Non-positive 
> value for record field 'interval': 0
> Jul 11 13:54:05 orangepipc weewx[17631]:   Exiting.
> root@orangepipc:~#
> 
> -.snap-
> 
> Anyone an idea ?!
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/d246c922-3d58-4130-a0b1-e512d9347b94%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/90370F9A-87D0-4065-9AC9-ED209FAC40D1%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: "wee_device" not working after new install

[Leon Shaner]

Christian,

 What happens if you change to the /usr/share/weewx/bin dir an run it as:

$ ./wee_device --help

Or use the fully-qualified path?

$ /usr/share/weewx/bin/wee_device --help

By default the current path (.) is not in your PATH environment variable.
You may have changed that in user pi's "dot" files on the RPI where things are 
working (~/.bashrc, ~/.bash_profile, /etc/bash.bashrc, ~/.profile, 
/etc/profile, etc.).

It's better to leave the current path (.) out of your PATH for security reasons 
and always pretend ./ to the command or use the fully-qualified path to the 
exec/script.

You can "simulate" the "not found" using any command that doesn't exist.  For 
example,

$ oog
-bash: oog: command not found

Don't forget if you're running from pkgs, most weewx commands need to be 
prefaced with sudo.

Other than that, I would use "sudo apt list --installed" on each RPI and then 
diff the outputs to see what packages you are missing.  Seems unlikely that you 
are missing python, but that's wee_device is using:

 $ head wee_device
#!/usr/bin/python

If you don't have /usr/bin/python then that would explain a lot, but it would 
also be surprising to me that an RPI could function without it.  ;-)

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad)

> On Jul 7, 2019, at 9:37 AM, Christian Nimmervoll 
>  wrote:
> 
> 
> For an example:
> 
> On the old Device I a get an answer when i type this:
> wee_device --help
> 
> On the new install - i get only the erros message - not found...
> 
> 
> 
> Am Sonntag, 7. Juli 2019 15:35:47 UTC+2 schrieb Christian Nimmervoll:
>> 
>> Hello,
>> 
>> I've run WEEWX on an existing Raspberry and installed a new one today.
>> Everything works so far on the new device again without problems.
>> Only one function doen't work...
>> 
>> wee_device works without problems on the old device but wee_device is not 
>> running on the new device.
>> I get the error message: "command not found".
>> 
>> In the weewx.conf the station_type is entered correctly:
>> station_type = Vantage
>> 
>> And the new Raspberry communicate also without errors with the Davis VP2 
>> Console.
>> 
>> What can be the reason that the function: "wee_device" is not working?
>> Is there another important setting in the software?
>> 
>> Thanks for infos.
>> Chris
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/d5efbb2c-80fd-45ba-80c6-9c8e9eefba11%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/DED8418C-4667-4433-A792-FBA3EB6AADBE%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: /dev/serial0 disappeared with RPi update

[Leon Shaner]

I took his note to be a heads up when upgrading from Raspbian Stretch to 
Raspbian Buster.
Sounds like they named it Buster, appropriately, because things are breaking.  
:S

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad)

> On Jul 5, 2019, at 12:00 PM, vince  wrote:
> 
> 
> Is there a question in there someplace ?
> 
> Perhaps you might try a google search for "No such file or directory: 
> '/dev/serial0'" and see if that helps any.
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/75adbc5c-a3cf-481a-941f-b6e3ada9610b%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/60DC7FF7-7D10-4ECD-8DC2-8CC0B68F6DD1%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] weewx suddenly stops

[Leon Shaner]

Werner,
You wouldn't happen to be using "tail -f /var/log/syslog" would you?
File may have rotated.  CTRL-C the tail and run it again or just grab both the 
plain syslog files a la:

$ cd /var/log
$ tar cf - syslog syslog.1 | gzip -9 > /tmp/syslog.tgz

And send the /tmp/syslog.tgz

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad)

> On Jul 5, 2019, at 11:37 AM, Werner  wrote:
> 
> 
> There are no further entries in the log. But the RPi does not crash because 
> it reacts to keyboard and mouse.
> After a "sudo /etc/init.d/weewx stop" and "sudo /etc/init.d/weewx start" 
> everything will be ok.
> Since I am an RPi newcomer, I will now start looking for other logs. Thank you
> 
> 
> Am Freitag, 5. Juli 2019 16:56:31 UTC+2 schrieb Thomas Keffer:
>> 
>> Absolutely nothing? Not even system entries?
>> 
>> If that is the case, then your RPi crashed for reasons having nothing to do 
>> with WeeWX.
>> 
>> If there are system entries, then it would be useful to see them.
>> 
>> -tk
>> 
>>> On Fri, Jul 5, 2019 at 7:22 AM Werner  wrote:
>>> Nothing! The log stops even if weewx stops.
>>> 
>>> Am Freitag, 5. Juli 2019 16:15:39 UTC+2 schrieb Thomas Keffer:
>>>> 
>>>> Hello, Werner
>>>> 
>>>> What comes after the Jul  5 10:20:28 entry in the log? Anything?
>>>> 
>>>>> On Fri, Jul 5, 2019 at 2:08 AM Werner  wrote:
>>>>> Hello everybody, I installed weewx on a raspberry pi zero to put the data 
>>>>> of my WMR300 online at AWEKAS.
>>>>> The Raspberry is running the current Raspian and weewx is the only 
>>>>> program installed.
>>>>> It works fine too, until weewx suddenly stops. It happens after different 
>>>>> periods of time.
>>>>> It's one hour or four hours. I just need to interrupt the USB connection 
>>>>> to the WMR300 and weewx will run again.
>>>>> Maybe someone has a tip for me? Many Thanks!
>>>>> 
>>>>> -- 
>>>>> You received this message because you are subscribed to the Google Groups 
>>>>> "weewx-user" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>>>> email to weewx...@googlegroups.com.
>>>>> To view this discussion on the web visit 
>>>>> https://groups.google.com/d/msgid/weewx-user/ccd77f3c-347a-458a-93db-a9ae0747a1b3%40googlegroups.com.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>> 
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "weewx-user" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to weewx...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/weewx-user/f531f57c-5465-49d2-8e32-7b119fcc498e%40googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/4e90241c-e602-463a-81cf-5ea2632148d7%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/C3AC71AE-9CF3-4310-A568-EACA41CA589E%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] WMR300: Reading from history

[Leon Shaner]

Hola, Miguel,

Por favor, confirme, ¿está ejecutando el último controlador, según el siguiente 
enlace?

Esta versión contiene múltiples mejoras: lectura / borrado del historial 
después del reinicio.

https://github.com/weewx/weewx/blob/master/bin/weewx/drivers/wmr300.py

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPhone)

> On Jul 3, 2019, at 12:42 PM, Miguel Iniesta  wrote:
> 
> Hola, 
> 
> Creo que existen un par de desajustes cuando al iniciar weewx se leen datos 
> desde el histórico de la estación:
> 1.- La hora del último dato leido no es la del reinicio, con lo que se crea 
> un intervalo en el que no hay datos. Pienso puede deberse a que no se ajusta 
> la hora al horario de verano GMT+2.
> 2.- En el gráfico de viento se observa que la línea de racha de viento 
> fluctua al mismo nivel que la línea de velocidad de viento, mientras que en 
> una lectura normal (no del histórico) la línea de racha de viento tiene 
> siempre valores muy superiores.
> 
> Incluyo dos gráficos de ejemplo.
> En el primero se muestra la parada del día 2-7-2019 a las 10:45 y el reinicio 
> a las 8:45 del 3-7-2019
> En el segundo se observa una parada el día 3-7-2019 entre las 10:30 h y 13 h. 
> aproximadamente.
> 
> No sé si alguien más ha observado esto.
> 
> 
> 
> 
> 
> 
> Gracias.
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/d8ad70c2-a739-47aa-a249-ac261ba2ccb9%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
> 
> 

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/EAA66929-7021-4598-8141-90335A4860DC%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] raspi4 users yet ?

[Leon Shaner]

Well, does it?
Pi's are power managed in that they draw what they need based on what they're 
doing.
Sure, the Pi 4 is capable of drawing more, but will it do so for the workload 
needs of WeeWx and related SW?

For one thing the Pi 4's new SoC is on paper 2-3x as efficient as a Pi 3, so 
things may either run ~2x as fast or waste fewer cycles getting the job done.
Only way to know is to run exact same OS version and software stack on two 
different units side-by-side, and measure how many watts they're actually 
pulling nominally and during peak times and actually compare the two.   YMMV.  
=D

I'm holding out for the 4 GB Pi 4, which is out of stock already, where I live.
Not needed for WeeWx, but as my main development platform, while a slower Pi 
Zero runs the code developed and unit tested on the Pi 4.

I sure do like having a faster box so compile times are faster.  =D
And to that end, eventually there will be a 64-bit Raspbian on the official 
support train, in which case the additional memory will get me past some 
hurdles where the 2 GB virtual memory per process currently has me shutting off 
SW features that gcc/ gplusplus can't manage to build with only 2 GB VM.
But I digress.  ;-)

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad)

> On Jun 27, 2019, at 11:05 AM, Robert Lorenzini  wrote:
> 
>  Be aware that the 4 draws more power and runs hotter.
> 
> Bob - wd6dod
> 
>> On 6/27/2019 8:02 AM, vince wrote:
>> Anybody have a raspi4 yet ?   I'm curious what the time to process skins is 
>> on this one vs. a pi3b+ since the new one is reportedly much faster in 
>> general.
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to weewx-user+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/weewx-user/a54f93ad-79c8-4898-9119-9a67ecbaa2e3%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/76dbfbb6-ce59-87e0-8c15-2c4916000473%40llorenzini.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/171A9BC2-0B0E-44BE-9038-0CD8F6BFC674%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: weewx hangs after update from 3.8 to 3.9.1

[Leon Shaner]

Just a hunch,

But this line looks suspect.
Perhaps just comment it out?

data_services = ,

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPhone)

> On Jun 20, 2019, at 10:31 AM, Michael Aschauer  
> wrote:
> 
> data_services = ,

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/8D54E013-9681-4687-988B-2221DDB00379%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: weewx stop working

[Leon Shaner]

FYI, on all RH/Fedora (including CantOS) there are multiple consoles, which can 
be accessed by pressing CTRL-ALT 1 thru =.

IIRC the debug messages go to console 12, the last one behind CRTL-ALT =, but 
having them in a file is preferred in my book.

So, good job! =D

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPhone)

> On Jun 15, 2019, at 11:09 AM, gjr80  wrote:
> 
> Thanks, so if I am reading the centos7.6 /etc/rsyslog.conf correctly only 
> info and above go to /var/log/messages. There is nothing covering debug at 
> all so I expect that is why debug is (effectively) discarded. We could send 
> them to another file but that will make debugging difficult so let's just 
> send debug to /var/log/messages as well.
> 
> Damjan. Try the following:
> 
> 1. edit /etc/rsyslog.conf and add the following to the bottom of the file 
> (you will need privileged access to edit the file):
> 
> # all debug to /var/log/messages
> *.=debug /var/log/messages
> 
> 2. restart rsyslog:
> 
> $ sudo systemctl restart rsyslog
> 
> 3. make sure /etc/weewx/weewx.conf has debug=1
> 
> 4. restart WeeWX:
> 
> $ sudo systemctl restart weewx
> 
> check /var/log/messages and you should see some debug output.
> 
> Gary
> 
>> On Sunday, 16 June 2019 00:36:45 UTC+10, Leon Shaner wrote:
>> Gary, good sleuthing.
>> 
>> Could it be that the debug messages simply go to a different file, such as 
>> /var/log/debug?
>> 
>> Check to see where the debug logs are pointed in /etc/rsyslog.conf.
>> 
>> Also, see here:
>> 
>> https://www.the-art-of-web.com/system/rsyslog-config/
>> 
>> Regards,
>> \Leon
>> --
>> Leon Shaner :: Dearborn, Michigan (iPhone)
>> 
>>> On Jun 15, 2019, at 10:19 AM, gjr80  wrote:
>>> 
>>> I guess the question is where to from here given that it appears you cannot 
>>> get any debug output when running under centos7.6. I think we can assume 
>>> that your system is running properly (debug issue aside) with the simulator 
>>> driver. Troubleshooting the interceptor driver is going to be difficult 
>>> without any debug output. One thing I did notice, some time ago Matthew 
>>> suggested running WeeWX (with the interceptor driver) directly. You could 
>>> try that again, remember what we are seeking when running WeeWX directly is 
>>> not the log output but rather the console output, ie what you see on the 
>>> screen. I notice you don't appear to have provided the console output, 
>>> rather you gave the log output.
>>> 
>>> Gary
>>> 
>>>> On Saturday, 15 June 2019 02:28:14 UTC+10, Damjan Hajsek wrote:
>>>> Ok I did logs again I hope this time better.
>>>> 
>>>> 
>>>> Dne petek, 14. junij 2019 14.44.03 UTC+2 je oseba Andrew Milner napisala:
>>>>> 
>>>>> Not really, no
>>>>> 
>>>>> The first part appears to show normal running with archive records every 
>>>>> minute - but no log for the startup process
>>>>> 
>>>>> The second part, manual running , shows us the startup but only loop data 
>>>>> - it runs for under a minute so does not give the coverage for two 
>>>>> archive periods (which in your case would be at least 2 minutes)
>>>>> 
>>>>> The third part says debug disabled - but there does not appear to be any 
>>>>> section where debug is enabled - so it is not clear if you are enabling 
>>>>> debug correctly or not.
>>>>> 
>>>>> it is much better to just attach the logfile directly rather than trying 
>>>>> to extract relevant portions.  you can identify sections by the 
>>>>> timestamps of when you did things
>>>>> 
>>>>> we are making slow progress - but have yet to see debug successfully 
>>>>> enabled!!  However, simulator does appear to be OK - and indeed the 
>>>>> website confirms that simulator is running just fine.
>>>>> 
>>>>> See if you can get us some log which shows debug enabled at startup!!
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Friday, 14 June 2019 15:13:07 UTC+3, Damjan Hajsek wrote:
>>>>>> here it is.
>>>>>> Is this ok what I attach?
>>>>>> regards
>>>>>> 
>>>>>> Dne petek, 14. junij 2019 12.45.41 UTC+2 je oseba Andrew Milner napisala:
>>>>>&

Re: [weewx-user] Re: weewx stop working

[Leon Shaner]

Gary, good sleuthing.

Could it be that the debug messages simply go to a different file, such as 
/var/log/debug?

Check to see where the debug logs are pointed in /etc/rsyslog.conf.

Also, see here:

https://www.the-art-of-web.com/system/rsyslog-config/

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPhone)

> On Jun 15, 2019, at 10:19 AM, gjr80  wrote:
> 
> I guess the question is where to from here given that it appears you cannot 
> get any debug output when running under centos7.6. I think we can assume that 
> your system is running properly (debug issue aside) with the simulator 
> driver. Troubleshooting the interceptor driver is going to be difficult 
> without any debug output. One thing I did notice, some time ago Matthew 
> suggested running WeeWX (with the interceptor driver) directly. You could try 
> that again, remember what we are seeking when running WeeWX directly is not 
> the log output but rather the console output, ie what you see on the screen. 
> I notice you don't appear to have provided the console output, rather you 
> gave the log output.
> 
> Gary
> 
>> On Saturday, 15 June 2019 02:28:14 UTC+10, Damjan Hajsek wrote:
>> Ok I did logs again I hope this time better.
>> 
>> 
>> Dne petek, 14. junij 2019 14.44.03 UTC+2 je oseba Andrew Milner napisala:
>>> 
>>> Not really, no
>>> 
>>> The first part appears to show normal running with archive records every 
>>> minute - but no log for the startup process
>>> 
>>> The second part, manual running , shows us the startup but only loop data - 
>>> it runs for under a minute so does not give the coverage for two archive 
>>> periods (which in your case would be at least 2 minutes)
>>> 
>>> The third part says debug disabled - but there does not appear to be any 
>>> section where debug is enabled - so it is not clear if you are enabling 
>>> debug correctly or not.
>>> 
>>> it is much better to just attach the logfile directly rather than trying to 
>>> extract relevant portions.  you can identify sections by the timestamps of 
>>> when you did things
>>> 
>>> we are making slow progress - but have yet to see debug successfully 
>>> enabled!!  However, simulator does appear to be OK - and indeed the website 
>>> confirms that simulator is running just fine.
>>> 
>>> See if you can get us some log which shows debug enabled at startup!!
>>> 
>>> 
>>> 
>>> 
>>> 
>>>> On Friday, 14 June 2019 15:13:07 UTC+3, Damjan Hajsek wrote:
>>>> here it is.
>>>> Is this ok what I attach?
>>>> regards
>>>> 
>>>> Dne petek, 14. junij 2019 12.45.41 UTC+2 je oseba Andrew Milner napisala:
>>>>> 
>>>>> Keep running the simulator for a while, but can you now give us the log 
>>>>> from startup for at least two archive intervals FOR SIMULATOR so that we 
>>>>> can be SURE everything is working as it should be.  Then, if that looks 
>>>>> ok, can you stop weewx, set debug = 1, restart weewx and again attach the 
>>>>> log from startup until at least two archive intervals.  This should show 
>>>>> that all is ok with debug also!!
>>>>> 
>>>>>  
>>>>> 
>>>>>> On Friday, 14 June 2019 12:36:32 UTC+3, Damjan Hajsek wrote:
>>>>>> https://github.com/matthewwall/weewx-interceptor
>>>>>> 
>>>>>>  this is what I have for interceptor in weewx.conf
>>>>>> 
>>>>>> [Interceptor]
>>>>>> # This section is for the network traffic interceptor driver.
>>>>>> 
>>>>>> # The driver to use:
>>>>>> driver = user.interceptor
>>>>>> 
>>>>>> # Specify the hardware device to capture.  Options include:
>>>>>> #   acurite-bridge - acurite internet bridge, smarthub, or access
>>>>>> #   observer - fine offset WH2600/HP1000/HP1003, ambient WS2902
>>>>>> #   lw30x - oregon scientific LW301/LW302
>>>>>> #   lacrosse-bridge - lacrosse GW1000U/C84612 internet bridge
>>>>>> #   wu-client - any hardware that uses the weather underground 
>>>>>> protocol
>>>>>> #device_type = acurite-bridge
>>>>>> device_type = observer
>>>>>> port = 990
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/2dc29325-db15-4831-899a-f256e2c024b4%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/AD3FDEC5-6CF4-4A58-9132-D1D9EE55BC44%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: weewx stop working

[Leon Shaner]

Damjan,
A long time ago you mentioned that your troubles all started when you updated 
some OS pkgs.
Have you seen this thread?  Since you are using interceptor, which must accept 
connections from remote, it could be relevant:

https://www.centos.org/forums/viewtopic.php?t=69009

To rule it out, perhaps you could temporarily disable firewalld and fail2ban?

$ sudo systemctl disable firewalld
$ sudo systemctl stop firewalld

$ sudo systemctl disable fail2ban
$ sudo systemctl stop fail2ban

If disabling those helps with weewx, then you may need to explicitly add rules 
for interceptor on 990 to be allowed, that way you can have the benefit of 
those firewalls without breaking weewx.

In the meantime you probably want to "enable" and "start" them so your system 
isn't left vulnerable.

Lemme know if disabling those helps and then I can dig out the proper syntax to 
add a rule to allow interceptor to connect even with firewalld active.   It may 
still not work, though, depending on the outcome of that thread mentioned far 
above.  That last comment calls an an issue for people upgrading from CentOS 
7.5 to 7.6, but doesn't explicitly say how he got it working again.  :-/

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad)

> On Jun 14, 2019, at 12:28 PM, Damjan Hajsek  wrote:
> 
> 
> Ok I did logs again I hope this time better.
> 
> 
> Dne petek, 14. junij 2019 14.44.03 UTC+2 je oseba Andrew Milner napisala:
>> 
>> Not really, no
>> 
>> The first part appears to show normal running with archive records every 
>> minute - but no log for the startup process
>> 
>> The second part, manual running , shows us the startup but only loop data - 
>> it runs for under a minute so does not give the coverage for two archive 
>> periods (which in your case would be at least 2 minutes)
>> 
>> The third part says debug disabled - but there does not appear to be any 
>> section where debug is enabled - so it is not clear if you are enabling 
>> debug correctly or not.
>> 
>> it is much better to just attach the logfile directly rather than trying to 
>> extract relevant portions.  you can identify sections by the timestamps of 
>> when you did things
>> 
>> we are making slow progress - but have yet to see debug successfully 
>> enabled!!  However, simulator does appear to be OK - and indeed the website 
>> confirms that simulator is running just fine.
>> 
>> See if you can get us some log which shows debug enabled at startup!!
>> 
>> 
>> 
>> 
>> 
>>> On Friday, 14 June 2019 15:13:07 UTC+3, Damjan Hajsek wrote:
>>> here it is.
>>> Is this ok what I attach?
>>> regards
>>> 
>>> Dne petek, 14. junij 2019 12.45.41 UTC+2 je oseba Andrew Milner napisala:
>>>> 
>>>> Keep running the simulator for a while, but can you now give us the log 
>>>> from startup for at least two archive intervals FOR SIMULATOR so that we 
>>>> can be SURE everything is working as it should be.  Then, if that looks 
>>>> ok, can you stop weewx, set debug = 1, restart weewx and again attach the 
>>>> log from startup until at least two archive intervals.  This should show 
>>>> that all is ok with debug also!!
>>>> 
>>>>  
>>>> 
>>>>> On Friday, 14 June 2019 12:36:32 UTC+3, Damjan Hajsek wrote:
>>>>> https://github.com/matthewwall/weewx-interceptor
>>>>> 
>>>>>  this is what I have for interceptor in weewx.conf
>>>>> 
>>>>> [Interceptor]
>>>>> # This section is for the network traffic interceptor driver.
>>>>> 
>>>>> # The driver to use:
>>>>> driver = user.interceptor
>>>>> 
>>>>> # Specify the hardware device to capture.  Options include:
>>>>> #   acurite-bridge - acurite internet bridge, smarthub, or access
>>>>> #   observer - fine offset WH2600/HP1000/HP1003, ambient WS2902
>>>>> #   lw30x - oregon scientific LW301/LW302
>>>>> #   lacrosse-bridge - lacrosse GW1000U/C84612 internet bridge
>>>>> #   wu-client - any hardware that uses the weather underground 
>>>>> protocol
>>>>> #device_type = acurite-bridge
>>>>> device_type = observer
>>>>> port = 990
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/57255078-0935-48db-b1c3-2ce7813b488a%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
> 
> 
> 

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/B2D4D668-35E7-4F99-ABA9-EB37134AAEB7%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] New to Weewx. Conf file

[Leon Shaner]

Or can use this form and it will choose the user's preferred editor (a la 
SUDO_EDITOR, VISUAL or EDITOR variables):

$ sudoedit /etc/weewx/weewx.conf

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad)

> On Jun 13, 2019, at 9:50 AM, Dave Webb KB1PVH  wrote:
> 
> 
> Oops... sudo nano /etc/weewx/weewx.conf
> 
> Dave-KB1PVH
> 
> 
> Sent from my Galaxy S9
> 
>> On Thu, Jun 13, 2019, 9:49 AM Dave Webb KB1PVH  wrote:
>> Use sudo /etc/weeex/weewx.conf
>> 
>> Dave-KB1PVH
>> 
>> 
>> Sent from my Galaxy S9
>> 
>>> On Thu, Jun 13, 2019, 9:47 AM Dale  wrote:
>>> Greeting I'm new to Linux.
>>> 
>>> Computer: Raspberry pi 3
>>> Software: Raspbian-stretch-full
>>> Weather Station: WMR968
>>> 
>>> I'm having problems understanding on how to gain access to the conf file to 
>>> make changes.
>>> I have used the terminal and typed: /etc/weewx/weewx.conf
>>> Comes back Permission denied
>>> 
>>> I have down loaded the weewx: installation on Debian systems and User's 
>>> Guide.
>>> 
>>> Yes , I have Verified to make sure the weathers station is running from the 
>>> web browser. file:///var/www/html/weewx/index.html 
>>> 
>>> Dale KJ6IX
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "weewx-user" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to weewx-user+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/weewx-user/4fd01c92-a44b-4644-8793-3512a39c3e1f%40googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/CAEMY9Fez4t678Z1CoWYgc1FFqbo8hNc4tzoSL8-B5Nd%3DBh9%2B%2BA%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/46FA5F53-E897-41C6-A7E9-92AE3EC107AC%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Thanks, Rod!  =D

Those are precisely the same tests I ran and exact same results that noted, 
before  before publishing.  =D
Plus a ton of checks against my own station, KMIDEARB5, west of UTC.

As before, even more important than these 'historical' API tests is to query 
against the current day at different times throughout the day, such as before 
and after your midnight and before and after UTC midnight.  That logic which 
uses the 'today' API (rapid) for today is the one we will likely keep no matter 
what IBM does with the well-known bugs in the 'historical' API.

(I can't test the 'today' logic 100% without changing my localtime on my system 
and I don't want to do that, which is why I appreciate help from others to 
validate the code).  =D

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On Jun 3, 2019, at 5:39 PM, Rod Yager  wrote:
> 
> Dear Leon,
> 
> Thanks for all your work on this. This works as well as we are going to be 
> able to manage unless and until WU fixes the bug. 
> 
> The remaining issue affects just one date, the  “change-over” date when WU 
> transitions to returning the correct data - and will only bite for stations 
> east of UTC, and only in the hours that they are ahead of UTC.
> 
> Currently, for me, that date is 2019-05-28. When it originally asks for 
> 2019-05-28, it returns the data for 2019-05-29. Wunderfixer then tries to 
> compensate by asking for 2019-05-27. But 2019-05-27 is a date before the WU 
> bug, and so it returns the data for 2019-05-27. We can’t work around this, 
> because no request to WU will actually return the data we want. The only fix 
> is to wait a few hours until the UTC date is aligned with the Sydney date.
> 
> Good news is that for stations west of UTC, this won’t happen as the bug 
> causes a duplicate day, not a missing day.
> 
> Here’s the output for the requests for my station from 2019-05-27 (before the 
> WU bug kicks in) to today 2019-06-04.
> 
> 
> [rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 
> 2019-05-27  --test --verbose | (head -n 11 ; tail -n 3)
> Using configuration file /home/weewx/weewx.conf.
> Weather Underground Station:   ISYDNEY155
> Date to check: 2019-05-27
> WU API obsTimeLocal:   2019-05-27
> epoch: 1558879200 date_epoch_local: 2019-05-27 00:00:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-26T14:00:00Z obsTimeLocal: 2019-05-27 00:00:00
> epoch: 1558879500 date_epoch_local: 2019-05-27 00:05:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-26T14:05:00Z obsTimeLocal: 2019-05-27 00:05:00
> epoch: 1558879800 date_epoch_local: 2019-05-27 00:10:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-26T14:10:00Z obsTimeLocal: 2019-05-27 00:10:00
> epoch: 1558880100 date_epoch_local: 2019-05-27 00:15:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-26T14:15:00Z obsTimeLocal: 2019-05-27 00:15:00
> epoch: 1558880400 date_epoch_local: 2019-05-27 00:20:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-26T14:20:00Z obsTimeLocal: 2019-05-27 00:20:00
> epoch: 1558880700 date_epoch_local: 2019-05-27 00:25:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-26T14:25:00Z obsTimeLocal: 2019-05-27 00:25:00
> epoch: 1558881000 date_epoch_local: 2019-05-27 00:30:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-26T14:30:00Z obsTimeLocal: 2019-05-27 00:30:00
> epoch: 1558965000 date_epoch_local: 2019-05-27 23:50:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-27T13:50:00Z obsTimeLocal: 2019-05-27 23:50:00
> epoch: 1558965300 date_epoch_local: 2019-05-27 23:55:00 tz: Australia/Sydney 
> obsTimeUtc: 2019-05-27T13:55:00Z obsTimeLocal: 2019-05-27 23:55:00
> Number of WU records:  288
> [rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 
> 2019-05-28  --test --verbose | (head -n 11 ; tail -n 3)
> 
> No results returned from Weather Underground (perhaps a bad station name??).
> Using configuration file /home/weewx/weewx.conf.
> Weather Underground Station:   ISYDNEY155
> Date to check: 2019-05-28
> WU API obsTimeLocal:   2019-05-29
> WU API RETURNED WRONG DATE!!!
> WU API COMPENSATION DATE:  2019-05-27
> WU API obsTimeLocal:   2019-05-27
> WU API COMPENSATION FAILURE! ABORTING!!!
> Number of WU records:  0
> [rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 
> 2019-05-29  --test --verbose | (head -n 11 ; tail -n 3)
> Using configuration file /home/weewx/weewx.conf.
> Weather Underground Station:   ISYDNEY155
> Date to check: 2019-05-29
> WU API obsTimeLocal:   2019-05-30
> WU API RETURNED WRONG DATE!!!
> WU API COMPENSATION DATE:  2019-05-28
> WU API obsTimeLocal:   2019-05-29
> epoch: 1559052000 date_epoch_local: 2019-05-29 00:00:00 tz: Aust

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

All,
There are updates to the wunderdates utilities, to implement the following:

Add +/- 1-day compensation logic to workaround WU 'historical' API bug, 
depending on whether station is east/west of UTC, and whether it is within X 
hours of UTC midnight, relative to the local station time.

These wunderdates utilities are for debug purposes only to aid in testing the 
WU behavior, while IBM works to fix the actual bug (no intention of adding this 
logic to wunderfixer, which will remain broken until IBM fixes their API).

The bugs are well known / well documented.
The continued use of wunderdates is just to keep an eye on IBM's progress in 
fixing the bugs, e.g. keep an eye out for if the behavior changes / improves.

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On Jun 1, 2019, at 1:17 PM, Leon Shaner  wrote:
> 
> All,
> 
> I'm testing a new approach.  Below you will find links to an updated 
> wunderdates utility that can be used to validate whether I am on the right 
> track.
> The wunderdates utility simply dumps out timestamp related records from what 
> WU returns from the query.
> 
> We're looking mainly at the obsTimeUtc and obsTimeLocal values, which were 
> demonstrated under the prior approach to be returning the wrong dates when 
> within the stations localtime +/- offset from UTC.
> 
> The new approach is to detect if the requested date is 'today' and if so, use 
> a different API URL that already assumes 'today' and will hopefully not be 
> subject to the UTC offset bugs we've been chasing with the historical data 
> API URL.
> 
> I have my crontab set to do another test approaching my UTC offset, just 
> after coming within the offset, and then again just before and after midnight 
> localtime.
> (Same test I did before, but now with the new approach in place).
> 
> Here is the utility for anyone else that wants to check out the behavior:
> 
> https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates3
> https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates4
> 
> Which version you pull will depend on which base weewx you are running.
> Pull the one that matches your weewx version and place it in bin, next to 
> wunderfixer, and it will take the same arguments as wunderfixer.
> 
> You can just try wunderfixer as you normally would (with --apikey) and then 
> run wunderdates(3 or 4) with exact same arguments to be able to see what 
> actual timestamps WU is sending back for the date queried.
> Parameters like --epsilon don't have any effect in the case of wunderdates, 
> but I left it there so you don't have to change options when running the util 
> to get the debug output.
> 
> Regards,
> \Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPad Pro)
> 
>> On May 30, 2019, at 12:40 AM, Leon Shaner  wrote:
>> 
>> Rod,
>> Thanks again for this.
>> 
>> Since the in-progress version of wunderfixer doesn't really show you the 
>> dates that come back from WU, I have written a tool just to debug the dates.
>> 
>> The command-line input and basing of defaults on weewx.conf works the same 
>> as wunderfixer, except it doesn't look at your DB at all.  It only prints 
>> out datestamps in various incarnations coming back from WU and as compared 
>> to your system's localtime.
>> 
>> It would be helpful to see the "wunderdates" output at times like you've 
>> shown below, a la before and after your localtime rolls around to UTC 
>> midnight.
>> 
>> Since you are at UTC + 10, another interesting time would be 11+ hours on 
>> either side of UTC midnight, in addition to within 9 or less hours.  This is 
>> just to make sure we're covering all the corner cases.
>> 
>> Gary reported a difference for stations that are east vs. west of GMT, and I 
>> expect we're really chasing the same bug in that there is some bad math WU 
>> is doing based on UTC offset, but since an offset can be +/-, the effects go 
>> in opposite directions date-wise, depending on which side of the UTC 
>> dateline your station is located.
>> 
>> At least that is what I surmise may be happening, but the wunderdates 
>> utility should shed light one way or the other.  =D
>> 
>> The wunderdates utility is available at the links below.
>> 
>> https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates3
>> https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates4
>> 
>> Which version you pull will depend on which base weewx you are running.
>> Pull the one that matches your weewx version and place it in bin, next to 
>> wunderfixer, and it will take the same arguments as wunderfixer.
>> 
>> You can just try wu

Re: [weewx-user] We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Rod,

Thanks again for the added validation of what I saw earlier.
Never minding for now the issues with last week's data, what I am now most 
concerned to know is whether there is any point throughout the PRESENT day 
(your local time) that the latest wunderdates is showing the wrong results?

Recall that for the present day I am using a 'today' API which is for the 
'today' results.
That method is working for my station at all times during the PRESENT day (even 
within my UTC offset of local midnight).

I thought I understood you to say that those changes were no good for you at 
certain times of the PRESENT day, such as before 10 a.m. your local time, was 
it?

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On Jun 2, 2019, at 5:50 PM, Rod Yager  wrote:
> 
> Well, it turns out that the East of the UTC date line bug only occurs for the 
> latest week. (I’d assumed that testing involving requests for the last few 
> days would show the behaviour, but that turned out to be wrong.)
> 
> At the moment, I’m a day ahead of UTC.  Here is the output for requests for 
> 2019-05-27 (within the last week and giving data for the 2019-05-28) and for 
> 2019-05-26 (not within the last week and giving the correct data for 
> 2019-05-06)  [I repeated the 2019-05-27 responses to show that they hadn’t 
> happened coincidentally at a time where WU fixed the problem.]  All dates 
> I’ve tested prior to 2019-05-27 also work as we would like. All dates within 
> the last week give data for the day following the day in the request.
> 
> [rodyager@moses ~]$ date
> Mon Jun  3 07:32:55 AEST 2019
> [rodyager@moses ~]$ /home/weewx/bin/wunderdates --epsilon=125 --date 
> 2019-05-27 --station ISYDNEY155 --test --verbose | (head -n 8 ; tail -n 1)
> Using configuration file /home/weewx/weewx.conf.
> Weather Underground Station:   ISYDNEY155
> Date to check: 2019-05-27
> epoch: 1558965600 date_epoch_local: 2019-05-28 00:00:00 utcepoch: 1558964600 
> date_epoch_utc: 2019-05-27 14:00:00 tz: Australia/Sydney obsTimeUtc: 
> 2019-05-27T14:00:00Z obsTimeLocal: 2019-05-28 00:00:00
> epoch: 1558965900 date_epoch_local: 2019-05-28 00:05:00 utcepoch: 1558964900 
> date_epoch_utc: 2019-05-27 14:05:00 tz: Australia/Sydney obsTimeUtc: 
> 2019-05-27T14:05:00Z obsTimeLocal: 2019-05-28 00:05:00
> epoch: 1558966200 date_epoch_local: 2019-05-28 00:10:00 utcepoch: 1558965200 
> date_epoch_utc: 2019-05-27 14:10:00 tz: Australia/Sydney obsTimeUtc: 
> 2019-05-27T14:10:00Z obsTimeLocal: 2019-05-28 00:10:00
> epoch: 1558966500 date_epoch_local: 2019-05-28 00:15:00 utcepoch: 1558965500 
> date_epoch_utc: 2019-05-27 14:15:00 tz: Australia/Sydney obsTimeUtc: 
> 2019-05-27T14:15:00Z obsTimeLocal: 2019-05-28 00:15:00
> epoch: 1558966800 date_epoch_local: 2019-05-28 00:20:00 utcepoch: 1558965800 
> date_epoch_utc: 2019-05-27 14:20:00 tz: Australia/Sydney obsTimeUtc: 
> 2019-05-27T14:20:00Z obsTimeLocal: 2019-05-28 00:20:00
> Number of WU records:  288
> [rodyager@moses ~]$ /home/weewx/bin/wunderdates --epsilon=125 --date 
> 2019-05-26 --station ISYDNEY155 --test --verbose | (head -n 8 ; tail -n 1)
> Using configuration file /home/weewx/weewx.conf.
> Weather Underground Station:   ISYDNEY155
> Date to check: 2019-05-26
> epoch: 1558792800 date_epoch_local: 2019-05-26 00:00:00 utcepoch: 1558791800 
> date_epoch_utc: 2019-05-25 14:00:00 tz: Australia/Sydney obsTimeUtc: 
> 2019-05-25T14:00:00Z obsTimeLocal: 2019-05-26 00:00:00
> epoch: 1558793100 date_epoch_local: 2019-05-26 00:05:00 utcepoch: 1558792100 
> date_epoch_utc: 2019-05-25 14:05:00 tz: Australia/Sydney obsTimeUtc: 
> 2019-05-25T14:05:00Z obsTimeLocal: 2019-05-26 00:05:00
> epoch: 1558793400 date_epoch_local: 2019-05-26 00:10:00 utcepoch: 1558792400 
> date_epoch_utc: 2019-05-25 14:10:00 tz: Australia/Sydney obsTimeUtc: 
> 2019-05-25T14:10:00Z obsTimeLocal: 2019-05-26 00:10:00
> epoch: 1558793700 date_epoch_local: 2019-05-26 00:15:00 utcepoch: 1558792700 
> date_epoch_utc: 2019-05-25 14:15:00 tz: Australia/Sydney obsTimeUtc: 
> 2019-05-25T14:15:00Z obsTimeLocal: 2019-05-26 00:15:00
> epoch: 1558794000 date_epoch_local: 2019-05-26 00:20:00 utcepoch: 1558793000 
> date_epoch_utc: 2019-05-25 14:20:00 tz: Australia/Sydney obsTimeUtc: 
> 2019-05-25T14:20:00Z obsTimeLocal: 2019-05-26 00:20:00
> Number of WU records:  288
> [rodyager@moses ~]$ /home/weewx/bin/wunderdates --epsilon=125 --date 
> 2019-05-27 --station ISYDNEY155 --test --verbose | (head -n 8 ; tail -n 1)
> Using configuration file /home/weewx/weewx.conf.
> Weather Underground Station:   ISYDNEY155
> Date to check: 2019-05-27
> epoch: 1558965600 date_epoch_local: 2019-05-28 00:00:00 utcepoch: 1558964600 
> date_epoch_utc: 2019-05-27 14:00:00 tz: Australia/Sydney obsTimeUtc: 

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

019-05-28
WU API obsTimeLocal:   2019-05-29
epoch: 1559052000 date_epoch_local: 2019-05-28 10:00:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-28T14:00:00Z obsTimeLocal: 2019-05-29 00:00:00
epoch: 1559052300 date_epoch_local: 2019-05-28 10:05:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-28T14:05:00Z obsTimeLocal: 2019-05-29 00:05:00
Number of WU records:  288

pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates 
--epsilon=125 --date 2019-05-28 --station ISYDNEY155 --test --verbose | (head 
-n 9 ; tail -n 1)
Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check: 2019-05-28
WU API obsTimeLocal:   2019-05-29
WU API WRONG DATE !!!
WU API COMPENSATION DATE:  2019-05-27
WU API obsTimeLocal:   2019-05-28
epoch: 1558965600 date_epoch_local: 2019-05-27 10:00:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-27T14:00:00Z obsTimeLocal: 2019-05-28 00:00:00
epoch: 1558965900 date_epoch_local: 2019-05-27 10:05:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-27T14:05:00Z obsTimeLocal: 2019-05-28 00:05:00
Number of WU records:  288

/usr/share/weewx4/bin/wunderdates --epsilon=125 --date 2019-05-27 --station 
ISYDNEY155 --test --verbose | (head -n 9 ; tail -n 1)
Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check: 2019-05-27
WU API obsTimeLocal:   2019-05-28
WU API WRONG DATE !!!
WU API COMPENSATION DATE:  2019-05-26
WU API obsTimeLocal:   2019-05-26
WU API COMPSENSATION FAILURE! ABORTING!!!
Number of WU records:  0

No results returned from Weather Underground (perhaps a bad station name??).


pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates 
--epsilon=125 --date 2019-05-26 --station ISYDNEY155 --test --verbose | (head 
-n 9 ; tail -n 1)
Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check: 2019-05-26
WU API obsTimeLocal:   2019-05-26
epoch: 1558792800 date_epoch_local: 2019-05-25 10:00:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-25T14:00:00Z obsTimeLocal: 2019-05-26 00:00:00
epoch: 1558793100 date_epoch_local: 2019-05-25 10:05:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-25T14:05:00Z obsTimeLocal: 2019-05-26 00:05:00
epoch: 1558793400 date_epoch_local: 2019-05-25 10:10:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-25T14:10:00Z obsTimeLocal: 2019-05-26 00:10:00
epoch: 1558793700 date_epoch_local: 2019-05-25 10:15:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-25T14:15:00Z obsTimeLocal: 2019-05-26 00:15:00
epoch: 1558794000 date_epoch_local: 2019-05-25 10:20:00 tz: Australia/Sydney 
obsTimeUtc: 2019-05-25T14:20:00Z obsTimeLocal: 2019-05-26 00:20:00
Number of WU records:  288

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On Jun 2, 2019, at 10:20 AM, Leon Shaner  wrote:
> 
> Rod,
> 
> It's interesting that the "workaround" that I've already done, is actually 
> working 100% for me.   I got good results when combining the 'today' API when 
> asking for the current date, and only using the 'historical' result when 
> asking for dates other than the current date -- relative to station local 
> time.
> 
> Here is my KMIDEARB5 station's transition from 7:59 p.m. to 8:00 p.m, where 
> time moves to within the UTC-4 offset:
> 
> $ diff 59:19:1:6_wu.txt 0:20:1:6_wu.txt
> 241c241,242
> < Number of WU records:  239
> ---
> > epoch: 1559433599 date_epoch_local: 2019-06-01 19:59:59 utcepoch: 
> > 1559433999 date_epoch_utc: 2019-06-01 23:59:59 tz: America/New_York 
> > obsTimeUtc: 2019-06-01T23:59:59Z obsTimeLocal: 2019-06-01 19:59:59
> > Number of WU records:  240
> 
> The above is exactly correct -- the second query is one minute later and is 
> after/on the 5-minute record "boundary" hence one additional record.
> 
> And then as my station transition from 11:59 p.m. to midnight next day.  
> Before midnight the 'today' API is used, but after midnight the 'historical' 
> API will be used, which was always working for me after midnight (station 
> local time):
> 
> $ tail -n 4 59:23:1:6_wu.txt 0:0:2:6_wu.txt
> ==> 59:23:1:6_wu.txt <==
> epoch: 1559447399 date_epoch_local: 2019-06-01 23:49:59 utcepoch: 1559447799 
> date_epoch_utc: 2019-06-02 03:49:59 tz: America/New_York obsTimeUtc: 
> 2019-06-02T03:49:59Z obsTimeLocal: 2019-06-01 23:49:59
> epoch: 1559447699 date_epoch_local: 2019-06-01 23:54:59 utcepoch: 1559448099 
> date_epoch_utc: 2019-06-02 03:54:59 tz: America/New_York obsTimeUtc: 
> 2019-06-02T03:54:59Z obsTimeLocal: 2019-06-01 23:54:59
> epoch: 1559447701 date_epoch_local: 2019-06-01 23:55:01 utcepoch: 1559448101 
> date_epoch_utc: 2019-06-02 03:55:01 tz: America/New_Y

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Rod,

Thanks for testing.  Well, WU's approach is flawed, not mine, really.  :S

It means that both API's, the one I just added for 'today" and the one from 
before that was for 'historical' (including today), both have the same bug.

For now, the best workaround I can think of is to only ever run the new 
wunderfixer only ever for the previous day, only ever after UTC has rolled over 
midnight.   :-(
E.g. when it is 2019-06-02 in UTC, you can get the right results with 
wunderfixer if you ask for 2019-06-01 only.In other words, always delay 
your wunderfixer fix ups by a day, operating always on yesterday's date 
(relative to UTC) after UTC has rolled over to the next day.

Scroll to see the output of me running wunderdates on my local machine in UTC-4 
against your station in UTC+10.

Because I am asking for 2019-06-01, which is also my local date, the logic will 
use the URL that queries for 'today' and instead of returning obsTimeLocal 
records, it is returning obsTimeUtc records for the requested date.  We can see 
that it's returning 2019-06-02 relative to your station, even though I asked 
for 2019-06-01.

Sure, I could go a bit further and use a related API query to check the 
timezone of your station and use date conversions to check if the date 
requested is 'today' relative to the station requested, but really this is 
intended to be run on the same machine as weewx, so the localtime date should 
match that of the station.  It means there isn't any real reason for me to 
workaround their bug.  In fact even if I did, then the logic would just fall 
through to the historical data URL, which we already know is buggy.

For now, I'll just report this to IBM a bug in the 'today' API, similar to the 
bug in the 'historical' API.

$ ./wunderdates --epsilon=125 --date 2019-06-01 --station ISYDNEY155 --test 
--verbose | more

Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check: 2019-06-01
epoch: 1559397600 date_epoch_local: 2019-06-01 10:00:00 utcepoch: 1559398000 
date_epoch_utc: 2019-06-01 14:00:00 tz: Australia/Sydney obsTimeUtc: 
2019-06-01T14:00:00Z obsTimeLocal: 2019-06-02 00:00:00
epoch: 1559397900 date_epoch_local: 2019-06-01 10:05:00 utcepoch: 1559398300 
date_epoch_utc: 2019-06-01 14:05:00 tz: Australia/Sydney obsTimeUtc: 
2019-06-01T14:05:00Z obsTimeLocal: 2019-06-02 00:05:00
...


Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On Jun 1, 2019, at 6:03 PM, Rod Yager  wrote:
> 
> Dear Leon,
> 
> Your approach is flawed. It won’t work for historical data.
> 
> If you run
> 
> wunderdates —date=2019-01-01   at 9pm your time   you will receive the data 
> for 2018-12-31  (because your station is a day behind UTC at the time of the 
> request, so it gives you the data for the previous day).
> 
> On the other hand, if you run 
> 
> wunderdates —date=2019-01-01   at 9am your time   you will receive the data 
> for 2019-01-01  (because your station is in the same day as UTC at the time 
> of the request).
> 
> If I do the same thing (I’m in UTC+10)
> 
> wunderdates —date=2019-01-01   at 9pm my time  produces the data for 
> 2019-01-01 (because my station is in the same day as UTC at the time of the 
> request)  but
> wunderdates —date=2019-01-01   at 9am my time produces the data for 
> 2019-01-02 (because my station is a day ahead of UTC at the time of the 
> request) 
> 
> Rod
> 
>> On 2 Jun 2019, at 3:17 am, Leon Shaner  wrote:
>> 
>> All,
>> 
>> I'm testing a new approach.  Below you will find links to an updated 
>> wunderdates utility that can be used to validate whether I am on the right 
>> track.
>> The wunderdates utility simply dumps out timestamp related records from what 
>> WU returns from the query.
>> 
>> We're looking mainly at the obsTimeUtc and obsTimeLocal values, which were 
>> demonstrated under the prior approach to be returning the wrong dates when 
>> within the stations localtime +/- offset from UTC.
>> 
>> The new approach is to detect if the requested date is 'today' and if so, 
>> use a different API URL that already assumes 'today' and will hopefully not 
>> be subject to the UTC offset bugs we've been chasing with the historical 
>> data API URL.
>> 
>> I have my crontab set to do another test approaching my UTC offset, just 
>> after coming within the offset, and then again just before and after 
>> midnight localtime.
>> (Same test I did before, but now with the new approach in place).
>> 
>> Here is the utility for anyone else that wants to check out the behavior:
>> 
>> https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates3
>> https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates4
>>

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

All,

I'm testing a new approach.  Below you will find links to an updated 
wunderdates utility that can be used to validate whether I am on the right 
track.
The wunderdates utility simply dumps out timestamp related records from what WU 
returns from the query.

We're looking mainly at the obsTimeUtc and obsTimeLocal values, which were 
demonstrated under the prior approach to be returning the wrong dates when 
within the stations localtime +/- offset from UTC.

The new approach is to detect if the requested date is 'today' and if so, use a 
different API URL that already assumes 'today' and will hopefully not be 
subject to the UTC offset bugs we've been chasing with the historical data API 
URL.

I have my crontab set to do another test approaching my UTC offset, just after 
coming within the offset, and then again just before and after midnight 
localtime.
(Same test I did before, but now with the new approach in place).

Here is the utility for anyone else that wants to check out the behavior:

https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates3
https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates4

Which version you pull will depend on which base weewx you are running.
Pull the one that matches your weewx version and place it in bin, next to 
wunderfixer, and it will take the same arguments as wunderfixer.

You can just try wunderfixer as you normally would (with --apikey) and then run 
wunderdates(3 or 4) with exact same arguments to be able to see what actual 
timestamps WU is sending back for the date queried.
Parameters like --epsilon don't have any effect in the case of wunderdates, but 
I left it there so you don't have to change options when running the util to 
get the debug output.

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 30, 2019, at 12:40 AM, Leon Shaner  wrote:
> 
> Rod,
> Thanks again for this.
> 
> Since the in-progress version of wunderfixer doesn't really show you the 
> dates that come back from WU, I have written a tool just to debug the dates.
> 
> The command-line input and basing of defaults on weewx.conf works the same as 
> wunderfixer, except it doesn't look at your DB at all.  It only prints out 
> datestamps in various incarnations coming back from WU and as compared to 
> your system's localtime.
> 
> It would be helpful to see the "wunderdates" output at times like you've 
> shown below, a la before and after your localtime rolls around to UTC 
> midnight.
> 
> Since you are at UTC + 10, another interesting time would be 11+ hours on 
> either side of UTC midnight, in addition to within 9 or less hours.  This is 
> just to make sure we're covering all the corner cases.
> 
> Gary reported a difference for stations that are east vs. west of GMT, and I 
> expect we're really chasing the same bug in that there is some bad math WU is 
> doing based on UTC offset, but since an offset can be +/-, the effects go in 
> opposite directions date-wise, depending on which side of the UTC dateline 
> your station is located.
> 
> At least that is what I surmise may be happening, but the wunderdates utility 
> should shed light one way or the other.  =D
> 
> The wunderdates utility is available at the links below.
> 
> https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates3
> https://github.com/UberEclectic/weewx/blob/wuapi/bin/wunderdates4
> 
> Which version you pull will depend on which base weewx you are running.
> Pull the one that matches your weewx version and place it in bin, next to 
> wunderfixer, and it will take the same arguments as wunderfixer.
> 
> You can just try wunderfixer as you normally would (with --apikey) and then 
> run wunderdates(3 or 4) with exact same arguments to be able to see what 
> actual timestamps WU is sending back for the date queried.
> Parameters like --epsilon don't have any effect in the case of wunderdates, 
> but I left it there so you don't have to change options when running the util 
> to get the debug output.
> 
> What I've been doing is saving the output to files for use with sdiff 
> (side-by-side) diff.
> Or you can just compare the head and tail of each file individually.
> 
> Example for my system before and after my local time on 5-28 once it was 
> at/after 8 p.m. here, which is within 4 hours of UTC midnight (I am at UTC-4).
> 
> This output is optimized for screen widths 203 or wider.  Sorry.  :S
> Mainly the last two data fields in the output tell what we need to know.
> 
> 
> pi@nixie:/var/tmp $ head -n 3 59:19:28:5_wu.txt  0:20:28:5_wu.txt
> ==> 59:19:28:5_wu.txt <==
> Using configuration file /usr/share/weewx4/weewx.conf.
> epoch: 1559016299 date_epoch_local: 2019-05-28 00:04:59 utcepoch: 1559016699 
> date_epoch_utc: 2019-05-28 04:04:59 tz: America/New_Yo

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Andrew,

We can't use this as a workaround, but what you're suggesting is essentially 
what WU should be doing with the date that we provide in the query URL.
WU should be doing the conversion to station localtime and returning the values 
for that date, relative to the station.
But they seem to be returning the values for the date relative to UTC instead.

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 31, 2019, at 1:14 AM, Andrew Milner  
> wrote:
> 
> . so does this just simplify to the fact that in the api the date is 
> taken as being the UTC date?
> ie if one requests data for 20190529, and timezone is +3 you get back data 
> for UTC00:00 29 may to 23:59 29 may 
> which is 03:00 29 may local to 23:59 29 may local AND 00:00 30 may local - 
> 02:59 30 may local
> 
> so to get the data for 29 may local it would be necessary to 
> 1. get data for utc 28 may and discard utc 00:00 - 20:59
> 2. get data for utc 29 may and discard utc 21:00 - 23:59
> 
> or have i not understood correctly?
> 
> 
>> On Friday, 31 May 2019 05:48:09 UTC+3, Rod Yager wrote:
>> 
>> I constructed a cron job which exectued  wunderdates --date=2019-05-30  
>> twice every hour, once 2 minutes before the hour and once 2 minutes after 
>> the hour.
>> 
>> Consistent with my earlier post, between 00:00+10 and 09:59+10 the reply 
>> from the api had data which belonged to 2019-05-31 - the day after the 
>> requested date. Before midnight, and after 10am local time, the reply from 
>> the api had data which belonged to 2019-05-30 (as requested).
>> 
>> I've pasted below some representative samples of the last 3 lines of the 
>> wunderdates output.
>> 
>> Before midnight on May 30: Request is for May 30 data. Replies are May 30 
>> data  Local Date = UTC date.
>> 
>> Thu May 30 20:58:02 2019 UTC+10
>> epoch: 1559213100 date_epoch_local: 2019-05-30 20:45:00 utcepoch: 1559212100 
>> date_epoch_utc: 2019-05-30 10:45:00 tz: Australia/Sydney obsTimeUtc: 
>> 2019-05-30T10:45:00Z obsTimeLocal: 2019-05-30 20:45:00
>> epoch: 1559213400 date_epoch_local: 2019-05-30 20:50:00 utcepoch: 1559212400 
>> date_epoch_utc: 2019-05-30 10:50:00 tz: Australia/Sydney obsTimeUtc: 
>> 2019-05-30T10:50:00Z obsTimeLocal: 2019-05-30 20:50:00
>> Number of WU records:  251
>> 
>> 
>> Thu May 30 22:58:03 2019 UTC+10
>> epoch: 155922 date_epoch_local: 2019-05-30 22:40:00 utcepoch: 1559219000 
>> date_epoch_utc: 2019-05-30 12:40:00 tz: Australia/Sydney obsTimeUtc: 
>> 2019-05-30T12:40:00Z obsTimeLocal: 2019-05-30 22:40:00
>> epoch: 1559220300 date_epoch_local: 2019-05-30 22:45:00 utcepoch: 1559219300 
>> date_epoch_utc: 2019-05-30 12:45:00 tz: Australia/Sydney obsTimeUtc: 
>> 2019-05-30T12:45:00Z obsTimeLocal: 2019-05-30 22:45:00
>> epoch: 1559220600 date_epoch_local: 2019-05-30 22:50:00 utcepoch: 1559219600 
>> date_epoch_utc: 2019-05-30 12:50:00 tz: Australia/Sydney obsTimeUtc: 
>> 2019-05-30T12:50:00Z obsTimeLocal: 2019-05-30 22:50:00
>> Number of WU records:  275
>> 
>> 
>> Thu May 30 23:58:02 2019 UTC+10
>> epoch: 1559223900 date_epoch_local: 2019-05-30 23:45:00 utcepoch: 1559222900 
>> date_epoch_utc: 2019-05-30 13:45:00 tz: Australia/Sydney obsTimeUtc: 
>> 2019-05-30T13:45:00Z obsTimeLocal: 2019-05-30 23:45:00
>> epoch: 1559224200 date_epoch_local: 2019-05-30 23:50:00 utcepoch: 1559223200 
>> date_epoch_utc: 2019-05-30 13:50:00 tz: Australia/Sydney obsTimeUtc: 
>> 2019-05-30T13:50:00Z obsTimeLocal: 2019-05-30 23:50:00
>> Number of WU records:  287
>> 
>> 
>> 10 hour period after midnight on May 31. Request is for May 30 data. Replies 
>> are May 31 data.  Local Date is 1 day ahead of UTC date
>> 
>> Fri May 31 00:02:02 2019 UTC+10
>> Could not get Weather Underground data.
>> Reason: No JSON object could be decoded
>> Could not get Weather Underground data. Exiting. 
>> 
>> 
>> Fri May 31 00:58:02 2019  UTC+10
>> epoch: 1559227500 date_epoch_local: 2019-05-31 00:45:00 utcepoch: 1559226500 
>> date_epoch_utc: 2019-05-30 14:45:00 tz: Australia/Sydney obsTimeUtc: 
>> 2019-05-30T14:45:00Z obsTimeLocal: 2019-05-31 00:45:00
>> epoch: 1559227800 date_epoch_local: 2019-05-31 00:50:00 utcepoch: 1559226800 
>> date_epoch_utc: 2019-05-30 14:50:00 tz: Australia/Sydney obsTimeUtc: 
>> 2019-05-30T14:50:00Z obsTimeLocal: 2019-05-31 00:50:00
>> Number of WU records:  11
>> 
>> 
>> Fri May 31 02:58:03 2019 UTC+10
>> epoch: 1559234700 date_epoch_local: 2019-05-31 02:45:00 utcepoch: 1559233700 
>> date_epoch_utc: 2019-05-30 16:45:00 tz: Austral

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Hi, Rod,

Yes and thanks for adding yet another confirmation of the issue.  =D

I can show that if I do the query within X hours of my offset of UTC, what 
actually happens is they report 288 records from the day PRIOR to the one I am 
asking about.
For example, I ask for 20190528 and they give me records for 20190527, so 
*that* is why wunderfixer "thinks" it needs to re-upload everything.

I am in contact with IBM about it and have shown them irrefutable proof of the 
issue.
They didn't respond back yet, which I expect is because the proof was 
irrefutable.  Ha!  ;-)

I expect that they're investigating and would rather respond from a position of 
understanding, or with any luck maybe even a quick fix.  =D

I meant to follow-up with IBM again this morning, but got waylaid, so I'll do 
that now.
Thanks again, and for the reminder.  =D 

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 29, 2019, at 6:14 PM, Rod Yager  wrote:
> 
> There is definitely a time zone issue. I am in the Sydney Australia timezone 
> (UTC +10 hours).
> 
> It is currently 8am local time on May 30, 2019.   (10pm May 29, 2019 UTC)
> 
> If I execute
> 
> ./wunderfixer --verbose --date=2019-05-29 --epsilon=125
> 
> 
> 
> I get
> 
> 
> 
> Using configuration file /home/weewx/weewx.conf.
> 
> Using database binding 'wx_binding', which is bound to database 
> 'archive_mysql'
> 
> Weather Underground Station:   xxx
> 
> Date to check: 2019-05-29
> 
> Number of archive records: 288
> 
> Number of WU records:  97
> 
> Number of missing records: 288
> 
> 
> 
> Now WU actually has 288 records for 2019-05-29.
> 
> But it only has 97 records for 2019-05-30.
> 
> 
> 
> So it is clear that wunderfixer is downloading the record data for 2019-05-30 
> from WeatherUnderground and trying to match them with the local records for 
> 2019-30-29.
> 
> Of course, they all mismatch, and so wunderfixer concludes that it must 
> upload all the data for 2019-05-29.
> 
> 
> 
> Hope this narrows down the search for a solution.
> 
> 
> 
> Rod
> 
> 
> 
> 
>> On Monday, May 27, 2019 at 9:35:25 PM UTC+10, Leon Shaner wrote:
>> 
>>> On May 27, 2019, at 12:12 AM, gjr80  wrote:
>>> 
>>>> On Monday, 27 May 2019 13:16:53 UTC+10, Leon Shaner wrote:
>>>> [snip]
>>>> If you can see any shorter paths to a more reliable outcome than I have 
>>>> achieved so far, then you know know know I will be very grateful.  =D
>>> 
>>> I am not sure what local/UTC issue you refer to. When I do a 
>>> api.weather.com/v2/pws/history query on a station to the east of Greenwich 
>>> I am returned all records for the date specified (eg 20190525 gives me all 
>>> records for 25 May 2019), each record contains an epoch timestamp which is 
>>> correct and consistent with 25 May 2019. Everything is as I would expect. 
>>> However, when I perform the same query on a station to the west of 
>>> Greenwich I am returned records for the day before the date specified (ie 
>>> 20190525 gives me all records for 24 May 2019 not 25 May 2019), again each 
>>> record contains an epoch timestamp but the timestamp is for the previous 
>>> day Ie 24 May 2019. I have checked a number of data records in the stations 
>>> history table and WU is definitely returning the midnight to midnight data 
>>> for the day before. I have confirmed this behaviour with a number of 
>>> stations both east and west of Greenwich.
>>> 
>>> I don't think there is a local/UTC time issue, I think WU is having some 
>>> implementation issues and for stations west of Greenwich they are returning 
>>> the wrong day of data.
>> 
>> Thanks, Gary!  This was all very helpful.
>> In addition to what you've described across the east vs west of GMT, I get 
>> similar behavior if I am within X hours of my local UTC offset when querying 
>> my own station.
>> Last night as soon as localtime rolled over midnight, the queries for the 
>> previous day were correct.
>> 
>> --Leon
>> 
>>> 
>>> Gary
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/683a28af-35e4-474e-95a0-f684b9926af0%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/FD8C60DC-D403-4411-8A6D-B088AFFC90FC%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

> On May 27, 2019, at 12:12 AM, gjr80  wrote:
> 
>> On Monday, 27 May 2019 13:16:53 UTC+10, Leon Shaner wrote:
>> [snip]
>> If you can see any shorter paths to a more reliable outcome than I have 
>> achieved so far, then you know know know I will be very grateful.  =D
> 
> I am not sure what local/UTC issue you refer to. When I do a 
> api.weather.com/v2/pws/history query on a station to the east of Greenwich I 
> am returned all records for the date specified (eg 20190525 gives me all 
> records for 25 May 2019), each record contains an epoch timestamp which is 
> correct and consistent with 25 May 2019. Everything is as I would expect. 
> However, when I perform the same query on a station to the west of Greenwich 
> I am returned records for the day before the date specified (ie 20190525 
> gives me all records for 24 May 2019 not 25 May 2019), again each record 
> contains an epoch timestamp but the timestamp is for the previous day Ie 24 
> May 2019. I have checked a number of data records in the stations history 
> table and WU is definitely returning the midnight to midnight data for the 
> day before. I have confirmed this behaviour with a number of stations both 
> east and west of Greenwich.
> 
> I don't think there is a local/UTC time issue, I think WU is having some 
> implementation issues and for stations west of Greenwich they are returning 
> the wrong day of data.

Thanks, Gary!  This was all very helpful.
In addition to what you've described across the east vs west of GMT, I get 
similar behavior if I am within X hours of my local UTC offset when querying my 
own station.
Last night as soon as localtime rolled over midnight, the queries for the 
previous day were correct.

--Leon

> 
> Gary

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/7BBF146A-7408-44AD-8676-FDE99ED6532C%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Gary,

It's late, so I'll respond to the rest later, but...

The problem here is that if we compare our local every 1-minute records to WU's 
query that only shows every 5-minute records, then we'll keep re-uploading the 
the "missing" 1-minute records every time wunderfixer is called.  This amounts 
to pointless hits against WU's infrastructure, since pragmatically it's very 
clear they almost always drop all records not closely aligned to 5-minute 
"buckets."

I'm trying to get to the heart of the matter re: what wunderfixer was really 
designed to do, which is to make sure what we have locally is consistent with 
what WU will actually REPORT, and when there are gaps in what WU *should* 
report, and we have local data to fill in those gaps, then re-upload.  BUT to 
not gratuitously re-upload data that WU generally throws away (they just don't 
seem to care about storing data at < 5-minute intervals).

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 27, 2019, at 12:12 AM, gjr80  wrote:
> 
>> On Monday, 27 May 2019 13:16:53 UTC+10, Leon Shaner wrote:
>> Gary,
>> 
>> In practice, WU seems to discard data that is not close to their APPARENTly 
>> preferred 5-minute "normalization buckets."
>> 
>> I upload via rapidfire *and* regular loop on 1-minute intervals, and 
>> irregardless of same, the queries only ever show the records most closely 
>> aligned to their "preferred" 5-minute "buckets."
>> 
>> I would love to see them preserve the same interval that I upload, but 
>> either by design or omission or bug, they simply don't.  :-(
>> Could be the layers in-between are dropping data by code-exception.
>> Could be it's deliberate.
>> I'm not here to debug their code unless they want to start paying me a 
>> worthy salary. ;-)
>> 
>> Well...  MOST of the time the behavior I have described seems to be true.  
>> :-/
>> 
>> While WU normally SEEMS to only keeps records on 5-minute boundaries, I have 
>> seen where if wunderfixer is used persistently enough (say more than 10 
>> times spread out every 20 minutes across a total span of 200 minutes) then 
>> it will keep records that are more frequent than every 5-minutes.
>> 
>> Right now / especially lately, things are so sporadic with WU that I don't 
>> even want to spend cycles chasing what's really going on there.  I would 
>> rather just only have wunderfixer ignore most records except those nearest 
>> the 5-minute boundaries that WU seems to most care about.
>> 
> 
> Show Quoted Content
>> Gary,
>> 
>> In practice, WU seems to discard data that is not close to their APPARENTly 
>> preferred 5-minute "normalization buckets."
>> 
>> I upload via rapidfire *and* regular loop on 1-minute intervals, and 
>> irregardless of same, the queries only ever show the records most closely 
>> aligned to their "preferred" 5-minute "buckets."
>> 
>> I would love to see them preserve the same interval that I upload, but 
>> either by design or omission or bug, they simply don't.  :-(
>> Could be the layers in-between are dropping data by code-exception.
>> Could be it's deliberate.
>> I'm not here to debug their code unless they want to start paying me a 
>> worthy salary. ;-)
>> 
>> Well...  MOST of the time the behavior I have described seems to be true.  
>> :-/
>> 
>> While WU normally SEEMS to only keeps records on 5-minute boundaries, I have 
>> seen where if wunderfixer is used persistently enough (say more than 10 
>> times spread out every 20 minutes across a total span of 200 minutes) then 
>> it will keep records that are more frequent than every 5-minutes.
>> 
>> Right now / especially lately, things are so sporadic with WU that I don't 
>> even want to spend cycles chasing what's really going on there.  I would 
>> rather just only have wunderfixer ignore most records except those nearest 
>> the 5-minute boundaries that WU seems to most care about.
>> 
> I am not suggesting we/you/anyone should try to solve WU 
> issues/problems/whatever, merely making the point that WU is a blackbox, 
> nobody seems to really know what it does with the data that it is fed. If we 
> are going to have WU recreate/create 'missing' data then the logical approach 
> would be to feed it with the data it was orignally fed rather than trying to 
> guess what it wants.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/8B0BFD31-7D91-4A3D-BDED-8923E8AF0C04%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Gary,

In practice, WU seems to discard data that is not close to their APPARENTly 
preferred 5-minute "normalization buckets."

I upload via rapidfire *and* regular loop on 1-minute intervals, and 
irregardless of same, the queries only ever show the records most closely 
aligned to their "preferred" 5-minute "buckets."

I would love to see them preserve the same interval that I upload, but either 
by design or omission or bug, they simply don't.  :-(
Could be the layers in-between are dropping data by code-exception.
Could be it's deliberate.
I'm not here to debug their code unless they want to start paying me a worthy 
salary. ;-)

Well...  MOST of the time the behavior I have described seems to be true.  :-/

While WU normally SEEMS to only keeps records on 5-minute boundaries, I have 
seen where if wunderfixer is used persistently enough (say more than 10 times 
spread out every 20 minutes across a total span of 200 minutes) then it will 
keep records that are more frequent than every 5-minutes.

Right now / especially lately, things are so sporadic with WU that I don't even 
want to spend cycles chasing what's really going on there.  I would rather just 
only have wunderfixer ignore most records except those nearest the 5-minute 
boundaries that WU seems to most care about.

I have just now solved the "align to 5-minute boundaries" exercise with:

_gen_rows =  dbmanager.genSql("""SELECT dateTime FROM archive WHERE dateTime>=? 
AND dateTime On May 26, 2019, at 10:59 PM, gjr80  wrote:
> 
>> On Saturday, 25 May 2019 22:54:25 UTC+10, Leon Shaner wrote:
>> 
>> There is no point re-uploading 00:00:00, then 00:01:00, then 00:02:00, 
>> because WU is only going to keep 00:00:00, and then later it will keep 
>> whatever is closest to 00:05:00.   That's been the behavior of wunderfixer 
>> vs. WU for as long as I've been working with it, however.  :-/
> 
> Maybe, maybe not. The problem is WU acts as a black box that accepts your 
> data at (more or less) whatever interval you choose WU then does who knows 
> what with the data and (usually) 5 minute interval records appear. You could 
> argue that re-uploading of data should be done in the same manner as it was 
> originally uploaded.
> 
> Gary
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/c3476bc2-674f-4928-bf58-55f2150159b2%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/309313C3-2AD2-43B4-941C-7BBD90DB35C8%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Tested on 4.x (python 2 and 3) and 3.9.x. (python 2).

Ultimately went with apikey instead of apiKey in weewx.conf, which was to be 
more consistent with other parameter names.

If you are willing to pass "--apikey=0123456789012345678901" on wunderfixer 
command-line, then you only need wunderfixer.

Unrelated to these changes, FYI you probably also want --epsilon=125 because of 
silly WU "rounding" errors on fractional minutes.

If you wish to put "apikey=0123456789012345678901" in the WU section of 
weewx.conf, then you also need restx.py.

API Key is generated in the "Member Settings" section of wunderground.com ("My 
Profile" > "Member Settings" > "API Keys").

3.9.x:
https://github.com/UberEclectic/weewx/blob/master/bin/wunderfixer

https://github.com/UberEclectic/weewx/blob/master/bin/weewx/restx.py

4.x:
https://github.com/UberEclectic/weewx/blob/development/bin/wunderfixer

https://github.com/UberEclectic/weewx/blob/development/bin/weewx/restx.py

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 25, 2019, at 8:54 AM, Leon Shaner  wrote:
> 
> So this is where I am...
> These all seem reasonable (but then scroll for the problem which starts with 
> yesterday's date):
> 
> pi@nixie:/usr/share/weewx4/bin $ ./wunderfixer --epsilon=125 
> --date=2019-05-21 --test --verbose | more
> Using configuration file /usr/share/weewx4/weewx.conf.
> Using database binding 'wx_binding', which is bound to database 
> 'archive_sqlite'
> Weather Underground Station:   KMIDEARB5
> Date to check: 2019-05-21
> Number of archive records: 1440
> Number of WU records:  288
> Number of missing records: 4
> 
> Missing records:
> 2019-05-21 00:00:00 EDT (1558411200); 29.379";  49.8F;  78%; 0.0 mph; N/A 
> deg; 0.0 mph gust;  43.2F; 0.00" rain  ... skipped.
> 2019-05-21 00:01:00 EDT (1558411260); 29.379";  49.7F;  78%; 0.0 mph; N/A 
> deg; 0.0 mph gust;  43.2F; 0.00" rain  ... skipped.
> 2019-05-21 00:02:00 EDT (1558411320); 29.379";  49.6F;  78%; 0.0 mph; N/A 
> deg; 0.0 mph gust;  43.1F; 0.00" rain  ... skipped.
> 2019-05-21 22:42:00 EDT (1558492920); 29.406";  55.4F;  67%; 1.3 mph;  68 
> deg; 0.0 mph gust;  44.6F; 0.00" rain  ... skipped.
> pi@nixie:/usr/share/weewx4/bin $ ./wunderfixer --epsilon=125 
> --date=2019-05-22 --test --verbose | more
> Using configuration file /usr/share/weewx4/weewx.conf.
> Using database binding 'wx_binding', which is bound to database 
> 'archive_sqlite'
> Weather Underground Station:   KMIDEARB5
> Date to check: 2019-05-22
> Number of archive records: 1440
> Number of WU records:  288
> Number of missing records: 3
> 
> Missing records:
> 2019-05-22 00:00:00 EDT (1558497600); 29.406";  54.0F;  65%; 2.0 mph;  76 
> deg; 3.6 mph gust;  42.5F; 0.00" rain  ... skipped.
> 2019-05-22 00:01:00 EDT (1558497660); 29.406";  54.0F;  65%; 2.3 mph;  64 
> deg; 2.5 mph gust;  42.5F; 0.00" rain  ... skipped.
> 2019-05-22 00:02:00 EDT (1558497720); 29.406";  54.0F;  65%; 2.5 mph;  61 
> deg; 2.9 mph gust;  42.5F; 0.00" rain  ... skipped.
> pi@nixie:/usr/share/weewx4/bin $ ./wunderfixer --epsilon=125 
> --date=2019-05-23 --test --verbose | more
> Using configuration file /usr/share/weewx4/weewx.conf.
> Using database binding 'wx_binding', which is bound to database 
> 'archive_sqlite'
> Weather Underground Station:   KMIDEARB5
> Date to check: 2019-05-23
> Number of archive records: 1438
> Number of WU records:  288
> Number of missing records: 4
> 
> Missing records:
> 2019-05-23 00:00:00 EDT (1558584000); 29.244";  59.9F;  90%; 1.3 mph; 158 
> deg; 1.8 mph gust;  56.9F; 0.00" rain  ... skipped.
> 2019-05-23 00:01:00 EDT (1558584060); 29.244";  59.8F;  90%; 1.3 mph; 159 
> deg; 1.8 mph gust;  56.8F; 0.00" rain  ... skipped.
> 2019-05-23 00:02:00 EDT (1558584120); 29.244";  59.7F;  90%; 1.3 mph; 160 
> deg; 1.3 mph gust;  56.8F; 0.00" rain  ... skipped.
> 2019-05-23 14:52:00 EDT (1558637520); 29.161";  83.8F;  46%; 3.6 mph; 248 
> deg;12.3 mph gust;  60.8F; 0.00" rain  ... skipped.
> 
> 
> Using configuration file /usr/share/weewx4/weewx.conf.
> Using database binding 'wx_binding', which is bound to database 
> 'archive_sqlite'
> Weather Underground Station:   KMIDEARB5
> Date to check: 2019-05-24
> Number of archive records: 1436
> Number of WU records:  260
> Number of missing records: 149
> 
> Missing records:
> 2019-05-24 00:00:00 EDT (1558670400); 29.320";  63.0F;  70%; 0.0 mph; N/A 
> deg; 0.0 mph gust;  52.9F; 0.00" rain  .

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

ecords:
2019-05-25 00:00:00 EDT (1558756800); 29.329";  61.4F;  75%; 0.0 mph; N/A deg; 
0.0 mph gust;  53.4F; 0.00" rain  ... skipped.
2019-05-25 00:01:00 EDT (1558756860); 29.329";  61.3F;  75%; 0.0 mph; N/A deg; 
3.1 mph gust;  53.3F; 0.00" rain  ... skipped.
2019-05-25 00:02:00 EDT (1558756920); 29.329";  61.3F;  75%; 0.0 mph; N/A deg; 
2.5 mph gust;  53.3F; 0.00" rain  ... skipped.


Once I figure out what really happened with yesterday's data, late last night, 
and
If I manage to find some long-lost brain cells over the weekend, I'd still like 
to "normalize" the local records to 5-minute boundaries, and not bother to flag 
the in-between ones for re-upload, since WU seems to drop them on purpose.

There is no point re-uploading 00:00:00, then 00:01:00, then 00:02:00, because 
WU is only going to keep 00:00:00, and then later it will keep whatever is 
closest to 00:05:00.   That's been the behavior of wunderfixer vs. WU for as 
long as I've been working with it, however.  :-/

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 25, 2019, at 8:08 AM, Leon Shaner  wrote:
> 
> Hey, Gary,
> 
> I figured this out last night with a cue from Tom.  But it requires modifying 
> two different classes in restx.py, similar to the way you mentioned and I am 
> a little uncomfortable with it, since I don't know all the different touch 
> points / ramifications for other code.
> 
> Also, something strange started happening toward the end last night and the 
> list of missing records grew to pretty much 100%.
> I double checked and the new query is still pulling back the expected ~288 
> records (every 5 mins, is 12 an hour, so 288 in a day), but I now need to 
> scrutinize the "epoch" values, because it's  almost as if IBM changed them by 
> some offset in the middle of the day, so now none of them "line up" with my 
> local data.
> 
> I am heading out of town, so I will have to pick this up on Monday.
> 
> Thanks for the help! =D
> 
> Regards,
> \Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPhone)
> 
>> On May 25, 2019, at 1:24 AM, gjr80  wrote:
>> 
>> No, all get_site_dict does is check that the listed config options exist and 
>> are not the install defaults. The real issue is that _ambient_dict is used 
>> to pass keyword arguments to class AmbientThread init, apiKey is not a 
>> parameter in the init signature so hence the error. One approach will be to 
>> simply add an apiKey=None parameter to the AmbientClass init signature, it 
>> won't be used but who knows where WU may end up, maybe it could be thought 
>> of as a bit of future proofing/planning. Another approach would be to have a 
>> separate [Wunderfixer] stanza in weewx.conf, seems a waste to me and it 
>> makes sense to have the WU parameters together in the one place. Maybe leave 
>> it as a command line parameter only for wunderfixer, to me that seems almost 
>> the same as having a separate [Wunderfixer] stanza. Maybe introduce whatever 
>> approach in 4.0 and leave it as a wunderfixer command line parameter for the 
>> (expected) 3.9.2 release. Probably one for guidance from Tom.
>> 
>> Gary
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to weewx-user+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/weewx-user/68bfe21d-d0a2-4c55-9e71-2d65a69eea03%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/CAA23789-2FE9-4DCA-A2EC-386451439DC4%40isylum.org.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/4F4B3374-A489-4DF2-8E7B-3E2ABD71B676%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Hey, Gary,

I figured this out last night with a cue from Tom.  But it requires modifying 
two different classes in restx.py, similar to the way you mentioned and I am a 
little uncomfortable with it, since I don't know all the different touch points 
/ ramifications for other code.

Also, something strange started happening toward the end last night and the 
list of missing records grew to pretty much 100%.
I double checked and the new query is still pulling back the expected ~288 
records (every 5 mins, is 12 an hour, so 288 in a day), but I now need to 
scrutinize the "epoch" values, because it's  almost as if IBM changed them by 
some offset in the middle of the day, so now none of them "line up" with my 
local data.

I am heading out of town, so I will have to pick this up on Monday.

Thanks for the help! =D

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPhone)

> On May 25, 2019, at 1:24 AM, gjr80  wrote:
> 
> No, all get_site_dict does is check that the listed config options exist and 
> are not the install defaults. The real issue is that _ambient_dict is used to 
> pass keyword arguments to class AmbientThread init, apiKey is not a parameter 
> in the init signature so hence the error. One approach will be to simply add 
> an apiKey=None parameter to the AmbientClass init signature, it won't be used 
> but who knows where WU may end up, maybe it could be thought of as a bit of 
> future proofing/planning. Another approach would be to have a separate 
> [Wunderfixer] stanza in weewx.conf, seems a waste to me and it makes sense to 
> have the WU parameters together in the one place. Maybe leave it as a command 
> line parameter only for wunderfixer, to me that seems almost the same as 
> having a separate [Wunderfixer] stanza. Maybe introduce whatever approach in 
> 4.0 and leave it as a wunderfixer command line parameter for the (expected) 
> 3.9.2 release. Probably one for guidance from Tom.
> 
> Gary
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/68bfe21d-d0a2-4c55-9e71-2d65a69eea03%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/CAA23789-2FE9-4DCA-A2EC-386451439DC4%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Hey, Gary,

Below is the error on startup, simply because I added the "apiKey = blah" right 
after the "password = oog" in the [[Wunderground]] section.

This led me to believe that I need to somehow "extend" the ambient_dict, but 
for the life of me I am not seeing how to do that.  I looked every where for 
station and password in the restx.py and weecfg/__init__.py, and so far I am 
absolutely stumped.

Of course I appended to what seemed like an obvious place in restx.py, but 
nope.  No good:

_ambient_dict = get_site_dict(
config_dict, 'Wunderground', 'station', 'password', 'apiKey')
if _ambient_dict is None:
return

Same exact error without or with that added.  :-(

Any insights very much appreciated.  =D

May 24 19:59:03 nixie weewx[17779]: restx: WU essentials: {}
May 24 19:59:03 nixie weewx[17779]: engine: Caught unrecoverable exception in 
engine:
May 24 19:59:03 nixie weewx[17779]:   __init__() got an unexpected 
keyword argument 'apiKey'
May 24 19:59:03 nixie weewx[17779]:   : Traceback (most recent call 
last):
May 24 19:59:03 nixie weewx[17779]:   :   File 
"/usr/share/weewx4/bin/weewx/engine.py", line 892, in main
May 24 19:59:03 nixie weewx[17779]:   : engine = 
engine_class(config_dict)
May 24 19:59:03 nixie weewx[17779]:   :   File 
"/usr/share/weewx4/bin/weewx/engine.py", line 79, in __init__
May 24 19:59:03 nixie weewx[17779]:   : 
self.loadServices(config_dict)
May 24 19:59:03 nixie weewx[17779]:   :   File 
"/usr/share/weewx4/bin/weewx/engine.py", line 143, in loadServices
May 24 19:59:03 nixie weewx[17779]:   : 
self.service_obj.append(weeutil.weeutil._get_object(svc)(self, config_dict))
May 24 19:59:03 nixie weewx[17779]:   :   File 
"/usr/share/weewx4/bin/weewx/restx.py", line 607, in __init__
May 24 19:59:03 nixie weewx[17779]:   : **_ambient_dict)
May 24 19:59:03 nixie weewx[17779]:   : TypeError: __init__() got an 
unexpected keyword argument 'apiKey'
May 24 19:59:03 nixie weewx[17779]:   Exiting.
May 24 19:59:03 nixie systemd[1]: weewx.service: Main process exited, 
code=exited, status=1/FAILURE
May 24 19:59:03 nixie systemd[1]: weewx.service: Unit entered failed state.
May 24 19:59:03 nixie systemd[1]: weewx.service: Failed with result 'exit-code'.


Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 24, 2019, at 7:29 PM, gjr80  wrote:
> 
> What does the error trace show? WeeWX shouldn't be concerned about extra 
> config options. We don't need to re-invent wheels, forecast extension (among 
> others) already uses a WU API key, not appropriate to use that config option 
> here but using a consistent config option naming convention makes it easier 
> for users.
> 
> Gary
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/b069c856-8c59-4864-b724-e5f88ca6908e%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/68384321-D14F-4525-B301-4BF38B75B9E1%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Well shoot.

Found an issue that surfaced only upon weewx restart.
Don't add the apiKey to weewx.conf.
Found out engine.py is picky about properties it isn't expecting to find in the 
weewx.conf file.  Thought it would just ignore them.
So I guess I need to extend the dictionary of expected properties now, too.  :S

But you can still run this new wunderfixer on the side by passing the --apikey 
argument.

You could for example just call it wunderfixer_apikey and always pass your key 
on command-line:

$ wunderfixer_apikey --apikey=1234567890123456789012345678901
(With your real key there)...

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 24, 2019, at 5:01 PM, Leon Shaner  wrote:
> 
> Hi, WeeWX'ers!  =D
> 
> For those on WeeWX 4.0 / development, ONLY...
> I am done with changing wunderfixer to use the new wunderground API KEY way 
> of doing the WU query to see what records they have for comparison to those 
> in the archive DB.   Yay!  =D
> 
> Please see the comments in the pull request over here:
> 
> https://github.com/weewx/weewx/pull/416
> 
> And until that gets merged you can grab a copy from here -- again this is the 
> 4.0 / development version, ONLY:
> 
> https://raw.githubusercontent.com/UberEclectic/weewx/development/bin/wunderfixer
> 
> 
> Now, I'm off to work on the backport to WeeWX 3.9.1.
> Fun fun!  =D
> 
> Regards,
> \Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPad Pro)
> 
>> On May 24, 2019, at 1:35 PM, Leon Shaner  wrote:
>> 
>> FYI, all, I am currently hot on the trail to converting the wunderfixer 
>> query to use the new API instead.  The new API is going to be much easier 
>> than the old query, because the output is neatly formatted as JSON, and 
>> crucially for each record they even include the "epoch" i.e. Unix timestamp, 
>> which is all we care about in that query anyway.
>> The use of "epoch" means I won't even have to convert the date strings to do 
>> the matching.  =D
>> Even the input date parameter is cleaner a la 20190524.  =D
>> 
>> What I am slightly undecided about is how/where to store the API key.
>> I think for compatibility reasons I'll add a new field in the weewx.conf for 
>> the API key.
>> That way existing code based on a password can still use that method, and 
>> over time newer code can use the API key explicitly instead of a password.
>> 
>> Once I have the new wunderfixer working, for weewx 4.0, I'll work to 
>> backport the changes to weewx 3.9.1.
>> 
>> Hopefully I don't forget also to update wee_debug to obfuscate the API key, 
>> before I'm done with all of this.  LOL
>> 
>> Docs on the new API are over here...  This is for people like us who have a 
>> PWS, as opposed to other people who have to pay for API usage:
>> 
>> https://docs.google.com/document/d/1eKCnKXI9xnoMGRRzOL1xPCBihNV2rOet08qpE_gArAY/edit
>> The API Key is generated in the "Member Settings" section of 
>> wunderground.com ("My Profile" > "Member Settings" > "API Keys").
>> 
>> 
>> With any luck I'll have this banged out in a few hours.  =D
>> 
>> Regards,
>> \Leon
>> --
>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>> 
>>> On May 23, 2019, at 9:33 PM, Leon Shaner  wrote:
>>> 
>>> I am in contact with IBM.  This whole interface is entirely for THEIR 
>>> benefit.
>>> I am persistent SOB. It'll be fixed.  LOL
>>> 
>>> Regards,
>>> \Leon
>>> --
>>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>>> 
>>>> On May 23, 2019, at 9:22 PM, Jarom Hatch  wrote:
>>>> 
>>>> Even after that none of the backfilled data made it in.  If they are no 
>>>> longer accepting backfilled data then fixing wunderfixer might not be 
>>>> possible.
>>>> 
>>>> -- 
>>>> You received this message because you are subscribed to the Google Groups 
>>>> "weewx-user" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>>> email to weewx-user+unsubscr...@googlegroups.com.
>>>> To view this discussion on the web visit 
>>>> https://groups.google.com/d/msgid/weewx-user/76f9aa83-37ac-4014-aa04-7f675d63a6e1%40googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>> 
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "weewx-user" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to weewx-user+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/weewx-user/36EA4E1E-EEAE-4210-A18E-5024653E6F09%40isylum.org.
>>> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/D2429CA0-DD88-4316-8B8C-2D863F798477%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Hi, WeeWX'ers!  =D

For those on WeeWX 4.0 / development, ONLY...
I am done with changing wunderfixer to use the new wunderground API KEY way of 
doing the WU query to see what records they have for comparison to those in the 
archive DB.   Yay!  =D

Please see the comments in the pull request over here:

https://github.com/weewx/weewx/pull/416

And until that gets merged you can grab a copy from here -- again this is the 
4.0 / development version, ONLY:

https://raw.githubusercontent.com/UberEclectic/weewx/development/bin/wunderfixer


Now, I'm off to work on the backport to WeeWX 3.9.1.
Fun fun!  =D

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 24, 2019, at 1:35 PM, Leon Shaner  wrote:
> 
> FYI, all, I am currently hot on the trail to converting the wunderfixer query 
> to use the new API instead.  The new API is going to be much easier than the 
> old query, because the output is neatly formatted as JSON, and crucially for 
> each record they even include the "epoch" i.e. Unix timestamp, which is all 
> we care about in that query anyway.
> The use of "epoch" means I won't even have to convert the date strings to do 
> the matching.  =D
> Even the input date parameter is cleaner a la 20190524.  =D
> 
> What I am slightly undecided about is how/where to store the API key.
> I think for compatibility reasons I'll add a new field in the weewx.conf for 
> the API key.
> That way existing code based on a password can still use that method, and 
> over time newer code can use the API key explicitly instead of a password.
> 
> Once I have the new wunderfixer working, for weewx 4.0, I'll work to backport 
> the changes to weewx 3.9.1.
> 
> Hopefully I don't forget also to update wee_debug to obfuscate the API key, 
> before I'm done with all of this.  LOL
> 
> Docs on the new API are over here...  This is for people like us who have a 
> PWS, as opposed to other people who have to pay for API usage:
> 
> https://docs.google.com/document/d/1eKCnKXI9xnoMGRRzOL1xPCBihNV2rOet08qpE_gArAY/edit
> The API Key is generated in the "Member Settings" section of wunderground.com 
> ("My Profile" > "Member Settings" > "API Keys").
> 
> 
> With any luck I'll have this banged out in a few hours.  =D
> 
> Regards,
> \Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPad Pro)
> 
>> On May 23, 2019, at 9:33 PM, Leon Shaner  wrote:
>> 
>> I am in contact with IBM.  This whole interface is entirely for THEIR 
>> benefit.
>> I am persistent SOB. It'll be fixed.  LOL
>> 
>> Regards,
>> \Leon
>> --
>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>> 
>>> On May 23, 2019, at 9:22 PM, Jarom Hatch  wrote:
>>> 
>>> Even after that none of the backfilled data made it in.  If they are no 
>>> longer accepting backfilled data then fixing wunderfixer might not be 
>>> possible.
>>> 
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "weewx-user" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to weewx-user+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/weewx-user/76f9aa83-37ac-4014-aa04-7f675d63a6e1%40googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to weewx-user+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/weewx-user/36EA4E1E-EEAE-4210-A18E-5024653E6F09%40isylum.org.
>> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/2C6BD081-290F-4FE7-81F7-76D9FCE230DF%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

FYI, all, I am currently hot on the trail to converting the wunderfixer query 
to use the new API instead.  The new API is going to be much easier than the 
old query, because the output is neatly formatted as JSON, and crucially for 
each record they even include the "epoch" i.e. Unix timestamp, which is all we 
care about in that query anyway.
The use of "epoch" means I won't even have to convert the date strings to do 
the matching.  =D
Even the input date parameter is cleaner a la 20190524.  =D

What I am slightly undecided about is how/where to store the API key.
I think for compatibility reasons I'll add a new field in the weewx.conf for 
the API key.
That way existing code based on a password can still use that method, and over 
time newer code can use the API key explicitly instead of a password.

Once I have the new wunderfixer working, for weewx 4.0, I'll work to backport 
the changes to weewx 3.9.1.

Hopefully I don't forget also to update wee_debug to obfuscate the API key, 
before I'm done with all of this.  LOL

Docs on the new API are over here...  This is for people like us who have a 
PWS, as opposed to other people who have to pay for API usage:

https://docs.google.com/document/d/1eKCnKXI9xnoMGRRzOL1xPCBihNV2rOet08qpE_gArAY/edit
The API Key is generated in the "Member Settings" section of wunderground.com 
("My Profile" > "Member Settings" > "API Keys").


With any luck I'll have this banged out in a few hours.  =D

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 23, 2019, at 9:33 PM, Leon Shaner  wrote:
> 
> I am in contact with IBM.  This whole interface is entirely for THEIR benefit.
> I am persistent SOB. It'll be fixed.  LOL
> 
> Regards,
> \Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPad Pro)
> 
>> On May 23, 2019, at 9:22 PM, Jarom Hatch  wrote:
>> 
>> Even after that none of the backfilled data made it in.  If they are no 
>> longer accepting backfilled data then fixing wunderfixer might not be 
>> possible.
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to weewx-user+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/weewx-user/76f9aa83-37ac-4014-aa04-7f675d63a6e1%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/36EA4E1E-EEAE-4210-A18E-5024653E6F09%40isylum.org.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/6095CF30-8D08-4A43-A445-0265FEA0850D%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: Fresh install of weewx 3.9.1 with WMR300A on RPI, Non-positive value for record field 'interval': 0.0

[Leon Shaner]

Hi, Miguel,

I see you are running the latest wmr300.py version 0.19rc6+f3+l2, which has 
Cameron D's history clearing fixes any the rain counter warning enhancement.  
This is good.

However, I've never seen this type of error before:

May 24 08:49:31 raspberrypi weewx[10767]: wmr300: Excessive heartbeat delay: 
90s, restarting

Which raspberry pi do you have and which kernel?
Also, are there any other USB devices connected?
How long is your USB cable that connects to the WMR300?
Are you using an official Raspberry Pi power supply or another brand?
(Newer RPI's draw up to 2.5amps, so it matters)...

Here is what I am running on...

 $ dmesg | grep "Machine model"
[0.00] OF: fdt: Machine model: Raspberry Pi Zero W Rev 1.1

$ uname -a
Linux nixie 4.19.42+ #1219 Tue May 14 21:16:38 BST 2019 armv6l GNU/Linux

$ cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 9 (stretch)"
NAME="Raspbian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/;
SUPPORT_URL="http://www.raspbian.org/RaspbianForums;
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs;

$ cat /proc/cpuinfo
processor   : 0
model name  : ARMv6-compatible processor rev 7 (v6l)
BogoMIPS: 997.08
Features: half thumb fastmult vfp edsp java tls
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part: 0xb76
CPU revision: 7

Hardware: BCM2835
Revision: 9000c1
Serial  : f7e29d3c


$ head -3 /proc/meminfo
MemTotal: 443132 kB
MemFree:      182188 kB
MemAvailable: 272340 kB


Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 24, 2019, at 4:52 AM, Miguel Angel Peña Morales 
>  wrote:
> 
> May 24 08:33:51 raspberrypi systemd[1]: Started LSB: weewx weather system.
> May 24 08:33:51 raspberrypi weewx[10767]: engine: Using configuration file 
> /etc/weewx/weewx.conf
> May 24 08:33:51 raspberrypi weewx[10767]: engine: Loading station type WMR300 
> (weewx.drivers.wmr300)
> May 24 08:33:51 raspberrypi weewx[10767]: wmr300: driver version is 
> 0.19rc6+f3+l2
> May 24 08:33:51 raspberrypi weewx[10767]: wmr300: usb info: 
> pyusb_version=1.0.0
> May 24 08:33:51 raspberrypi weewx[10767]: wmr300: sensor map is 
> {'outHumidity': 'humidity_1', 'extraDewpoint6': 'dewpoint_7', 'windchill': 
> 'windchill', 'extraDewpoint4': 'dewpoint_5', 'rainRate': 'rain_rate', 
> 'heatindex': 'heatindex_1', 'inTemp': 'temperature_0', 'windGustDir': 
> 'wind_gust_dir', 'extraDewpoint2': 'dewpoint_3', 'extraDewpoint3': 
> 'dewpoint_4', 'extraDewpoint1': 'dewpoint_2', 'barometer': 'barometer', 
> 'extraDewpoint7': 'dewpoint_8', 'dewpoint': 'dewpoint_1', 'extraDewpoint5': 
> 'dewpoint_6', 'extraHumid6': 'humidity_7', 'pressure': 'pressure', 
> 'extraHumid4': 'humidity_5', 'extraHumid5': 'humidity_6', 'extraHumid2': 
> 'humidity_3', 'extraHumid3': 'humidity_4', 'extraHumid1': 'humidity_2', 
> 'extraTemp6': 'temperature_7', 'extraTemp7': 'temperature_8', 'extraTemp4': 
> 'temperature_5', 'extraTemp5': 'temperature_6', 'extraTemp2': 
> 'temperature_3', 'extraTemp3': 'temperature_4', 'extraTemp1': 
> 'temperature_2', 'extraHeatindex3': 'heatindex_4', 'extraHeatindex2': 
> 'heatindex_3', 'extraHeatindex1': 'heatindex_2', 'extraHeatindex7': 
> 'heatindex_8', 'extraHeatindex6': 'heatindex_7', 'extraHeatindex5': 
> 'heatindex_6', 'extraHumid7': 'humidity_8', 'extraHeatindex4': 'heatindex_5', 
> 'windDir': 'wind_dir', 'outTemp': 'temperature_1', 'windSpeed': 'wind_avg', 
> 'inHumidity': 'humidity_0', 'windGust': 'wind_gust'}
> May 24 08:33:51 raspberrypi weewx[10767]: wmr300: history limit is 20% at 
> index 6572
> May 24 08:33:52 raspberrypi kernel: [39422.245849] usb 1-1.5: reset 
> full-speed USB device number 8 using dwc_otg
> May 24 08:33:52 raspberrypi weewx[10767]: wmr300: communication established: 
> {'history_end_index': 32735, 'station_model': 'A004', 'station_type': 
> 'WMR300', 'mystery1': 44, 'mystery0': 73, 'history_cleared': False, 'magic1': 
> 255, 'magic0': 255, 'packet_type': 87}
> May 24 08:33:52 raspberrypi weewx[10767]: engine: StdConvert target unit is 
> 0x1
> May 24 08:33:52 raspberrypi weewx[10767]: wxcalculate: The following values 
> will be calculated: barometer=prefer_hardware, windchill=hardware, 
> dewpoint=software, appTemp=prefer_hardware, rainRate=hardware, 
> windrun=prefer_hardware, heatindex=hardware, maxSolarRad=prefer_hardware, 
> humidex=prefer_hardware, pressure=prefer_hardware, 
> inDewpoint=prefer_hardware, ET=prefer_hardware, altimeter=prefer_hardware, 
> cloudbase=prefer_hardware
> May 24 08:33:52 raspberrypi weewx[10767]: wxcalculate: The following 
> algorithms will be used for calculations: altimeter=aaNOAA, maxSolarRad=RS
> May 24 08:33:52 raspbe

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

I am in contact with IBM.  This whole intersection is entirely for THEIR 
benefit.
I am persistent SOB. It'll be fixed.  LOL

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 23, 2019, at 9:22 PM, Jarom Hatch  wrote:
> 
> Even after that none of the backfilled data made it in.  If they are no 
> longer accepting backfilled data then fixing wunderfixer might not be 
> possible.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/76f9aa83-37ac-4014-aa04-7f675d63a6e1%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/36EA4E1E-EEAE-4210-A18E-5024653E6F09%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Jarom,

So you got 503's with my updated code, but not 403's?

Yeah, I have found that it is normal for WU to lazily process re-uploaded 
records, and it seems as if they drop them...as if some backend process is 
dying.

I found through trial and error that after a weewx "outage" I need to re-upload 
exactly 10 times, but not back to back.  I do one upload every 20 minutes, 
which means spreading the 10 re-uploads out over a 200 minute period.
After that wunderfixer reports no missing records.
Well, at least it used to work before all these 404's and 503's.  :-(

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 23, 2019, at 11:06 AM, Jarom Hatch  wrote:
> 
> Those 503's are coming from Akamai.  When I have used Akamai in the past 
> those errors are because the origin is not responding correctly.
> 
> Related sorta, when I took out the code to pull the timestamps and just 
> force-ran all my records as a big backfill I got success messages back 
> however none of the records actually made their way in.
> 
>> On Thursday, May 23, 2019 at 8:27:37 AM UTC-6, Leon Shaner wrote:
>> Jarom,
>> 
>> Thanks so much!  I see what I did wrong and I was able to make a stubbed 
>> down version for basic testing to prove it's at least trying to connect.
>> 
>> Same location (for WeeWX 3.9.1):
>> 
>> https://raw.githubusercontent.com/UberEclectic/weewx/master/bin/wunderfixer
>> 
>> The thing is, I'm still constantly getting 404 (Not Found) even with CURL, 
>> and just a bit ago the site started throwing 503 (Service Not Available).
>> So...  It's kinda hard to test under these conditions.   But as long as you 
>> don't get a 403, then at least my User-Agent "hack" will be "proven."   :-/
>> 
>> Regards,
>> \Leon
>> --
>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>> 
>>> On May 23, 2019, at 1:57 AM, Jarom Hatch  wrote:
>>> 
>>> I tried the 3.9.1 version and I get Could not get Weather Underground data. 
>>> Exiting.
>>> 
>>> Curl still works, even for yesterday's data.  Tracing the script it doesn't 
>>> appear to be actually attempting the download.  
>>> 
>>> 
>>>> On Wednesday, May 22, 2019 at 7:03:49 PM UTC-6, Leon Shaner wrote:
>>>> Say, we need a tester who is still on 3.9.1 or there abouts to try this 
>>>> out:
>>>> 
>>>> https://raw.githubusercontent.com/UberEclectic/weewx/master/bin/wunderfixer
>>>> 
>>>> Can't do anything to workaround WU's sporadic 404 and 503 errors, but at 
>>>> least the 403 error should be gone.
>>>> 
>>>> I was able to test the 4.0 / development version myself on both Python2 
>>>> and 3, so hopefully Tom will merge that soon.  It's over here if you are 
>>>> impatient.  =D
>>>> 
>>>> https://raw.githubusercontent.com/UberEclectic/weewx/development/bin/wunderfixer
>>>> 
>>>> Regards,
>>>> \Leon
>>>> --
>>>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>>>> 
>>>>> On May 22, 2019, at 7:24 PM, Leon Shaner  wrote:
>>>>> 
>>>>> Hey WeeWX'ers!!!  =D
>>>>> 
>>>>> I have a fix in the hopper.
>>>>> 
>>>>> There's nothing that can be done for the occasional HTTP 404, or even 
>>>>> 503's I am now seeing, but the HTTP 403 was due to a change on WU's part 
>>>>> where they are rejecting certain HTTP User-Agent strings.  The fact that 
>>>>> they are putting Akamai in the middle is almost certainly a great thing 
>>>>> re: their scalability issues; however, they probably inherited some 
>>>>> default settings that filter "bots" and malware and such, which is likely 
>>>>> why the HTTP User-Agent now matters.
>>>>> 
>>>>> I have set the User-Agent to "CURL" and it works.
>>>>> I have set it to "Mozilla" and it works.  I'm going with that one, since 
>>>>> it means Mosaic Killer, both of which were among the the very first 
>>>>> User-Agents I ever worked with, circa 1993 back before there was such as 
>>>>> thing as Netscape.  =D
>>>>> 
>>>>> /ye-olde-farte mode off  ;-)
>>>>> 
>>>>> My testing has so far been under Python3, but coincidentally (and not a 
>>>>> causation), WU started throwing HTTP 503's around the time that I tried 
>>>>> validating my code also under Python2.
>>&

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Jarom,

Thanks so much!  I see what I did wrong and I was able to make a stubbed down 
version for basic testing to prove it's at least trying to connect.

Same location (for WeeWX 3.9.1):

https://raw.githubusercontent.com/UberEclectic/weewx/master/bin/wunderfixer

The thing is, I'm still constantly getting 404 (Not Found) even with CURL, and 
just a bit ago the site started throwing 503 (Service Not Available).
So...  It's kinda hard to test under these conditions.   But as long as you 
don't get a 403, then at least my User-Agent "hack" will be "proven."   :-/

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 23, 2019, at 1:57 AM, Jarom Hatch  wrote:
> 
> I tried the 3.9.1 version and I get Could not get Weather Underground data. 
> Exiting.
> 
> Curl still works, even for yesterday's data.  Tracing the script it doesn't 
> appear to be actually attempting the download.  
> 
> 
>> On Wednesday, May 22, 2019 at 7:03:49 PM UTC-6, Leon Shaner wrote:
>> Say, we need a tester who is still on 3.9.1 or there abouts to try this out:
>> 
>> https://raw.githubusercontent.com/UberEclectic/weewx/master/bin/wunderfixer
>> 
>> Can't do anything to workaround WU's sporadic 404 and 503 errors, but at 
>> least the 403 error should be gone.
>> 
>> I was able to test the 4.0 / development version myself on both Python2 and 
>> 3, so hopefully Tom will merge that soon.  It's over here if you are 
>> impatient.  =D
>> 
>> https://raw.githubusercontent.com/UberEclectic/weewx/development/bin/wunderfixer
>> 
>> Regards,
>> \Leon
>> --
>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>> 
>>> On May 22, 2019, at 7:24 PM, Leon Shaner  wrote:
>>> 
>>> Hey WeeWX'ers!!!  =D
>>> 
>>> I have a fix in the hopper.
>>> 
>>> There's nothing that can be done for the occasional HTTP 404, or even 503's 
>>> I am now seeing, but the HTTP 403 was due to a change on WU's part where 
>>> they are rejecting certain HTTP User-Agent strings.  The fact that they are 
>>> putting Akamai in the middle is almost certainly a great thing re: their 
>>> scalability issues; however, they probably inherited some default settings 
>>> that filter "bots" and malware and such, which is likely why the HTTP 
>>> User-Agent now matters.
>>> 
>>> I have set the User-Agent to "CURL" and it works.
>>> I have set it to "Mozilla" and it works.  I'm going with that one, since it 
>>> means Mosaic Killer, both of which were among the the very first 
>>> User-Agents I ever worked with, circa 1993 back before there was such as 
>>> thing as Netscape.  =D
>>> 
>>> /ye-olde-farte mode off  ;-)
>>> 
>>> My testing has so far been under Python3, but coincidentally (and not a 
>>> causation), WU started throwing HTTP 503's around the time that I tried 
>>> validating my code also under Python2.
>>> 
>>> Everything is working against today's date.
>>> It's when I go after yesterday's date that I get the HTTP server error 503.
>>> 
>>> I expect the 404's and 503's to go away eventually, but at least for now I 
>>> have a fix for the 403 (forbidden)'s, just based on the User-Agent string.
>>> 
>>> I'll submit a change for wunderfixer both to the 3.9.x "master" and 4.0.x 
>>> "development" branches in a moment and reply back with direct links for 
>>> anyone who wants a fix sooner.  =D
>>> 
>>> Isn't this fun?  =D
>>> 
>>> Regards,
>>> \Leon
>>> --
>>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>>> 
>>>> On May 22, 2019, at 4:20 PM, Leon Shaner  wrote:
>>>> 
>>>> I'm still working on this.
>>>> CURL is telling me they are not only using https, but also TLSv1.2.
>>>> Here is a transcript, in case one of y'all beats me to the fix.  =D
>>>> 
>>>> -- 
>>>> You received this message because you are subscribed to the Google Groups 
>>>> "weewx-user" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>>> email to weewx...@googlegroups.com.
>>>> To view this discussion on the web visit 
>>>> https://groups.google.com/d/msgid/weewx-user/DA01E425-B99A-4959-8FB2-B564A61B3E77%40isylum.org.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>> 
>>>> 
>>>> 
>>>> 
>>>&g

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Hey WeeWX'ers!!!  =D

I have a fix in the hopper.

There's nothing that can be done for the occasional HTTP 404, or even 503's I 
am now seeing, but the HTTP 403 was due to a change on WU's part where they are 
rejecting certain HTTP User-Agent strings.  The fact that they are putting 
Akamai in the middle is almost certainly a great thing re: their scalability 
issues; however, they probably inherited some default settings that filter 
"bots" and malware and such, which is likely why the HTTP User-Agent now 
matters.

I have set the User-Agent to "CURL" and it works.
I have set it to "Mozilla" and it works.  I'm going with that one, since it 
means Mosaic Killer, both of which were among the the very first User-Agents I 
ever worked with, circa 1993 back before there was such as thing as Netscape.  
=D

/ye-olde-farte mode off  ;-)

My testing has so far been under Python3, but coincidentally (and not a 
causation), WU started throwing HTTP 503's around the time that I tried 
validating my code also under Python2.

Everything is working against today's date.
It's when I go after yesterday's date that I get the HTTP server error 503.

I expect the 404's and 503's to go away eventually, but at least for now I have 
a fix for the 403 (forbidden)'s, just based on the User-Agent string.

I'll submit a change for wunderfixer both to the 3.9.x "master" and 4.0.x 
"development" branches in a moment and reply back with direct links for anyone 
who wants a fix sooner.  =D

Isn't this fun?  =D

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 22, 2019, at 4:20 PM, Leon Shaner  wrote:
> 
> I'm still working on this.
> CURL is telling me they are not only using https, but also TLSv1.2.
> Here is a transcript, in case one of y'all beats me to the fix.  =D
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/DA01E425-B99A-4959-8FB2-B564A61B3E77%40isylum.org.
> For more options, visit https://groups.google.com/d/optout.
> 
> 
> 
> 
> Working from here:
> https://docs.python.org/2/library/ssl.html
> 
> So far I have tried this, to no avail.
> Really just doing the "import ssl" and using https in the URL, and adding 
> context=ssl_context to the urllib.request.
> 
> A snippet of that looks as follows, but still getting 403 forbidden.  :-(
> 
> # For new WU interface which uses SSL and TLSv1.2
> import ssl
> 
> ...
> 
> _url = 
> "https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=%s; \
>"=%d=%d=%d=1" % (self.station, 
> dayRequested_tt[1],
>   dayRequested_tt[2], 
> dayRequested_tt[0])
> 
> # specify TLSv1.2 and SSLv2, but not SSLv3
> ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
> ssl_context.options |= ssl.PROTOCOL_SSLv23
> ssl_context.options |= ssl.OP_NO_SSLv3
> 
> try :
> # Hit the weather underground site:
> _wudata = urllib.request.urlopen(_url, context=ssl_context)
> 
> 
> 
> Regards,
> \Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPad Pro)
> 
>> On May 22, 2019, at 2:42 PM, Leon Shaner  wrote:
>> 
>> Jarom,
>> 
>> CURL is pretty sophisticated in its ability to emulate browser state in 
>> pretty much any way but JavaScript.  When it worked this morning, I saw some 
>> cookies were involved.
>> It may well be that the python way isn't handling that part.
>> I don't know enough about how python fetches pages to work that out, but I 
>> am very familiar with CURL, so if I can find a path that works consistently, 
>> then I'll go back to the python to see about how to implement same.
>> 
>> I was getting 404's in the browser even, when I looked at it earlier.
>> 
>> I'll keep working on it, but not too hard, so as to not get on their radar 
>> in any unwanted sort of way.  ;-)
>> 
>> Regards,
>> \Leon
>> --
>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>> 
>>> On May 22, 2019, at 2:04 PM, Jarom Hatch  wrote:
>>> 
>>> Interesting, using curl sometimes I can it fine, but wunderfixer is always 
>>> getting a 403 Forbidden, as if it is actively being blocked...  When it 
>>> doesn't work in curl I get `HTTP/1.1 404 Not Found` and when it does work I 
>>> get `HTTP/1.1 200 OK`.  Curl never gets a 403 error.
>>> 
>>>>

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Hey, Doug,
Creating a front end to search your own local data would certainly be doable.
For sure, directing end-users to our own sites and letting them search our 
historical data would be useful.

But that wouldn't fix the problem at hand.  :-/
The wunderfixer utility is there to query what data WU has on record from your 
station and to re-upload any missing data.
Basically if we can't fix this then it means WU will often be missing data from 
time to time for various reasons.
Really only WU should care and end-users who go to their site to look at 
historical data.   So yeah, they should be invented to help, since they're the 
ones missing the data, and the less reliable their data, the more likely even 
end-users will begin to abandon using them.

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 22, 2019, at 4:33 PM, Doug Bo  wrote:
> 
> Sort of a noob question or maybe a feature request: how about adding a web 
> interface to search the database for historic weather data?  I grab the 
> wunderground data that I have uploaded to add to various farm records a few 
> months or even a year after the fact.  If WU is getting worse perhaps I need 
> another solution?
> 
> Thanks,
> Doug B.
> 
>> On Tuesday, May 21, 2019 at 11:09:14 AM UTC-7, tds wrote:
>> For the last three days, my daily cron job of wunderfixer have returned "503 
>> Unavaialbe" (May 18), "404 Not Found" (May 19) and "403 Forbidden" (May 20).
>> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/22591326-058f-4717-bdb4-2f91cd099e68%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/2B320D7A-2F1D-475D-810E-BCD69BB562E8%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

.9,5.4,91,0.03,,,0.15,weewx-4.0.0a4,2019-05-22 
16:45:00,

2019-05-22 
12:49:59,54,51.7,29.29,East,101,2.9,3.6,92,0.04,,,0.16,weewx-4.0.0a4,2019-05-22 
16:49:59,

2019-05-22 
12:54:59,54,51.7,29.29,ESE,108,4,5.4,92,0.04,,,0.16,weewx-4.0.0a4,2019-05-22 
16:54:59,

2019-05-22 
13:00:00,54.1,51.9,29.29,ESE,108,4,7.4,92,0.03,,,0.16,weewx-4.0.0a4,2019-05-22 
17:00:00,

2019-05-22 
13:05:00,54.3,51.7,29.28,ESE,110,3.6,2.5,91,0.03,,,0.16,weewx-4.0.0a4,2019-05-22
 17:05:00,

2019-05-22 
13:09:59,54.5,52.2,29.28,ESE,110,3.6,4.3,92,0.03,,,0.16,weewx-4.0.0a4,2019-05-22
 17:09:59,

2019-05-22 
13:14:59,54.7,52.4,29.28,SE,132,3.6,1.8,92,0.03,,,0.16,weewx-4.0.0a4,2019-05-22 
17:14:59,

2019-05-22 
13:19:59,55,52.8,29.27,SE,132,3.6,5.1,92,0.03,,,0.16,weewx-4.0.0a4,2019-05-22 
17:19:59,

2019-05-22 13:24:59,55,52.2,29.27,ESE,111,3.1,5.1,90,0.0
100 19032  100 190320 0   3165  0  0:00:06  0:00:06 --:--:--  3757* 
Curl_http_done: called premature == 0

100 19032  100 190320 0   3162  0  0:00:06  0:00:06 --:--:--  4796
* Connection #0 to host www.wunderground.com left intact
1,,,0.16,weewx-4.0.0a4,2019-05-22 17:24:59,

2019-05-22 
13:30:00,55.2,52.6,29.27,ESE,111,3.1,0,91,0.02,,,0.17,weewx-4.0.0a4,2019-05-22 
17:30:00,

2019-05-22 
13:34:59,55.6,53,29.30,SE,135,2.9,4.7,91,0.02,,,0.17,weewx-4.0.0a4,2019-05-22 
17:34:59,

2019-05-22 
13:39:59,55.6,52.7,29.30,SE,135,2.9,4,90,0.02,,,0.17,weewx-4.0.0a4,2019-05-22 
17:39:59,

2019-05-22 
13:44:59,55.6,52.7,29.30,SSE,157,2.5,1.3,90,0.02,,,0.17,weewx-4.0.0a4,2019-05-22
 17:44:59,

2019-05-22 
13:50:00,55.6,52.7,29.31,SSE,157,2.5,0,90,0.01,,,0.17,weewx-4.0.0a4,2019-05-22 
17:50:00,

2019-05-22 
13:54:59,55.8,53.5,29.31,SSE,156,1.8,0,92,0.01,,,0.17,weewx-4.0.0a4,2019-05-22 
17:54:59,

2019-05-22 
13:59:59,56.1,53.8,29.31,SSE,156,1.8,0,92,0.01,,,0.17,weewx-4.0.0a4,2019-05-22 
17:59:59,

2019-05-22 
14:04:59,56.7,54.4,29.29,North,,0,0,92,0.01,,,0.17,weewx-4.0.0a4,2019-05-22 
18:04:59,

2019-05-22 
14:10:00,57,54.4,29.29,North,,0,2,91,0.01,,,0.17,weewx-4.0.0a4,2019-05-22 
18:10:00,

2019-05-22 
14:14:59,57.7,54.2,29.29,East,87,1.8,2.9,88,0.01,,,0.17,weewx-4.0.0a4,2019-05-22
 18:14:59,

2019-05-22 
14:19:59,58.3,54.7,29.29,East,87,1.8,3.1,88,0.01,,,0.17,weewx-4.0.0a4,2019-05-22
 18:19:59,

2019-05-22 
14:24:59,58.6,54.5,29.29,SE,128,2,1.8,86,0.01,,,0.17,weewx-4.0.0a4,2019-05-22 
18:24:59,

2019-05-22 
14:29:59,59.4,55.8,29.29,SE,128,2,3.1,88,0,,,0.17,weewx-4.0.0a4,2019-05-22 
18:29:59,

2019-05-22 
14:35:00,60.1,55.9,29.29,SSE,161,2,4,86,0,,,0.17,weewx-4.0.0a4,2019-05-22 
18:35:00,

2019-05-22 
14:39:59,61,55.8,29.29,SSE,161,2,2,83,0,,,0.17,weewx-4.0.0a4,2019-05-22 
18:39:59,

2019-05-22 
14:45:00,61.5,55.3,29.29,SSE,151,2,3.1,80,0,,,0.17,weewx-4.0.0a4,2019-05-22 
18:45:00,

2019-05-22 
14:49:59,61.7,55.1,29.29,SSE,151,2,3.6,79,0,,,0.17,weewx-4.0.0a4,2019-05-22 
18:49:59,

2019-05-22 
14:55:00,61.9,55.6,29.29,SE,144,2.5,3.6,80,0,,,0.17,weewx-4.0.0a4,2019-05-22 
18:55:00,

2019-05-22 
14:59:59,62.1,54.8,29.29,SE,144,2.5,2.9,77,0.01,,,0.18,weewx-4.0.0a4,2019-05-22 
18:59:59,

2019-05-22 
15:04:59,62.4,55.5,29.28,SE,126,2.9,1.8,78,0.01,,,0.18,weewx-4.0.0a4,2019-05-22 
19:04:59,

2019-05-22 
15:09:59,62.8,55.8,29.28,SE,126,2.9,3.6,78,0.01,,,0.18,weewx-4.0.0a4,2019-05-22 
19:09:59,

2019-05-22 
15:14:59,63.3,56,29.28,SE,138,2.9,1.8,77,0.01,,,0.18,weewx-4.0.0a4,2019-05-22 
19:14:59,

2019-05-22 
15:20:00,64,56,29.28,SE,138,2.9,5.8,75,0.01,,,0.18,weewx-4.0.0a4,2019-05-22 
19:20:00,

2019-05-22 
15:24:59,64.6,56.1,29.28,SSE,156,2.5,3.6,74,0.01,,,0.18,weewx-4.0.0a4,2019-05-22
 19:24:59,

2019-05-22 
15:30:00,65.1,56.2,29.28,SSE,156,2.5,3.6,73,0.01,,,0.18,weewx-4.0.0a4,2019-05-22
 19:30:00,

Working from here:https://docs.python.org/2/library/ssl.htmlSo far I have tried this, to no avail.Really just doing the "import ssl" and using https in the URL, and adding context=ssl_context to the urllib.request.A snippet of that looks as follows, but still getting 403 forbidden.  :-(# For new WU interface which uses SSL and TLSv1.2import ssl...        _url = "https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=%s" \               "=%d=%d=%d=1" % (self.station, dayRequested_tt[1],                                                      dayRequested_tt[2], dayRequested_tt[0])        # specify TLSv1.2 and SSLv2, but not SSLv3        ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)        ssl_context.options |= ssl.PROTOCOL_SSLv23        ssl_context.options |= ssl.OP_NO_SSLv3        try :            # Hit the weather underground site:            _wudata = urllib.request.urlopen(_url, context=ssl_context)Regards,\Leon--Leon Shaner :: Dearborn, Michigan (iPad Pro)On May 22, 2019, at 2:42 PM, Leon Shaner <l...@isylum.org> wrote:Jarom,CURL is pretty sophisticated in its ability to emulate browser state in pretty much any way but _javascript_.  When it worked this morning, I saw some cookies were involved.It may well be that the python way isn't ha

Re: [weewx-user] Probably an OT Linux question, but:

[Leon Shaner]

Hey, Steve,

You're logged in as user pi.
When (re)starting weewx via a system service, such as "/etc/init.d/weewx 
restart" or "systemctl restart weewx" you have to elevate to root by one means 
or another.
The fact that it prompted you to pick a method of running under root is just a 
convenience.

Most people will just straight out run such a command under sudo which elevates 
to root for you, provided your user is allowed (which on RPI, the pi user, 
generally is allowed).

So what you wanted was:

$ sudo /etc/init.d/weewx restart

So if you never used to run it via "sudo" then you must have had an 
installation with some fancy permissions set to allow everything to be accessed 
by the pi user.
It's doable, but definitely not an out of the box config so if you had that 
working, you must have put a lot of effort into it a long time ago and maybe 
some "apt" kinds of commands reverted to the default "runs as root" paradigm?

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 22, 2019, at 12:20 PM, Steve2Q  wrote:
> 
> I SSH'd into my Pi to do a weewx restart;
> 
> I used to type /etc/init.d/weewx restart and the command just worked
> 
> Now it says the following:  
> 
> pi@raspberrypi:~ $ /etc/init.d/weewx restart
> [] Restarting weewx (via systemctl): weewx.service AUTHENTICATING FOR 
> org.freedesktop.systemd1.manage-units ===
> Authentication is required to restart 'weewx.service'.
> Multiple identities can be used for authentication:
>  1.  ,,, (pi)
>  2.  root
> Choose identity to authenticate as (1-2): 1
> Password:
> 
> when I choose #1 and type my password the command  executes
> 
> This just suddenly started with no changes that I made.
> 
> I know this may be way OT, but I have looked and could not find an answer 
> elsewhere.
> 
> Thanks
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/6a38b5b3-1296-42ba-bea8-aeaa245fd0c1%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/680FCF93-378F-4E5F-B408-7D6E11FCC1E2%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

Jarom,

CURL is pretty sophisticated in its ability to emulate browser state in pretty 
much any way but JavaScript.  When it worked this morning, I saw some cookies 
were involved.
It may well be that the python way isn't handling that part.
I don't know enough about how python fetches pages to work that out, but I am 
very familiar with CURL, so if I can find a path that works consistently, then 
I'll go back to the python to see about how to implement same.

I was getting 404's in the browser even, when I looked at it earlier.

I'll keep working on it, but not too hard, so as to not get on their radar in 
any unwanted sort of way.  ;-)

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 22, 2019, at 2:04 PM, Jarom Hatch  wrote:
> 
> Interesting, using curl sometimes I can it fine, but wunderfixer is always 
> getting a 403 Forbidden, as if it is actively being blocked...  When it 
> doesn't work in curl I get `HTTP/1.1 404 Not Found` and when it does work I 
> get `HTTP/1.1 200 OK`.  Curl never gets a 403 error.
> 
>> On Wednesday, May 22, 2019 at 11:48:08 AM UTC-6, Jarom Hatch wrote:
>> I was able to get it to work twice in my web browser, but as you said, it is 
>> sporadic.  I don't ever recall them using Akamai before so that may very 
>> well be a contributing factor.
>> 
>> I wonder if we can find out the origin address and see what happens if we 
>> can bypass Akamai...
>> 
>>> On Wednesday, May 22, 2019 at 7:35:18 AM UTC-6, Leon Shaner wrote:
>>> For one thing, the URL of this form:
>>> 
>>> http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=SOMESTATION=5=22=2019=1
>>> 
>>> Is now redirecting to one using HTTPS:
>>> 
>>> https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=SOMESTATION=5=22=2019=1
>>> 
>>> Also, the redirect itself takes an excruciatingly long time.
>>> So I just changed the URL to https directly...
>>> 
>>> The first time I tried any of the above using CURL this morning it worked, 
>>> but then after that I started getting:
>>> 
>>> An error occurred while processing your request.
>>> Reference #30.6f451160.1558531514.16ced4f6
>>> 
>>> It looks as if they've put some kind of Akamai proxy in the middle, which 
>>> is fine for static content, but not so fine for a query of this nature.  
>>> Strange that it worked for me the very first time.  It's almost as if the 
>>> Akamai "farm" has lost some "state" information and not all nodes have the 
>>> same content, so if you get stuck going through a bad node you get a bogus 
>>> response.
>>> 
>>> Attached is a transcript of a failed attempt.  I put SOMESTATION there only 
>>> after the fact.  The actual query was for my actual station, which used to 
>>> work.
>>> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/07ac6f86-ae4d-4854-8398-ce4ab8d846c1%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/3260F8E7-37D4-4737-999B-97522E324370%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: Second crash after 11 days

[Leon Shaner]

Hey, Steve,

I must have missed the context, but if you are looking at overall system memory 
growth, well, it's a normal / healthy thing for a Linux kernel to try to use as 
much memory as possible at all times, to keep from having to go back to disk 
for code pages as different processes context switch in/out of the scheduler.   
Even as memory is freed by a program, the kernel won't bother to go back and 
re-use free'd pages unless/until there are no more previously unused memory 
pages available.
At that point the kernel will run the free memory "scanner" to go through the 
list of freed pages to find some to allocate to some other program requesting 
memory.

In a healthy system with plenty of RAM the free memory scanner should rarely 
run.
If the scanner runs a lot, then that means memory pressure is high and so THERE 
would be your indication that you're running low on memory, not "free -m" 
indicating a high number (because a high number is a good/normal thing in 
Linux).

What you should be most concerned about is if the system is starting to 
actually use swap, because then that means that not only is the physical RAM 
full, and freed pages are constantly being turned over / re-used, but the 
kernel is ALSO having to dip into virtual memory by swapping out (to disk) 
things not currently running, and then swapping those things back in when the 
associated processes are brought back to the running state by the process 
scheduler.

I like to use "top" to see what's going on with swap usage, but you can also 
get the info via other ways, such as:

$ cat /proc/meminfo
$ vmstat

(Etc etc)...

As for how to track the *real* indicator of memory pressure, that being the 
free memory scanner running too frequently, I'd look to "sar" for that.

I didn't see mention of what platform you are on, but for me on RPI, I get sar 
via:

$ sudo apt-get install sysstat
(And enable it in "/etc/default/sysstat").

And then these go in root's cron, a la "sudo crontab -e"

# Enable SAR:
# Collect measurements at 10-minute intervals
0,10,20,30,40,50   * * * *   /usr/lib/sysstat/sa1
# Create daily reports and purge old files
0  0 * * *   /usr/lib/sysstat/sa2 -A



After sa1 has been running a few times (say, give it an hour) then you can use 
"sar -B" to see the "pgscand/s" statistics (and related).

Of course check "man sar" for more details.


Like I said I missed the context, but did you previously capture some data to 
suggest that weewxd is growing and growing in its memory consumption?

Here is a handy command you can use to check it:

$ ps -C weewxd -o comm,size,rss,vsize,%mem
COMMAND  SIZE   RSSVSZ %MEM
weewxd  88616 71816 102640 16.1

You can use "man ps" to understand what size and RSS is, but if those grow and 
grow and grow, then that's usually an indicator of a memory leak in the code.

If you run that ps command in a loop, checking it say once every 10 minutes you 
can do some math to gauge how much it is growing.

A la:

$ while [ 1 ] ; do
ps -C weewxd -o comm,size,rss,vsize,%mem
sleep 600
done | tee -a /var/tmp/weewxd_size.txt

Wouldn't be too hard to just show the delta in the numbers.
Lemme know if you're still reading this far and I'll take a few seconds to show 
such an example.  ;-)

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 22, 2019, at 12:14 PM, Steve2Q  wrote:
> 
> A little follow up. As the memory growth is still happening, I implemented a 
> script which runs the program free -m and does a little calculation to 
> determine if memory use is over 80% (or any % I want to use). If mem >80% 
> weewx is restarted. So at least it now will run unattended.
> 
> A new question with Steel Gauges. I did the re-installation of RTG on April 
> 9. I just noticed that the gauges themselves are working perfectly, the pop 
> up charts are showing April 9; they have not updated. I probably have the 
> wrong syntax somewhere?
> 
> Steve
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/aa2ae09d-6a65-4c1d-aff5-46332084ade7%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/170C7D38-46B5-4285-BFC4-C82807F8DB7B%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x1724ea8)
} [5 bytes data]
> GET 
> /weatherstation/WXDailyHistory.asp?ID=SOMESTATION=5=22=2019=1
>  HTTP/1.1
> Host: www.wunderground.com
> User-Agent: curl/7.52.1
> Accept: */*
> 
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
} [5 bytes data]

  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0< 
HTTP/2 503 
< server: AkamaiGHost
< mime-version: 1.0
< content-type: text/html
< content-length: 176
< cache-control: max-age=0
< expires: Wed, 22 May 2019 13:29:04 GMT
< date: Wed, 22 May 2019 13:29:04 GMT
< set-cookie: speedpin=4G; expires=Wed, 22-May-2019 13:59:04 GMT; path=/; 
domain=.wunderground.com; secure
< set-cookie: 
ci=TWC-Connection-Speed=4G=US=desktop=dna=wifi=US=42.3055=-83.1724=1000+=exempt;
 path=/; domain=.wunderground.com; secure
< property-id: drupal-prod
< x-requestsource: AkamaiProdProxy
< twc-privacy: exempt
< twc-geoip-latlong: 42.3055,-83.1724
< twc-geoip-country: US
< twc-device-class: desktop
< twc-locale-group: US
< twc-connection-speed: 4G
< x-requestsource: AkamaiDefaultDNA
< 
{ [176 bytes data]
* Curl_http_done: called premature == 0

100   176  100   1760 0108  0  0:00:01  0:00:01 --:--:--   261
* Connection #1 to host www.wunderground.com left intact
Error
An error occurred while processing your request.
Reference305c45116015585317441e332837

Also, it doesn't appear to be an authentication issue.I just used a web browser and logged in to WU as I would to manage my station, then went to the WXXDailyHistory URL and it still failed.Mind you, I am still using the old old old style password, not any so-called api key.  :-/Regards,\Leon--Leon Shaner :: Dearborn, Michigan (iPad Pro)On May 21, 2019, at 11:53 PM, gjr80 <gjroder...@gmail.com> wrote:Hmm, I notice that WXDailyHistory is working as usual now (well for me anyway) but wunderfixer still gives a 403 error.GaryOn Wednesday, 22 May 2019 13:02:24 UTC+10, gjr80  wrote:Wunderfixer (as it stands now) relies on WUs WXDailyHistory.asp page. If WXDailyHistory doesn't work then wunderfixer will not work. Period.I guess depending on where you sit on the conspiracy theory scale will determine whether you believe the current issues with WXDailyHistory are short term glitches or a silent closure of the service. There are a few threads on wxforum.net that touch on this issue, and of course many others that touch on the WU issues over the last not so long period. There seems to have been a few posts from WU staff indicating that 'all the old stuff is very fragile', so that could mean short term glitch, but on the other hand WU has been pushing folks towards their 'new' API which could mean the end for WXDailyHistory.The 'new' WU API that is available to PWS owners/uploaders does have a '1 day rapid history request' that provides what seems like the same sort of info as WXDailyHistory, albeit in JSON format and with a whole pile more cruft, well at least as far as wunderfixer is concerned. So in theory there is no reason wunderfixer could not be reworked to utilise the '1 day rapid history request' data. Of note to use the new API you need to have a (new) API key, whereas WXDailyHistory/wunderfixer does not require an API key. One shortcoming I did notice with the '1 day rapid history request' is there appears to be no way to obtain a given days data; you get the current day only, whereas with WXDailyHistory you can choose the date required, so that would mean no more historical wunderfixing.In any case, those who may consider WU as a de facto backup may want to start brushing up on their bash/MySQL skills and implement a robust local backup arrangement (though on WUs form you should have done that ages ago).Gary



-- 
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/fac9e205-aca1-4376-92d6-b1ddcb889d19%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.




-- 
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/D629ECBA-956A-48D1-97F6-662D8A4EE70E%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: We have finally put the last nail on the coffin for Wunderfixer?

[Leon Shaner]

FWIW, I had the odd 404 the other day, but then hours later it worked.
They may be just moving infrastructure around and not getting all the load 
balancer rules correct on the first go. Hope that's all it is...

Could it be that we need to request an API key and use that as a password, 
instead?

Here is what I see:

/usr/share/weewx/log/weewx.log.1:Thu 16 May 13:17:01 EDT 2019 Unable to open 
Weather Underground station KMIDEARB5  or  HTTP Error 404: Not Found
/usr/share/weewx/log/weewx.log.1:Thu 16 May 13:17:01 EDT 2019 Reason: HTTP 
Error 404: Not Found
/usr/share/weewx/log/weewx.log.1:Thu 16 May 13:17:01 EDT 2019 Unable to open 
Weather Underground station KMIDEARB5  or  HTTP Error 404: Not Found
/usr/share/weewx/log/weewx.log.1:Thu 16 May 13:17:01 EDT 2019 Reason: HTTP 
Error 404: Not Found
/usr/share/weewx/log/weewx.log.1:Fri 17 May 13:17:01 EDT 2019 Unable to open 
Weather Underground station KMIDEARB5  or  HTTP Error 404: Not Found
/usr/share/weewx/log/weewx.log.1:Fri 17 May 13:17:01 EDT 2019 Reason: HTTP 
Error 404: Not Found
/usr/share/weewx/log/weewx.log.1:Fri 17 May 13:17:01 EDT 2019 Unable to open 
Weather Underground station KMIDEARB5  or  HTTP Error 404: Not Found
/usr/share/weewx/log/weewx.log.1:Fri 17 May 13:17:01 EDT 2019 Reason: HTTP 
Error 404: Not Found
/usr/share/weewx/log/weewx.log.1:Sat 18 May 03:02:03 EDT 2019 Unable to open 
Weather Underground station KMIDEARB5  or  HTTP Error 404: Not Found
/usr/share/weewx/log/weewx.log.1:Sat 18 May 03:02:03 EDT 2019 Reason: HTTP 
Error 404: Not Found
/usr/share/weewx/log/weewx.log:Mon 20 May 13:17:01 EDT 2019 Unable to open 
Weather Underground station KMIDEARB5  or  HTTP Error 403: Forbidden
/usr/share/weewx/log/weewx.log:Mon 20 May 13:17:01 EDT 2019 Reason: HTTP Error 
403: Forbidden
/usr/share/weewx/log/weewx.log:Mon 20 May 13:17:01 EDT 2019 Unable to open 
Weather Underground station KMIDEARB5  or  HTTP Error 403: Forbidden
/usr/share/weewx/log/weewx.log:Mon 20 May 13:17:01 EDT 2019 Reason: HTTP Error 
403: Forbidden
/usr/share/weewx/log/weewx.log:Tue 21 May 01:17:01 EDT 2019 Unable to open 
Weather Underground station KMIDEARB5  or  HTTP Error 403: Forbidden
/usr/share/weewx/log/weewx.log:Tue 21 May 01:17:01 EDT 2019 Reason: HTTP Error 
403: Forbidden
/usr/share/weewx/log/weewx.log:Tue 21 May 01:17:01 EDT 2019 Unable to open 
Weather Underground station KMIDEARB5  or  HTTP Error 403: Forbidden
/usr/share/weewx/log/weewx.log:Tue 21 May 01:17:01 EDT 2019 Reason: HTTP Error 
403: Forbidden
/usr/share/weewx/log/weewx.log:Tue 21 May 13:17:01 EDT 2019 Unable to open 
Weather Underground station KMIDEARB5  or  HTTP Error 403: Forbidden
/usr/share/weewx/log/weewx.log:Tue 21 May 13:17:01 EDT 2019 Reason: HTTP Error 
403: Forbidden
/usr/share/weewx/log/weewx.log:Tue 21 May 13:17:01 EDT 2019 Unable to open 
Weather Underground station KMIDEARB5  or  HTTP Error 403: Forbidden
/usr/share/weewx/log/weewx.log:Tue 21 May 13:17:01 EDT 2019 Reason: HTTP Error 
403: Forbidden


Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 21, 2019, at 4:33 PM, Thomas Keffer  wrote:
> 
> It looks like the WU completely changed their API without warning. For 
> example, to access my station, it used to be
> https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KORHOODR3
> but now it's
> 
> https://www.wunderground.com/dashboard/pws/KORHOODR3
> You used to be able to download a day's worth of data, but now I'm not sure 
> how to do that without screen scraping. Wunderfixer may no longer be possible 
> to do, at least not accurately
> 
> -tk
> 
>> On Tue, May 21, 2019 at 12:46 PM Jarom Hatch  wrote:
>> I'm getting 403.  I entered an issue but upon looking further it appears to 
>> me that wunderground has either broken that url or intentionally deactivated 
>> it.  Hoping it isn't dead forever.
>> 
>>> On Tuesday, May 21, 2019 at 12:09:14 PM UTC-6, tds wrote:
>>> For the last three days, my daily cron job of wunderfixer have returned 
>>> "503 Unavaialbe" (May 18), "404 Not Found" (May 19) and "403 Forbidden" 
>>> (May 20).
>>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "weewx-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to weewx-user+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/weewx-user/54718d99-b68d-4f34-8213-4786a2578265%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@google

Re: [weewx-user] Re: Fresh install of weewx 3.9.1 with WMR300A on RPI, Non-positive value for record field 'interval': 0.0

[Leon Shaner]

Yeah, good catch.

I expect it was running the old wmr300.pyc since the html one named wmr300.py 
was invalid.

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPhone)

> On May 19, 2019, at 7:36 AM, gjr80  wrote:
> 
> Hi, 
> 
> If you are downloading a source file from github you need to get the raw 
> source otherwise you end up with the github htmml used to display the file. 
> Try using the following commands to download and install wmr300.py:
> 
> $ wget -P /tmp 
> https://raw.githubusercontent.com/weewx/weewx/master/bin/weewx/drivers/wmr300.py
> $ sudo cp /tmp/wmr300.py /usr/share/weewx/weewx/drivers/wmr300.py
> 
> Then restart WeeWX.
> 
> Gary
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/8281aa25-d717-4df6-8f61-7f4ca8ef4564%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/8600EB2A-E8FD-45A4-80B8-49E5D9B60DC0%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] upgrade from 3.8.2 to 3.9.1 failure

[Leon Shaner]

Hey, Ian,

You can use "wee_debug --info > /var/tmp/wee_debug.txt 2>&1" to write a helpful 
file at /var/tmp/wee_debug.txt for attaching to an e-mail.

You may want to scan the output anyway, in case something important didn't get 
obfuscated.  It does a good job with passwords, but ya never know... 
software... bugs happen.  :-/

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 18, 2019, at 9:02 PM, Ian Prescott  wrote:
> 
> sorry
> the station is an Aercus 3080
> and I am trying to find the howto on posting the config file
> 
>> On Sun, 19 May 2019 at 10:40, Leon Shaner  wrote:
>> Logs too abbreviated, or at least I am not able to pick out which driver / 
>> weatherstation version of are running.  Please expound.
>> WMR200, WMR300, both have this issue.  Maybe others.
>> Which model is yours?
>> 
>> Regards,
>> \Leon
>> --
>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>> 
>>> On May 18, 2019, at 8:05 PM, Ian Prescott  wrote:
>>> 
>>> Hi
>>> I did an upgrade on my pi and by default weewx got upgraded as well.
>>> After a reboot the syslog shows the following.
>>> Any help would be appreciated.
>>> Thanks
>>> 
>>> 
>>> May 19 09:18:31 weatherpi systemd[1]: Starting LSB: weewx weather system...
>>> May 19 09:18:31 weatherpi weewx[2257]: engine: Initializing weewx version 
>>> 3.9.1
>>> May 19 09:18:31 weatherpi weewx[2257]: engine: Using Python 2.7.9 (default, 
>>> Sep 26 2018, 05:58:52) #012[GCC 4.9.2]
>>> May 19 09:18:31 weatherpi weewx[2257]: engine: Platform 
>>> Linux-4.14.78-v7+-armv7l-with-debian-8.0
>>> May 19 09:18:31 weatherpi weewx[2257]: engine: Locale is 'en_AU.UTF-8'
>>> May 19 09:18:31 weatherpi weewx[2257]: engine: pid file is 
>>> /var/run/weewx.pid
>>> May 19 09:18:31 weatherpi weewx[2247]: Starting weewx weather system: weewx.
>>> May 19 09:18:31 weatherpi systemd[1]: Started LSB: weewx weather system.
>>> May 19 09:18:31 weatherpi weewx[2261]: engine: Using configuration file 
>>> /etc/weewx/weewx.conf
>>> May 19 09:18:31 weatherpi weewx[2261]: engine: Debug is 1
>>> May 19 09:18:31 weatherpi weewx[2261]: engine: Initializing engine
>>> May 19 09:18:31 weatherpi weewx[2261]: engine: Loading station type 
>>> FineOffsetUSB (weewx.drivers.fousb)
>>> May 19 09:18:31 weatherpi weewx[2261]: fousb: driver version is 1.9
>>> May 19 09:18:31 weatherpi weewx[2261]: fousb: power cycling enabled for 
>>> port 4 on hub 001:004
>>> May 19 09:18:31 weatherpi weewx[2261]: fousb: polling mode is PERIODIC
>>> May 19 09:18:31 weatherpi weewx[2261]: fousb: polling interval is 60
>>> May 19 09:18:31 weatherpi weewx[2261]: fousb: found station on USB bus=001 
>>> device=006
>>> May 19 09:18:31 weatherpi weewx[2261]: engine: Loading service 
>>> weewx.engine.StdTimeSynch
>>> May 19 09:18:31 weatherpi weewx[2261]: engine: Finished loading service 
>>> weewx.engine.StdTimeSynch
>>> May 19 09:18:31 weatherpi weewx[2261]: engine: Loading service 
>>> weewx.engine.StdConvert
>>> May 19 09:18:31 weatherpi weewx[2261]: engine: StdConvert target unit is 0x1
>>> May 19 09:18:31 weatherpi weewx[2261]: engine: Finished loading service 
>>> weewx.engine.StdConvert
>>> May 19 09:18:31 weatherpi weewx[2261]: engine: Loading service 
>>> weewx.engine.StdCalibrate
>>> May 19 09:18:31 weatherpi weewx[2261]: engine: Finished loading service 
>>> weewx.engine.StdCalibrate
>>> May 19 09:18:31 weatherpi weewx[2261]: engine: Loading service 
>>> weewx.engine.StdQC
>>> May 19 09:18:31 weatherpi weewx[2261]: engine: Finished loading service 
>>> weewx.engine.StdQC
>>> May 19 09:18:31 weatherpi weewx[2261]: engine: Loading service 
>>> weewx.wxservices.StdWXCalculate
>>> May 19 09:18:31 weatherpi weewx[2261]: wxcalculate: The following values 
>>> will be calculated: barometer=prefer_hardware, windchill=prefer_hardware, 
>>> dewpoint=prefer_hardware, appTemp=prefer_hardware, 
>>> rainRate=prefer_hardware, windrun=prefer_hardware, 
>>> heatindex=prefer_hardware, maxSolarRad=prefer_hardware, 
>>> humidex=prefer_hardware, pressure=prefer_hardware, 
>>> inDewpoint=prefer_hardware, ET=prefer_hardware, altimeter=prefer_hardware, 
>>> cloudbase=prefer_hardware
>>> May 19 09:18:31 weatherpi weewx[2261]: wxcalculate: The following 
>>> algorithms will be used for calculations: altimeter=aaNOAA, maxSolarRad=RS
>>> May 19 09:18:31 weatherpi weewx[226

Re: [weewx-user] upgrade from 3.8.2 to 3.9.1 failure

[Leon Shaner]

Logs too abbreviated, or at least I am not able to pick out which driver / 
weatherstation version of are running.  Please expound.
WMR200, WMR300, both have this issue.  Maybe others.
Which model is yours?

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 18, 2019, at 8:05 PM, Ian Prescott  wrote:
> 
> Hi
> I did an upgrade on my pi and by default weewx got upgraded as well.
> After a reboot the syslog shows the following.
> Any help would be appreciated.
> Thanks
> 
> 
> May 19 09:18:31 weatherpi systemd[1]: Starting LSB: weewx weather system...
> May 19 09:18:31 weatherpi weewx[2257]: engine: Initializing weewx version 
> 3.9.1
> May 19 09:18:31 weatherpi weewx[2257]: engine: Using Python 2.7.9 (default, 
> Sep 26 2018, 05:58:52) #012[GCC 4.9.2]
> May 19 09:18:31 weatherpi weewx[2257]: engine: Platform 
> Linux-4.14.78-v7+-armv7l-with-debian-8.0
> May 19 09:18:31 weatherpi weewx[2257]: engine: Locale is 'en_AU.UTF-8'
> May 19 09:18:31 weatherpi weewx[2257]: engine: pid file is /var/run/weewx.pid
> May 19 09:18:31 weatherpi weewx[2247]: Starting weewx weather system: weewx.
> May 19 09:18:31 weatherpi systemd[1]: Started LSB: weewx weather system.
> May 19 09:18:31 weatherpi weewx[2261]: engine: Using configuration file 
> /etc/weewx/weewx.conf
> May 19 09:18:31 weatherpi weewx[2261]: engine: Debug is 1
> May 19 09:18:31 weatherpi weewx[2261]: engine: Initializing engine
> May 19 09:18:31 weatherpi weewx[2261]: engine: Loading station type 
> FineOffsetUSB (weewx.drivers.fousb)
> May 19 09:18:31 weatherpi weewx[2261]: fousb: driver version is 1.9
> May 19 09:18:31 weatherpi weewx[2261]: fousb: power cycling enabled for port 
> 4 on hub 001:004
> May 19 09:18:31 weatherpi weewx[2261]: fousb: polling mode is PERIODIC
> May 19 09:18:31 weatherpi weewx[2261]: fousb: polling interval is 60
> May 19 09:18:31 weatherpi weewx[2261]: fousb: found station on USB bus=001 
> device=006
> May 19 09:18:31 weatherpi weewx[2261]: engine: Loading service 
> weewx.engine.StdTimeSynch
> May 19 09:18:31 weatherpi weewx[2261]: engine: Finished loading service 
> weewx.engine.StdTimeSynch
> May 19 09:18:31 weatherpi weewx[2261]: engine: Loading service 
> weewx.engine.StdConvert
> May 19 09:18:31 weatherpi weewx[2261]: engine: StdConvert target unit is 0x1
> May 19 09:18:31 weatherpi weewx[2261]: engine: Finished loading service 
> weewx.engine.StdConvert
> May 19 09:18:31 weatherpi weewx[2261]: engine: Loading service 
> weewx.engine.StdCalibrate
> May 19 09:18:31 weatherpi weewx[2261]: engine: Finished loading service 
> weewx.engine.StdCalibrate
> May 19 09:18:31 weatherpi weewx[2261]: engine: Loading service 
> weewx.engine.StdQC
> May 19 09:18:31 weatherpi weewx[2261]: engine: Finished loading service 
> weewx.engine.StdQC
> May 19 09:18:31 weatherpi weewx[2261]: engine: Loading service 
> weewx.wxservices.StdWXCalculate
> May 19 09:18:31 weatherpi weewx[2261]: wxcalculate: The following values will 
> be calculated: barometer=prefer_hardware, windchill=prefer_hardware, 
> dewpoint=prefer_hardware, appTemp=prefer_hardware, rainRate=prefer_hardware, 
> windrun=prefer_hardware, heatindex=prefer_hardware, 
> maxSolarRad=prefer_hardware, humidex=prefer_hardware, 
> pressure=prefer_hardware, inDewpoint=prefer_hardware, ET=prefer_hardware, 
> altimeter=prefer_hardware, cloudbase=prefer_hardware
> May 19 09:18:31 weatherpi weewx[2261]: wxcalculate: The following algorithms 
> will be used for calculations: altimeter=aaNOAA, maxSolarRad=RS
> May 19 09:18:31 weatherpi weewx[2261]: engine: Finished loading service 
> weewx.wxservices.StdWXCalculate
> May 19 09:18:31 weatherpi weewx[2261]: engine: Loading service 
> weewx.engine.StdArchive
> May 19 09:18:31 weatherpi weewx[2261]: engine: Archive will use data binding 
> wx_binding
> May 19 09:18:31 weatherpi weewx[2261]: engine: Record generation will be 
> attempted in 'software'
> May 19 09:18:31 weatherpi weewx[2261]: engine: Using archive interval of 300 
> seconds (software record generation)
> May 19 09:18:31 weatherpi weewx[2261]: engine: Use LOOP data in hi/low 
> calculations: 1
> May 19 09:18:31 weatherpi weewx[2261]: manager: Daily summary version is 2.0
> May 19 09:18:31 weatherpi weewx[2261]: engine: Using binding 'wx_binding' to 
> database 'weewx.sdb'
> May 19 09:18:31 weatherpi weewx[2261]: manager: Starting backfill of daily 
> summaries
> May 19 09:18:32 weatherpi weewx[2261]: engine: Caught unrecoverable exception 
> in engine:
> May 19 09:18:32 weatherpi weewx[2261]:   Non-positive value for 
> record field 'interval': 0
> May 19 09:18:32 weatherpi weewx[2261]:   Traceback (most recent call 
> last):
> May 19 09:18:32 weatherpi weewx[2261]: File 
> "

Re: [weewx-user] Re: Time Tags and UTC

[Leon Shaner]

We're all fine with the timestamps until 2038.  ;-)

https://en.m.wikipedia.org/wiki/Year_2038_problem

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 16, 2019, at 7:23 PM, gjr80  wrote:
> 
> Hi,
> 
> It was a deliberate decision for WeeWX to be timezone unaware, but 
> fortunately we can do some limited utc manipulation using some of the inbuilt 
> utc capabilities of the python datetime module. In this case the easiest 
> approach is likely just putting a little inline python code in your template. 
> Something like:
> 
> #import datetime
> #set $local_dt = datetime.datetime.utcfromtimestamp($current.dateTime.raw)
> #set $utc_formatted = $local_dt.strftime("%d/%m/%Y %H:%M:%S")
> 
> and then when you want to display the utc time just use the $utc_formatted 
> tag in your template, something like:
> 
> The current time is: $current.dateTime (local) or $utc_formatted (UTC)
> 
> would give something like:
> 
> The current time is: 17/05/19 09:18:00 (local) or 16/05/2019 23:18:00 (UTC)
> 
> The only requirement is the code needs to be included at a location in the 
> template before the tag is used. The format of the output can be changed by 
> using the format codes from here. You could go to more trouble and create a 
> tag (or an SLE that returns a tag) that accepts the format string as a 
> parameter but given your intended use this is the simplest approach.
> 
> Gary
> 
>> On Friday, 17 May 2019 08:48:11 UTC+10, Eric Mears wrote:
>> Hi,
>> 
>> I'm running weewx on a raspberry pi connected to a Vantage weather station.
>> 
>> I create a custom report that that National Weather Service downloads hourly.
>> 
>> The National Weather Service requires the times be in UTC.
>> 
>> 
>> I'd like to run my raspberry pi on local time so the weewx reports display 
>> local time.
>> 
>> Is there an option on a Time Tag to display UTC?
>> 
>> I'm currently using variations of 
>> 
>> $current.dateTime.format("%H:%M %Z")
>> 
>> in my custom report, but these all display local time.
>> 
>> 
>> Do I have any options other than running the raspberry pi on UTC?
>> 
>> Thanks,
>> 
>> Eric
>> 
>> 
>> 
>> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/af9ae8c1-a26a-4ee3-a305-63f83d439fcc%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/4D5481F4-CABF-4295-8FE5-572492E9C7E8%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: WS3080 Lockup - FineOffsetUsb workaround not working.

[Leon Shaner]

Wow!  This is great to know!  Thanks!

I will try this and if it works in my case, I will add it as the primary 
remediation step in my weewx_watchdog.  =D

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPhone)

> On May 14, 2019, at 9:40 AM, HoracioDos  wrote:
> 
> Hello!
> 
> I also had problems to reset port with usb_control. I own the same WS. I was 
> able to reset raspberry pi port 2 without a hub with uhubctl
> https://github.com/mvp/uhubctl
> 
> sudo ./uhubctl -a cycle -p 2
> Current status for hub 1-1 [0424:9514, USB 2.00, 5 ports]
>   Port 2: 0303 power lowspeed enable connect [1941:8021]
> Sent power off request
> New status for hub 1-1 [0424:9514, USB 2.00, 5 ports]
>   Port 2:  off
> Current status for hub 1-1 [0424:9514, USB 2.00, 5 ports]
>   Port 2:  off
> Sent power on request
> New status for hub 1-1 [0424:9514, USB 2.00, 5 ports]
>   Port 2: 0301 power lowspeed connect [1941:8021]
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/a7237ce7-a910-430c-812e-fb95af77b81b%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/F34F3388-02C8-486D-9F0D-78DCBBEF24D6%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] On this date

[Leon Shaner]

Hey, Robin,

I don't know the weewx way, but here is a sql way that may lead some smart 
person to a better solution.  :-/
I used the system date command up front to get the month-day, which is to avoid 
asking SQLite to do that computation for every row within the query.

$ today=$(date +%\m-%\d); sqlite3 -header /var/lib/weewx/weewx.sdb "select 
date(dateTime, 'unixepoch', 'localtime') as day, min, max from 
archive_day_outTemp where strftime('%m-%d', day)='$today' group by day;"
day|min|max
2017-05-11|46.9|70.0
2018-05-11|43.5|54.9
2019-05-11|39.02|52.7

Oh, and in case outTemp is not the name of your sensor per archive_day_outTemp, 
above, you can glean the table you need from here:

$ sqlite3 /var/lib/weewx/weewx.sdb ".tables"

And of course to see the column names available, it's:

$ sqlite3 /var/lib/weewx/weewx.sdb ".schema archive_day_outTemp"
CREATE TABLE archive_day_outTemp (dateTime INTEGER NOT NULL UNIQUE PRIMARY KEY, 
min REAL, mintime INTEGER, max REAL, maxtime INTEGER, sum REAL, count INTEGER, 
wsum REAL, sumtime INTEGER);

Or the more human readable way:

$ sqlite3 -header /var/lib/weewx/weewx.sdb "select * from archive_day_outTemp 
limit 1;"
dateTime|min|mintime|max|maxtime|sum|count|wsum|sumtime
1325394000|0.0|0|0.0|0

HTH.  =D

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 11, 2019, at 9:30 AM, Robin  wrote:
> 
> I apologise if this has been asked before or if there is a simple and obvious 
> way to do this, but I can't see it.
> 
> I want to display the temperature (min,max) for today's date for each year 
> since we started keeping records.
> 
> Can somebody point me in the right direction?
> 
> Thanks people.
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/75abaada-5cb6-47ec-9d3e-0b3046f4f9a3%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/A548DC9B-FF2C-4C11-ABE2-F99F13EF8640%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] On this date

[Leon Shaner]

Hey, Robin!

I just posted a reply, but the alias didn't echo it back to my inbox.
I think it must have been eaten by the AEther.  :-(
If it doesn't show up soon, I'll try posting it via the web interface to 
googlegroups.com

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 11, 2019, at 9:30 AM, Robin  wrote:
> 
> I apologise if this has been asked before or if there is a simple and obvious 
> way to do this, but I can't see it.
> 
> I want to display the temperature (min,max) for today's date for each year 
> since we started keeping records.
> 
> Can somebody point me in the right direction?
> 
> Thanks people.
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/75abaada-5cb6-47ec-9d3e-0b3046f4f9a3%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/017227BA-D43B-4BDE-B103-48550B0C956C%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: "Bad USB report received" from WMR88

[Leon Shaner]

Wysiwyg,

Oh, then never mind.  I am surprised reboot is no help.

Do you have any other USB devices plugged in, such as any hard drives?
If so, do you have a HUB in between to supply additional power?
That can be a common issue with RPI and since all ports are on a hub behind 
what is really a single port, if any other devices are drawing too much power 
it could effect your weather station's comms.

Other than that, how long is your USB cable?
15 feet is the limit for a basic cable.  Longer than that and you need a 
powered "repeater" of sorts.  Can you try a different cable?

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 12, 2019, at 1:32 PM, wysiwyg  wrote:
> 
> Thanks for your answer Leon, but do you mean the watchdog will reboot in case 
> of usb errors?
> Because currently, reboot seems not helping in my situation. 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/bf72b3e0-4ca6-404c-a9e1-8c1ce4bcf2ea%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/E2708CE6-EC29-45DF-800F-ACEF6670BA1B%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: "Bad USB report received" from WMR88

[Leon Shaner]

Hey, Wysiwyg,

Lots of people have reported USB issues on Raspberry Pi, myself included.
This is not a weewx issue, rather it's something wrong somewhere between the 
Raspbian kernel, USB driver, and the weatherstation firmware.  Apparently much 
older kernel releases didn't have the problem but any of the newer ones do.
The issue is also seen sometimes on full desktop-class linuxes.

Seems the issue is really out of our control, so that's ultimately why I wrote 
the weewx_watchdog utility that I've mentioned before.

It's over here if you are interested:

https://github.com/UberEclectic/weewx/tree/watchdog/examples/watchdog

The latest version now spreads the wunderfixer calls out over twice the time, 
which is to allow WU more time to process the uploads, so that ultimately any 
loss of data -> WU is minimized.

If your system is like mine you can more than likely disable the restart 
remediation and just go with the reboot remediation.  In my case when the USB 
comms are wedged, restarting weewx is no help, so it's just a waste of over a 
minute before moving ahead to the reboot.

All behavior is easily customizable via a few flags at the top of the script.
Lemme know, off alias, if you have any questions.

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 12, 2019, at 11:40 AM, wysiwyg  wrote:
> 
> I forget to tell it's a WMR88 station, using WMR100 driver.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/385d0961-4d2f-4d99-8b22-e36246749936%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/9136CC3F-8A52-4944-A022-DA1F710DE7A8%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] On this date

[Leon Shaner]

Hey, Robin!

I just posted a reply to the group (via e-mail), but didn't echo it back to my 
inbox.
I think it must have been eaten by the AEther.  :-(
Then I tried a more simple message and it didn't post, either.
So now I am trying from the web interface to google groups.
Something seems to be wrong with the alias, or is it just me?

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/76b5886d-ccb6-4671-a318-9fc00d40a68e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] On this date

[Leon Shaner]

Posting from the web interface did get echoed to my inbox.  So whatever is 
wrong is on the e-mail reply side.
Here is my original reply.  Hopefully we don't get a duplicate whenever Google 
fixes whatever is wrong with the e-mail interface to the Group.  :-/

FWD:

Hey, Robin,

I don't know the weewx way, but here is a sql way that may lead some smart 
person to a better solution.  :-/
I used the system date command up front to get the month-day, which is to avoid 
asking SQLite to do that computation for every row within the query.

$ today=$(date +%\m-%\d); sqlite3 -header /var/lib/weewx/weewx.sdb "select 
date(dateTime, 'unixepoch', 'localtime') as day, min, max from 
archive_day_outTemp where strftime('%m-%d', day)='$today' group by day;"
day|min|max
2017-05-11|46.9|70.0
2018-05-11|43.5|54.9
2019-05-11|39.02|52.7

Oh, and in case outTemp is not the name of your sensor per archive_day_outTemp, 
above, you can glean the table you need from here:

$ sqlite3 /var/lib/weewx/weewx.sdb ".tables"

And of course to see the column names available, it's:

$ sqlite3 /var/lib/weewx/weewx.sdb ".schema archive_day_outTemp"
CREATE TABLE archive_day_outTemp (dateTime INTEGER NOT NULL UNIQUE PRIMARY KEY, 
min REAL, mintime INTEGER, max REAL, maxtime INTEGER, sum REAL, count INTEGER, 
wsum REAL, sumtime INTEGER);

Or the more human readable way:

$ sqlite3 -header /var/lib/weewx/weewx.sdb "select * from archive_day_outTemp 
limit 1;"
dateTime|min|mintime|max|maxtime|sum|count|wsum|sumtime
1325394000|0.0|0|0.0|0

HTH.  =D

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/b9d8a97c-17db-4a7b-80aa-7d3354e97485%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Attempting to switch to non-root user for rsync

[Leon Shaner]

Steve,

Probably a leftover.
I expect this should do it:

$ sudo chown weewx /var/run/weewx.pid

I would expect that file to be deleted on host reboot, except of course it is 
pronounced reboot, but spelled:

$ sudo shutdown -r now

If you use actual "reboot" then it skips shutdown scripts, so it's a big no-no.

What about the other files, did you change their owner, too, such as for the DB?

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 7, 2019, at 5:42 PM, Steve Chiz  wrote:
> 
> Guess I spoke too soon. After a reboot, weewx won't start. IOError: [Errno 
> 13] Permission denied: '/var/run/weewx.pid'
> 
>> On Tuesday, May 7, 2019 at 4:37:44 PM UTC-4, Steve Chiz wrote:
>> Basically, it was my lack of understanding on how the .rules files work. I 
>> appreciate the explanation of the granular permissions as it helped me 
>> understand the 'why'. I am not worried about others plugging USB devices 
>> into the pi, so I went ahead and edited the 99-usb.rules and added my newly 
>> created weewx user to the plugdev group. I am successfully running weewx as 
>> non-root, thanks again!  WeeWx still complains that my key verification 
>> fails, but I can directly ssh successfully to my remote host as the weewx 
>> user without a password, so I'm close. 
>> 
>> Oh, and thanks for updating the wiki. I had run the "lsusb", I just wasn't 
>> entirely sure what to do with the output. The edit makes it more clear.
>> 
>>> On Tuesday, May 7, 2019 at 11:28:35 AM UTC-4, Leon Shaner wrote:
>>> Steve,
>>> Hope it works!  =D
>>> 
>>> I just updated the wiki.  That section now reads:
>>> 
>>> First find the idVendor and idProduct of your weatherstation with lsusb 
>>> command then add a rules file in /etc/udev/rules.d/ with this content:
>>> 
>>> SUBSYSTEM=="usb", ATTR{idVendor}=="your_value", 
>>> ATTR{idProduct}=="your_value", ACTION=="add", GROUP="weewx", MODE="0664"
>>> Name the udev rules file something descriptive, such as an abbreviation of 
>>> your weatherstation model or just weewx.rules, a la 
>>> /etc/udev/rules.d/weewx.rules (extension must be .rules and filename should 
>>> be simple, no spaces or special characters other than '-' and/or '_' and 
>>> should not contain more than one period '.').
>>> 
>>> 
>>> Regards,
>>> \Leon
>>> --
>>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>>> 
>>>> On May 7, 2019, at 10:39 AM, Leon Shaner  wrote:
>>>> 
>>>> Steve,
>>>> 
>>>> In my first reply, I failed to answer your first question.
>>>> 
>>>> Yes, if you use the first form with idVendor, idProduct explicitly filled 
>>>> in, you can call the UDEV rules file anything you like, as long as the 
>>>> extension is .rules and you place it in the /etc/udev/rules.d directory.
>>>> 
>>>> I used a more generic /etc/udev/rules.d/99-usb.rules in my example, 
>>>> because my example is very generic, not tied to weewx, but would work for 
>>>> weewx provided weewx user is in the plugdev group.
>>>> 
>>>> The (optional) number prefixes on the UDEV .rules files establish an order 
>>>> of precedence with later rules overriding earlier rules.  Really it's 
>>>> ordered lexicographically, so files that start with letters, such as 
>>>> weewx.rules will be evaluated after (take precedence over) the files that 
>>>> do start with numbers.
>>>> 
>>>> Regards,
>>>> \Leon
>>>> --
>>>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>>>> 
>>>>> On May 7, 2019, at 10:31 AM, Leon Shaner  wrote:
>>>>> 
>>>>> Hey, Steve,
>>>>> 
>>>>> That first wiki looks pretty complete.
>>>>> Did you in fact try the "lsusb" command to get the values you need for 
>>>>> the first form of the udev rules?
>>>>> Using the first form with the idVendor and idProduct for your weather 
>>>>> station is preferred.
>>>>> 
>>>>> As an alternative, and if it's just you with physical access to the host 
>>>>> and USB devices, e.g. you aren't too worried about other people 
>>>>> connecting USB devices and accessing them as non-root, you can also just 
>>>>> do this:
>>>>> 
>>>>> File:  /etc/udev/rules.d/99-usb.rules
>>>>

Re: [weewx-user] Attempting to switch to non-root user for rsync

[Leon Shaner]

Steve,
Hope it works!  =D

I just updated the wiki.  That section now reads:

First find the idVendor and idProduct of your weatherstation with lsusb command 
then add a rules file in /etc/udev/rules.d/ with this content:

SUBSYSTEM=="usb", ATTR{idVendor}=="your_value", ATTR{idProduct}=="your_value", 
ACTION=="add", GROUP="weewx", MODE="0664"
Name the udev rules file something descriptive, such as an abbreviation of your 
weatherstation model or just weewx.rules, a la /etc/udev/rules.d/weewx.rules 
(extension must be .rules and filename should be simple, no spaces or special 
characters other than '-' and/or '_' and should not contain more than one 
period '.').


Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 7, 2019, at 10:39 AM, Leon Shaner  wrote:
> 
> Steve,
> 
> In my first reply, I failed to answer your first question.
> 
> Yes, if you use the first form with idVendor, idProduct explicitly filled in, 
> you can call the UDEV rules file anything you like, as long as the extension 
> is .rules and you place it in the /etc/udev/rules.d directory.
> 
> I used a more generic /etc/udev/rules.d/99-usb.rules in my example, because 
> my example is very generic, not tied to weewx, but would work for weewx 
> provided weewx user is in the plugdev group.
> 
> The (optional) number prefixes on the UDEV .rules files establish an order of 
> precedence with later rules overriding earlier rules.  Really it's ordered 
> lexicographically, so files that start with letters, such as weewx.rules will 
> be evaluated after (take precedence over) the files that do start with 
> numbers.
> 
> Regards,
> \Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPad Pro)
> 
>> On May 7, 2019, at 10:31 AM, Leon Shaner  wrote:
>> 
>> Hey, Steve,
>> 
>> That first wiki looks pretty complete.
>> Did you in fact try the "lsusb" command to get the values you need for the 
>> first form of the udev rules?
>> Using the first form with the idVendor and idProduct for your weather 
>> station is preferred.
>> 
>> As an alternative, and if it's just you with physical access to the host and 
>> USB devices, e.g. you aren't too worried about other people connecting USB 
>> devices and accessing them as non-root, you can also just do this:
>> 
>> File:  /etc/udev/rules.d/99-usb.rules
>> Contents:
>> SUBSYSTEM=="usb", GROUP="plugdev", MODE="0660"
>> 
>> Then be sure to put the wxuser and any other users in the "plugdev" group in 
>> /etc/group, a la:
>> 
>> plugdev:x:46:steve,pi,weewx
>> 
>> (Or whatever usernames you care to be allowed to access USB ports).
>> (Your GID may differ from 46)...
>> 
>> Notice that for perms, above, I put 0660.  I can't think why "others" / 
>> "nobody" should even need to read the USB ports.  Anybody that needs to 
>> read(or write) USB ports should be in the "plugdev" group.
>> 
>> You could of course put GROUP="weewx" in my example above, but then any user 
>> would need to be in the weewx port to use any USB device, even those 
>> unrelated to weewx.  The "plugdev" group is commonly used for other USB 
>> devices, such as auto-mounting removable media, so that is why I chose it in 
>> my example.  If you used my example and put GROUP="weewx" it would likely 
>> break auto-mounting of removable media (maybe you don't care; maybe you 
>> don't use the usbmount service, etc.).
>> 
>> Note that changes in /etc/group take a log out / log in to take effect.
>> Check group membership via "id -a" ...
>> 
>> Of course the explicit method, per the wiki, using the idVendor and 
>> idProduct values for your specific USB device avoids any conflict, because 
>> then assigning group weewx would only ever happen to that one device that 
>> exactly matches the idVendor and idProduct values from "lsusb" output.
>> 
>> Hope that helps!  =D
>> 
>> Regards,
>> \Leon
>> --
>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>> 
>>> On May 7, 2019, at 9:37 AM, Steve Chiz  wrote:
>>> 
>>> I've been trying to use the wiki to resolve this on my own, but can't seem 
>>> to sort it out. This page suggests I create a rules file, but no indication 
>>> on what that file should be named...  weewx.rules?  
>>> https://github.com/weewx/weewx/wiki/systemd 
>>> 
>>> I hunted up an older page 
>>> https://github.com/weewx/weewx/wiki/Run-as-a-non-root-user that cites an 
>

Re: [weewx-user] Attempting to switch to non-root user for rsync

[Leon Shaner]

Steve,

In my first reply, I failed to answer your first question.

Yes, if you use the first form with idVendor, idProduct explicitly filled in, 
you can call the UDEV rules file anything you like, as long as the extension is 
.rules and you place it in the /etc/udev/rules.d directory.

I used a more generic /etc/udev/rules.d/99-usb.rules in my example, because my 
example is very generic, not tied to weewx, but would work for weewx provided 
weewx user is in the plugdev group.

The (optional) number prefixes on the UDEV .rules files establish an order of 
precedence with later rules overriding earlier rules.  Really it's ordered 
lexicographically, so files that start with letters, such as weewx.rules will 
be evaluated after (take precedence over) the files that do start with numbers.

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 7, 2019, at 10:31 AM, Leon Shaner  wrote:
> 
> Hey, Steve,
> 
> That first wiki looks pretty complete.
> Did you in fact try the "lsusb" command to get the values you need for the 
> first form of the udev rules?
> Using the first form with the idVendor and idProduct for your weather station 
> is preferred.
> 
> As an alternative, and if it's just you with physical access to the host and 
> USB devices, e.g. you aren't too worried about other people connecting USB 
> devices and accessing them as non-root, you can also just do this:
> 
> File:  /etc/udev/rules.d/99-usb.rules
> Contents:
> SUBSYSTEM=="usb", GROUP="plugdev", MODE="0660"
> 
> Then be sure to put the wxuser and any other users in the "plugdev" group in 
> /etc/group, a la:
> 
> plugdev:x:46:steve,pi,weewx
> 
> (Or whatever usernames you care to be allowed to access USB ports).
> (Your GID may differ from 46)...
> 
> Notice that for perms, above, I put 0660.  I can't think why "others" / 
> "nobody" should even need to read the USB ports.  Anybody that needs to 
> read(or write) USB ports should be in the "plugdev" group.
> 
> You could of course put GROUP="weewx" in my example above, but then any user 
> would need to be in the weewx port to use any USB device, even those 
> unrelated to weewx.  The "plugdev" group is commonly used for other USB 
> devices, such as auto-mounting removable media, so that is why I chose it in 
> my example.  If you used my example and put GROUP="weewx" it would likely 
> break auto-mounting of removable media (maybe you don't care; maybe you don't 
> use the usbmount service, etc.).
> 
> Note that changes in /etc/group take a log out / log in to take effect.
> Check group membership via "id -a" ...
> 
> Of course the explicit method, per the wiki, using the idVendor and idProduct 
> values for your specific USB device avoids any conflict, because then 
> assigning group weewx would only ever happen to that one device that exactly 
> matches the idVendor and idProduct values from "lsusb" output.
> 
> Hope that helps!  =D
> 
> Regards,
> \Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPad Pro)
> 
>> On May 7, 2019, at 9:37 AM, Steve Chiz  wrote:
>> 
>> I've been trying to use the wiki to resolve this on my own, but can't seem 
>> to sort it out. This page suggests I create a rules file, but no indication 
>> on what that file should be named...  weewx.rules?  
>> https://github.com/weewx/weewx/wiki/systemd 
>> 
>> I hunted up an older page 
>> https://github.com/weewx/weewx/wiki/Run-as-a-non-root-user that cites an 
>> example for Vantage (name the file vpro.rules) but what about other devices? 
>> In any event, the contents of the rules file is different than the more 
>> recently edited page. Which should I use? 
>> 
>> SUBSYSTEM=="usb", ATTR{idVendor}=="your_value", 
>> ATTR{idProduct}=="your_value", ACTION=="add", GROUP="weewx", MODE="0664"
>>  or 
>> SUBSYSTEM=="usb", ATTRS{interface}=="CP2102 USB to UART Bridge Controller", 
>> MODE: = "664", GROUP = "wxuser"
>> 
>> I get that one page is about systemd specifically, which I am using, but 
>> both address the need to run weewx as a non-root user. If someone could 
>> point me to some documentation on how to switch from running weewx as root 
>> to a non-root user, that would be great! I probably should have set it up 
>> that way initially, regardless of rsync, as running as root always seems 
>> like a risky idea.
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "weewx-user" group.
>> To unsubscribe

Re: [weewx-user] Attempting to switch to non-root user for rsync

[Leon Shaner]

Hey, Steve,

That first wiki looks pretty complete.
Did you in fact try the "lsusb" command to get the values you need for the 
first form of the udev rules?
Using the first form with the idVendor and idProduct for your weather station 
is preferred.

As an alternative, and if it's just you with physical access to the host and 
USB devices, e.g. you aren't too worried about other people connecting USB 
devices and accessing them as non-root, you can also just do this:

File:  /etc/udev/rules.d/99-usb.rules
Contents:
SUBSYSTEM=="usb", GROUP="plugdev", MODE="0660"

Then be sure to put the wxuser and any other users in the "plugdev" group in 
/etc/group, a la:

plugdev:x:46:steve,pi,weewx

(Or whatever usernames you care to be allowed to access USB ports).
(Your GID may differ from 46)...

Notice that for perms, above, I put 0660.  I can't think why "others" / 
"nobody" should even need to read the USB ports.  Anybody that needs to read(or 
write) USB ports should be in the "plugdev" group.

You could of course put GROUP="weewx" in my example above, but then any user 
would need to be in the weewx port to use any USB device, even those unrelated 
to weewx.  The "plugdev" group is commonly used for other USB devices, such as 
auto-mounting removable media, so that is why I chose it in my example.  If you 
used my example and put GROUP="weewx" it would likely break auto-mounting of 
removable media (maybe you don't care; maybe you don't use the usbmount 
service, etc.).

Note that changes in /etc/group take a log out / log in to take effect.
Check group membership via "id -a" ...

Of course the explicit method, per the wiki, using the idVendor and idProduct 
values for your specific USB device avoids any conflict, because then assigning 
group weewx would only ever happen to that one device that exactly matches the 
idVendor and idProduct values from "lsusb" output.

Hope that helps!  =D

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 7, 2019, at 9:37 AM, Steve Chiz  wrote:
> 
> I've been trying to use the wiki to resolve this on my own, but can't seem to 
> sort it out. This page suggests I create a rules file, but no indication on 
> what that file should be named...  weewx.rules?  
> https://github.com/weewx/weewx/wiki/systemd 
> 
> I hunted up an older page 
> https://github.com/weewx/weewx/wiki/Run-as-a-non-root-user that cites an 
> example for Vantage (name the file vpro.rules) but what about other devices? 
> In any event, the contents of the rules file is different than the more 
> recently edited page. Which should I use? 
> 
> SUBSYSTEM=="usb", ATTR{idVendor}=="your_value", 
> ATTR{idProduct}=="your_value", ACTION=="add", GROUP="weewx", MODE="0664"
>  or 
> SUBSYSTEM=="usb", ATTRS{interface}=="CP2102 USB to UART Bridge Controller", 
> MODE: = "664", GROUP = "wxuser"
> 
> I get that one page is about systemd specifically, which I am using, but both 
> address the need to run weewx as a non-root user. If someone could point me 
> to some documentation on how to switch from running weewx as root to a 
> non-root user, that would be great! I probably should have set it up that way 
> initially, regardless of rsync, as running as root always seems like a risky 
> idea.
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/weewx-user/aaab2dd1-376f-4f89-82a6-8ff03d032c9e%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/0130621D-4F28-4F79-8036-1EF1743D9A95%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] WeeWX Watchdog, latest improvements

[Leon Shaner]

Hey, WeeWX'ers!  =D

Lots of improvements made to my watchdog script(s).

1)   wunderfixer is now decoupled, except in conjunction with an outage.

It means that even if dowufixer=1 (enabled), it will only run if it is within a 
certain time-frame after an outage (watchdogsecs * repeatwufixer).
With the defaults, wunderfixer runs every 10 minutes, six times, i.e. spread 
over an hour after an outage.

2)  A separate weewx_wunderfixer wrapper is provided to run separately twice a 
day.
See readme for recommendations.  This change and #1 above is in the spirit of 
lowering the amount of "gratuitous" calls to WU infrastructure, while still 
attempting to keep WU up-to-date a soon as possible after an outage.  The main 
purpose of the weewx_wunderfixer is to compute today's and yesterday's dates 
and run against both, just to be extra sure that there are no gaps on the WU 
side.  Decoupling now means those actions only occur twice a day, instead of 
every watchdogsecs (e.g. every 10 minutes by default), plus a default of 6 more 
times after an outage.

3)  The running status of weewx is now explicitly checked, which is in part to 
catch an outage sooner, in case weewx crashed very recently after a cron 
interval.
A similar check is now added after a weewx restart attempt, which avoids a 
double watchdogsecs wait (allows back to back weewx restart and host reboot 
remediations, which is especially nice in the case of a USB / firmware hang).

4)  Improved and more consistent logging with a running history of status and 
remediation steps the beginning of the current pass of weewx_watchdog.

That one proved more challenging than expected due to a 1024 line-length 
limitation somewhere in the middle between the host and my inbox.  A simple 
"fmt -s -w 1024" did the trick.  I woulda had this update out sooner, were it 
not for that one!  LOL

The latest 1.1.0 version is over here:

https://github.com/UberEclectic/weewx/tree/watchdog/examples/watchdog

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/weewx-user/475020EA-09A6-44E4-B282-502B8C64F37F%40isylum.org.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Interesting wunderfixer behavior

[Leon Shaner]

;know" to keep uploading 
those 1-minute boundary records, when --epsilon is 125?

OH!  I get it!  It's because in fact there was a ~30 minute actual gap in data 
and all those in between records are more than 125 seconds either side of the 
before/after outage timestamps.   The --epsilon {secs} isn't trying to bound 
the records to 5-minute intervals, it's just comparing to the timestamps of 
whatever data is there, and there was a huge gap, so ALL the in-between 
records, even the ones on 1-minute intervals were deemed as needing to be 
re-uploaded.

BUT, then still...  How is it that WU will keep 1-minute records, when normally 
it apparently throws away everything other than those that are on the 5-minute 
boundaries?

Strange that WU eventually kept those 1-minute records, but only after 
persistent re-uploads.  In this case after 16 wunderfixer uploads, the 17th 
stuck!

Or perhaps it is just that there is a huge lag from the time the records are 
uploaded until they are posted.
Anyone know?

Maybe by way of an enhancement, wunderfixer could further filter out records 
that are not within --epsilon {seconds} of a 5-minute boundary, just to save 
extra re-uploading / strain on the WU side.   Maybe I'll try to find time to 
take that one on.  HA!  =D

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Watchdog example to warn when station is not reporting for any reason

[Leon Shaner]

Hey, Mikael,

If I am understanding what you did, yes it sounds like you've proven my logic 
works correctly.  You're a good tester! =D  Thanks!  =D

The question about RPI and usb automount w/out graphical login is off-topic, 
but to close that out, try the following, and if you get stuck, contact me 
directly (off-alias).

Firstly, if you will ever use the graphical login on your RPI, you will want to 
disable the built-in File Manager feature that auto-mounts USB drives, or it 
will likely conflict with the usbmount / system service way.  You can still use 
File Manager to access drives that get mounted by the system (independent of 
the graphical login).

Go to the RPI menu -> Accessories -> File Manager
In File Manager, go to Edit -> Preferences -> Volume Management
Disable all three "auto-mount" options and click CLOSE.


Now for the actual instructions.  =D

How to use the usbmount utility + system service to automatically mount USB 
drives on RPI:

1) If you have already mounted the USB drive manually, then umount and 
DISCONNECT it now.


2)  Install the usbmount package:

$ sudo apt-get install usbmount


3) OPTIONAL step...

If you are using FLASH storage (like a basic USB thumb-drive) you probably want 
to disable the "sync" option in /etc/usbmount/usbmount.conf -- see the comments 
there, about why...

The default settings are as follows:

MOUNTOPTIONS="sync,noexec,nodev,noatime,nodiratime"

For FLASH drives, remove the sync part as follows:

MOUNTOPTIONS="noexec,nodev,noatime,nodiratime"

In that case, keep in mind NEVER just yank the USB cable out without first 
syncing and umounting, or use the "pumount" command to handle both in one step 
without the need of root/sudo.

Always use "pumount /media/usbN" (as a regular user) and it will automatically 
do a "sync" and will then umount the drive safely.

HINT:  /media/usbN is just an example, use whatever the "mount" command shows 
for the desired device

If you need to make the above change mentioned, use the following method:

$ sudoedit /etc/usbmount/usbmount.conf

Either way, to continue with the usbmount installation...


4)  Edit the systemd-udevd service control file and change:

MountFlags=slave
to
MountFlags=shared

Make the change using the following method:

$ sudoedit /lib/systemd/system/systemd-udevd.service


5)  Reload the systemd-udevd service for the above changes to take effect:

$ sudo systemctl daemon-reload
$ sudo systemctl restart systemd-udevd


6)  Plug in your USB drive, and look for it under the following (most likely) 
path:

/media/usb0

It could be /dev/usb1, etc.   You can always use the following command after 
connecting a drive to find the path:

$ grep /media/usb /var/log/syslog

Remember, if you are using FLASH storage and you disabled "sync" in the 
usbmount.conf, then...

Always use "pumount /media/usbN" (as a regular user) and it will automatically 
do a "sync" and will then umount the drive safely.

HINT:  /media/usbN is just an example, use whatever the "mount" command shows 
for the desired device

NOTE:  The pumount also accepts the sd?N device name, such as sdb1 instead of 
/media/usbN.  Again check "mount" listing for /media devices so you know which 
one to use with pumount before disconnecting a drive.

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 3, 2019, at 5:39 PM, pliggen...@gmail.com wrote:
> 
> Hi Leon,
> 
> Ok I will stick with 720 watchdogsecs, and today I have stopped weewx service 
> several times just to see if the script starts the service again, and it did 
> everytime! 
> One time it did a reboot and that was because it directly stopped weewx after 
> a restart of weewx before it had reached the archive time so no new post was 
> made and your script did a reboot as intended, right I suppose?
> 
> Installed mailutils and now the mail works :)
> For people like me it's great to have setup instructions as describing as 
> possible so yes, put it in the readme and maybe also a in the script.
> 
> So I will now let it run as it is and I will get back here if something 
> happens that seems to be wrong, and hopefully the script will have nothing to 
> do as Weewx are running fine now!
> 
> Also, thanks for looking in to the usb question I had, and we'll see if you 
> find something that can manage that. 
> 
> Again, great to have people like you here to help out :)
> 
> BR Mikael
> 
> 
> 
> 
> 
> 
> 
>> Den fredag 3 maj 2019 kl. 00:43:42 UTC+2 skrev Leon Shaner:
>> Mikael,
>> 
>> I would say watchdogsecs=720 is more than conservative.  It is just to catch 
>> 

Re: [weewx-user] Watchdog example to warn when station is not reporting for any reason

[Leon Shaner]

Mikael,

I would say watchdogsecs=720 is more than conservative.  It is just to catch a 
case where loss of comms happens very close to the cron execution and another 
cron event fires just before your first record is posted after a reboot.  Even 
that seems unlikely because the cron interval aligns with the reboot and you 
should have a record within 2 mins after that.  So 720 will always be fine, but 
even 600 should almost always be fine.  I'd have to think a lot harder for a 
definite case where exactly 600 would cause a problem.  Rare to never, most 
likely.

As for mailx it would get pretty complicated if I had to add support for lots 
of different mail utilities, none of which agree on the arguments or how to 
pass the body.
It's kinda the point of mailx, to abstract away from whatever is happening 
underneath.

Easier for me to just to recommend:

$ sudo apt-get install mailutils

=D

See if that's enough for you and I can add it to the readme.  =D
I can put a check for it in the script, too.  =D

About the USB auto mount, I expect there is a media manager package tied to the 
graphical login on the RPI.   If you are running without the graphical login, 
there may be other options.  I will do some research and contact you off alias.

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPhone)




Regards,
Leon
--
Leon Shaner :: Dearborn, Michigan (iPhone)
https://twitter.com/AEtherZcythe
> On May 2, 2019, at 3:56 PM, pliggen...@gmail.com wrote:
> 
> Hi Leon, 
> 
> Ok, good to hear that it was a left-over and that you updated it!
> 
> I did a reboot of the system and found that it took aprox. 2min to get the 
> first "added" message from weewx, so I set the watchdogsecs to 720. Would 
> that be ok? What could happen if the time is set to early och to late? 
> 
> I had toogled them both to 1 at the same time.
> 
> Here's the part of var/log/weewx.log when it occured:
> 
> tor  2 maj 2019 15:45:06 CEST Missing records:
> tor  2 maj 2019 15:45:06 CEST 2019-05-02 12:37:05 CEST (1556793425); 29.224"; 
>  47.1F;  65%; 3.8 mph; 135 deg; 5.4 mph gust;  36.0F;  N/A  rain  
> ...published.
> tor  2 maj 2019 15:45:06 CEST 2019-05-02 12:47:05 CEST (1556794025); 29.231"; 
>  46.6F;  63%; 6.0 mph; 225 deg; 7.6 mph gust;  34.7F; 0.00" rain  
> ...published.
> tor  2 maj 2019 15:45:06 CEST 2019-05-02 13:16:53 CEST (1556795813); 29.231"; 
>  48.2F;  57%;10.7 mph;  45 deg;14.5 mph gust;  33.7F;  N/A  rain  
> ...published.
> tor  2 maj 2019 15:45:06 CEST 2019-05-02 13:36:47 CEST (1556797007); 29.240"; 
>  48.2F;  51%; 8.3 mph; 315 deg; 9.8 mph gust;  30.9F;  N/A  rain  
> ...published.
> tor  2 maj 2019 15:45:06 CEST 2019-05-02 13:46:47 CEST (1556797607); 29.241"; 
>  48.0F;  51%; 5.4 mph; 225 deg; 7.6 mph gust;  30.8F; 0.00" rain  
> ...published.
> tor  2 maj 2019 15:45:06 CEST 2019-05-02 14:17:05 CEST (1556799425); 29.251"; 
>  49.3F;  45%; 3.8 mph;   0 deg; 6.0 mph gust;  28.8F;  N/A  rain  
> ...published.
> tor  2 maj 2019 15:45:06 CEST 2019-05-02 14:37:05 CEST (1556800625); 29.249"; 
>  50.0F;  44%; 3.1 mph; 270 deg; 6.9 mph gust;  29.0F;  N/A  rain  
> ...published.
> tor  2 maj 2019 15:45:06 CEST 2019-05-02 14:47:05 CEST (1556801225); 29.248"; 
>  49.3F;  43%; 6.0 mph;   0 deg; 8.3 mph gust;  27.7F; 0.00" rain  
> ...published.
> tor  2 maj 2019 15:45:06 CEST 2019-05-02 15:17:00 CEST (1556803020); 29.244"; 
>  51.3F;  41%; 2.2 mph; 315 deg; 3.8 mph gust;  28.4F;  N/A  rain  
> ...published.
> tor  2 maj 2019 15:45:06 CEST 2019-05-02 15:40:00 CEST (1556804400); 29.247"; 
>  51.5F;  40%; 5.4 mph; 267 deg;10.7 mph gust;  28.0F; 0.00" rain  
> ...published.
> tor  2 maj 2019 16:12:01 CEST WeeWX: Watchdog warning notification sent to 
> ...@xx.xxx
> tor  2 maj 2019 16:12:01 CEST WeeWX: Watchdog Restart event timestamp: 
> (1556806321)
> tor  2 maj 2019 16:12:01 CEST WeeWX: Watchdog Stop message: Stopping weewx 
> (via systemctl): weewx.service.
> tor  2 maj 2019 16:12:01 CEST WeeWX: Watchdog Start message: Restarting weewx 
> (via systemctl): weewx.service.
> tor  2 maj 2019 16:31:01 CEST WeeWX: Watchdog warning notification sent to 
> ...@xx.xxx
> tor  2 maj 2019 16:31:01 CEST WeeWX: Watchdog Restart was NOT attempted (last 
> restart was too recent).
> tor  2 maj 2019 16:31:01 CEST WeeWX: Watchdog REBOOT event timestamp: 
> (1556807461)
> tor  2 maj 2019 16:31:01 CEST WeeWX: Watchdog REBOOT notification sent to 
> ...@xx.xxx and REBOOT initiated.
> tor  2 maj 2019 16:39:01 CEST WeeWX: Watchdog warning notification sent to 
> ...@xx.xxx
> tor  2 maj 2019 16:39:01 CEST WeeWX: Watchdog Restart event timestamp: 
> (1556807941)
> tor  2 maj 2019 16:39:01 CEST WeeWX: Watchdog Stop message: Stopping weewx 
&

Re: [weewx-user] Watchdog example to warn when station is not reporting for any reason

[Leon Shaner]

Hey, Mikael,

Nice catch.  That message about 1556807461 seconds is a left-over from the 
earlier version.  It is not only redundant, but also incorrect, due to what 
amounts to re-use of an earlier variable.  :S   I have removed it.

https://github.com/UberEclectic/weewx/tree/watchdog/examples/watchdog

Your weewx.conf archive interval of 600 seconds is not really a problem, it's 
up to you what to put there, but in that case you probably want to increase my 
watchdogsecs by a factor that approximates how long it takes your system to 
boot and for weewx to post the first record.  My RPI boots in under a minute, 
so if yours is similar, even watchdogsecs=660 should suffice.

Hmm.  Maybe longer.  Please check your syslog and see if you can approximate 
how long after a reboot it takes before this type of message to appear from 
weewx:

weewx[9809]: manager: Added record 2019-05-02 06:24:00 EDT (1556792640) to 
database

To your next questions...
It isn't 100% clear to me which way you have the toggles set.
Did you ultimately enable both remediation actions steps, before seeing the 
erroneous message?

doweewxrestart=1
dohostreboot=1

Even that erroneous message shouldn't have appeared unless you have 
dohostreboot=1 but you are mentioning mainly that you tried the 
doweewxrestart=1  first.  Not sure if you mean that after trying the restart 
option you then went on to also try the dohostreboot=1 and saw the erroneous 
message?

Just want to be sure, because the logic is the most important thing, and I did 
test that extensively.  =D

I didn't notice the erroneous message because it only goes to syslog and I was 
mainly watching the more interesting log in the case of weewx_watchdog, which 
is at /var/log/weewx.log.  Would be helpful to see that log if there are any 
other issues.

Cheers! =D
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)



Regards,
Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)
> On May 2, 2019, at 10:56 AM, pliggen...@gmail.com wrote:
> 
> Hi Leon,
> 
> Just installed the updated script now.
> 
> Seems to be running fine dispite on thing that occured, see part of log here:
> 
> May  2 16:21:08 raspberrypi weewx[2209]: cheetahgenerator: Generated 10 files 
> for report Sofaskin-FW2205-master in 6.93 seconds
> May  2 16:21:10 raspberrypi weewx[2209]: imagegenerator: Generated 9 images 
> for Sofaskin-FW2205-master in 1.03 seconds
> May  2 16:21:10 raspberrypi weewx[2209]: copygenerator: copied 8 files to 
> /var/www/html/weewx/Sofaskin-FW2205-master
> May  2 16:21:10 raspberrypi weewx[2209]: GaugeGenerator: Generated 6 images 
> for Bootstrap in 0.55 seconds
> May  2 16:21:10 raspberrypi weewx[2209]: historygenerator.pyc: Generated 9 
> tables in 0.18 seconds
> May  2 16:21:13 raspberrypi weewx[2209]: cheetahgenerator: Generated 10 files 
> for report Bootstrap in 2.86 seconds
> May  2 16:21:13 raspberrypi weewx[2209]: copygenerator: copied 3 files to 
> /var/www/html/weewx/Bootstrap
> May  2 16:21:17 raspberrypi weewx[2209]: cheetahgenerator: Generated 6 files 
> for report Bjurdammen in 4.13 seconds
> May  2 16:21:17 raspberrypi weewx[2209]: copygenerator: copied 3 files to 
> /var/www/html/weewx/Bjurdammen
> May  2 16:21:19 raspberrypi weewx[2209]: cheetahgenerator: Generated 6 files 
> for report SeasonsReport in 1.46 seconds
> May  2 16:21:19 raspberrypi weewx[2209]: copygenerator: copied 3 files to 
> /var/www/html/weewx
> May  2 16:22:28 raspberrypi vncserver-x11[438]: Connections: rejecting 
> blacklisted connection: 217.61.106.100
> May  2 16:27:01 raspberrypi CRON[2616]: (root) CMD 
> (/var/www/html/scripts/Watchdog/weewx_watchdog2)
> May  2 16:29:14 raspberrypi crontab[2636]: (root) BEGIN EDIT (root)
> May  2 16:29:24 raspberrypi crontab[2636]: (root) REPLACE (root)
> May  2 16:29:24 raspberrypi crontab[2636]: (root) END EDIT (root)
> May  2 16:29:41 raspberrypi systemd[1]: Stopping LSB: weewx weather system...
> May  2 16:29:41 raspberrypi weewx[2209]: engine: Main loop exiting. Shutting 
> engine down.
> May  2 16:29:41 raspberrypi weewx[2209]: engine: Shutting down StdReport 
> thread
> May  2 16:29:41 raspberrypi weewx[2209]: engine: Terminating weewx version 
> 3.9.1
> May  2 16:29:46 raspberrypi weewx[2683]: Stopping weewx weather system: 
> weewx..
> May  2 16:29:46 raspberrypi systemd[1]: Stopped LSB: weewx weather system.
> May  2 16:30:01 raspberrypi cron[304]: (root) RELOAD (crontabs/root)
> May  2 16:30:01 raspberrypi CRON[2727]: (pliggen) CMD (/usr/bin/php7.0 
> /var/www/html/weewx/smhi_warnings_bjurdammen.php > /dev/null 2>&1)
> May  2 16:31:01 raspberrypi CRON[2741]: (root) CMD 
> (/var/www/html/scripts/Watchdog/weewx_watchdog2)
> May  2 16:31:01 raspberrypi root[2748]: weewx_watchdog: 661 seconds have 
> passed since weewx logged any records!
> May  2 16:31:01 raspberrypi ro

Re: [weewx-user] Watchdog example to warn when station is not reporting for any reason

[Leon Shaner]

Hi, Mikael,

I've added the simple toggles at the top of the script to control remediation 
actions.
The script will always notify, but the restart weewx and reboot host actions 
are completely optional and controlled simply by the variables at the top of 
the script.

If both restart weewx and reboot host actions are enabled, these become nested 
in that first the restart weewx will be tried one time and if after 
watchdogsecs the station is still not reporting, then the host reboot will 
occur.  You can enable one or the other or both.

I also added logic to check if the host was rebooted within 2x watchdog secs to 
avoid reboot loops.   I strongly recommend not lowering the watchdogsecs below 
600 (10 minutes).

It's important to note that my approach isn't actually checking whether the 
weewx process is running, rather, the loss of communications checking is based 
on whether weewx has written any records to the database lately. 

You'll find the update over here:

https://github.com/UberEclectic/weewx/tree/watchdog/examples/watchdog

BTW, if you use my script don't run another kind of weewx restarter script, or 
they will step on each other.  (Or at least be sure to disable the weewx 
restart toggle in mine).

Regards,
Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 1, 2019, at 2:24 PM, pliggen...@gmail.com wrote:
> 
> Hi,
> 
> I now have this script running. 
> Indeed this shouldn't be necessary but I've had both issues (Weewx crash and 
> USB issue) so I think these scripts are nice to have in case something goes 
> wrong until these issue are ironed out. Of course we should provide TF with 
> logs from these events.
> 
> I still have commented out the settings that initiate a reboot, so I will 
> have to do that manually in that case, for now.
> So it's good to have those scripts sending email at errors.
> 
> Yes it would be great to have these functions in the same script, with 
> toggle's, so I would appreciate it. Lets see if there are someone else that 
> would like it. 
> 
> Well, thank you so far for all help! Learning all the time now :)
> 
> BR Mikael
> 
> 
> 
> 
> 
>> Den onsdag 1 maj 2019 kl. 16:26:33 UTC+2 skrev Leon Shaner:
>> Hi, Mikael,
>> 
>> My watchdog script could certainly be "complimentary" to Constantine's 
>> script, and with minor modifications could potentially replace it.   IIRC, 
>> isn't mine at least the third script recently mentioned on the alias to 
>> address nearly the same issues?
>> 
>> For sure these scripts shouldn't be necessary, but things happen, and bugs 
>> that can be fixed may eventually be fixed, but in the mean-time, I'd rather 
>> have a script notify / remediate / workaround issues than keep randomly 
>> discovering that my station isn't reporting only hours after it "went 
>> offline."   :-/
>> 
>> So, anyway, in my weewx_watchdog script I am merely sending an e-mail 
>> notification when the station hasn't reported, but there is an example 
>> included (commented out) that initiates a reboot, which is the only remedy I 
>> have found for when my WMR300 stops communicating on my RPI.
>> 
>> Instead of the "shutdown -r now" it would be very easy to instead use 
>> "systemctl status weewx" to check if the service is running and then 
>> "systemctl start weewx" to start it (or "systemctl restart weewx").
>> 
>> With a little more work, I could do a series of remediation steps, like 
>> first attempting to restart weewx, then IFF the station still is not 
>> reporting, do the reboot the next time.   I didn't bother, because never 
>> once ever did a simple restart of weewx fix the USB issues with my WMR300 on 
>> RPI.   Also, I didn't bother to check if weewx was running, because my weewx 
>> has never actually stopped running.  It's always the USB issue at fault in 
>> my case.  :-/
>> 
>> If there is interest I could add a weewx restart remediation and even put 
>> some simple toggle's at the top to control which remediation steps are 
>> desired, such as restart_weewx vs. reboot_host, and maybe even do the logic 
>> to first try one and then the other if both are enabled.   Could put a 
>> toggle for whether to do the wunderfixer steps, too.   I'm going to write a 
>> separate post about that in a minute.  ;-)
>> 
>> Incidentally, I had to move the script.  It's in its own branch now:
>> 
>> https://github.com/UberEclectic/weewx/tree/watchdog/examples/watchdog
>> 
>> Regards,
>> Leon
>> --
>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>> 
>>> On May 1, 2019, at 4:13 AM, plig...@gmail.com wrote:
>>>

Re: [weewx-user] WU "capacity" issues, wunderfixer constantly finding missing records, et al

[Leon Shaner]

Bingo!  Okay, sorry for the noise.
Yes, "wunderfixer --epsilon 180" was exactly what was needed.
In this case even just "wunderfixer --epsilon 125" would have been enough!

I hope this was as educational to someone as it was to me.  LOL

pi@nixie:/usr/share/weewx $ /usr/share/weewx/wunderfixer --date `date 
+\%Y-\%m-\%d` --epsilon 120 --verbose --test
Using configuration file /etc/weewx/weewx.conf.
Using database binding 'wx_binding', which is bound to database 'archive_sqlite'
Weather Underground Station:   KMIDEARB5
Date to check: 2019-05-01
Number of archive records: 757
Number of WU records:  172
Number of missing records: 10

Missing records:
2019-05-01 09:57:00 EDT (1556719020); 29.226";  50.4F;  93%; 2.9 mph; 111 deg; 
6.9 mph gust;  48.4F; 0.00" rain  ... skipped.
2019-05-01 10:17:00 EDT (1556720220); 29.223";  52.0F;  92%; 2.0 mph; 136 deg; 
3.6 mph gust;  49.9F; 0.00" rain  ... skipped.
2019-05-01 10:27:00 EDT (1556720820); 29.214";  52.9F;  92%; 2.5 mph; 115 deg; 
6.5 mph gust;  50.6F; 0.00" rain  ... skipped.
2019-05-01 10:32:00 EDT (1556721120); 29.214";  53.2F;  91%; 2.5 mph; 113 deg; 
2.5 mph gust;  50.7F; 0.00" rain  ... skipped.
2019-05-01 11:02:00 EDT (1556722920); 29.196";  56.5F;  89%; 3.1 mph; 141 deg; 
3.1 mph gust;  53.3F; 0.00" rain  ... skipped.
2019-05-01 11:47:00 EDT (1556725620); 29.184";  61.0F;  84%; 3.6 mph; 156 deg; 
3.6 mph gust;  56.1F; 0.00" rain  ... skipped.
2019-05-01 11:52:00 EDT (1556725920); 29.179";  61.3F;  84%; 2.0 mph; 164 deg; 
5.1 mph gust;  56.4F; 0.00" rain  ... skipped.
2019-05-01 12:12:00 EDT (1556727120); 29.182";  61.9F;  84%; 3.6 mph; 184 deg; 
5.1 mph gust;  56.9F; 0.00" rain  ... skipped.
2019-05-01 12:17:00 EDT (1556727420); 29.182";  62.1F;  84%; 3.6 mph; 184 deg; 
4.7 mph gust;  57.2F; 0.00" rain  ... skipped.
2019-05-01 12:32:00 EDT (1556728320); 29.184";  62.7F;  84%; 2.0 mph; 191 deg; 
4.0 mph gust;  57.8F; 0.00" rain  ... skipped.


pi@nixie:/usr/share/weewx $ /usr/share/weewx/wunderfixer --date `date 
+\%Y-\%m-\%d` --epsilon 180 --verbose --test
Using configuration file /etc/weewx/weewx.conf.
Using database binding 'wx_binding', which is bound to database 'archive_sqlite'
Weather Underground Station:   KMIDEARB5
Date to check: 2019-05-01
Number of archive records: 758
Number of WU records:  172
Number of missing records: 0


pi@nixie:/usr/share/weewx $ /usr/share/weewx/wunderfixer --date `date 
+\%Y-\%m-\%d` --epsilon 125 --verbose --test
Using configuration file /etc/weewx/weewx.conf.
Using database binding 'wx_binding', which is bound to database 'archive_sqlite'
Weather Underground Station:   KMIDEARB5
Date to check: 2019-05-01
Number of archive records: 761
Number of WU records:  173
Number of missing records: 0


Regards,
Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 1, 2019, at 12:36 PM, Leon Shaner  wrote:
> 
> Hmm.  Looking more closely at that query:
> 
> https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KMIDEARB5=5=1=2019=1
> 
> Although timestamp are often a little off, WU appears to only ever report 
> data normalized to every 5-minute bucket.
> 
> So this morning 9:50, ~9:55, and 10:00 are all they're going to show, and 
> those data points are already there in the query.
> 
> Therefore, I expect that the record locally stamped at 9:57 is likely to 
> NEVER show in the query.
> 
> Is this a case where increasing the wunderfixer --epsilon to say 180 seconds 
> would suppress the attempt to upload the record from 9:57?
> 
> Regards,
> Leon
> --
> Leon Shaner :: Dearborn, Michigan (iPad Pro)
> 
>> On May 1, 2019, at 12:10 PM, Leon Shaner  wrote:
>> 
>> Here is a bit more analysis on one record that has been found missing 
>> consistently / won't stick.   Seeking a second opinion on my analysis.  =D
>> 
>> The missing record has timestamp 1556719020 and has been found missing every 
>> 10 minutes when wunderfixer runs against "today".
>> That record is from from 9:57 a.m. EDT (my local time).
>> 
>> Further down, I show the WU query via a similar method that wunderfixer uses.
>> 
>> There are no records from between ~9:54 a.m. and 10:00 a.m. EDT.
>> So I trust wunderfixer is correct, here.  =D
>> 
>> The WU record from 10:00 a.m. has values "close" to the missing 9:57 a..m 
>> record, but the values are just far enough off that I do not suspect the 120 
>> second "epsilon" is causing the "miss" on validation (the 9:57 record is not 
>> being mis-matched against 10:00 record).
>> 
>> Are we in agreement, that wunderfixer is correct and WU is simply failing to 
>&

Re: [weewx-user] WU "capacity" issues, wunderfixer constantly finding missing records, et al

[Leon Shaner]

Hmm.  Looking more closely at that query:

https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KMIDEARB5=5=1=2019=1

Although timestamp are often a little off, WU appears to only ever report data 
normalized to every 5-minute bucket.

So this morning 9:50, ~9:55, and 10:00 are all they're going to show, and those 
data points are already there in the query.

Therefore, I expect that the record locally stamped at 9:57 is likely to NEVER 
show in the query.

Is this a case where increasing the wunderfixer --epsilon to say 180 seconds 
would suppress the attempt to upload the record from 9:57?

Regards,
Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 1, 2019, at 12:10 PM, Leon Shaner  wrote:
> 
> Here is a bit more analysis on one record that has been found missing 
> consistently / won't stick.   Seeking a second opinion on my analysis.  =D
> 
> The missing record has timestamp 1556719020 and has been found missing every 
> 10 minutes when wunderfixer runs against "today".
> That record is from from 9:57 a.m. EDT (my local time).
> 
> Further down, I show the WU query via a similar method that wunderfixer uses.
> 
> There are no records from between ~9:54 a.m. and 10:00 a.m. EDT.
> So I trust wunderfixer is correct, here.  =D
> 
> The WU record from 10:00 a.m. has values "close" to the missing 9:57 a..m 
> record, but the values are just far enough off that I do not suspect the 120 
> second "epsilon" is causing the "miss" on validation (the 9:57 record is not 
> being mis-matched against 10:00 record).
> 
> Are we in agreement, that wunderfixer is correct and WU is simply failing to 
> show the record even up multiple attempts to re-upload?
> 
> Here are the wunderfixer reported misses:
> 
> pi@nixie:/usr/share/weewx $ grep 1556719020 /var/log/weewx.log
> Wed  1 May 10:05:01 EDT 2019 2019-05-01 09:57:00 EDT (1556719020); 29.226";  
> 50.4F;  93%; 2.9 mph; 111 deg; 6.9 mph gust;  48.4F; 0.00" rain  ...published.
> Wed  1 May 10:15:02 EDT 2019 2019-05-01 09:57:00 EDT (1556719020); 29.226";  
> 50.4F;  93%; 2.9 mph; 111 deg; 6.9 mph gust;  48.4F; 0.00" rain  ...published.
> Wed  1 May 10:25:01 EDT 2019 2019-05-01 09:57:00 EDT (1556719020); 29.226";  
> 50.4F;  93%; 2.9 mph; 111 deg; 6.9 mph gust;  48.4F; 0.00" rain  ...published.
> Wed  1 May 10:35:01 EDT 2019 2019-05-01 09:57:00 EDT (1556719020); 29.226";  
> 50.4F;  93%; 2.9 mph; 111 deg; 6.9 mph gust;  48.4F; 0.00" rain  ...published.
> Wed  1 May 10:45:01 EDT 2019 2019-05-01 09:57:00 EDT (1556719020); 29.226";  
> 50.4F;  93%; 2.9 mph; 111 deg; 6.9 mph gust;  48.4F; 0.00" rain  ...published.
> Wed  1 May 10:55:02 EDT 2019 2019-05-01 09:57:00 EDT (1556719020); 29.226";  
> 50.4F;  93%; 2.9 mph; 111 deg; 6.9 mph gust;  48.4F; 0.00" rain  ...published.
> Wed  1 May 11:05:02 EDT 2019 2019-05-01 09:57:00 EDT (1556719020); 29.226";  
> 50.4F;  93%; 2.9 mph; 111 deg; 6.9 mph gust;  48.4F; 0.00" rain  ...published.
> Wed  1 May 11:15:02 EDT 2019 2019-05-01 09:57:00 EDT (1556719020); 29.226";  
> 50.4F;  93%; 2.9 mph; 111 deg; 6.9 mph gust;  48.4F; 0.00" rain  ...published.
> Wed  1 May 11:25:02 EDT 2019 2019-05-01 09:57:00 EDT (1556719020); 29.226";  
> 50.4F;  93%; 2.9 mph; 111 deg; 6.9 mph gust;  48.4F; 0.00" rain  ...published.
> Wed  1 May 11:35:01 EDT 2019 2019-05-01 09:57:00 EDT (1556719020); 29.226";  
> 50.4F;  93%; 2.9 mph; 111 deg; 6.9 mph gust;  48.4F; 0.00" rain  ...published.
> Wed  1 May 11:45:01 EDT 2019 2019-05-01 09:57:00 EDT (1556719020); 29.226";  
> 50.4F;  93%; 2.9 mph; 111 deg; 6.9 mph gust;  48.4F; 0.00" rain  ...published.
> Wed  1 May 11:55:01 EDT 2019 2019-05-01 09:57:00 EDT (1556719020); 29.226";  
> 50.4F;  93%; 2.9 mph; 111 deg; 6.9 mph gust;  48.4F; 0.00" rain  ...published.
> 
> 
> Here is what WU shows around that time...
> 
> pi@nixie:/usr/share/weewx $ curl -L 
> "http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KMIDEARB5=5=1=2019=1;
>  2>/dev/null | grep '^2019-05-01 09:'
> 2019-05-01 
> 09:00:00,48.6,46.6,29.26,ESE,112,3.1,4,93,0,,,0.91,weewx-3.9.1,2019-05-01 
> 13:00:00,
> 2019-05-01 
> 09:05:00,48.7,46.8,29.25,ESE,108,2,2.5,93,0,,,0.91,weewx-3.9.1,2019-05-01 
> 13:05:00,
> 2019-05-01 
> 09:10:00,48.7,46.8,29.25,ESE,108,2,0,93,0,,,0.91,weewx-3.9.1,2019-05-01 
> 13:10:00,
> 2019-05-01 
> 09:15:00,48.9,47,29.25,ESE,102,2,5.4,93,0,,,0.91,weewx-3.9.1,2019-05-01 
> 13:15:00,
> 2019-05-01 
> 09:20:00,48.9,47,29.25,ESE,102,2,0,93,0,,,0.91,weewx-3.9.1,2019-05-01 
> 13:20:00,
> 2019-05-01 
> 09:25:00,48.9,47,29.25,ESE,108,2.5,5.4,93,0,,,0.91,weewx-3.9.1,2019-05-01 
> 13:25:00,
> 2019-05-01 
> 09:30:00,4

Re: [weewx-user] WU "capacity" issues, wunderfixer constantly finding missing records, et al

[Leon Shaner]

,2019-05-01 
14:14:59,
2019-05-01 
10:20:00,52.2,49.9,29.21,SE,136,2,4.3,92,0,,,0.91,weewx-3.9.1,2019-05-01 
14:20:00,
2019-05-01 
10:24:58,52.5,50.3,29.21,ESE,115,2.5,3.6,92,0,,,0.91,weewx-3.9.1,2019-05-01 
14:24:58,
2019-05-01 
10:29:59,53.1,50.5,29.21,ESE,115,2.5,1.3,91,0,,,0.91,weewx-3.9.1,2019-05-01 
14:29:59,
2019-05-01 
10:35:00,53.4,50.9,29.20,ESE,113,2.5,5.8,91,0,,,0.91,weewx-3.9.1,2019-05-01 
14:35:00,
2019-05-01 
10:40:00,53.8,51.2,29.20,ESE,113,2.5,3.1,91,0,,,0.91,weewx-3.9.1,2019-05-01 
14:40:00,
2019-05-01 
10:45:00,54.3,51.4,29.20,ESE,111,2,5.1,90,0,,,0.91,weewx-3.9.1,2019-05-01 
14:45:00,
2019-05-01 
10:50:00,54.9,52.3,29.20,ESE,111,2,5.1,91,0,,,0.91,weewx-3.9.1,2019-05-01 
14:50:00,
2019-05-01 
10:55:00,55.6,53,29.20,SE,126,2.9,4.7,91,0,,,0.91,weewx-3.9.1,2019-05-01 
14:55:00,
2019-05-01 
10:59:59,56.1,52.9,29.20,SE,126,2.9,3.6,89,0,,,0.91,weewx-3.9.1,2019-05-01 
14:59:59,



Regards,
Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 1, 2019, at 11:07 AM, Leon Shaner  wrote:
> 
> Hey, WeeWX'ers,
> 
> I wonder how many folks are experiencing issues with WU timing out / not 
> responding /  failing to accept station updates from weewx?
> 
> I've resorted to running wunderfixer roughly every 10 minutes against "today" 
> and "yesterday" and it's almost always finding missing records throughout 
> "today" which after enough wunderfixer re-uploads, they do eventually "stick."
> 
> I can see from my "weewx_watchdog" logs that wunderfixer was timing out 
> almost every attempt to run against the "yesterday" date, but was usually 
> fine when running against the "today" date.
> I'm guessing that WU may have some optimizations in place to respond quickly 
> to "current" day queries, and maybe they have intentionally allocated less 
> "capacity" to historical queries.   That might explain the timeouts I was 
> seeing mainly when querying the prior day.
> 
> I found that increasing the wunderfixer timeout from 10 seconds to 20 seconds 
> was helpful in working around the WU connection issues.   There is an updated 
> wunderfixer in the that accepts a --timeout option, over here.  Thanks, Tom!  
> =D
> 
> https://github.com/weewx/weewx/blob/master/bin/wunderfixer
> 
> 
> Now, because wunderfixer is pretty much always finding missing records, I am 
> wondering of weewx itself is falling victim to the WU "capacity issues" 
> during the normal uploading of records?
> As in could the records be missing on the WU side due to similar timeouts 
> that wunderfixer is experiencing?
> 
> I have debug = 1, but there are no issues being reported, so do I need a 
> higher debug, such as debut = 3?
> 
> Could it be that the connections are going through but WU is losing the data, 
> anyway?
> 
> And/or could it be that running wunderfixer every 10 minutes isn't allowing 
> WU enough time to process the data, such that is is available by the next 
> query, 10 minutes later?
> 
> 
> Here are some logs of what I am seeing re: the trend of missing WU data.
> You can see the "overlap" in that multiple runs of wunderfixer 10 minutes 
> apart are finding and re-uploading the exact same missing records until 
> eventually they do "stick" in that they are not shown missing / not 
> re-uploaded.
> 
> In the below, you can see that a record from 1556709420 did get posted 
> properly, because it was not mentioned again 10 minutes later.   However, 
> there are three other records in bold, which did not "stick" between those 
> wunderfixer runs, 10 minutes apart.  And then 10 minutes later, two more 
> overlapping re-uploads, which didn't "stick."
> 
> 
> Wed  1 May 10:15:11 EDT 2019 Weather Underground Station:   KMIDEARB5
> Wed  1 May 10:15:11 EDT 2019 Date to check: 2019-04-30
> Wed  1 May 10:15:11 EDT 2019 Number of archive records: 1438
> Wed  1 May 10:15:11 EDT 2019 Number of WU records:  381
> Wed  1 May 10:15:11 EDT 2019 Number of missing records: 0
> Wed  1 May 10:25:01 EDT 2019 Using configuration file /etc/weewx/weewx.conf.
> Wed  1 May 10:25:01 EDT 2019 Using database binding 'wx_binding', which is 
> bound to database 'archive_sqlite'
> Wed  1 May 10:25:01 EDT 2019 Weather Underground Station:   KMIDEARB5
> Wed  1 May 10:25:01 EDT 2019 Date to check: 2019-05-01
> Wed  1 May 10:25:01 EDT 2019 Number of archive records: 624
> Wed  1 May 10:25:01 EDT 2019 Number of WU records:  144
> Wed  1 May 10:25:01 EDT 2019 Number of missing records: 5
> Wed  1 May 10:25:01 EDT 2019
> Wed  1 May 10:25:01 EDT 2019 Missing records:
> Wed  1 May 10:25:01 EDT 2019 2019-05-01 07:17:00 EDT (1556709420); 29.2

[weewx-user] WU "capacity" issues, wunderfixer constantly finding missing records, et al

[Leon Shaner]

 Weather Underground data. Exiting.
Tue 30 Apr 16:15:19 EDT 2019 Could not get Weather Underground data. Exiting.
Tue 30 Apr 16:25:14 EDT 2019 Could not get Weather Underground data. Exiting.
Tue 30 Apr 16:35:19 EDT 2019 Could not get Weather Underground data. Exiting.
Tue 30 Apr 16:45:02 EDT 2019 Could not get Weather Underground data. Exiting.
Tue 30 Apr 16:45:15 EDT 2019 Could not get Weather Underground data. Exiting.
### timeout changed from 10 seconds to 20 seconds; far fewer timeouts occurring
Tue 30 Apr 21:55:23 EDT 2019 Could not get Weather Underground data. Exiting.
Tue 30 Apr 23:15:33 EDT 2019 Could not get Weather Underground data. Exiting.
Wed  1 May 01:15:20 EDT 2019 Could not get Weather Underground data. Exiting.
Wed  1 May 01:25:14 EDT 2019 Could not get Weather Underground data. Exiting.
Wed  1 May 01:35:15 EDT 2019 Could not get Weather Underground data. Exiting.
### no timeouts seen since that time, so far, today

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Watchdog example to warn when station is not reporting for any reason

[Leon Shaner]

Hi, Mikael,

My watchdog script could certainly be "complimentary" to Constantine's script, 
and with minor modifications could potentially replace it.   IIRC, isn't mine 
at least the third script recently mentioned on the alias to address nearly the 
same issues?

For sure these scripts shouldn't be necessary, but things happen, and bugs that 
can be fixed may eventually be fixed, but in the mean-time, I'd rather have a 
script notify / remediate / workaround issues than keep randomly discovering 
that my station isn't reporting only hours after it "went offline."   :-/

So, anyway, in my weewx_watchdog script I am merely sending an e-mail 
notification when the station hasn't reported, but there is an example included 
(commented out) that initiates a reboot, which is the only remedy I have found 
for when my WMR300 stops communicating on my RPI.

Instead of the "shutdown -r now" it would be very easy to instead use 
"systemctl status weewx" to check if the service is running and then "systemctl 
start weewx" to start it (or "systemctl restart weewx").

With a little more work, I could do a series of remediation steps, like first 
attempting to restart weewx, then IFF the station still is not reporting, do 
the reboot the next time.   I didn't bother, because never once ever did a 
simple restart of weewx fix the USB issues with my WMR300 on RPI.   Also, I 
didn't bother to check if weewx was running, because my weewx has never 
actually stopped running.  It's always the USB issue at fault in my case.  :-/

If there is interest I could add a weewx restart remediation and even put some 
simple toggle's at the top to control which remediation steps are desired, such 
as restart_weewx vs. reboot_host, and maybe even do the logic to first try one 
and then the other if both are enabled.   Could put a toggle for whether to do 
the wunderfixer steps, too.   I'm going to write a separate post about that in 
a minute.  ;-)

Incidentally, I had to move the script.  It's in its own branch now:

https://github.com/UberEclectic/weewx/tree/watchdog/examples/watchdog

Regards,
Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 1, 2019, at 4:13 AM, pliggen...@gmail.com wrote:
> 
> Hi Leon,
> 
> could this be used as a complement to Constantine Samaklis script you helped 
> me with yesterday? 
> Sometimes the wh1080 freezes USB-connection to my raspberry pi but the weewx 
> service is still running so Constantines script wont have any effect in this 
> case?
> I think this issue with wh is well known.
> 
> BR Mikael
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: Small Python 'daemon' script to check if weewx is running

[Leon Shaner]

Hi, hi,

I didn't look at the mechanics of the script too closely, but before you had 
shown:

flagLogDir=/home/pliggen/scripts

And just now you showed:

logOutput=/var/log/syslog.txt

Which I glean from the error is being concatenated to:

 /home/pliggen/scripts//var/log/syslog.txt

And that can't work if you don't first "mkdir -p  
/home/pliggen/scripts//var/log/" except I doubt very much you really want such 
a convoluted path down to the checkWeewx.log.  ;-)

If you want to keep the log in your home dir, you probably want:

logOutput=checkWeewx.log

But it will be owned by root in your home dir.

I wouldn't be messing with the syslog directly, but if you want the 
checkWeewx.log to be next to syslog, you could do this:

flagLogDir=/var/log
logOutput=checkWeewx.log

So if I have gleaned correctly, that will give you a log owned by root, here:

/var/log/checkWeewx.log

The author of the script can validate the above, if I have it right or wrong.  
;-)

Regards,
Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On Apr 30, 2019, at 5:10 PM, pliggen...@gmail.com wrote:
> 
> Hi Leon, 
> 
> and thanks for the help! I was running my own non-root crontab. So I tried to 
> put the command in sudo crontab -e. 
> Stopped the service and tried it and now the service started as expected. 
> Great! And it worked with /etc/init.d/weewx start as startcommand.
> I'm not sure I understood everything you wrote, I'm quite new at this, but 
> you made it clear what to try.
> 
> One more question, it seems that when I run it via sudo crontab there were 
> some issue with the logging,
> hade to remove home/pliggen/scripts from the "logDir=" line.
> 
> Got this email message:
> 
> sh: 1: cannot create /home/pliggen/scripts//var/log/syslog.txt: Directory 
> nonexistent
> Starting program...
> Traceback (most recent call last):
>  File "/var/www/html/scripts/checkWeewx/checkWeewx.py", line 159, in 
>sendMailFailure(logDir + "/" + logOutput, config, additionalText)
>  File "/var/www/html/scripts/checkWeewx/checkWeewx.py", line 86, in 
> sendMailFailure
>f = file(filename)
> IOError: [Errno 2] No such file or directory: 
> '/home/pliggen/scripts//var/log/syslog.txt'
> 
> 
> So I removed the logDir file-path so it looks like this now:
> 
> [logging]
> logDir=
> logFile=checkWeewx.log
> logOutput=/var/log/syslog.txt
> linesToTail=200
> logToTail=/var/log/messages
> logRotationDayInterval=1
> logDaysToKeep=5
> 
> And now I get this message in the email when the cronjob runs if the service 
> is stopped:
> 
> sh: echo: I/O error
> sh: echo: I/O error
> Starting program...
> 
> It doesn't matter as long as the service now seems to start but if you see 
> something obvious I'm glad to hear!
> 
> Thanks, and sorry for my bad english. 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: Small Python 'daemon' script to check if weewx is running

[Leon Shaner]

Hi, hi,

The sudo implies you have this in your own / non-root crontab.  Is that correct?
If instead, you use "sudo crontab -e" so root is calling your script, then you 
can omit the sudo in the script.  That much alone may be the solution to your 
problem.

Meanwhile, you didn't say what OS you're running, but on newer systems, those 
RC / init.d scripts are often presented via legacy wrappers for use by 
systemctl.  In that case the systemctl mechanism for restarting the service 
might be less susceptible to foreground vs. background / disconnected usage.   
Meaning where you are using "sudo /etc/init.d/weewx start" from command-line 
and it works, but doesn't work from cron, substituting the systemctl equivalent 
may be a way out.

You could try these from command line to verify your system has the legacy RC / 
init.d wrappers, then substitute the the "sudo systemctl start weewx" of 
calling the init.d script directly...

Here again, however, putting any of these in a script that runs from the root 
crontab will avoid needing to prefix with sudo...

# Check status:
$ sudo systemctl status weewx

# Restart
$ sudo systemctl restart weewx

# Stop
$ sudo systemctl stop weewx

# Start
$ sudo systemctl start weewx

Regards,
Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On Apr 30, 2019, at 3:10 PM, pliggen...@gmail.com wrote:
> 
> I got this running as a cronjob, and I get a mail when weewx is not running 
> when the cronjob is executed. But I don't get the script to start weewx again.
> 
> This is the last part of checkweewx.config:
> 
> [weewx]
> #how many times to try and restart the weewx process
> restarts=3
> flagLogDir=/home/pliggen/scripts
> flagFile=weewxNotRunning.txt
> startCommand=sudo /etc/init.d/weewx start
> 
> If I run /etc/init.d/weewx start in terminal it starts.
> Any tips where to check to get this working?
> 
>> Den söndag 11 oktober 2015 kl. 02:45:50 UTC+2 skrev Constantine Samaklis:
>> The script file and configuration file can be anywhere you want.
>> 
>> You can launch the script by typing: python /checkWeewx.py 
>> /checkWeewx.config
>> 
>> There are no changes required in the weewx.conf. You will need to just fill 
>> the pertinent information for your setup in checkWeewx.config as it is the 
>> configuration file that drives the python script.
>> 
>>> On Saturday, October 10, 2015 at 5:22:12 PM UTC-4, Arild Halvorsen wrote:
>>> This looks exciting. How do I implement this solution in weewx? Where do I 
>>> put the files. Which information is nødvenidig in weewx.conf?
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weewx-user] Watchdog example to warn when station is not reporting for any reason

[Leon Shaner]

Hey, WeeWX'ers.

I've written a watchdog to notify via e-mail if WeeWX is not getting data from 
the station for any reason.  This is entirely based on weewx log records, so 
even if weewx has crashed out entirely, you can get notified.

This arose In my case out of a re-occurring issue on Raspberry Pi with my 
WMR300A, where weewx is generally running fine, but some combination of Raspian 
USB bug vs. flakey WMR300 firmware bug leads to loss of communication with the 
station, so records just stop coming in.

In addition to detecting lack of records / weewx crash, the watchdog script 
also runs wunderfixer periodically.  I haven't figured out why WU is often 
missing records -- I expect it is WU flakiness in that sometimes wunderfixer 
reports REST connection issues, and the WU website itself says there is an 
issue ("dark clouds pass" etc.).
I think WU is often just overwhelmed.  :-/

Since weewx_watchdog is just a script, of course you can tweak it all you need 
to remove any features that don't apply to you.  =D

Seeking feedback.

And if you need help getting the pre-requisite "mailx" working a la exim4 
(default "sendmail" on Raspian), feel free reaching out to me off-alias.

The weewx_watchdog is over here, for now:

https://github.com/UberEclectic/weewx/tree/master/examples/watchdog

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: Noob help setting up acurite bridges and indoor sensors

[Leon Shaner]

Ken,

Sorry, I failed to delete the second "driver = user.interceptor" line.  That 
was the cause of the duplicate keyword complaint.
There was the one right under [Interceptor] and the one further down after the 
comments.  I see you cleaned that up and you've got past the parsing issue.

Sorry I don't know how those inTemp mappings work.  I'm sure someone on the 
alias will know...

You do realize all those China / Swiss ones are commented out, right?
So you really only have the one inTemp set:

>   inTemp = temperature.*.*

Regards,
Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On Apr 27, 2019, at 10:33 AM, Ken Booth  wrote:
> 
>> On Saturday, 27 April 2019 15:58:57 UTC+2, Leon Shaner wrote:
>> Hey, Ken,
>> 
>> You've got two [Interceptor] sections, run together, using two different IP 
>> addresses.
>> 
>> So you should end up with the following for the [Interceptor] section, but 
>> check that the IP is correct:
>> 
>> ##
>> 
>> [Interceptor]
>> # This section is for the network traffic interceptor driver.
>> 
>> # The driver to use:
>> driver = user.interceptor
>> 
>> # Specify the hardware device to capture.  Options include:
>> #   acurite-bridge - acurite internet bridge, smarthub, or access
>> #   observer - fine offset WH2600/HP1000/HP1003, ambient WS2902
>> #   lw30x - oregon scientific LW301/LW302
>> #   lacrosse-bridge - lacrosse GW1000U/C84612 internet bridge
>> #   wu-client - any hardware that uses the weather underground protocol
>> 
>> driver = user.interceptor
>> device_type = acurite-bridge
>> mode = listen
>> iface = ens3
>> pcap_filter = src 192.168.2.10 and dst port 80
>> 
>> [[sensor_map]]
>> inTemp = temperature.8565.*  
>> inTemp = temperature.12520.* 
>> inTemp = temperature.2190.*  
>> 
>> ##
>> 
>> 
>> Regards,
>> Leon
>> --
>> Leon Shaner :: Dearborn, Michigan (iPhone)
> 
> Thanks Leon,
> 
> However, it didn't like that:
> Apr 27 15:39:52 hostname weewx[20326]: engine: Error while parsing 
> configuration file /etc/weewx/weewx.conf
> Apr 27 15:39:52 hostname weewx[20326]: Reason: 'Duplicate keyword 
> name at line 79.'
> 
> Now I've got:
> [Interceptor]
> driver = user.interceptor
> device_type = acurite-bridge
> mode = listen
> iface = ens3
> pcap_filter = src 192.168.180.0/22 and dst port 80
> [[sensor_map]]
> inTemp = temperature.*.*
> inHumidity = inHumidity.*.*
> barometer = barometer.*.*
> rxCheckPercent = rxCheckPercent.*.*
> txBatteryStatus = txBatteryStatus.*.*
> 
> #inTemp = temperature.12387.* china room 1
> #inTemp = temperature.15283.* china room 2
> #inTemp = temperature.5422.* china room 3
> #inTemp = temperature.2266.* china room 4
> #inTemp = temperature.8565.* swiss room 1
> #inTemp = temperature.12520.* swiss room 2
> #inTemp = temperature.2190.* swiss room 3
> 
> But, looking at the my webpage, I'm only getting one value for inside 
> temperature, so I'm guessing that I need to define each sensor with a 
> different name other than inTemp
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Weewx crashing with error associated with ScalarStats.addHiLo

[Leon Shaner]

Mike,

That number doesn't look too big for type int on most modern day systems.  :-/

I am on the same platform.  But my Raspian or python version could be different 
(I am on latest latest Raspian and python points to python2).

See if you can follow along with these tests and report back if the results 
differ for you.
Where I put cat, of course you need to edit those files and paste what I showed.

pi@nixie:~ $ cat /var/tmp/maxsize.py
import sys
print sys.maxsize
pi@nixie:~ $ python /var/tmp/maxsize.py
2147483647
pi@nixie:~ $ python2 /var/tmp/maxsize.py
2147483647
pi@nixie:~ $ python3 /var/tmp/maxsize.py
  File "/var/tmp/maxsize.py", line 2
print sys.maxsize
^
SyntaxError: Missing parentheses in call to 'print'
pi@nixie:~ $ cat /var/tmp/maxsize3.py
import sys
print (sys.maxsize)
pi@nixie:~ $ python3 /var/tmp/maxsize3.py
2147483647

Regards,
Leon
--
Leon Shaner :: Dearborn, Michigan (iPhone)

> On Apr 26, 2019, at 6:57 AM, Redanman  wrote:
> 
> Thank you for any possible insights into what might be causing Weewx to 
> occasionally crash.  The relevant part of the syslog is :
> 
> Apr 26 08:20:32 raspberrypi weewx[7715]: File 
> "/usr/share/weewx/weewx/manager.py", line 1216, in _addSingleRecord
> Apr 26 08:20:32 raspberrypi weewx[7715]:   
> _day_summary.addRecord(record, weight=_weight)
> Apr 26 08:20:32 raspberrypi weewx[7715]: File 
> "/usr/share/weewx/weewx/accum.py", line 256, in addRecord
> Apr 26 08:20:32 raspberrypi weewx[7715]:   func(self, record, 
> obs_type, add_hilo, weight)
> Apr 26 08:20:32 raspberrypi weewx[7715]: File 
> "/usr/share/weewx/weewx/accum.py", line 314, in add_value
> Apr 26 08:20:32 raspberrypi weewx[7715]:   
> self[obs_type].addHiLo(val, record['dateTime'])
> Apr 26 08:20:32 raspberrypi weewx[7715]: File 
> "/usr/share/weewx/weewx/accum.py", line 77, in addHiLo
> Apr 26 08:20:32 raspberrypi weewx[7715]:   raise 
> ValueError("accum: ScalarStats.addHiLo expected float or int, got %s" % val)
> Apr 26 08:20:32 raspberrypi weewx[7715]:   ValueError: accum: 
> ScalarStats.addHiLo expected float or int, got 645138
> Apr 26 08:20:32 raspberrypi weewx[7715]:   Exiting.
> 
> I have Weewx running on a Raspberry Pi Zero and my weather station is an 
> Aercus WeatherSleuth, so I am using the interceptor driver.  The installation 
> was very stable until I upgraded to rev 3.9.1 a few weeks ago.  Now, every 
> few days Weewx crashes on me.
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [weewx-user] Re: weewx stop working

[Leon Shaner]

Damian,

Still very strange that we do not see debug = 1 echoed in the logs.

At this point I am at a loss other than, perhaps you could preface the command 
with strace so we can see what is going on.  I expect it gets to a point where 
it is just more or less sleeping, doing nothing.

I say that based on your previous "ps -elf" output where it showed 00:00:00 cpu 
time and was sleeping.   It is normal for the process to sleep when there is no 
input, but it should have to be doing at least some work from time to time.

>From the ps man page:

PROCESS STATE CODES
   Here are the different values that the s, stat and state output 
specifiers (header "STAT" or "S") will display to describe the state of a 
process:

   Duninterruptible sleep (usually IO)
   Rrunning or runnable (on run queue)
   Sinterruptible sleep (waiting for an event to complete)
   Tstopped by job control signal
   tstopped by debugger during the tracing
   Wpaging (not valid since the 2.6.xx kernel)
   Xdead (should never be seen)
   Zdefunct ("zombie") process, terminated but not reaped by 
its parent

Here is an example of how to strace the process and get the debug output into a 
file:

$ sudo strace -o /tmp/weewxd_strace.txt weewxd

You can "tail -f /tmp/weewxd_strace.txt" to see what is going on.
It will likely get to a point where it is quiet for long periods.  You can stop 
the strace after you see those a couple of times.

The output could be quite large so you may want to compress it:

$ gzip /tmp/weewxd_strace.txt

Regards,
Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On Apr 25, 2019, at 10:30 AM, Damjan Hajsek  wrote:
> 
> Hi
> I have done this and it is all the same. I created some logs from manually 
> starting weewx.
> when I start weewx manually sudo weewxd /etc/weewx/weewx.conf'
> it didn't crash, I stopped it.
> here it is.
> http://zerobin.povej.net/?b06b1ad942bc51a2#34Nj1ed+8odSg7jHbjz7AU+WsqZ3yxPjZDB1EPHyVAk=
> 
> 
> 
> Dne sreda, 24. april 2019 15.48.25 UTC+2 je oseba Leon Shaner napisala:
>> 
>> Damian,
>> You have to give wee_config a command.
>> From the usage, you want this:
>> 
>> $ sudo wee_config --reconfigure --config=/etc/weewx/weewx.conf
>> 
>> (Although you can leave the --config part off, since your conf file is 
>> located in the default location).
>> 
>> But do be sure to back up the config file first...
>> 
>> Regards,
>> Leon
>> --
>> Leon Shaner :: Dearborn, Michigan (iPad Pro)
>> 
>>> On Apr 24, 2019, at 6:20 AM, Damjan Hajsek  wrote:
>>> 
>>> Hi
>>> What I did is first use some old weewx.conf file from last year and doesn't 
>>> work either.
>>> So I tried to reconfigure it but can't
>>> when I run wee_config --help I get list of help
>>> but when I do this I get
>>> wee_config /etc/weewx/weewx.conf
>>> No command specified.
>>> wee_config --config=/etc/weewx/weewx.conf
>>> No command specified.
>>> 
>>> 
>>> Dne torek, 23. april 2019 15.17.24 UTC+2 je oseba Leon Shaner napisala:
>>>> 
>>>> Damian,
>>>> 
>>>> These logs still do not state debug = 1.
>>>> It still could be that the weewx.conf isn't being parsed correctly, and 
>>>> isn't even able to read the debug = 1 setting.
>>>> 
>>>> Did you try my suggestions for how to "rebuild" the conf file?
>>>> 
>>>> Regards,
>>>> Leon
>>>> --
>>>> Leon Shaner :: Dearborn, Michigan (iPhone)
>>>> 
>>>> 
>>>>> On Apr 23, 2019, at 8:45 AM, Damjan Hajsek  wrote:
>>>>> 
>>>>> Hi
>>>>> I have followed your instruction so I did all as described
>>>>> first turn off monit
>>>>> stop weewx
>>>>> debug = 1 was already set
>>>>> than I run weewx manually
>>>>> here is a log from that
>>>>> 
>>>>> http://zerobin.povej.net/?a63b7ac09e222bcc#9Y5+B9WWtxXgUJWTKZWmzElusp9QRgEK5Pde606SvQc=
>>>>> 
>>>>> http://zerobin.povej.net/?7d382c267025fd54#0p/b4Z0y5k7U3qFc8T/2Sgl0CbbU8ZkYxdSFCRh/I4I=
>>>>> 
>>>>> 
>>>>> Dne nedelja, 21. april 2019 15.41.04 UTC+2 je oseba mwall napisala:
>>>>>> 
>>>>>> at this point in the thread it is not clear what is happening or what is 
>>>>>> the state of your system.
>>>>>> 
>&

Re: [weewx-user] Re: Interceptor not sniffing packets, router configured correctly

[Leon Shaner]

Kev,

MAYBE you need a good old fashioned hub in the middle.

A switch does jack to jack / port to port optimizations such that not every 
packet is seen on every jack.   Also, if WiFi is involved and you have more 
than one access point, and the weather station and your weewx host are not 
connected via the same access point (or one is wired and the other is WiFi and 
there is a switch in the middle), then they too will be subject to the jack to 
jack / port to port optimizations at the switch.

I say this because your weather station is sending to the server and your weewx 
interceptor is a "third-party" and your switch has no reason to think the 
conversation between the weather station and the server should be "shared" with 
your weewx host.

Regards,
Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On Apr 24, 2019, at 3:54 PM, Kev D  wrote:
> 
> Another update: To eliminate any possible interference, I spun up an Ubuntu 
> VM to continue testing. This is the current setup:
> 
> Weather station: 192.168.0.7
> Unbuntu/WeeWx/Interceptor: 192.168.0.8
> 
> I can confirm the router is sending data from the weather station to the 
> server as I when I run TCDUMP, you can see the data coming from 192.168.0.7
> 
> However, when I call the interceptor driver directly it does not capture any 
> of this data. This is both in sniff and listen modes. Does anyone know what I 
> missing?
> 
> 
> 
> 
> 
> Thanks in advance,
> 
> Kev
> 
>> On Tuesday, April 23, 2019 at 9:56:12 AM UTC-4, Kev D wrote:
>> One thing I am confused on, the Weewx logs appear to be seeing data but the 
>> interceptor is not. I assume since I routed all data coming from the weather 
>> station IP to weewx this would have to be data from the WS right? When I 
>> disable this routing it will just return "empty queue". I feel like I am 
>> missing something here.
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>>> On Tuesday, April 23, 2019 at 9:30:23 AM UTC-4, Kevin De Lucca wrote:
>>> The goal is to continue to send to wu while sniffing from weewx. I had the 
>>> router configured to the point where anything from the weather station IP 
>>> was sent to the observer and still would not sniff (WU site even had it 
>>> showing offline because of this). Maybe I should go the DNS hijack route 
>>> then just have weewx send the data to wu. Would this mean I need to change 
>>> the observer to listen mode rather than sniff?
>>> 
>>> Thanks,
>>> 
>>> Kev
>>> 
>>>> On Tuesday, April 23, 2019 at 9:24:17 AM UTC-4, mwall wrote:
>>>> 
>>>> 
>>>>> On Monday, April 22, 2019 at 3:09:22 PM UTC-4, Kev D wrote:
>>>>> I am confident that the router is configured properly, but no matter what 
>>>>> I try I simply cannot get the interceptor driver to capture any data. On 
>>>>> a side note, I am also running PIHole on this device, but I had changed 
>>>>> the admin console listening port away from port 80. Does anyone have any 
>>>>> ideas for me? 
>>>> 
>>>> do you want the observer to send directly to wu, with weewx just sniffing?
>>>> 
>>>> or do you want the observer to send directly to weewx?
>>>> 
>>>> if you want the former, then the interceptor should be in sniff mode, and 
>>>> you need to configure the router so that weewx can see the traffic from 
>>>> the observer.
>>>> 
>>>> if you want the latter, then you need to hijack dns so that queries for 
>>>> the weather underground servers resolve to the machine running weewx.
>>>> 
>>>> m 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to weewx-user+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
> 

-- 
You received this message because you are subscribed to the Google Groups 
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to weewx-user+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.