[Bug-wget] Overly permissive hostname matching

[Jeffrey Walton]

I believe wget has a security flaw in its certificate hostname matching code.

In the attached server certificate, the hostname is provided via a
Subject Alt Name (SAN). The only SAN entry is a DNS name for *.com.
Also attached is the default CA, which was used to sign the server's
certificate.

Effectively, wget accepts a single certificate for the gTLD of .COM.
That's probably bad. If a CA is compromised, then the compromised CA
could issue a super certificate and cover the entire top level
domain space.

I suspect wget also accepts certificates for .COM's friends, like
.NET, .ORG, .MIL, etc.

Its probably not limited to gTLDs. Mozilla maintains a list of
effective TLDs at https://wiki.mozilla.org/Public_Suffix_List. The
1600+ effective TLDs are probably accepted, too.

Attached are the certificates, keys, and commands to set up a test rig
with OpenSSL's s_server. The certificates are issued for example.com,
and require a modification to /etc/hosts to make things work as
(un)expected.

Jeffrey Walton
Baltimore, MD, US


hostname-verification.tar.gz
Description: GNU Zip compressed data

Re: [Bug-wget] Overly permissive hostname matching

[Jeffrey Walton]

On Wed, Mar 19, 2014 at 10:59 AM, Daniel Kahn Gillmor
d...@fifthhorseman.net wrote:
 On 03/19/2014 06:19 AM, Tim Ruehsen wrote:
 As a programmer, I want to have control. E.g. the option to load from a
 different file, or to switch off loading. Why ? e.g. for testing purposes, or
 simply imagine a swiss army knife client for experts - maybe they want to
 have control via CLI args. Or you are in a controlled environment and simply
 don't want to waste CPU cycles when downloading a single file from a trusted
 server. Just some examples.
 And than, clients like Wget would like to have access, at least for checking
 cookies.

 i understand, and i think we're probably not disagreeing -- you want the
 ability to control it; i want sane defaults so that people who don't
 touch it get sensible behavior.

 I just took a quick look but I am not sure about the API (i did not have this
 'aha' effect). But what I don't like is the dependency on PHP which is used 
 to
 'compile' the PSL before the C functions can use it. While the idea of
 compilation/preprocessing is a good one, it should at least be optional.

 pre-compilation/preprocessing is probably a reasonable performance
 optimization for heavy use; we might even want a C library to embed a
 precompiled version of the most recent known list at time of
 compilation, so that it can be used with no initialization step or when
 no file is available.
This may help with seeding thoughts for an implementation. I'm
fortunate because I work in C++.

I have a 'precooked' list with, com, mil, ...  ak.us, co.uk,
etc. One entry for each line.

There can be multiple dots. For example, sekikawa.niigata.jp.

I read the list into a vector, sort it in n*log(n), and then get
log(n) lookups for the lifetime of the program. I pay the cost of the
sort because I make frequent lookups.

When I match names with wild cards, I take a DNS name like
*.example.com. I change it to example.com, and see if its banned. Its
a simple algorithm but its effective.

I embed the list in my executable with GNU's assembler (*.S file). Its
essentially a string with both a length and a NULL terminator:

;; eff_tld_list.S
.section .rodata

;; Mozilla's Effective TLD list
.global eff_tld_list
.type   eff_tld_list, @object
.align  8
eff_tld_list:
eff_tld_list_start:
.incbin res/eff_tld_list.lst
eff_tld_list_end:
.byte 0

;; The string's size (if needed)
.global eff_tld_list_size
.type   eff_tld_list_size, @object
.align  4
eff_tld_list_size:
.inteff_tld_list_end - eff_tld_list_start

Below is the script I use to fetch Mozilla's list.

Jeff

**

#! /bin/bash

MOZILLA_LIST=MOZILLA_LIST=eff_tld_list.lst

wget http://publicsuffix.org/list/effective_tld_names.dat; -O $MOZILLA_LIST

# Remove comments
sed /^\/\//d $MOZILLA_LIST  temp-1.txt
mv temp-1.txt $MOZILLA_LIST

# Remove empty lines
sed /^$/d $MOZILLA_LIST  temp-2.txt
mv temp-2.txt $MOZILLA_LIST

# Remove lines that begin with !
sed s/^!//g $MOZILLA_LIST  temp-3.txt
mv temp-3.txt $MOZILLA_LIST

# Remove lines that begin with *.
sed s/^\*\.//g $MOZILLA_LIST  temp-4.txt
mv temp-4.txt $MOZILLA_LIST

# Pre-sort it
cat $MOZILLA_LIST | sort  temp-8.txt
mv temp-8.txt $MOZILLA_LIST

# Copy it to resources
cp $MOZILLA_LIST ../res

Re: [Bug-wget] Overly permissive hostname matching

[Jeffrey Walton]

On Wed, Mar 19, 2014 at 11:37 AM, Jeffrey Walton noloa...@gmail.com wrote:
 On Wed, Mar 19, 2014 at 11:30 AM, Daniel Stenberg dan...@haxx.se wrote:
 On Wed, 19 Mar 2014, Jeffrey Walton wrote:

 # Remove lines that begin with !


 That sounds wrong:

   A rule may begin with a ! (exclamation mark). If it does, it is labelled
   as a exception rule and then treated as if the exclamation mark is not
 present.
 Oh well. I'm too aggressive on the ban. I'd rather fail closed than open :)

 Anyway, I'll try to find the meaning of that bang. I seem to recall I
 could not find the meaning of it in the past.
After reading that again, I don't mean to sound rude. Sorry about
that. Thanks for pointing it out.

And it does bring up a good point: the data structure needs two thing:
(1) a name, and (2) a flag for white/black. White is white listed
while black is black listed.

The API needs (at minimum): (1) take a name, and (2) return
white/black/no entry. Everything else is just frills.

Jeff

Re: [Bug-wget] Overly permissive hostname matching

[Jeffrey Walton]

On Wed, Mar 19, 2014 at 11:30 AM, Daniel Stenberg dan...@haxx.se wrote:
 On Wed, 19 Mar 2014, Jeffrey Walton wrote:

 # Remove lines that begin with !


 That sounds wrong:

   A rule may begin with a ! (exclamation mark). If it does, it is labelled
   as a exception rule and then treated as if the exclamation mark is not
 present.
Oh well. I'm too aggressive on the ban. I'd rather fail closed than open :)

Anyway, I'll try to find the meaning of that bang. I seem to recall I
could not find the meaning of it in the past.

Jeff

Re: [Bug-wget] Overly permissive hostname matching

[Jeffrey Walton]

On Wed, Mar 19, 2014 at 11:45 AM, Jeffrey Walton noloa...@gmail.com wrote:
 On Wed, Mar 19, 2014 at 11:37 AM, Jeffrey Walton noloa...@gmail.com wrote:
 On Wed, Mar 19, 2014 at 11:30 AM, Daniel Stenberg dan...@haxx.se wrote:
 On Wed, 19 Mar 2014, Jeffrey Walton wrote:

 # Remove lines that begin with !


 That sounds wrong:

   A rule may begin with a ! (exclamation mark). If it does, it is labelled
   as a exception rule and then treated as if the exclamation mark is not
 present.
 Oh well. I'm too aggressive on the ban. I'd rather fail closed than open :)

 Anyway, I'll try to find the meaning of that bang. I seem to recall I
 could not find the meaning of it in the past.
 After reading that again, I don't mean to sound rude. Sorry about
 that. Thanks for pointing it out.

 And it does bring up a good point: the data structure needs two thing:
 (1) a name, and (2) a flag for white/black. White is white listed
 while black is black listed.

 The API needs (at minimum): (1) take a name, and (2) return
 white/black/no entry. Everything else is just frills.

Something else you may want in the API is a way to determine how,
exactly, a name matched if its a failure. So it might be usefult to
include a Generic Top Level Domain (gTLD), a Country Code Top Level
Domain (ccTLD), or an Effective Top Level Domain from a PSL.

I mention it because my code has some diagnostics in debug builds that
logs the info.

Jeff

Re: [Bug-wget] Read error at byte ...

[Jeffrey Walton]

On Wed, Mar 19, 2014 at 3:18 PM, Ángel González keis...@gmail.com wrote:
 On 19/03/14 12:52, Tim Ruehsen wrote:

 On Tuesday 18 March 2014 20:05:07 Daniel Kahn Gillmor wrote:

 On 03/18/2014 05:31 PM, Tim Rühsen wrote:

 $ wget -d --ca-certificate=ca-rsa-cert.pem
 --private-key=ca-rsa-key-plain.pem https://example.com:8443
 2014-03-18 21:48:04 (1.88 GB/s) - Read error at byte 5116 (The TLS
 connection was non-properly terminated.).Retrying.

 There seems to be a problem in Wget 1.15 (on Debian SID)...

 ...
 In our case, the connection shutdown by the server generates an error at the
 Wget side. (I guess this is a difference between SSL and plain TCP
 connections.)
 Wget assumes the transfer being incomplete and tries it again and again.
 Not really a bug, but also not the result a user would expect...
 Saying the server is buggy doesn't help either.


 In order to close a ssl session, the server should send a close_notify
 message,
 which seems to be what the server is not doing (in addition of not providing
 the Content-Length).
 IIS is a well-known server not doing it:
 https://bugs.php.net/bug.php?id=23220
That's actually OpenSSL's s_server (unless Tim's tests were carried
out with a different platform).

Jeff

Re: [Bug-wget] Overly permissive hostname matching

[Jeffrey Walton]

On Wed, Mar 19, 2014 at 3:03 PM, Ángel González keis...@gmail.com wrote:
 On 19/03/14 16:37, Jeffrey Walton wrote:

 ...
 Also note that by removing the *. from the beginning of the lines*, you
 are acepting more hosts than
 you should, such as a certificate for *.com.bd (represented as *.bd in the
 PSL) which should have been
 rejected.
Oh, that's bad. I'll have to check that. Thanks.

Jeff

Re: [Bug-wget] Overly permissive hostname matching

[Jeffrey Walton]

On Thu, Mar 20, 2014 at 5:52 PM, Tim Rühsen tim.rueh...@gmx.de wrote:
 Am Mittwoch, 19. März 2014, 10:59:05 schrieb Daniel Kahn Gillmor:
 I'm imagining a C library API that has a public suffix list context
 object that can do efficient lookups (however we define the lookups),
 and the library would bundle a pre-compiled context, based on the
 currently-known public suffix list.

 something like:

 ---
 struct psl_ctx;
 typedef struct psl_ctx * psl_ctx_t;
 const psl_ctx_t psl_builtin;

 psl_ctx_t psl_new_ctx_from_filename(const char* filename);
 psl_ctx_t psl_new_ctx_from_fd(int fd);
 void psl_free_ctx(psl_ctx_t ctx);

 /*
   query forms, very rough draft -- do we need both?
   need to consider memory allocation responsibilities and
   DNS internationalization/canonicalization issues
 */

 const char* psl_get_public_suffix(const psl_ctx_t, const char* domain);
 const char* psl_get_registered_domain(const psl_ctx_t, const char* d);
 ---

 I broke out the public suffix code together and created a first go (really 
 very
 quick, distcheck fails - couldn't figure out this evening).

 https://github.com/rockdaboot/libpsl

 The first step was a psl_is_tld() function.
 There is a test case for some major things (wildcards, exceptions).

 I hope there will be some interest and some contributions...
Yes, I'd be interested. Especially since Angel pointed out failures in
my use of the PSL (the close-open failures are troubling to me).

I had a sidebar with one of the OpenSSL devs because OpenSSL is
cutting in hostname matching in version 1.0.2.

He shared a link to a IETF working group on the subject:
https://www.ietf.org/mailman/listinfo/dbound.

Jeff

Re: [Bug-wget] Overly permissive hostname matching

[Jeffrey Walton]

On Thu, Mar 20, 2014 at 6:11 PM, Daniel Stenberg dan...@haxx.se wrote:
 On Thu, 20 Mar 2014, Tim Rühsen wrote:

 I broke out the public suffix code together and created a first go (really
 very quick, distcheck fails - couldn't figure out this evening).

 https://github.com/rockdaboot/libpsl


 Ok, I'll be the first to rain on the parade. Sorry but it seems fit to do
 this early.

 You do realize that with a *GPL license on the thing, you won't get adopted
 by OpenSSL, curl and possibly others?
+1

 I can't prevent you of course and the decision is yours to make, but I'd
 prefer a BSD style license as then I could really consider basing future
 enhancements of curl on this effort.
Does GNU have a permissive license? I know permissive does not meet
all of Dr. Stallman's goals, but it will allow GNU more intellectual
property in the arena.

Jeff

Re: [Bug-wget] Overly permissive hostname matching

[Jeffrey Walton]

On Thu, Mar 20, 2014 at 7:11 PM, Ángel González keis...@gmail.com wrote:
 On 20/03/14 23:16, Jeffrey Walton wrote:


 I can't prevent you of course and the decision is yours to make, but I'd
 prefer a BSD style license as then I could really consider basing future
 enhancements of curl on this effort.

 Does GNU have a permissive license? I know permissive does not meet
 all of Dr. Stallman's goals, but it will allow GNU more intellectual
 property in the arena.

 The LGPL would be an option. I don't see why is intellectual property
 related.
 The license and the copyright owner seem quite orthogonal to me.
Isn't copyright assigned to GNU or FSF?

(Sorry, I don't know. I'm not a lawyer, so my solution is usually to
avoid GPL code all together).

Jeff

Re: [Bug-wget] Overly permissive hostname matching

[Jeffrey Walton]

On Thu, Mar 20, 2014 at 8:12 PM, Ángel González keis...@gmail.com wrote:
 On 21/03/14 00:21, Daniel Stenberg wrote:

 ...
 (Sorry, I don't know. I'm not a lawyer, so my solution is usually to
 avoid GPL code all together).

 That's a solution. Although it's a sad result from usage of a license
 intended to preserve freedoms.
For what its worth, I agree with you.

I can't afford lawyers on retainer to untangle things or to defend a
suite. Hence the reason would be happy to use a permissive GPL license
and assign any IP to GNU or FSF.

I don't take the position due to philosophy or perceived moral high
ground. Its simply economic for me. Anyone who has not experienced the
economics of a technology lawsuit is in for a shock.

In the past, I spent 10,000's on a lawyer in a technology case. I'm
not going through that again, unless I'm independently wealthy. (I
think I had the moral high ground since I was suing a chronic spammer
who harassed me for nearly 15 years. I tried for years to get off the
lists, and finally had to resort to the courts).

Jeff

[Bug-wget] More hostname matching goodness

[Jeffrey Walton]

Hi Gentleman,

Here's another that looks illegal per the RFCs and CA/B Baseline.

Create a server cert with a single SAN of WWW.*.COM:

$ openssl x509 -in server-rsa-cert.pem -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 9008050290962543110 (0x7d0306034fad3206)
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=US, O=Example, LLC, CN=Example CA
Validity
Not Before: Jan  1 00:00:00 2014 GMT
Not After : Jan  1 00:00:00 2024 GMT
Subject: O=Example, LLC, CN=Example Certificate
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:b0:3b:86:b8:17:4e:0f:b7:d5:ff:9b:4a:16:32:
...
aa:7a:2e:24:75:25:20:e6:5e:5c:c2:67:56:0f:14:
dd:0b
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Alternative Name:
DNS:www.*.com
X509v3 Basic Constraints:
CA:FALSE
X509v3 Key Usage:
Digital Signature, Key Encipherment, Key Agreement
X509v3 Subject Key Identifier:
B5:AF:38:82:0C:C4:32:6E:9F:F5:F1:97:83:49:26:8D:AB:CB:3C:88
X509v3 Authority Key Identifier:

keyid:B1:77:69:71:06:C6:25:90:28:B8:BA:49:70:A1:2F:3F:0F:32:C0:3C
...

Start the server:

$ openssl s_server -accept 8443 -www -certform PEM -cert
server-rsa-cert-2.pem -keyform PEM -key server-rsa-key-plain.pem -tls1
-cipher HIGH:-EDH:-DHE

Make a client request trusting the exemplary CA:

$ wget https://www.example.com:8443 --ca-certificate ca-rsa-cert.pem
--2014-03-21 23:05:24--  https://www.example.com:8443/
Resolving www.example.com (www.example.com)... 127.0.0.1
Connecting to www.example.com (www.example.com)|127.0.0.1|:8443... connected.
HTTP request sent, awaiting response... 200 ok
Length: unspecified [text/html]
Saving to: `index.html'
...

$ wget https://www.foo.com:8443 --ca-certificate ca-rsa-cert.pem
--2014-03-21 23:06:46--  https://www.foo.com:8443/
Resolving www.foo.com (www.foo.com)... 127.0.0.1
Connecting to www.foo.com (www.foo.com)|127.0.0.1|:8443... connected.
HTTP request sent, awaiting response... 200 ok
Length: unspecified [text/html]
Saving to: `index.html.1'
...

$ wget https://www.bar.com:8443 --ca-certificate ca-rsa-cert.pem
--2014-03-21 23:06:54--  https://www.bar.com:8443/
Resolving www.bar.com (www.bar.com)... 127.0.0.1
Connecting to www.bar.com (www.bar.com)|127.0.0.1|:8443... connected.
HTTP request sent, awaiting response... 200 ok
Length: unspecified [text/html]
Saving to: `index.html.2'
...

*

I also found a server certificate with two SANs is most useful: one
*.COM, and one WWW.*.COM.

I've also got a really cool one cert to rule them all. Its got the
top levels (*.COM, *.NET, etc) and the named host variants (WWW.*.COM,
WWW.*.NET, MAIL.*.COM, MAIL.*.NET, FTP.*.COM, FTP.*.NET).

*

Attached is the cert with WWW.*.COM. All the other parameters (CA
key and cert, server key) are the same, but they were attached for
convenience.

*

This was also sent this to the GnuTLS folks, but I wanted to give the
Wget project a heads up.

Jeffrey Walton


server-rsa-cert.pem
Description: application/x509-ca-cert


server-rsa-key-plain.pem
Description: application/x509-ca-cert


ca-rsa-cert.pem
Description: application/x509-ca-cert

[Bug-wget] Wget cannot validate https://ftp.gnu.org?

[Jeffrey Walton]

I'm having trouble downloading tarballs from ftp.gnu.org using wget.

wget --ca-certificate="$HOME/.cacert/lets-encrypt-root-x3.pem"
"https://ftp.gnu.org/gnu/libunistring/libunistring-0.9.7.tar.gz; -O
libunistring-0.9.7.tar.gz
--2017-10-14 17:59:40--
https://ftp.gnu.org/gnu/libunistring/libunistring-0.9.7.tar.gz
Resolving ftp.gnu.org (ftp.gnu.org)... 208.118.235.20, 2001:4830:134:3::b
Connecting to ftp.gnu.org (ftp.gnu.org)|208.118.235.20|:443... connected.
ERROR: cannot verify ftp.gnu.org's certificate, issued by 'CN=Let\'s
Encrypt Authority X3,O=Let\'s Encrypt,C=US':
  unable to get issuer certificate
To connect to ftp.gnu.org insecurely, use `--no-check-certificate'.

The CA file lets-encrypt-root-x3.pem is provided at
https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem.txt,
and it is shown below. It has the CA bit set, basic constraints are
present, and proper key usage are present. It appears to be a valid ca
cert.

The thing that looks unusual to me is, the addition of characters in
the distinguished name. For example, it appears Wget add a slash to
escape the single apostrophe in the common name.

Does anyone have an idea what I might be doing wrong? Or if things are
working as expected, then how do I use the certificate to download the
file using Wget?

**

$ wget -V
GNU Wget 1.19.1 built on solaris2.11.

-cares +digest -gpgme +https +ipv6 +iri +large-file -metalink +nls
+ntlm +opie -psl +ssl/openssl

Wgetrc:
/usr/local/etc/wgetrc (system)
Locale:
/usr/local/share/locale
Compile:
gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/usr/local/etc/wgetrc"
-DLOCALEDIR="/usr/local/share/locale" -I. -I../lib -I../lib
-I/usr/local/include -DNDEBUG -D_REENTRANT -I/usr/include/pcre
-DNDEBUG -m64
Link:
gcc -I/usr/include/pcre -DNDEBUG -m64 -m64
-Wl,-rpath,/usr/local/lib64 -L/usr/local/lib64 -lpcre -luuid -lidn2
/usr/local/lib64/libssl.so /usr/local/lib64/libcrypto.so
-R/usr/local/lib64 -ldl -lz -lssl -lcrypto -ldl -lpthread
ftp-opie.o openssl.o http-ntlm.o ../lib/libgnu.a -lsocket -lnsl
-lnsl -lnsl -lsocket -lsocket /usr/local/lib64/libiconv.so
-R/usr/local/lib64 /usr/local/lib64/libunistring.so
/usr/local/lib64/libiconv.so -ldl -lpthread -R/usr/local/lib64
-lsocket

**

$ openssl x509 -in $HOME/cacert/lets-encrypt-root-x3.pem -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
0a:01:41:42:00:00:01:53:85:73:6a:0b:85:ec:a7:08
Signature Algorithm: sha256WithRSAEncryption
Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
Validity
Not Before: Mar 17 16:40:46 2016 GMT
Not After : Mar 17 16:40:46 2021 GMT
Subject: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:9c:d3:0c:f0:5a:e5:2e:47:b7:72:5d:37:83:b3:
68:63:30:ea:d7:35:26:19:25:e1:bd:be:35:f1:70:
92:2f:b7:b8:4b:41:05:ab:a9:9e:35:08:58:ec:b1:
2a:c4:68:87:0b:a3:e3:75:e4:e6:f3:a7:62:71:ba:
79:81:60:1f:d7:91:9a:9f:f3:d0:78:67:71:c8:69:
0e:95:91:cf:fe:e6:99:e9:60:3c:48:cc:7e:ca:4d:
77:12:24:9d:47:1b:5a:eb:b9:ec:1e:37:00:1c:9c:
ac:7b:a7:05:ea:ce:4a:eb:bd:41:e5:36:98:b9:cb:
fd:6d:3c:96:68:df:23:2a:42:90:0c:86:74:67:c8:
7f:a5:9a:b8:52:61:14:13:3f:65:e9:82:87:cb:db:
fa:0e:56:f6:86:89:f3:85:3f:97:86:af:b0:dc:1a:
ef:6b:0d:95:16:7d:c4:2b:a0:65:b2:99:04:36:75:
80:6b:ac:4a:f3:1b:90:49:78:2f:a2:96:4f:2a:20:
25:29:04:c6:74:c0:d0:31:cd:8f:31:38:95:16:ba:
a8:33:b8:43:f1:b1:1f:c3:30:7f:a2:79:31:13:3d:
2d:36:f8:e3:fc:f2:33:6a:b9:39:31:c5:af:c4:8d:
0d:1d:64:16:33:aa:fa:84:29:b6:d4:0b:c0:d8:7d:
c3:93
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints: critical
CA:TRUE, pathlen:0
X509v3 Key Usage: critical
Digital Signature, Certificate Sign, CRL Sign
Authority Information Access:
OCSP - URI:http://isrg.trustid.ocsp.identrust.com
CA Issuers - URI:http://apps.identrust.com/roots/dstrootcax3.p7c

X509v3 Authority Key Identifier:

keyid:C4:A7:B1:A4:7B:2C:71:FA:DB:E1:4B:90:75:FF:C4:15:60:85:89:10

X509v3 Certificate Policies:
Policy: 2.23.140.1.2.1
Policy: 1.3.6.1.4.1.44947.1.1.1
  CPS: http://cps.root-x1.letsencrypt.org

X509v3 CRL Distribution Points:

Full Name:
  URI:http://crl.identrust.com/DSTROOTCAX3CRL.crl

X509v3 Subject Key Identifier:
 

Re: [Bug-wget] Wget cannot validate https://ftp.gnu.org?

[Jeffrey Walton]

So it looks like the behavior below is inherited from OpenSSL:

$ openssl s_client -connect ftp.gnu.org:443 -servername ftp.gnu.org
-CAfile ~/.cacert/lets-encrypt-root-x3.pem
CONNECTED(0003)
...
Verify return code: 2 (unable to get issuer certificate)

However, OpenSSL also has -partial-chain (thanks to Dave Thompson) so
we can pin trust at the cross-certified Let's Encrypt X3 root:

$ openssl s_client -connect ftp.gnu.org:443 -servername ftp.gnu.org
-CAfile ~/.cacert/lets-encrypt-root-x3.pem -partial_chain
CONNECTED(0003)
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = ftp.gnu.org
verify return:1
---
Certificate chain
 0 s:/CN=ftp.gnu.org
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
   ...
   Verify return code: 0 (ok)
---
read:errno=0

Loog thorugh the Wget 1.18 manual
(https://www.gnu.org/software/wget/manual/wget.html) I don't see a
similar option.

So my question is, does Wget allow us to do the same? If so, then how
do we do it?

Jeff

On Sat, Oct 14, 2017 at 6:53 PM, Jeffrey Walton <noloa...@gmail.com> wrote:
> I'm having trouble downloading tarballs from ftp.gnu.org using wget.
>
> wget --ca-certificate="$HOME/.cacert/lets-encrypt-root-x3.pem"
> "https://ftp.gnu.org/gnu/libunistring/libunistring-0.9.7.tar.gz; -O
> libunistring-0.9.7.tar.gz
> --2017-10-14 17:59:40--
> https://ftp.gnu.org/gnu/libunistring/libunistring-0.9.7.tar.gz
> Resolving ftp.gnu.org (ftp.gnu.org)... 208.118.235.20, 2001:4830:134:3::b
> Connecting to ftp.gnu.org (ftp.gnu.org)|208.118.235.20|:443... connected.
> ERROR: cannot verify ftp.gnu.org's certificate, issued by 'CN=Let\'s
> Encrypt Authority X3,O=Let\'s Encrypt,C=US':
>   unable to get issuer certificate
> To connect to ftp.gnu.org insecurely, use `--no-check-certificate'.
>
> The CA file lets-encrypt-root-x3.pem is provided at
> https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem.txt,
> and it is shown below. It has the CA bit set, basic constraints are
> present, and proper key usage are present. It appears to be a valid ca
> cert.
>
> The thing that looks unusual to me is, the addition of characters in
> the distinguished name. For example, it appears Wget add a slash to
> escape the single apostrophe in the common name.
>
> Does anyone have an idea what I might be doing wrong? Or if things are
> working as expected, then how do I use the certificate to download the
> file using Wget?
>
> **
>
> $ wget -V
> GNU Wget 1.19.1 built on solaris2.11.
>
> -cares +digest -gpgme +https +ipv6 +iri +large-file -metalink +nls
> +ntlm +opie -psl +ssl/openssl
>
> Wgetrc:
> /usr/local/etc/wgetrc (system)
> Locale:
> /usr/local/share/locale
> Compile:
> gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/usr/local/etc/wgetrc"
> -DLOCALEDIR="/usr/local/share/locale" -I. -I../lib -I../lib
> -I/usr/local/include -DNDEBUG -D_REENTRANT -I/usr/include/pcre
> -DNDEBUG -m64
> Link:
> gcc -I/usr/include/pcre -DNDEBUG -m64 -m64
> -Wl,-rpath,/usr/local/lib64 -L/usr/local/lib64 -lpcre -luuid -lidn2
> /usr/local/lib64/libssl.so /usr/local/lib64/libcrypto.so
> -R/usr/local/lib64 -ldl -lz -lssl -lcrypto -ldl -lpthread
> ftp-opie.o openssl.o http-ntlm.o ../lib/libgnu.a -lsocket -lnsl
> -lnsl -lnsl -lsocket -lsocket /usr/local/lib64/libiconv.so
> -R/usr/local/lib64 /usr/local/lib64/libunistring.so
> /usr/local/lib64/libiconv.so -ldl -lpthread -R/usr/local/lib64
> -lsocket
>
> **
>
> $ openssl x509 -in $HOME/cacert/lets-encrypt-root-x3.pem -text -noout
> Certificate:
> Data:
> Version: 3 (0x2)
> Serial Number:
> 0a:01:41:42:00:00:01:53:85:73:6a:0b:85:ec:a7:08
> Signature Algorithm: sha256WithRSAEncryption
> Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
> Validity
> Not Before: Mar 17 16:40:46 2016 GMT
> Not After : Mar 17 16:40:46 2021 GMT
> Subject: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3
> Subject Public Key Info:
> Public Key Algorithm: rsaEncryption
> Public-Key: (2048 bit)
> Modulus:
> 00:9c:d3:0c:f0:5a:e5:2e:47:b7:72:5d:37:83:b3:
> 68:63:30:ea:d7:35:26:19:25:e1:bd:be:35:f1:70:
> 92:2f:b7:b8:4b:41:05:ab:a9:9e:35:08:58:ec:b1:
> 2a:c4:68:87:0b:a3:e3:75:e4:e6:f3:a7:62:71:ba:
> 79:81:60:1f:d7:91:9a:9f:f3:d0:78:67:71:c8:69:
> 0e:95:91:cf:fe:e6:99:e9:60:3c:48:cc:7e:ca:4d:
> 77:12

Re: [Bug-wget] Wget cannot validate https://ftp.gnu.org?

[Jeffrey Walton]

On Mon, Oct 16, 2017 at 4:52 AM, Tim Rühsen  wrote:
> ...
>
> Caveat: wget has been build with GnuTLS (3.5.15). The OpenSSL (1.1.0f)
> code seems not to support --ca-directory !? It succeeds with both the
> above tests. While we only actively support GnuTLS, we accept OpenSSL
> code patches (if you like to provide one).

I believe this is most of the patch you need. You or Simon will still
need to touch it up. For example, I did not know how to handle a
failure in OpenSSL from X509_VERIFY_PARAM_new().

$ git diff > openssl.c.diff
$ cat openssl.c.diff
diff --git a/src/openssl.c b/src/openssl.c
index 0404d2d0..62d8b084 100644
--- a/src/openssl.c
+++ b/src/openssl.c
@@ -269,11 +269,36 @@ ssl_init (void)
* Since we want a good protection, we also use HIGH (that excludes
MD4 ciphers and some more)
*/
   if (opt.secure_protocol == secure_protocol_pfs)
-SSL_CTX_set_cipher_list (ssl_ctx,
"HIGH:MEDIUM:!RC4:!SRP:!PSK:!RSA:!aNULL@STRENGTH");
+SSL_CTX_set_cipher_list (ssl_ctx, "HIGH:!aNULL:!kRSA:!RC4:!MD5:!SRP:!PSK");

   SSL_CTX_set_default_verify_paths (ssl_ctx);
   SSL_CTX_load_verify_locations (ssl_ctx, opt.ca_cert, opt.ca_directory);

+  if (opt.ca_cert)
+{
+  /* Set X509_V_FLAG_PARTIAL_CHAIN to allow the client to anchor trust in
+   * a non-self-signed certificate. This defies RFC 4158 (Path Building)
+   * which defines a trust anchor in terms of a self-signed certificate.
+   * However, it substantially reduces attack surface by prunning the tree
+   * of unneeded trust points. For example, the cross-certified
+   * Let's Encrypt X3 CA, which protects gnu.org, appears as an
+   * intermediate CA to clients, can be used as a trust anchor without
+   * the entire IdentTrust PKI.
+   */
+  X509_VERIFY_PARAM *param = X509_VERIFY_PARAM_new();
+  if (!param)
+   {
+  /* TODO: How does Wget handle a malloc failure? */
+   }
+ else
+   {
+ /* Return value is the old options */
+ (void)X509_VERIFY_PARAM_set_flags(param,
X509_V_FLAG_PARTIAL_CHAIN);
+ SSL_CTX_set1_param(ssl_ctx, param);
+ X509_VERIFY_PARAM_free(param);
+   }
+}
+
   if (opt.crl_file)
 {
   X509_STORE *store = SSL_CTX_get_cert_store (ssl_ctx);


Jeff
diff --git a/src/openssl.c b/src/openssl.c
index 0404d2d0..62d8b084 100644
--- a/src/openssl.c
+++ b/src/openssl.c
@@ -269,11 +269,36 @@ ssl_init (void)
* Since we want a good protection, we also use HIGH (that excludes MD4 
ciphers and some more)
*/
   if (opt.secure_protocol == secure_protocol_pfs)
-SSL_CTX_set_cipher_list (ssl_ctx, 
"HIGH:MEDIUM:!RC4:!SRP:!PSK:!RSA:!aNULL@STRENGTH");
+SSL_CTX_set_cipher_list (ssl_ctx, "HIGH:!aNULL:!kRSA:!RC4:!MD5:!SRP:!PSK");
 
   SSL_CTX_set_default_verify_paths (ssl_ctx);
   SSL_CTX_load_verify_locations (ssl_ctx, opt.ca_cert, opt.ca_directory);
 
+  if (opt.ca_cert)
+{
+  /* Set X509_V_FLAG_PARTIAL_CHAIN to allow the client to anchor trust in
+   * a non-self-signed certificate. This defies RFC 4158 (Path Building)
+   * which defines a trust anchor in terms of a self-signed certificate.
+   * However, it substantially reduces attack surface by prunning the tree
+   * of unneeded trust points. For example, the cross-certified
+   * Let's Encrypt X3 CA, which protects gnu.org, appears as an
+   * intermediate CA to clients, can be used as a trust anchor without
+   * the entire IdentTrust PKI.
+   */ 
+  X509_VERIFY_PARAM *param = X509_VERIFY_PARAM_new();
+  if (!param)
+   {
+  /* TODO: How does Wget handle a malloc failure? */
+   }
+ else
+   {
+ /* Return value is the old options */
+ (void)X509_VERIFY_PARAM_set_flags(param, 
X509_V_FLAG_PARTIAL_CHAIN);
+ SSL_CTX_set1_param(ssl_ctx, param);
+ X509_VERIFY_PARAM_free(param);
+   }
+}
+
   if (opt.crl_file)
 {
   X509_STORE *store = SSL_CTX_get_cert_store (ssl_ctx);

Re: [Bug-wget] cipher_list string when using OpenSSL

[Jeffrey Walton]

On Thu, Oct 19, 2017 at 5:35 AM, Tim Rühsen  wrote:
> Hi Jeffrey,
>
> thanks for heads up !
>
> Does OpenSSL meanwhile have a PFS for their cipher list ?
>
> Currently it looks like that each and every client has to amend their
> cipher list from time to time. Instead, this should be done in the
> library. So that new versions automatically make the client code more
> secure. GnuTLS does it.
>
>
> That's one reason why we (wget developers) already discussed about
> dropping OpenSSL support completely. The background is that the OpenSSL
> code in Wget has no maintainer. We take (small) patches every now and
> then but there is no expert here for review or active progress.
>
> Having your random seeding issue in mind, there seems to be even more
> reasons to drop that OpenSSL code.
>
> If there is someone here who wants to maintain the OpenSSL code of Wget
> - you are very welcome (Let us know) ! In the meantime I'll ask the
> other maintainers about their opinion.

Ack, just decide what you want to do. I should not influence the
project's processes or bikeshed.

I favor OpenSSL because I've worked with it for so long, and I have
automated build scripts for it. On the other hand, I can switch to
GnuTLS if needed. I have not done so because its expedient to use
OpenSSL (another way of saying I'm lazy at times).

Jeff

[Bug-wget] OpenSSL, random seeding and init_prng

[Jeffrey Walton]

Hi Everyone,

Looking at https://git.savannah.gnu.org/cgit/wget.git/tree/src/openssl.c
and init_prng, there could be four small issues, but I don't think
they pose significant risk. I think its mostly splitting hairs.

First, checking RAND_status could setup a race condition for most
software. Wget is not general purpose, so I'm not sure if it applies
here. That is, will another Wget thread drain entropy just after this
thread initializes the prng:

if (RAND_status ())
  return;

Second, there are several of these sprinkled throughout init_prng.

if (RAND_status ())
  return;

If random_file is plentiful but fixed (maybe burned into ROM), then
entropy does not really change in the way one would expect on a
desktop. The early-out effectively means the same blob might be used
over and over again:

if (random_file && *random_file)
  RAND_load_file (random_file, 16384);

if (RAND_status ())
  return;

Third, if a users wants to add EGD entropy via --egd-file=file, then
the early out from item (2) means the user's choice may not be honored
in some instances. That is, the same blob is used over and over again
and the early out short circuits the desired EGD code:

if (RAND_status ())
  return;

  #ifdef HAVE_RAND_EGD
  /* Get random data from EGD if opt.egd_file was used.  */
if (opt.egd_file && *opt.egd_file)
  RAND_egd (opt.egd_file);
  #endif

Fourth, before an OpenSSL routine is called that uses a random
numbers, the prng could be re-seeded/stirred to help avoid some
entropy based attacks, like VM rollbacks. Also see the following
references. Even NIST is recommending a stir in their approved
generators for prediction resistance and back tracking resistance:

* When Virtual is Harder than Real: Security Challenges in Virtual
Machine Based Computing Environments,
https://www.usenix.org/legacy/event/hotos05/final_papers/full_papers/garfinkel/garfinkel.pdf

* When Good Randomness Goes Bad: Virtual Machine Reset Vulnerabilities
and Hedging Deployed Cryptography,
http://pages.cs.wisc.edu/~rist/papers/sslhedge.pdf

* NIST SP 800-90A,
http://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-90Ar1.pdf

Jeff

Re: [Bug-wget] Wget cannot validate https://ftp.gnu.org?

[Jeffrey Walton]

On Wed, Oct 18, 2017 at 7:58 PM, Jeffrey Walton <noloa...@gmail.com> wrote:
> On Mon, Oct 16, 2017 at 4:52 AM, Tim Rühsen <tim.rueh...@gmx.de> wrote:
>> ...
>>
>> Caveat: wget has been build with GnuTLS (3.5.15). The OpenSSL (1.1.0f)
>> code seems not to support --ca-directory !? It succeeds with both the
>> above tests. While we only actively support GnuTLS, we accept OpenSSL
>> code patches (if you like to provide one).
>
> I believe this is most of the patch you need. You or Simon will still
> need to touch it up. For example, I did not know how to handle a
> failure in OpenSSL from X509_VERIFY_PARAM_new().

Tim, hold-off on this at the moment. I want to talk to Matt Caswell
about it to ensure it meets requirements.

Mat is one of the OpenSSL devs. He is very helpful with his comments
and suggestions. In fact, he's the one who alerted me to
X509_V_FLAG_PARTIAL_CHAIN.

What I am wondering is... On a machine where both CAfile and CApath
are working as expected, are we only using CAfile; or are we using
CAPath and getting the CA Zoo if the directory is populated?

CAfile and CApath are names of s_client options that equate to the
parameters in calls like SSL_CTX_load_verify_locations. -CAfile is
actually equivalent to Wget's --ca-certificate. On some distros CApath
is broken so it appears OpenSSL does not trust anything by default.
But this is actually a bug in the utilities.

Jeff

Jeff

Re: [Bug-wget] cipher_list string when using OpenSSL

[Jeffrey Walton]

I think this is the change that would facilitate the cipher_list
discussed below:

--- a/src/openssl.c
+++ b/src/openssl.c
@@ -267,13 +267,42 @@ ssl_init (void)

   /* OpenSSL ciphers: https://www.openssl.org/docs/apps/ciphers.html
* Since we want a good protection, we also use HIGH (that excludes
MD4 ciphers and some more)
+   * !kRSA removes RSA key exchange (i.e., key transport), but allows
RSA digital signatures.
+   * With RSA key exchange removed, only key agreement schemes remain
(i.e., PFS schemes).
*/
   if (opt.secure_protocol == secure_protocol_pfs)
-SSL_CTX_set_cipher_list (ssl_ctx,
"HIGH:MEDIUM:!RC4:!SRP:!PSK:!RSA:!aNULL@STRENGTH");
+SSL_CTX_set_cipher_list (ssl_ctx, "HIGH:!aNULL:!kRSA:!RC4:!MD5:!SRP:!PSK");
+  else
+SSL_CTX_set_cipher_list (ssl_ctx, "HIGH:!aNULL:!RC4:!MD5:!SRP:!PSK");

Jeff


On Wed, Oct 18, 2017 at 6:57 PM, Jeffrey Walton <noloa...@gmail.com> wrote:
> Hi Everyone,
>
> I believe this has some room for improvement (from src/openssl.c):
>
> "HIGH:MEDIUM:!RC4:!SRP:!PSK:!RSA:!aNULL@STRENGTH"
>
> I think it would be a good idea to provide a `--cipher_list` option to
> allow the user to specify it. It might also be prudent to allow the
> string to be specified in `.wgetrc`.
>
> Regarding the default string, its 2017, and this is probably closer to
> what should be used by default:
>
> "HIGH:!aNULL:!RC4:!MD5:!SRP:!PSK:!kRSA"
>
> The "!kRSA" means RSA cannot be used for key exchange (i.e., RSA key
> transport), but can be used for digital signatures. MD5 is probably
> another algorithm that should be sunsetted at this point in time
> (though I am not aware of a HMAC/MD5 attack that can be carried out in
> TCP's 2MSL re-transmit time frame).
>
> I use the same cipher_list on the servers under my control. I've never
> received a complaint from them. They cipher_list also helps get one of
> those A+ reports from the various SSL scanners.
>
> Jeff

[Bug-wget] cipher_list string when using OpenSSL

[Jeffrey Walton]

Hi Everyone,

I believe this has some room for improvement (from src/openssl.c):

"HIGH:MEDIUM:!RC4:!SRP:!PSK:!RSA:!aNULL@STRENGTH"

I think it would be a good idea to provide a `--cipher_list` option to
allow the user to specify it. It might also be prudent to allow the
string to be specified in `.wgetrc`.

Regarding the default string, its 2017, and this is probably closer to
what should be used by default:

"HIGH:!aNULL:!RC4:!MD5:!SRP:!PSK:!kRSA"

The "!kRSA" means RSA cannot be used for key exchange (i.e., RSA key
transport), but can be used for digital signatures. MD5 is probably
another algorithm that should be sunsetted at this point in time
(though I am not aware of a HMAC/MD5 attack that can be carried out in
TCP's 2MSL re-transmit time frame).

I use the same cipher_list on the servers under my control. I've never
received a complaint from them. They cipher_list also helps get one of
those A+ reports from the various SSL scanners.

Jeff

Re: [Bug-wget] Wget cannot validate https://ftp.gnu.org?

[Jeffrey Walton]

On Mon, Oct 16, 2017 at 4:52 AM, Tim Rühsen  wrote:
> Hi Jeffrey,
> ...
> Caveat: wget has been build with GnuTLS (3.5.15). The OpenSSL (1.1.0f)
> code seems not to support --ca-directory !? It succeeds with both the
> above tests. While we only actively support GnuTLS, we accept OpenSSL
> code patches (if you like to provide one).

Thanks Tim.

A little help would be appreciated. I found where --ca-certifcate is
called out in src/main.c, but I'm having trouble following the
connection between the option and a variable. I did not see an opt.c
or similar.

What is the variable of interest here? Or, maybe the source file of interest?

I believe the change is about as simple as
https://github.com/openssl/openssl/blob/master/apps/opt.c#L593 , but
I'd like to get it tested before I ask you or Simon to look at it.

Jeff

Re: [Bug-wget] Failing to compile without depricated features from openssl

[Jeffrey Walton]

On Tue, Nov 21, 2017 at 1:07 AM, Matthew Thode
 wrote:
> Hi,
>
> It looks like openssl-1.1 support needs to be tweaked a bit to support
> building when openssl does not support depricated features.
>
> We are tracking the bug here, https://bugs.gentoo.org/604490 and have an
> attached patch here https://bugs.gentoo.org/attachment.cgi?id=498698

Forgive me if these don't really matter.

@@ -229,16 +238,31 @@
   ssl_options |= SSL_OP_NO_SSLv2 | SSL_OP_NO_SSLv3;

In my $day job$ we usually we want to see SSL_OP_NO_COMP too due to
BREACH and CRIME attacks. Does it matter here?

Regarding:

+#if !defined(LIBRESSL_VERSION_NUMBER) && (OPENSSL_VERSION_NUMBER >=
0x1010L)
+  meth = TLS_client_method();
+  ssl_proto_version = TLS1_VERSION;
+#else
   meth = TLSv1_client_method ();
+#endif

I think OpenSSL reworked them for 1.1.0, but I don't know what the new
pattern is.

Jeff

[Bug-wget] Self test results on OS X 10.9

[Jeffrey Walton]

Hi Everyone,

I'm building 1.19.2 from sources. Does anyone have an idea about the
"Use of uninitialized value $addr in concatenation ..."?

Thanks,

Jeff

*

hilbert:$ cat wget-1.19.2/tests/test-suite.log

===
   wget 1.19.2: tests/test-suite.log
===

# TOTAL: 93
# PASS:  69
# SKIP:  23
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

SKIP: Test-ftp-iri
==

Setting --no-config (noconfig) to 1
Skipped test: Wget misses feature 'iri'
Features available from 'wget --version' output:
gpgme=0
https=1
nls=0
opie=1
cares=0
digest=1
metalink=0
psl=0
ipv6=1
ntlm=1
iri=0
ssl/openssl=1
large-file=1
SKIP Test-ftp-iri.px (exit status: 77)

SKIP: Test-ftp-iri-fallback
===

Setting --no-config (noconfig) to 1
Skipped test: Wget misses feature 'iri'
Features available from 'wget --version' output:
ipv6=1
opie=1
nls=0
ssl/openssl=1
cares=0
gpgme=0
digest=1
https=1
ntlm=1
iri=0
large-file=1
psl=0
metalink=0
SKIP Test-ftp-iri-fallback.px (exit status: 77)

SKIP: Test-ftp-iri-recursive


Setting --no-config (noconfig) to 1
Skipped test: Wget misses feature 'iri'
Features available from 'wget --version' output:
ssl/openssl=1
https=1
ntlm=1
ipv6=1
cares=0
digest=1
gpgme=0
opie=1
iri=0
large-file=1
metalink=0
psl=0
nls=0
SKIP Test-ftp-iri-recursive.px (exit status: 77)

SKIP: Test-ftp-iri-disabled
===

Setting --no-config (noconfig) to 1
Skipped test: Wget misses feature 'iri'
Features available from 'wget --version' output:
psl=0
metalink=0
ntlm=1
opie=1
nls=0
ssl/openssl=1
cares=0
digest=1
large-file=1
gpgme=0
iri=0
ipv6=1
https=1
SKIP Test-ftp-iri-disabled.px (exit status: 77)

SKIP: Test-idn-headers
==

Setting --no-config (noconfig) to 1
Skipped test: Wget misses feature 'iri'
Features available from 'wget --version' output:
ntlm=1
nls=0
opie=1
metalink=0
digest=1
iri=0
https=1
cares=0
large-file=1
psl=0
gpgme=0
ssl/openssl=1
ipv6=1
SKIP Test-idn-headers.px (exit status: 77)

SKIP: Test-idn-meta
===

Setting --no-config (noconfig) to 1
Skipped test: Wget misses feature 'iri'
Features available from 'wget --version' output:
https=1
gpgme=0
cares=0
ipv6=1
opie=1
ntlm=1
nls=0
iri=0
ssl/openssl=1
digest=1
large-file=1
psl=0
metalink=0
SKIP Test-idn-meta.px (exit status: 77)

SKIP: Test-idn-cmd
==

Setting --no-config (noconfig) to 1
Skipped test: Wget misses feature 'iri'
Features available from 'wget --version' output:
digest=1
ntlm=1
opie=1
ssl/openssl=1
metalink=0
cares=0
nls=0
gpgme=0
large-file=1
psl=0
iri=0
https=1
ipv6=1
SKIP Test-idn-cmd.px (exit status: 77)

SKIP: Test-idn-cmd-utf8
===

Setting --no-config (noconfig) to 1
Skipped test: Wget misses feature 'iri'
Features available from 'wget --version' output:
https=1
psl=0
large-file=1
ipv6=1
ssl/openssl=1
cares=0
nls=0
digest=1
gpgme=0
ntlm=1
metalink=0
opie=1
iri=0
SKIP Test-idn-cmd-utf8.px (exit status: 77)

SKIP: Test-idn-robots
=

Setting --no-config (noconfig) to 1
Skipped test: Wget misses feature 'iri'
Features available from 'wget --version' output:
gpgme=0
ipv6=1
https=1
ntlm=1
metalink=0
cares=0
opie=1
iri=0
nls=0
psl=0
ssl/openssl=1
digest=1
large-file=1
SKIP Test-idn-robots.px (exit status: 77)

SKIP: Test-idn-robots-utf8
==

Setting --no-config (noconfig) to 1
Skipped test: Wget misses feature 'iri'
Features available from 'wget --version' output:
metalink=0
psl=0
cares=0
ntlm=1
gpgme=0
iri=0
nls=0
large-file=1
https=1
ipv6=1
ssl/openssl=1
opie=1
digest=1
SKIP Test-idn-robots-utf8.px (exit status: 77)

SKIP: Test-iri
==

Setting --no-config (noconfig) to 1
Skipped test: Wget misses feature 'iri'
Features available from 'wget --version' output:
psl=0
metalink=0
cares=0
nls=0
ntlm=1
digest=1
gpgme=0
ssl/openssl=1
large-file=1
iri=0
https=1
opie=1
ipv6=1
SKIP Test-iri.px (exit status: 77)

SKIP: Test-iri-percent
==

Setting --no-config (noconfig) to 1
Skipped test: Wget misses feature 'iri'
Features available from 'wget --version' output:
nls=0
psl=0
ipv6=1
digest=1
ssl/openssl=1
large-file=1
ntlm=1
gpgme=0
metalink=0
iri=0
https=1
opie=1
cares=0
SKIP Test-iri-percent.px (exit status: 77)

FAIL: Test-iri-disabled

Re: [Bug-wget] gcc: error: unrecognized command line option '-R'

[Jeffrey Walton]

On Sat, May 19, 2018 at 5:21 PM, Tim Rühsen <tim.rueh...@gmx.de> wrote:
> On 19.05.2018 20:53, Jeffrey Walton wrote:
>> On Sat, May 19, 2018 at 12:27 PM, Tim Rühsen <tim.rueh...@gmx.de> wrote:
>> ...
>> make[4]: Entering directory '/home/Build-Scripts/wget-1.19.5/src'
>> make[4]: Leaving directory '/home/jwalton/Build-Scripts/wget-1.19.5/src'
>> gcc   -Wno-unused-parameter -Wno-pedantic -I/usr/local/include
>> -I/usr/local/include  -I/usr/local/include  -DNDEBUG -g2 -O2 -m64
>> -march=native -fPIC  -L/usr/local/lib64 -m64 -Wl,-R,/usr/local/lib64
>> -Wl,--enable-new-dtags -o wget_css_fuzzer wget_css_fuzzer.o main.o
>> ../src/libunittest.a ../lib/libgnu.a   -L/usr/local/lib64 -liconv
>> -R/usr/local/lib64  -lpthread   -ldl  -L/usr/local/lib64 -lpcre
>> -lidn2 /usr/local/lib64/libssl.so /usr/local/lib64/libcrypto.so
>> -Wl,-rpath -Wl,/usr/local/lib64 -ldl -L/usr/local/lib64 -lz
>> -L/usr/local/lib64 -lpsl  -ldl -lpthread
>> gcc: error: unrecognized command line option '-R'
>> make[3]: *** [Makefile:1757: wget_css_fuzzer] Error 1
>>
>> Some of the command above looks a little unusual. For example:
>>
>> -Wl,-rpath -Wl,/usr/local/lib64
>>
>> Here are the variables I configure with:
>>
>> PKGCONFIG: /usr/local/lib64/pkgconfig
>>  CPPFLAGS: -I/usr/local/include -DNDEBUG
>>CFLAGS: -g2 -O2 -m64 -march=native -fPIC
>>  CXXFLAGS: -g2 -O2 -m64 -march=native -fPIC
>>   LDFLAGS: -L/usr/local/lib64 -m64 -Wl,-R,/usr/local/lib64
>> -Wl,--enable-new-dtags
>>LDLIBS: -ldl -lpthread
>>
>
> Could you please change $(LTLIBICONV) to $(LIBICONV) in
> fuzz/Makefile.am, then autoreconf -fi and ./configure, ...
>
> I think it's that wrong variable used there.

Yes, that was it. Thanks.

Jeff

[Bug-wget] gcc: error: unrecognized command line option '-R'

[Jeffrey Walton]

Hi Everyone,

This looks like a new issue with Wget 1.19.5:

make
...
  CC   libunittest_a-version.o
  AR   libunittest.a
gmake[3]: Leaving directory '/home/Build-Scripts/wget-1.19.5/src'
gmake[2]: Leaving directory '/home/Build-Scripts/wget-1.19.5/src'
Making check in doc
...

gmake[4]: Leaving directory '/home/Build-Scripts/wget-1.19.5/src'
  CCLD wget_css_fuzzer
gcc: error: unrecognized command line option '-R'
gmake[3]: *** [Makefile:1757: wget_css_fuzzer] Error 1

The config.log is attached. It looks like there is a bad interaction
with -Wl,-R,/usr/local/lib64, which is in my LDFLAGS. The complete
LDFLAGS used is:

-L/usr/local/lib64 -m64 -Wl,-R,/usr/local/lib64 -Wl,--enable-new-dtags

Jeff


config.log.tar.gz
Description: application/gzip

Re: [Bug-wget] gcc: error: unrecognized command line option '-R'

[Jeffrey Walton]

On Sat, May 19, 2018 at 12:27 PM, Tim Rühsen  wrote:
> Hi Jeff,
>
> could you 'cd fuzz', then 'make -j1 V=1' and send us the ouput ?
>
> It should include the full gcc command line.
>
> Please attach your config.log.
>

Thanks Tim.

$ cd wget-1.19.5
$ make check V=1
...

make[4]: Entering directory '/home/Build-Scripts/wget-1.19.5/src'
make[4]: Leaving directory '/home/jwalton/Build-Scripts/wget-1.19.5/src'
gcc   -Wno-unused-parameter -Wno-pedantic -I/usr/local/include
-I/usr/local/include  -I/usr/local/include  -DNDEBUG -g2 -O2 -m64
-march=native -fPIC  -L/usr/local/lib64 -m64 -Wl,-R,/usr/local/lib64
-Wl,--enable-new-dtags -o wget_css_fuzzer wget_css_fuzzer.o main.o
../src/libunittest.a ../lib/libgnu.a   -L/usr/local/lib64 -liconv
-R/usr/local/lib64  -lpthread   -ldl  -L/usr/local/lib64 -lpcre
-lidn2 /usr/local/lib64/libssl.so /usr/local/lib64/libcrypto.so
-Wl,-rpath -Wl,/usr/local/lib64 -ldl -L/usr/local/lib64 -lz
-L/usr/local/lib64 -lpsl  -ldl -lpthread
gcc: error: unrecognized command line option '-R'
make[3]: *** [Makefile:1757: wget_css_fuzzer] Error 1

Some of the command above looks a little unusual. For example:

-Wl,-rpath -Wl,/usr/local/lib64

Here are the variables I configure with:

PKGCONFIG: /usr/local/lib64/pkgconfig
 CPPFLAGS: -I/usr/local/include -DNDEBUG
   CFLAGS: -g2 -O2 -m64 -march=native -fPIC
 CXXFLAGS: -g2 -O2 -m64 -march=native -fPIC
  LDFLAGS: -L/usr/local/lib64 -m64 -Wl,-R,/usr/local/lib64
-Wl,--enable-new-dtags
   LDLIBS: -ldl -lpthread

Attached is config.log. It is the one for the Wget package. There is
no separate fuzz/config.log.

Jeff


config.log.tar.gz
Description: application/gzip

[Bug-wget] Wget not building with libpsl?

[Jeffrey Walton]

Hi Everyone,

I have a local copy of libpsl (https://github.com/rockdaboot/libpsl).
Wget was configured with:

  $ ./configure --prefix=/usr/local --libdir=/usr/local/lib64
--with-ssl=openssl --with-libssl-prefix=/usr/local
--with-libiconv-prefix=/usr/local
--with-libunistring-prefix=/usr/local --with-libidn=/usr/local

The configure line _lacks_ --without-libpsl. According to --help,
there is no corresponding --with-libpsl.

When I inspect Wget I don't see a libpsl dependency:

$ ldd /usr/local/bin/wget
linux-vdso.so.1 (0x7ffd87795000)
libiconv.so.2 => /usr/local/lib64/libiconv.so.2 (0x7f8eb7ef2000)
libunistring.so.2 => /usr/local/lib64/libunistring.so.2
(0x7f8eb7b5a000)
libpcre.so.1 => /usr/local/lib64/libpcre.so.1 (0x7f8eb790a000)
libidn2.so.0 => /usr/local/lib64/libidn2.so.0 (0x7f8eb76ec000)
libssl.so.1.0.0 => /usr/local/lib64/libssl.so.1.0.0 (0x7f8eb747d000)
libcrypto.so.1.0.0 => /usr/local/lib64/libcrypto.so.1.0.0
(0x7f8eb7004000)
libdl.so.2 => /lib64/libdl.so.2 (0x7f8eb6e0)
libz.so.1 => /usr/local/lib64/libz.so.1 (0x7f8eb6be2000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x7f8eb69c3000)
libc.so.6 => /lib64/libc.so.6 (0x7f8eb65ee000)
/lib64/ld-linux-x86-64.so.2 (0x7f8eb81e2000)

And the version string states it is missing. Adding --with-libpsl has
the same result:

$ wget --version | grep psl
+ntlm +opie -psl +ssl/openssl

Any ideas how to enable PSL?

Thanks in advance.

Jeff

Re: [Bug-wget] Wget not building with libpsl?

[Jeffrey Walton]

Cancel this... /usr/local/bin/pod2man was missing, so the script
bailed early. The program was not being updated with 'make install'
like I expected.

On Fri, Jan 5, 2018 at 4:47 AM, Jeffrey Walton <noloa...@gmail.com> wrote:
> Hi Everyone,
>
> I have a local copy of libpsl (https://github.com/rockdaboot/libpsl).
> Wget was configured with:
>
>   $ ./configure --prefix=/usr/local --libdir=/usr/local/lib64
> --with-ssl=openssl --with-libssl-prefix=/usr/local
> --with-libiconv-prefix=/usr/local
> --with-libunistring-prefix=/usr/local --with-libidn=/usr/local
>
> The configure line _lacks_ --without-libpsl. According to --help,
> there is no corresponding --with-libpsl.
>
> When I inspect Wget I don't see a libpsl dependency:
>
> $ ldd /usr/local/bin/wget
> linux-vdso.so.1 (0x7ffd87795000)
> libiconv.so.2 => /usr/local/lib64/libiconv.so.2 (0x7f8eb7ef2000)
> libunistring.so.2 => /usr/local/lib64/libunistring.so.2
> (0x7f8eb7b5a000)
> libpcre.so.1 => /usr/local/lib64/libpcre.so.1 (0x7f8eb790a000)
> libidn2.so.0 => /usr/local/lib64/libidn2.so.0 (0x7f8eb76ec000)
> libssl.so.1.0.0 => /usr/local/lib64/libssl.so.1.0.0 
> (0x7f8eb747d000)
> libcrypto.so.1.0.0 => /usr/local/lib64/libcrypto.so.1.0.0
> (0x7f8eb7004000)
> libdl.so.2 => /lib64/libdl.so.2 (0x7f8eb6e0)
> libz.so.1 => /usr/local/lib64/libz.so.1 (0x7f8eb6be2000)
> libpthread.so.0 => /lib64/libpthread.so.0 (0x7f8eb69c3000)
> libc.so.6 => /lib64/libc.so.6 (0x7f8eb65ee000)
> /lib64/ld-linux-x86-64.so.2 (0x7f8eb81e2000)
>
> And the version string states it is missing. Adding --with-libpsl has
> the same result:
>
> $ wget --version | grep psl
> +ntlm +opie -psl +ssl/openssl
>
> Any ideas how to enable PSL?
>
> Thanks in advance.
>
> Jeff

Re: [Bug-wget] Deprecate TLS 1.0 and TLS 1.1

[Jeffrey Walton]

On Tue, Jun 19, 2018 at 6:44 AM, Loganaden Velvindron  wrote:
> ...
> As per:
> https://tools.ietf.org/html/draft-moriarty-tls-oldversions-diediedie-00
>
> Attached is a tentative patch to disable TLS 1.0 and TLS 1.1 by
> default. No doubt that this will cause some discussions, I'm open to
> hearing all opinions on this.

What will users do?

I'm guessing most will turn to --no-check-certificate or HTTP, which
has the net effect of removing security, not improving it.

Stack Overflow is littered with the --no-check-certificate answer for
questions ranging from "how do I use wget to download a file" to "how
do I make my PHP work again".

Jeff

[Bug-wget] How to change SYSTEM_WGETRC?

[Jeffrey Walton]

When I check my locally installed wget --version it is showing the wrong wgetrc:

$ command -v wget
/usr/local/bin/wget

$ wget --version
GNU Wget 1.19.5 built on linux-gnu.
...
Wgetrc:
/etc/wgetrc (system)

I installed an updated wgetrc based on sample.wgetrc in $PREFIX/etc
but it appears it is not being used.

There does not appear to be a configure option to control it:

$ ./configure --help | grep -i wgetrc
$

I tried to add -DSYSTEM_WGETRC="$PREFIX/etc/wgetrc" to CFLAGS but it
resulted in a compile error.

How do I change SYSTEM_WGETRC?

Thanks in advance.

[Bug-wget] How specify a CA zoo file during configuration

[Jeffrey Walton]

Hi Everyone,

I'm working on an ancient system. I need to bootstrap an updated Wget.

I installed a new ca-certs.pem in /usr/local/share. I need to tell
Wget to use it. I don't see a configuration option:

$ ./configure --help | grep -i ca
  --cache-file=FILE   cache test results in FILE [disabled]
  -C, --config-cache  alias for `--cache-file=config.cache'
  [/usr/local]
`/usr/local/bin', `/usr/local/lib' etc.  You can specify
an installation prefix other than `/usr/local' using `--prefix',
  --localstatedir=DIR modifiable single-machine data [PREFIX/var]
  --runstatedir=DIR   modifiable per-process data [LOCALSTATEDIR/run]
  --localedir=DIR locale-dependent data [DATAROOTDIR/locale]
  --with-caresenable support for C-Ares DNS lookup.
  (use with caution on other systems).
  CARES_CFLAGS
  C compiler flags for CARES, overriding pkg-config
  CARES_LIBS  linker flags for CARES, overriding pkg-config

I don't want to overwrite/replace the existing one because I fear the
modern X509 extensions will break some things.

How do I tell configure where to look for the ca zoo file?

Jeff

Re: [Bug-wget] Failed 1.19.5 install on Solaris 11.3

[Jeffrey Walton]

On Wed, Jul 18, 2018 at 7:14 AM, Tim Rühsen  wrote:
> Maybe it's an bash/sh incompatibility. Anyways - what does 'make
> install' do !? It basically copies the 'wget' executable into a
> directory (e.g. /usr/local/bin/) that is listed in your PATH env variable.
>
> You can do that by hand. If you want the updated man file, copy wget.1
> into your man1 directory (e.g. /usr/local/share/man/man1/).

thanks.

It appears things are not getting as far as an install. My bad.

$ make
make  all-recursive
Making all in lib
make  all-recursive
Making all in src
make  all-am
Making all in doc
Making all in po
Making all in util
Making all in fuzz
 cd .. && /bin/sh
/export/home/jwalton/Build-Scripts/wget-1.19.5/build-aux/missing
automake-1.15 --gnu fuzz/Makefile
/export/home/jwalton/Build-Scripts/wget-1.19.5/build-aux/missing[81]:
automake-1.15: not found [No such file or directory]
WARNING: 'automake-1.15' is missing on your system.
 You should only need it if you modified 'Makefile.am' or
 'configure.ac' or m4 files included by 'configure.ac'.
 The 'automake' program is part of the GNU Automake package:
 
 It also requires GNU Autoconf, GNU m4 and Perl in order to run:
 
 
 
*** Error code 127

Re: [Bug-wget] Failed 1.19.5 install on Solaris 11.3

[Jeffrey Walton]

On Wed, Jul 18, 2018 at 7:14 AM, Tim Rühsen  wrote:
> Maybe it's an bash/sh incompatibility. Anyways - what does 'make
> install' do !? It basically copies the 'wget' executable into a
> directory (e.g. /usr/local/bin/) that is listed in your PATH env variable.
>
> You can do that by hand. If you want the updated man file, copy wget.1
> into your man1 directory (e.g. /usr/local/share/man/man1/).

I see what was happening. After unpacking this patch was applied:

sed -e 's|$(LTLIBICONV)|$(LIBICONV)|g' fuzz/Makefile.am >
fuzz/Makefile.am.fixed
mv fuzz/Makefile.am.fixed fuzz/Makefile.am

But it lacked this:

touch -t 19700101 fuzz/Makefile.am

So Autotools was trying to regenerate all of its shit. Autotools sucks
so bad I cringe when I have to work with it. What a miserable set of
programs.

Thanks for the help.

Jeff

Re: [Bug-wget] Failed 1.19.5 install on Solaris 11.3

[Jeffrey Walton]

On Wed, Jul 18, 2018 at 8:59 AM, Darshit Shah  wrote:
> Are you trying to compile Wget from git? Or are you using the tarballs?

Tarball.

> If you are using the tarballs, this should not happen unless you have modified
> some of the build files. In which case, I would ask you to share your changes
> with us so that we can fix the build for everyone.

Tim identified the issue several months ago. I think Tim already fixed
it in Master. The 1.19.5 tarball still needs manual patching.

Jeff

[Bug-wget] Failed 1.19.5 install on Solaris 11.3

[Jeffrey Walton]

Hi Everyone,

I'm working from the Wget 1.19.5 tarball. 'make install' is failing on
Solaris 11.3. Is there any way to avoid the automake version checks?
As it stands I'm in a DoS situation because I need an updated Wget.

It has been my experience it is nearly impossible to update Autotools
(I have never been able to do it on Linux or Solaris). I am stuck with
the tools Oracle ships with Solaris.

Thanks in advance.

Jeff

==

$ sudo make install
...

echo ' cd .. && /bin/sh
/export/home/build/wget-1.19.5/build-aux/missing automake-1.15 --gnu
fuzz/Makefile'; \
CDPATH="${ZSH_VERSION+.}:" && cd .. && \
  /bin/sh /export/home/build/wget-1.19.5/build-aux/missing
automake-1.15 --gnu fuzz/Makefile
make: Fatal error: Command failed for target `Makefile.in'
Current working directory /export/home/build/wget-1.19.5/fuzz
*** Error code 1
The following command caused the error:
fail=; \
if (target_option=k; case ${target_option-} in  ?) ;;  *) echo
"am__make_running_with_option: internal error: invalid"  "target
option '${target_option-}' specified" >&2;  exit 1;;  esac;
has_opt=no;  sane_makeflags=$MAKEFLAGS;  if {  if test -z '1'; then
false;  elif test -n ''; then  true;  elif test -n '' && test -n '';
then  true;  else  false;  fi;  }; then  sane_makeflags=$MFLAGS;  else
 case $MAKEFLAGS in  *\\[\ \  ]*)  bs=\\;  sane_makeflags=`printf
'%s\n' "$MAKEFLAGS"  | sed "s/$bs$bs[$bs $bs]*//g"`;;  esac;
fi;  skip_next=no;  strip_trailopt ()  {  flg=`printf '%s\n' "$flg" |
sed "s/$1.*$//"`;  };  for flg in $sane_makeflags; do  test $skip_next
= yes && { skip_next=no; continue; };  case $flg in  *=*|--*)
continue;;  -*I) strip_trailopt 'I'; skip_next=yes;;  -*I?*)
strip_trailopt 'I';;  -*O) strip_trailopt 'O'; skip_next=yes;;  -*O?*)
strip_trailopt 'O';;  -*l) strip_trailopt 'l'; skip_next=yes;;  -*l?*)
strip_trailopt 'l';;  -[dEDm]) skip_next=yes;;  -[JT]) skip_next=yes;;
 esac;  case $flg in  *$target_option*) has_opt=yes; break;;  esac;
done;  test $has_opt = yes); then \
  failcom='fail=yes'; \
else \
  failcom='exit 1'; \
fi; \
dot_seen=no; \
target=`echo install-recursive | sed s/-recursive//`; \
case "install-recursive" in \
  distclean-* | maintainer-clean-*) list='lib src doc po util fuzz
tests testenv' ;; \
  *) list='lib src doc po util fuzz tests testenv' ;; \
esac; \
for subdir in $list; do \
  echo "Making $target in $subdir"; \
  if test "$subdir" = "."; then \
dot_seen=yes; \
local_target="$target-am"; \
  else \
local_target="$target"; \
  fi; \
  (CDPATH="${ZSH_VERSION+.}:" && cd $subdir && make  $local_target) \
  || eval $failcom; \
done; \
if test "$dot_seen" = "no"; then \
  make  "$target-am" || exit 1; \
fi; test -z "$fail"
make: Fatal error: Command failed for target `install-recursive'
Current working directory /export/home/build/wget-1.19.5
*** Error code 1
make: Fatal error: Command failed for target `install'

Re: [Bug-wget] Docs missing info on ca_directory and ca_certfile

[Jeffrey Walton]

On Thu, Jan 3, 2019 at 12:23 PM Ander Juaristi  wrote:
>
> The patch looks good to me. As Tim says, I would also pass NULL as the
> second param in line 20.  If we provide --ca-directory what would happen
> is that OpenSSL will pick up the most suitable certificate from the
> directory based on the hash value of the name, and some other field I
> don't remember. GnuTLS will consider all of them. In the end it's the
> same behavior.
>
> Tim, could you merge the patch?

Feel free to knob turn on it. I'm fine with merciless editing.

The three use cases I was trying to capture is:

(1) wget ...  # no CA's specified; use defaults from wgetrc

(2) wget --ca-file=... # Use only this CA or collection of CAs

(3) wget --ca_directory=...   # Use only this collection of CAs

Cases (2) and (3) attempt to avoid unwanted additional CAs for those
who are trying to be strict about what they are willing to accept.

If I mis-parsed the Wget sources and what is happening, then my
apologies. That's just ignorance on my part and I apologize for it.

Jeff

[Bug-wget] Docs missing info on ca_directory and ca_certfile

[Jeffrey Walton]

The sample wgetrc is missing info on ca_directory . Also see
https://www.gnu.org/software/wget/manual/html_node/Sample-Wgetrc.html.

I also cannot figure out how to tell Wget to use cacert.pem. I've
tried ca_cert, ca_certs and ca_certfile but it produces:

wget: Unknown command ‘ca_file’ in /opt/bootstrap/etc/wgetrc at line 141
Parsing system wgetrc file failed.

Any information on configuring cacerts.pem at build time would be welcomed.

Jeff

Re: [Bug-wget] Docs missing info on ca_directory and ca_certfile

[Jeffrey Walton]

On Fri, Dec 28, 2018 at 10:07 PM Jeffrey Walton  wrote:
>
> The sample wgetrc is missing info on ca_directory . Also see
> https://www.gnu.org/software/wget/manual/html_node/Sample-Wgetrc.html.
>
> I also cannot figure out how to tell Wget to use cacert.pem. I've
> tried ca_cert, ca_certs and ca_certfile but it produces:
>
> wget: Unknown command ‘ca_file’ in /opt/bootstrap/etc/wgetrc at line 141
> Parsing system wgetrc file failed.

My bad... I found it. openssl.c used "opt.ca_cert", so I was trying to
use the same in rc file. The correct name is ca_certificate.

Tim, you may want this when Wget is built against OpenSSL. It makes
Wget/OpenSSL behave like Wget/GnuTLS:
https://github.com/noloader/Build-Scripts/blob/master/bootstrap/wget.patch
.

Jeff

[Bug-wget] --without-libunistring-prefix may not be honored

[Jeffrey Walton]

Hi Everyone,

I'm testing a bootstrap build of Wget. It only includes (1) Wget, and
(2) OpenSSL; and nothing more. Wget static links to OpenSSL to avoid
those festering path problems that plague Linux.

Wget was configured with --prefix=$HOME/bootstrap and
--without-libunistring-prefix, but it appears config is picking up a
copy of libunistring in /usr/local:

[jwalton@localhost]$ ldd $HOME/bootstrap/bin/wget
librt.so.1 => /lib/tls/librt.so.1 (0x00a46000)
libunistring.so.2 => not found
libc.so.6 => /lib/tls/libc.so.6 (0x00bb)
libpthread.so.0 => /lib/tls/libpthread.so.0 (0x0013d000)
/lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x00b98000)

The is no libunistring in /usr/lib.

Later, it causes a runtime failure because some genius thought it was
a good idea compile and link against a library, then lose the library
at runtime (or worse, link to the wrong library with the same name):

[jwalton@localhost]$ $HOME/bootstrap/bin/wget --version
/home/jwalton/bootstrap/bin/wget: error while loading shared libraries:
libunistring.so.2: cannot open shared object file: No such file or directory

If I rm /usr/local then things work as expected.

I believe --without-libunistring-prefix is not being honored by the
configure scripts.

Jeff

Re: [Bug-wget] --without-libunistring-prefix may not be honored

[Jeffrey Walton]

On Sat, Dec 29, 2018 at 2:18 AM Jeffrey Walton  wrote:
>
> I'm testing a bootstrap build of Wget. It only includes (1) Wget, and
> (2) OpenSSL; and nothing more. Wget static links to OpenSSL to avoid
> those festering path problems that plague Linux.
>
> Wget was configured with --prefix=$HOME/bootstrap and
> --without-libunistring-prefix, but it appears config is picking up a
> copy of libunistring in /usr/local:
>
> [jwalton@localhost]$ ldd $HOME/bootstrap/bin/wget
> librt.so.1 => /lib/tls/librt.so.1 (0x00a46000)
> libunistring.so.2 => not found
> libc.so.6 => /lib/tls/libc.so.6 (0x00bb)
> libpthread.so.0 => /lib/tls/libpthread.so.0 (0x0013d000)
> /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x00b98000)
>
> The is no libunistring in /usr/lib.
>
> Later, it causes a runtime failure because some genius thought it was
> a good idea compile and link against a library, then lose the library
> at runtime (or worse, link to the wrong library with the same name):
>
> [jwalton@localhost]$ $HOME/bootstrap/bin/wget --version
> /home/jwalton/bootstrap/bin/wget: error while loading shared libraries:
> libunistring.so.2: cannot open shared object file: No such file or 
> directory
>
> If I rm /usr/local then things work as expected.
>
> I believe --without-libunistring-prefix is not being honored by the
> configure scripts.

Here's config.log.

Jeff
<>

Re: [Bug-wget] Failed self tests on CubieTruck with Linaro 14.04

[Jeffrey Walton]

On Wed, Mar 20, 2019 at 9:29 AM Tim Rühsen  wrote:
>
> my guess: the perl daemon on your installation does not support IPv6.
>
> See also: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=887590

Yeah, that looks like that is it.

I can't find an updated image for the dev-board. I'm kind of stuck
with what I have.

At first blush, I thought to test Perl and disable self tests if IPv6
was not available. That seems to be a bit heavy handed since that
throws out the hundreds of other tests that work as expected.

Another option is to configure Wget with --disable-perl so the self
tests pass. That seems to cripple Wget when Wget is not really the
problem.

Might it be possible for Wget to determine if the IPv6 test can be
run, and only run the test if supported by Perl? (It is not a high
priority for me since I can explain the cause and move forward with an
install).

Jeff

Re: [Bug-wget] Docs missing info on ca_directory and ca_certfile

[Jeffrey Walton]

On Fri, Feb 22, 2019 at 7:06 AM Tim Rühsen  wrote:
>
> On 1/3/19 6:39 PM, Jeffrey Walton wrote:
> > On Thu, Jan 3, 2019 at 12:23 PM Ander Juaristi  wrote:
> >>
> >> The patch looks good to me. As Tim says, I would also pass NULL as the
> >> second param in line 20.  If we provide --ca-directory what would happen
> >> is that OpenSSL will pick up the most suitable certificate from the
> >> directory based on the hash value of the name, and some other field I
> >> don't remember. GnuTLS will consider all of them. In the end it's the
> >> same behavior.
> >>
> >> Tim, could you merge the patch?
> >
> > Feel free to knob turn on it. I'm fine with merciless editing.
> >
> > The three use cases I was trying to capture is:
> >
> > (1) wget ...  # no CA's specified; use defaults from wgetrc
> >
> > (2) wget --ca-file=... # Use only this CA or collection of CAs
> >
> > (3) wget --ca_directory=...   # Use only this collection of CAs
> >
> > Cases (2) and (3) attempt to avoid unwanted additional CAs for those
> > who are trying to be strict about what they are willing to accept.
>
> I just made up a first commit out of the 'partial trust chain' code.
>
> The second part (your points 1-3) would look like a bit different.
>
> For backwards compat we don't want to change wget's behavior when using
> --ca-file and/or --ca_directory (even not to fix a design flaw).
>
> But we could skip loading the default certs (via
> SSL_CTX_set_default_verify_paths()) when --ca-file=... *and*
> --ca_directory="" is given.
>
> Another (cleaner) option would be to add a new option --ca-skip-defaults.
>
> WDYT ?

Looks good to me.

I think it is important to maintain consistent behavior across
backends, so the changes to the patch are important.

--ca-skip-defaults may make sense. I often avoid the CA Zoo. I don't
know how many others do the same.

Jeff

[Bug-wget] Failed self tests on CubieTruck with Linaro 14.04

[Jeffrey Walton]

Hi Everyone,

I'm working on a CubieTruck. It is a Cortex-A7 running a  Linaro
14.04, which is a Ubuntu derivative. I'm building Wget 1.20.1 from
sources.

$ cat tests/test-suite.log
===
   wget 1.20.1: tests/test-suite.log
===

# TOTAL: 93
# PASS:  86
# SKIP:  1
# XFAIL: 0
# FAIL:  6
# XPASS: 0
# ERROR: 0

test-suite.log is attached. It includes errors like "Bad arg length
for Socket::inet_ntoa".

Jeff
<>

[Bug-wget] wget_css_fuzzer.c: error: lvalue required as left operand of assignment

[Jeffrey Walton]

Hi Everyone,

I'm working on Solaris 11.3 i86pc. I'm building Wget 1.20.1 from sources.

'make check' is dying at:

gcc -std=gnu11 -DHAVE_CONFIG_H -I. -I../src  -I../src -I. -I../lib
-I../lib -DSRCDIR=\"/export/home/jwalton/Build-Scripts/wget-1.20.1/fuzz\"
-DTEST_RUN -I/usr/local/include -DNDEBUG -D_REENTRANT
-Wno-unused-parameter -Wno-pedantic -I/usr/local/include
-I/usr/local/include   -I/usr/local/include   -I/usr/local/include
-DNDEBUG -g2 -O2 -march=native -fPIC -MT wget_css_fuzzer.o -MD -MP -MF
.deps/wget_css_fuzzer.Tpo -c -o wget_css_fuzzer.o wget_css_fuzzer.c
wget_css_fuzzer.c: In function 'LLVMFuzzerTestOneInput':
wget_css_fuzzer.c:95:9: error: lvalue required as left operand of assignment
  stderr = fopen("/dev/null", "w");
 ^
wget_css_fuzzer.c:106:9: error: lvalue required as left operand of assignment
  stderr = bak;
 ^
gmake[3]: *** [wget_css_fuzzer.o] Error 1
gmake[3]: Leaving directory
`/export/home/jwalton/Build-Scripts/wget-1.20.1/fuzz'
gmake[2]: *** [check-am] Error 2
gmake[2]: Leaving directory
`/export/home/jwalton/Build-Scripts/wget-1.20.1/fuzz'
gmake[1]: *** [check-recursive] Error 1
gmake[1]: Leaving directory `/export/home/jwalton/Build-Scripts/wget-1.20.1'
gmake: *** [check] Error 2
Failed to test Wget

It looks like  is included in wget_css_fuzzer.c
(https://stackoverflow.com/a/47437649/608639). But I am wondering if
the "macro" requirement is the thing going sideways.

Any thoughts on a workaround?

Re: [Bug-wget] wget_css_fuzzer.c: error: lvalue required as left operand of assignment

[Jeffrey Walton]

On Sat, Mar 9, 2019 at 10:16 PM Jeffrey Walton  wrote:
>
> Hi Everyone,
>
> I'm working on Solaris 11.3 i86pc. I'm building Wget 1.20.1 from sources.
>
> 'make check' is dying at:
>
> gcc -std=gnu11 -DHAVE_CONFIG_H -I. -I../src  -I../src -I. -I../lib
> -I../lib -DSRCDIR=\"/export/home/jwalton/Build-Scripts/wget-1.20.1/fuzz\"
> -DTEST_RUN -I/usr/local/include -DNDEBUG -D_REENTRANT
> -Wno-unused-parameter -Wno-pedantic -I/usr/local/include
> -I/usr/local/include   -I/usr/local/include   -I/usr/local/include
> -DNDEBUG -g2 -O2 -march=native -fPIC -MT wget_css_fuzzer.o -MD -MP -MF
> .deps/wget_css_fuzzer.Tpo -c -o wget_css_fuzzer.o wget_css_fuzzer.c
> wget_css_fuzzer.c: In function 'LLVMFuzzerTestOneInput':
> wget_css_fuzzer.c:95:9: error: lvalue required as left operand of assignment
>   stderr = fopen("/dev/null", "w");
>  ^
> wget_css_fuzzer.c:106:9: error: lvalue required as left operand of assignment
>   stderr = bak;
>  ^
> gmake[3]: *** [wget_css_fuzzer.o] Error 1
> gmake[3]: Leaving directory
> `/export/home/jwalton/Build-Scripts/wget-1.20.1/fuzz'
> gmake[2]: *** [check-am] Error 2
> gmake[2]: Leaving directory
> `/export/home/jwalton/Build-Scripts/wget-1.20.1/fuzz'
> gmake[1]: *** [check-recursive] Error 1
> gmake[1]: Leaving directory `/export/home/jwalton/Build-Scripts/wget-1.20.1'
> gmake: *** [check] Error 2
> Failed to test Wget
>
> It looks like  is included in wget_css_fuzzer.c
> (https://stackoverflow.com/a/47437649/608639). But I am wondering if
> the "macro" requirement is the thing going sideways.
>
> Any thoughts on a workaround?

 includes  and .
stdio_iso.h has the following (stdio_c99.h lacks them):

typedef __FILE FILE;
...

#if defined(__STDC__)
extern __FILE   __iob[_NFILE];
#define stdin   (&__iob[0])
#define stdout  (&__iob[1])
#define stderr  (&__iob[2])
#else
extern __FILE   _iob[_NFILE];
#define stdin   (&_iob[0])
#define stdout  (&_iob[1])
#define stderr  (&_iob[2])
#endif  /* __STDC__ */

Then, in stdio_tag.h:

#define __FILE_TAG  __FILE

And in stdio_impl.h:

#ifdef  _LP64
struct __FILE_TAG {
long__pad[16];
};
#else
struct __FILE_TAG   /* needs to be binary-compatible with old versions */
{
#ifdef _STDIO_REVERSE
unsigned char   *_ptr;  /* next character from/to here in buffer */
int _cnt;   /* number of available characters in buffer */
#else
int _cnt;   /* number of available characters in buffer */
unsigned char   *_ptr;  /* next character from/to here in buffer */
#endif
unsigned char   *_base; /* the buffer */
...
};
#endif

I don't really see what the problem is. It looks like a potential lvalue to me.

Re: [Bug-wget] /usr/bin/env: invalid option -- 'S'

[Jeffrey Walton]

On Thu, May 30, 2019 at 5:40 AM Tim Rühsen  wrote:
>
> The shebang line here reads
>
> #!/usr/bin/env -S perl -I .
>
> /usr/bin/env is part of coreutils ans should know '-S', even on Debian
> stretch.
>
> $ /usr/bin/env --version
> env (GNU coreutils) 8.30
>
> @Jeff, what does '/usr/bin/env --help' show ? Or 'man env' ?

$ /usr/bin/env --version
env (GNU coreutils) 8.26
Copyright (C) 2016 Free Software Foundation, Inc.
...

$ /usr/bin/env --help
Usage: /usr/bin/env [OPTION]... [-] [NAME=VALUE]... [COMMAND [ARG]...]
Set each NAME to VALUE in the environment and run COMMAND.

Mandatory arguments to long options are mandatory for short options too.
  -i, --ignore-environment  start with an empty environment
  -0, --null   end each output line with NUL, not newline
  -u, --unset=NAME remove variable from the environment
  --help display this help and exit
  --version  output version information and exit

A mere - implies -i.  If no COMMAND, print the resulting environment.

GNU coreutils online help: 
Full documentation at: 
or available locally via: info '(coreutils) env invocation'
jwalton@tinkerboard:~/cryptopp$

Re: [Bug-wget] /usr/bin/env: invalid option -- 'S'

[Jeffrey Walton]

On Thu, May 30, 2019 at 6:30 AM Tim Rühsen  wrote:
>
> Reverted 79be99aff (passes CI). Could you give it try, please ?

Where can we get that?

It is not on the mirror (yet?), and https://www.gnu.org/software/wget/
does not list Git or Savannah access.

Jeff

Re: [Bug-wget] /usr/bin/env: invalid option -- 'S'

[Jeffrey Walton]

On Thu, May 30, 2019 at 9:50 AM Tim Rühsen  wrote:
>
> On 30.05.19 12:49, Jeffrey Walton wrote:
> > On Thu, May 30, 2019 at 6:30 AM Tim Rühsen  wrote:
> >>
> >> Reverted 79be99aff (passes CI). Could you give it try, please ?
> >
> > Where can we get that?
> >
> > It is not on the mirror (yet?), and https://www.gnu.org/software/wget/
> > does not list Git or Savannah access.
>
> It's on Savannah and Gitlab, branch master,
> commit b3f86f90cc5b3d3d1c2d6f720818fda9486108ec

OK, thanks.

I used PERL5LIB to put teests/ on path for Perl. It looks like at
least one Debian machine I have is back to the Socket::inet_ntoa
problems.

I'm calling it good.

The Perl people need to fix Socket::inet_ntoa, and the Debian people
need to make it available. I'm guessing Debian is the holdup. They
will leave things broke rather than supplying an update. It is a waste
of time to file a Debian bug report.

Jeff

Re: [Bug-wget] /usr/bin/env: invalid option -- 'S'

[Jeffrey Walton]

On Thu, May 30, 2019 at 6:49 AM Jeffrey Walton  wrote:
>
> On Thu, May 30, 2019 at 6:30 AM Tim Rühsen  wrote:
> >
> > Reverted 79be99aff (passes CI). Could you give it try, please ?
>
> Where can we get that?
>
> It is not on the mirror (yet?), and https://www.gnu.org/software/wget/
> does not list Git or Savannah access.

I performed a sed on all the *.px files. (If this was wrong let me
know what you need).

for file in $(find "$PWD" -name '*.px')
do
sed -e 's|env -S perl -I .|env perl|g' "$file" > "$file.fixed"
chmod +w "$file"
mv "$file.fixed" "$file"
chmod +x "$file"
chmod -w "$file"
done

The +x was needed for a few of the files (most were OK). Otherwise got
permission denied.

After the change, most tests pass. There were some failures that remained:

FAIL: Test-https-pfs.px
FAIL: Test-https-tlsv1.px
FAIL: Test-https-tlsv1x.px
FAIL: Test-https-selfsigned.px
SKIP: Test-https-weboftrust.px
FAIL: Test-https-clientcert.px
FAIL: Test-https-crl.px
PASS: Test-https-badcerts.px

$ cd ./wget-1.20.3/tests
$ ./Test-https-pfs.px
Can't locate WgetFeature.pm in @INC (you may need to install the
WgetFeature module) (@INC contains: /etc/perl
/usr/local/lib/x86_64-linux-gnu/perl/5.24.1
/usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24
/usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24
/usr/share/perl/5.24 /usr/local/lib/site_perl
/usr/lib/x86_64-linux-gnu/perl-base) at ./Test-https-pfs.px line 6.
BEGIN failed--compilation aborted at ./Test-https-pfs.px line 6.

And

$ find . -name WgetFeature.pm
./WgetFeature.pm

Jeff

Re: [Bug-wget] /usr/bin/env: invalid option -- 'S'

[Jeffrey Walton]

On Thu, May 30, 2019 at 7:21 AM Jeffrey Walton  wrote:
>
> On Thu, May 30, 2019 at 6:49 AM Jeffrey Walton  wrote:
> >
> > On Thu, May 30, 2019 at 6:30 AM Tim Rühsen  wrote:
> > >
> > > Reverted 79be99aff (passes CI). Could you give it try, please ?
> >
> > Where can we get that?
> >
> > It is not on the mirror (yet?), and https://www.gnu.org/software/wget/
> > does not list Git or Savannah access.
>
> I performed a sed on all the *.px files. (If this was wrong let me
> know what you need).
>
> for file in $(find "$PWD" -name '*.px')
> do
> sed -e 's|env -S perl -I .|env perl|g' "$file" > "$file.fixed"
> chmod +w "$file"
> mv "$file.fixed" "$file"
> chmod +x "$file"
> chmod -w "$file"
> done
>
> The +x was needed for a few of the files (most were OK). Otherwise got
> permission denied.
>
> After the change, most tests pass. There were some failures that remained:
>
> FAIL: Test-https-pfs.px
> FAIL: Test-https-tlsv1.px
> FAIL: Test-https-tlsv1x.px
> FAIL: Test-https-selfsigned.px
> SKIP: Test-https-weboftrust.px
> FAIL: Test-https-clientcert.px
> FAIL: Test-https-crl.px
> PASS: Test-https-badcerts.px
>
> $ cd ./wget-1.20.3/tests
> $ ./Test-https-pfs.px
> Can't locate WgetFeature.pm in @INC (you may need to install the
> WgetFeature module) (@INC contains: /etc/perl
> /usr/local/lib/x86_64-linux-gnu/perl/5.24.1
> /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24
> /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24
> /usr/share/perl/5.24 /usr/local/lib/site_perl
> /usr/lib/x86_64-linux-gnu/perl-base) at ./Test-https-pfs.px line 6.
> BEGIN failed--compilation aborted at ./Test-https-pfs.px line 6.

I don't know Perl, but it looks like this is the problem:
https://perlmaven.com/cant-locate-in-inc . And this may be the
solution: https://perlmaven.com/how-to-add-a-relative-directory-to-inc
.

I can't help but laugh at the solutions. It never ceases to amazes me
at the f**k'd up convoluted solutions engineers come up with.
Apparently 'use ./WgetFeature.pm;' would have been way too easy.

And apparently including PWD by default would have been way too easy.
We have already accepted the risk of compiling and running
Test-https-pfs.px in /some/user/directory. Using WgetFeature.pm, which
is side-by-side and needed by Test-https-pfs.px, adds no additional
risk.

Instead we have to suffer the shit the Perl people came up with. More
Darwin awards...

Jeff

Re: [Bug-wget] /usr/bin/env: invalid option -- 'S'

[Jeffrey Walton]

On Wed, May 29, 2019 at 8:35 AM Darshit Shah  wrote:
>
> That's very weird, the shebang line in that file reads:
>
> ```
> #!/usr/bin/env perl
> ```
>
> No options are being passed to env there. I'm going to have to take another
> look at this later

Also see https://github.com/mirror/wget/blob/master/tests/Test-https-pfs.px .

Jeff

Re: [Bug-wget] /usr/bin/env: invalid option -- 'S'

[Jeffrey Walton]

On Wed, May 29, 2019 at 8:38 AM Jeffrey Walton  wrote:
>
> On Wed, May 29, 2019 at 8:35 AM Darshit Shah  wrote:
> >
> > That's very weird, the shebang line in that file reads:
> >
> > ```
> > #!/usr/bin/env perl
> > ```
> >
> > No options are being passed to env there. I'm going to have to take another
> > look at this later
>
> Also see https://github.com/mirror/wget/blob/master/tests/Test-https-pfs.px .

I did a quick bash -x to make sure my script was not sideways
(https://github.com/noloader/Build-Scripts/blob/master/build-wget.sh#L153):

** Wget **
+ echo

+ /home/jwalton/bootstrap/bin/wget -O wget-1.20.3.tar.gz --ca-certificate=/home/
jwalton/.cacert/lets-encrypt-root-x3.pem https://ftp.gnu.org/pub/gnu/wget/wget-1
.20.3.tar.gz
--2019-05-29 08:41:06--  https://ftp.gnu.org/pub/gnu/wget/wget-1.20.3.tar.gz
Resolving ftp.gnu.org... 209.51.188.20, 2001:470:142:3::b
Connecting to ftp.gnu.org|209.51.188.20|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4489249 (4.3M) [application/x-gzip]
Saving to: 'wget-1.20.3.tar.gz'

It looks like I am fetching the 1.20.3 tarball as expected.

Jeff

[Bug-wget] /usr/bin/env: invalid option -- 'S'

[Jeffrey Walton]

Hi Everyone/Tim,

Debian 9.9:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:Debian GNU/Linux 9.9 (stretch)
Release:9.9
Codename:   stretch

$ make check
...

PASS: Test-ftp-pasv-not-supported.px
FAIL: Test-https-pfs.px
FAIL: Test-https-tlsv1.px
FAIL: Test-https-tlsv1x.px
FAIL: Test-https-selfsigned.px
SKIP: Test-https-weboftrust.px
FAIL: Test-https-clientcert.px
FAIL: Test-https-crl.px
PASS: Test-https-badcerts.px

Trying to run manually:

$ ./wget-1.20.3/tests/Test-https-pfs.px
/usr/bin/env: invalid option -- 'S'
Try '/usr/bin/env --help' for more information.

And

$ /usr/bin/env --version
env (GNU coreutils) 8.26

[Bug-wget] Wget 1.20.3 on Solaris

[Jeffrey Walton]

Hi Everyone,

Below is the result of running 'make check' on Solaris 11.3. The
previous self tests were OK. The only problem are the ones below. I'm
hazarding a guess it is an ancient version of Perl:

$ perl --version
This is perl 5, version 12, subversion 5 (v5.12.5) built for
i86pc-solaris-64int
(with 7 registered patches, see perl -V for more detail)

Maybe Wget should disable the self tests if Perl is too old or Solaris
is detected.

...
gmake  check-TESTS
gmake[3]: Entering directory `/export/home/jwalton/Build-Scripts/wget-1.20.3/tes
tenv'
gmake[4]: Entering directory `/export/home/jwalton/Build-Scripts/wget-1.20.3/tes
tenv'
FAIL: Test-504.py
FAIL: Test-416.py
FAIL: Test-auth-basic-fail.py
FAIL: Test-auth-basic.py
FAIL: Test-auth-basic-netrc.py
FAIL: Test-auth-basic-netrc-user-given.py
FAIL: Test-auth-basic-netrc-pass-given.py
FAIL: Test-auth-basic-no-netrc-fail.py
FAIL: Test-auth-both.py
FAIL: Test-auth-digest.py
FAIL: Test-auth-no-challenge.py
FAIL: Test-auth-no-challenge-url.py
FAIL: Test-auth-retcode.py
FAIL: Test-auth-with-content-disposition.py
FAIL: Test-c-full.py
FAIL: Test-condget.py
FAIL: Test-Content-disposition-2.py
FAIL: Test-Content-disposition.py
FAIL: Test--convert-links--content-on-error.py
FAIL: Test-cookie-401.py
FAIL: Test-cookie-domain-mismatch.py
FAIL: Test-cookie-expires.py
FAIL: Test-cookie.py
FAIL: Test-Head.py
FAIL: Test-hsts.py
FAIL: Test--https.py
FAIL: Test--https-crl.py
FAIL: Test-missing-scheme-retval.py
FAIL: Test-O.py
FAIL: Test-pinnedpubkey-der-https.py
FAIL: Test-pinnedpubkey-der-no-check-https.py
FAIL: Test-pinnedpubkey-hash-https.py
FAIL: Test-pinnedpubkey-hash-no-check-fail-https.py
FAIL: Test-pinnedpubkey-pem-fail-https.py
FAIL: Test-pinnedpubkey-pem-https.py
FAIL: Test-Post.py
FAIL: Test-recursive-basic.py
FAIL: Test-recursive-include.py
FAIL: Test-recursive-redirect.py
FAIL: Test-redirect.py
FAIL: Test-redirect-crash.py
FAIL: Test--rejected-log.py
FAIL: Test-reserved-chars.py
FAIL: Test--spider-r.py

Testsuite summary for wget 1.20.3

# TOTAL: 44
# PASS:  0
# SKIP:  0
# XFAIL: 0
# FAIL:  44
# XPASS: 0
# ERROR: 0

See testenv/test-suite.log
Please report to bug-wget@gnu.org

Re: [Bug-wget] Wget 1.20.3 on Solaris

[Jeffrey Walton]

On Wed, May 1, 2019 at 3:51 PM Tim Rühsen  wrote:
>
> could you post e.g. the content of tests/Test-504.log ?

Yes, attached.

Do you want an account on the box. I keep it around for testing, and I
can make you admin. You can connect to it with 'ssh
trushen@151.196.22.177'. If so, send over your authorized_keys.

Jeff
<>
<>

Re: [Bug-wget] Wget 1.20.3 on Solaris

[Jeffrey Walton]

On Wed, May 1, 2019 at 6:56 PM Darshit Shah  wrote:
>
> The error seems to arise from a missing python3 binary on the test machine.
>
> I do remember that we added a check for this to our build script to deal with
> this issue. I wonder why it is broken on your machine. Could you also please
> share the config.log file that is generated?
>
> Anyways, a quick fix for you would be to install Python 3

Thanks Darshit .

You are also welcomed to an account on the box. All I ask is your
hobbies not include photography and little children.

Jeff
<>

Re: [Bug-wget] Wget 1.20.3 on Solaris

[Jeffrey Walton]

On Thu, May 2, 2019 at 4:00 AM Tim Rühsen  wrote:
>
> Hi Jeff,
>
> On 5/1/19 11:38 PM, Jeffrey Walton wrote:
> > On Wed, May 1, 2019 at 3:51 PM Tim Rühsen  wrote:
> >>
> >> could you post e.g. the content of tests/Test-504.log ?
> >
> > Yes, attached.
> >
> > Do you want an account on the box. I keep it around for testing, and I
> > can make you admin. You can connect to it with 'ssh
> > trushen@151.196.22.177'. If so, send over your authorized_keys.
>
> thanks for the offer, but this issue is not Solaris specific.
> But if the need arises, I have access to the OpenCSW Solaris boxes :-)
>
> Please check README.checkout which lists python3 as requirement for
> running the tests in testenv/ (as Darshit also pointed out).

Ack, thanks.

Since I got you on the line, how do I disable them. There is no need
to run them if all they are going to do is fail. I did not see a
configure option.

Jeff

Re: [Bug-wget] Wget 1.20.3 on Solaris

[Jeffrey Walton]

On Mon, May 6, 2019 at 2:01 PM Tim Rühsen  wrote:
>  ...
>
> Please try this patch and report back:
> diff --git a/testenv/Makefile.am b/testenv/Makefile.am
> index b5a39ad2..5150d4ac 100644
> --- a/testenv/Makefile.am
> +++ b/testenv/Makefile.am
> @@ -123,5 +123,5 @@ endif
>  EXTRA_DIST = certs conf exc misc server test README $(TESTS)
>
>  TEST_EXTENSIONS = .py
> -PY_LOG_COMPILER = python3
> +PY_LOG_COMPILER = $(PYTHON)
>  AM_PY_LOG_FLAGS = -O

No joy on Solaris 11.3. I got the same result.

However, I had to hack around old Autotools provided by Oracle:

sed -e 's|PY_LOG_COMPILER = python3|PY_LOG_COMPILER = $(PYTHON)|g'
testenv/Makefile.am > testenv/Makefile.am.fixed
mv testenv/Makefile.am.fixed testenv/Makefile.am
touch -t 19700101 testenv/Makefile.am

If I don't set the filetime in the past then Autotools wants to do the
full reconf, which dies because of Solaris' ancient tools. The same
trick worked as expected for LTLIBICONV at Wget 1.20.1.

I did not test other platforms.

Jeff

[Bug-wget] FAIL: 6

[Jeffrey Walton]

Hi Everyone/Tim,

I'm testing Wget 1.20.3. Debian 9.8 (Stretch) is OK. Fedora 29 is
catching some self-test failures.

$ cat wget-1.20.3/tests/test-suite.log
===
   wget 1.20.3: tests/test-suite.log
===

# TOTAL: 93
# PASS:  86
# SKIP:  1
# XFAIL: 0
# FAIL:  6
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: Test-https-pfs


Setting --no-config (noconfig) to 1
Setting --no-config (noconfig) to 1
Running test Test-https-pfs
Calling /home/test/wget-1.20.3/tests/../src/wget -d --no-config
--secure-protocol=PFS
--ca-certificate=/home/test/wget-1.20.3/tests/certs/test-ca-cert.pem
https://WgetTestingServer:24443/somefile.txt
Setting --no-config (noconfig) to 1
Setting --no-config (noconfig) to 1
Setting --secure-protocol (secureprotocol) to PFS
Setting --secure-protocol (secureprotocol) to PFS
Setting --ca-certificate (cacertificate) to
/home/test/wget-1.20.3/tests/certs/test-ca-cert.pem
Setting --ca-certificate (cacertificate) to
/home/test/wget-1.20.3/tests/certs/test-ca-cert.pem
DEBUG output created by Wget 1.20.3 on linux-gnu.

Reading HSTS entries from /home/jwalton/.wget-hsts
URI encoding = 'ANSI_X3.4-1968'
iconv UTF-8 -> ANSI_X3.4-1968
iconv outlen=88 inlen=44
converted 'https://WgetTestingServer:24443/somefile.txt'
(ANSI_X3.4-1968) -> 'https://WgetTestingServer:24443/somefile.txt'
(UTF-8)
Converted file name 'somefile.txt' (UTF-8) -> 'somefile.txt' (ANSI_X3.4-1968)
--2019-04-18 20:43:03--  https://wgettestingserver:24443/somefile.txt
Resolving wgettestingserver (wgettestingserver)... ::1, 127.0.0.1
Caching wgettestingserver => ::1 127.0.0.1
Connecting to wgettestingserver (wgettestingserver)|::1|:24443... connected.
Created socket 3.
Releasing 0x5561cf524560 (new refcount 1).
Initiating SSL handshake.
Handshake successful; connected socket 3 to SSL handle 0x5561cf526810
certificate:
  subject: C=Freeland,ST=Sunshine,O=GNU,OU=Wget,CN=Wget
  issuer:  C=Freeland,ST=Sunshine,O=GNU,OU=Wget,CN=Wget
X509 certificate successfully verified and matches host wgettestingserver

---request begin---
GET /somefile.txt HTTP/1.1
User-Agent: Wget/1.20.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
Host: wgettestingserver:24443
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response... Bad arg length for
Socket::inet_ntoa, length is 16, should be 4 at
/usr/share/perl5/HTTP/Daemon.pm line 51.
No data received.
Closed 3/SSL 0x5561cf526810
Retrying.

--2019-04-18 20:43:04--  (try: 2)  https://wgettestingserver:24443/somefile.txt
Found wgettestingserver in host_name_addresses_map (0x5561cf524560)
Connecting to wgettestingserver (wgettestingserver)|::1|:24443... Closed fd 3
failed: Connection refused.
Connecting to wgettestingserver
(wgettestingserver)|127.0.0.1|:24443... Closed fd 3
failed: Connection refused.
Releasing 0x5561cf524560 (new refcount 1).
Releasing 0x5561cf524560 (new refcount 0).
Deleting unused 0x5561cf524560.
Resolving wgettestingserver (wgettestingserver)... ::1, 127.0.0.1
Caching wgettestingserver => ::1 127.0.0.1
Connecting to wgettestingserver (wgettestingserver)|::1|:24443... Closed fd 3
failed: Connection refused.
Connecting to wgettestingserver
(wgettestingserver)|127.0.0.1|:24443... Closed fd 3
failed: Connection refused.
Releasing 0x5561cf53c150 (new refcount 1).
Test failed: wrong code returned (was: 4, expected: 0)
FAIL Test-https-pfs.px (exit status: 1)

FAIL: Test-https-tlsv1
==

Setting --no-config (noconfig) to 1
Setting --no-config (noconfig) to 1
Running test Test-https-tlsv1
Calling /home/test/wget-1.20.3/tests/../src/wget -d --no-config
--secure-protocol=TLSv1
--ca-certificate=/home/test/wget-1.20.3/tests/certs/test-ca-cert.pem
https://WgetTestingServer:28443/somefile.txt
Setting --no-config (noconfig) to 1
Setting --no-config (noconfig) to 1
Setting --secure-protocol (secureprotocol) to TLSv1
Setting --secure-protocol (secureprotocol) to TLSv1
Setting --ca-certificate (cacertificate) to
/home/test/wget-1.20.3/tests/certs/test-ca-cert.pem
Setting --ca-certificate (cacertificate) to
/home/test/wget-1.20.3/tests/certs/test-ca-cert.pem
DEBUG output created by Wget 1.20.3 on linux-gnu.

Reading HSTS entries from /home/jwalton/.wget-hsts
URI encoding = 'ANSI_X3.4-1968'
iconv UTF-8 -> ANSI_X3.4-1968
iconv outlen=88 inlen=44
converted 'https://WgetTestingServer:28443/somefile.txt'
(ANSI_X3.4-1968) -> 'https://WgetTestingServer:28443/somefile.txt'
(UTF-8)
Converted file name 'somefile.txt' (UTF-8) -> 'somefile.txt' (ANSI_X3.4-1968)
--2019-04-18 20:43:04--  https://wgettestingserver:28443/somefile.txt
Resolving wgettestingserver (wgettestingserver)... ::1, 127.0.0.1
Caching wgettestingserver => ::1 127.0.0.1
Connecting to wgettestingserver (wgettestingserver)|::1|:28443... connected.
Created socket 3.
Releasing 0x55cb6347de10 (new refcount 1).
Initiating SSL handshake.
Handshake successful; connected socket 3 to SSL handle 

Re: [Bug-wget] Wget 1.20.3 on Solaris

[Jeffrey Walton]

On Thu, May 2, 2019 at 8:12 AM Darshit Shah  wrote:
> ...
> However, this _is_ a bug. Since if you see testenv/Makefile.am, we have a
> `HAVE_PYTHON3` conditional block. And configure also checks for it.
>
> This is why I am interested in seeing the contents of your config.log file. I
> would like to see what went wrong causing configure to believe that you indeed
> have Python3 installed.

My bad, attached.

I also put it on a web server at
https://www.cryptopp.com/config.log.zip in case the mailer strips it.

Jeff
<>

Re: [Bug-wget] Wget 1.20.3 on Solaris

[Jeffrey Walton]

On Thu, May 2, 2019 at 11:34 AM Jeffrey Walton  wrote:
>
> On Thu, May 2, 2019 at 9:13 AM Tim Rühsen  wrote:
> >
> > On 5/2/19 2:15 PM, Darshit Shah wrote:
> > > That is a terrible suggestion. Please don't do that or suggest it.
> > > I'm sure Jeffrey here is capable of knowing what this is and what the
> > > implications of such a binary are.
> >
> > I wouldn't call it 'terrible', but it's definitely not a good idea for
> > multi-user production environments :-)
> >
> > It demonstrates the principle for a system where absolutely nothing else
> > uses python3 - that's what Jeff was claiming. And so nothing breaks.
> > It is even not easily possible (or just unwanted !?) to install python3;
> > if it was easy, Jeff wouldn't have posted the question, I guess.
>
> Yeah, so this is Solaris. Oracle dismantled the packages system to get
> folks to upgrade or buy a support contract. I can't even run 'pkg
> update' without errors because Oracle deleted their signing keys.
>
> I did try to build Python from sources about a year ago. I had some
> had trouble (but don't recall what it was). Then I learned Solaris
> wasn't really a supported platform for Python. They were not really
> interested in testing the platform and fixing the problems.

By the way I did try to install Python3. And here's what it looks like
trying to run update:

$ sudo pkg update
Creating Plan (Package planning:  1/10): /
pkg update: Chain was rooted in an untrusted self-signed certificate.
The package involved is pkg://solaris/consolidation/ddt/ddt-incorporation@18.3.1
8.7.13,0.5.11-11.4.0.0.1.11.0:20180718T212443Z

Oracle deleted their keys and removed access to their 11.3 repos.

At this point about all I can do is make the best of things and truck on.

Jeff

Re: [Bug-wget] Wget 1.20.3 on Solaris

[Jeffrey Walton]

On Thu, May 2, 2019 at 9:13 AM Tim Rühsen  wrote:
>
> On 5/2/19 2:15 PM, Darshit Shah wrote:
> > That is a terrible suggestion. Please don't do that or suggest it.
> > I'm sure Jeffrey here is capable of knowing what this is and what the
> > implications of such a binary are.
>
> I wouldn't call it 'terrible', but it's definitely not a good idea for
> multi-user production environments :-)
>
> It demonstrates the principle for a system where absolutely nothing else
> uses python3 - that's what Jeff was claiming. And so nothing breaks.
> It is even not easily possible (or just unwanted !?) to install python3;
> if it was easy, Jeff wouldn't have posted the question, I guess.

Yeah, so this is Solaris. Oracle dismantled the packages system to get
folks to upgrade or buy a support contract. I can't even run 'pkg
update' without errors because Oracle deleted their signing keys.

I did try to build Python from sources about a year ago. I had some
had trouble (but don't recall what it was). Then I learned Solaris
wasn't really a supported platform for Python. They were not really
interested in testing the platform and fixing the problems.

Jeff

Re: [Bug-wget] Wget 1.20.3 on Solaris

[Jeffrey Walton]

On Fri, May 3, 2019 at 4:11 AM Tim Rühsen  wrote:
>
> On 5/2/19 6:12 PM, Jeffrey Walton wrote:
> > On Thu, May 2, 2019 at 8:12 AM Darshit Shah  wrote:
> >> ...
> >> However, this _is_ a bug. Since if you see testenv/Makefile.am, we have a
> >> `HAVE_PYTHON3` conditional block. And configure also checks for it.
>
> You are right, there is /usr/bin/python3.4 detected, so we have
>
> PYTHON=/usr/bin/python3.4
>
> and HAVE_PYTHON3 being defined / true.
>
> So what about 'ln -s /usr/bin/python3.4 /usr/bin/python3' ?
>
> Meanwhile I check how we can make use of $PYTHON in the shebang line.

Here is some more information. I'm not a Python guy so I don't mess
with it. It is in a default state from Oracle.

$ command -v python3
$

$ ls /opt/csw/bin/py*
/opt/csw/bin/pydoc   /opt/csw/bin/python-config
/opt/csw/bin/pydoc2  /opt/csw/bin/python2
/opt/csw/bin/pydoc2.6/opt/csw/bin/python2-config
/opt/csw/bin/python  /opt/csw/bin/python2.6

$ ls /bin/py*
/bin/pydoc/bin/python-config
/bin/pydoc-2.6/bin/python2.6
/bin/pydoc-2.7/bin/python2.6-config
/bin/pydoc3.4 /bin/python2.7
/bin/pygobject-codegen-2.0/bin/python2.7-config
/bin/pygobject27-codegen-2.0  /bin/python3.4
/bin/pygtk-codegen-2.0/bin/python3.4-config
/bin/pygtk27-codegen-2.0  /bin/python3.4m
/bin/pyrexc   /bin/python3.4m-config
/bin/pyrexc2.6/bin/pyvenv
/bin/pyrexc2.7/bin/pyvenv-3.4
/bin/python

I think the takeaway is, use 'command -v python3' to test for the
presence of Python. As far as I know it is POSIX and should work
everywhere.

I know another option is to try to configure Python. But I am not real
happy with Python breaking basic I/O and I don't want to try to unfuck
all the errors it could cause. I don't have enough experience to fix
all of the errors. I'd rather leave it unconfigured.

Jeff

Re: [Bug-wget] FAIL: 6

[Jeffrey Walton]

On Fri, Apr 19, 2019 at 11:44 AM Tim Rühsen  wrote:
>
> we had this issue coming up several times now. I guess, it's that your
> Daemon.pm can't handle IPv6. Debian got a patch/update several weeks
> ago. Maybe Fedora 29 doesn't have it (yet).

Thanks Tim.

It looks like Perl took an update on Fedora. perl-IO-Socket-SSL got an
update. I'll be re-testing soon.

I filed a bug report at
https://bugzilla.redhat.com/show_bug.cgi?id=1701797, but it was pretty
shitty. I don't know the internals Wget testing and I don't know a
damn thing about Perl, so it was awful. I hope one of the maintainers
recognizes the problem and remembered what they did on Debian/Ubuntu.

$ sudo ~/do-update.sh
[sudo] password for jwalton:
Last metadata expiration check: 0:02:36 ago on Wed 24 Apr 2019 12:08:02 AM EDT.
Dependencies resolved.

 Package   ArchVersion  Repository Size

Installing:
 kernelx86_64  5.0.8-200.fc29   updates30 k
 kernel-core   x86_64  5.0.8-200.fc29   updates25 M
 kernel-modulesx86_64  5.0.8-200.fc29   updates28 M
 kernel-modules-extra  x86_64  5.0.8-200.fc29   updates   2.1 M
Upgrading:
 opus  x86_64  1.3.1-1.fc29 updates   209 k
 perl-IO-Socket-SSLnoarch  2.066-1.fc29 updates   245 k
 python3-urllib3   noarch  1.24.2-1.fc29updates   170 k
Removing:
 kernelx86_64  5.0.5-200.fc29   @updates0
 kernel-core   x86_64  5.0.5-200.fc29   @updates   61 M
 kernel-modulesx86_64  5.0.5-200.fc29   @updates   28 M
 kernel-modules-extra  x86_64  5.0.5-200.fc29   @updates  2.1 M

Jeff

Re: [Bug-wget] FAIL: 6

[Jeffrey Walton]

On Wed, Apr 24, 2019 at 12:19 AM Jeffrey Walton  wrote:
>
> On Fri, Apr 19, 2019 at 11:44 AM Tim Rühsen  wrote:
> >
> > we had this issue coming up several times now. I guess, it's that your
> > Daemon.pm can't handle IPv6. Debian got a patch/update several weeks
> > ago. Maybe Fedora 29 doesn't have it (yet).
>
> Thanks Tim.
>
> It looks like Perl took an update on Fedora. perl-IO-Socket-SSL got an
> update. I'll be re-testing soon.

Yep, that update cleared the issue.

Thanks again for your help.

Jeff

[Bug-wget] OpenSSL

[Jeffrey Walton]

Hi Everyone,

I'm testing a new configuration. Wget 1.20.3 and OpenSSL 1.1.1d. One
minor warning:

gcc -DHAVE_CONFIG_H
-DSYSTEM_WGETRC=\"/home/jwalton/bootstrap/etc/wgetrc\"
-DLOCALEDIR=\"/home/jwalton/bootstrap/share/locale\" -I.  -I../lib
-I../lib -I/home/jwalton/bootstrap/include
-I/home/jwalton/bootstrap/include -DHAVE_LIBSSL -DNDEBUG   -MT
version.o -MD -MP -MF .deps/version.Tpo -c -o version.o version.c
mv -f .deps/version.Tpo .deps/version.Po
openssl.c: In function 'ssl_init':
openssl.c:178:7: warning: 'OPENSSL_config' is deprecated
[-Wdeprecated-declarations]
   OPENSSL_config (NULL);
   ^~
In file included from /home/jwalton/bootstrap/include/openssl/e_os2.h:13:0,
 from /home/jwalton/bootstrap/include/openssl/ssl.h:15,
 from openssl.c:40:
/home/jwalton/bootstrap/include/openssl/conf.h:91:25: note: declared here
 DEPRECATEDIN_1_1_0(void OPENSSL_config(const char *config_name))
 ^
/home/jwalton/bootstrap/include/openssl/opensslconf.h:127:37: note: in
definition of macro 'DECLARE_DEPRECATED'
 #   define DECLARE_DEPRECATED(f)f __attribute__ ((deprecated));
 ^
/home/jwalton/bootstrap/include/openssl/conf.h:91:1: note: in
expansion of macro 'DEPRECATEDIN_1_1_0'
 DEPRECATEDIN_1_1_0(void OPENSSL_config(const char *config_name))
 ^~

Wget2 and ssl_openssl.c:93: thread-local storage not supported for this target

[Jeffrey Walton]

Hi Everyone,

I cut-over from OpenSSL 1.0.2 to OpenSSL 1.1.1 recently. Building
Wget2 1.99.2 on an old PowerMac results in:

ssl_openssl.c:93: error: thread-local storage not supported for this target
ssl_openssl.c:94: error: thread-local storage not supported for this target
make[2]: *** [libwget_la-ssl_openssl.lo] Error 1
make[2]: *** Waiting for unfinished jobs

I've got a feeling Wget2 will eventually build here. I noticed
1.99.2's ssl_openssl.c has this:

static int _init;
static __thread int _ex_data_idx;
static __thread CRYPTO_EX_DATA _crypto_ex_data;
static wget_thread_mutex _mutex;

While Master ssl_openssl.c has this:

static int init;
static wget_thread_mutex mutex;

static SSL_CTX *_ctx;
static int store_userdata_idx;

Is there a work around that can be used until the next Wget2 is released?

Jeff

OPENSSL_config deprecation warning

[Jeffrey Walton]

Hi Everyone,

I'm building Wget 1.20.3 from the source tarball against OpenSSL 1.1.1d.

I'm catching a warning that looks like low hanging fruit:

gcc -DHAVE_CONFIG_H
-DSYSTEM_WGETRC=\"/home/jwalton/tmp/build-test/etc/wgetrc\"
-DLOCALEDIR=\"/home/jwalton/tmp/build-test/share/locale\" -I.
-I../lib -I../lib -I/home/jwalton/tmp/build-test/include -DNDEBUG
-I/home/jwalton/tmp/build-test/include
-I/home/jwalton/tmp/build-test/include
-I/home/jwalton/tmp/build-test/include
-I/home/jwalton/tmp/build-test/include -DNDEBUG -g2 -O2 -march=native
-fPIC -pthread -MT version.o -MD -MP -MF .deps/version.Tpo -c -o
version.o version.c
openssl.c: In function 'ssl_init':
openssl.c:178:7: warning: 'OPENSSL_config' is deprecated
[-Wdeprecated-declarations]
   OPENSSL_config (NULL);
   ^~

I believe the call to OPENSSL_config can be guarded:

#include 
...

#if OPENSSL_VERSION_NUMBER < 0x10001000L
OPENSSL_config(NULL);
#else
OPENSSL_init_crypto(OPENSSL_INIT_LOAD_CONFIG |
OPENSSL_INIT_ADD_ALL_CIPHERS |
OPENSSL_INIT_ADD_ALL_DIGESTS);
#endif

Jeff

Wget2 and FAIL: wget_options_fuzzer

[Jeffrey Walton]

Hi Everyone/Tim,

I'm building Wget2 master on Fedora 31. The fuzzer tickled a self test failure:

PASS: libwget_xml_parse_buffer_fuzzer
../build-aux/test-driver: line 107: 246733 Segmentation fault
(core dumped) "$@" > $log_file 2>&1
FAIL: wget_options_fuzzer

Attached are config.log and test-suite.log.
<>
<>

Wget2, OS X 10.12 and two elf test issues

[Jeffrey Walton]

Hi Everyone/Tim,

Here are testing results of Wegt2 master on OS X 10.12. OS X 10.12 has
System Integrity Protection (SIP). It may have something to do with
the failures.

make  check-TESTS
../build-aux/test-driver: line 107: 48775 Abort trap: 6   "$@"
> $log_file 2>&1
FAIL: test
../build-aux/test-driver: line 107: 48795 Abort trap: 6   "$@"
> $log_file 2>&1
FAIL: test-parse-html
PASS: test-cond
PASS: test-decompress
PASS: test-dl

Attached are config.log and test-suite.log.
<>
<>

Wget2 and Error: 'gettext' version == 0.16.1 is too old

[Jeffrey Walton]

Hi Everyone/Tim,

I'm testing Wget2 on Solaris 11.3.

$ git clone ...
Receiving objects: 100% (50791/50791), 11.35 MiB | 4.29 MiB/s, done.
Resolving deltas: 100% (17123/17123), done.
$ cd wget2
$ ./bootstrap
./bootstrap: Error: 'gettext' version == 0.16.1 is too old
./bootstrap:'gettext' version >= 0.18.2 is required
./bootstrap[255]: lzip: not found [No such file or directory]
./bootstrap: Error: 'lzip' not found
./bootstrap: Please install the prerequisite programs

I actually have two copies of Gettext that are usable. One is in
/usr/local, and the other is in $HOME/tmp/ok_to_delete (where I am
testing Wget2).

This is producing the same error:

if ! PKG_CONFIG_PATH="$HOME/tmp/ok_to_delete/lib/pkgconfig" ./bootstrap;
then
echo "Failed to bootstrap Wget2"
exit 1
fi

Is it possible to relax the test until I can run configure using
--with-gettext-prefix or setup a PKG_CONFIG_PATH?

Jeff

Wget2 result on OS X 10.9

[Jeffrey Walton]

Hi Everyone,

I have some testing results for OS X 10.9. This jumped out at me. It
is fairly prevalent.

(I'm going to enable self tests next).

Jeff

In file included from websequencediagram_high.c:28:
../include/wget/wget.h:453:44: warning: unknown attribute '__alloc_size__'
  ignored [-Wattributes]
RETURNS_NONNULL LIBWGET_WARN_UNUSED_RESULT WGET_GCC_ALLOC_SIZE(1) WGET_G...
   ^
../include/wget/wget.h:165:49: note: expanded from macro 'WGET_GCC_ALLOC_SIZE'
#   define WGET_GCC_ALLOC_SIZE(a) __attribute__ ((__alloc_size__(a)))
  ^
../include/wget/wget.h:459:44: warning: unknown attribute '__alloc_size__'
  ignored [-Wattributes]
RETURNS_NONNULL LIBWGET_WARN_UNUSED_RESULT WGET_GCC_ALLOC_SIZE2(1,2) WGE...
   ^
../include/wget/wget.h:166:53: note: expanded from macro 'WGET_GCC_ALLOC_SIZE2'
#   define WGET_GCC_ALLOC_SIZE2(a, b) __attribute__ ((__alloc_size__(a, b)))
  ^
../include/wget/wget.h:465:44: warning: unknown attribute '__alloc_size__'
  ignored [-Wattributes]
RETURNS_NONNULL LIBWGET_WARN_UNUSED_RESULT WGET_GCC_ALLOC_SIZE(2)
   ^
../include/wget/wget.h:165:49: note: expanded from macro 'WGET_GCC_ALLOC_SIZE'
#   define WGET_GCC_ALLOC_SIZE(a) __attribute__ ((__alloc_size__(a)))
  ^
../include/wget/wget.h:477:28: warning: unknown attribute '__alloc_size__'
  ignored [-Wattributes]
LIBWGET_WARN_UNUSED_RESULT WGET_GCC_ALLOC_SIZE(2)
   ^
../include/wget/wget.h:165:49: note: expanded from macro 'WGET_GCC_ALLOC_SIZE'
#   define WGET_GCC_ALLOC_SIZE(a) __attribute__ ((__alloc_size__(a)))
  ^
../include/wget/wget.h:485:28: warning: unknown attribute '__alloc_size__'
  ignored [-Wattributes]
LIBWGET_WARN_UNUSED_RESULT WGET_GCC_ALLOC_SIZE(2)
   ^
../include/wget/wget.h:165:49: note: expanded from macro 'WGET_GCC_ALLOC_SIZE'
#   define WGET_GCC_ALLOC_SIZE(a) __attribute__ ((__alloc_size__(a)))
  ^
../include/wget/wget.h:558:49: warning: unknown attribute '__alloc_size__'
  ignored [-Wattributes]
wget_buffer_alloc(size_t size) WGET_GCC_MALLOC WGET_GCC_ALLOC_SI...
   ^
../include/wget/wget.h:165:49: note: expanded from macro 'WGET_GCC_ALLOC_SIZE'
#   define WGET_GCC_ALLOC_SIZE(a) __attribute__ ((__alloc_size__(a)))
  ^
In file included from check_url_types.c:30:
../include/wget/wget.h:453:44: warning: unknown attribute '__alloc_size__'
  ignored [-Wattributes]
RETURNS_NONNULL LIBWGET_WARN_UNUSED_RESULT WGET_GCC_ALLOC_SIZE(1) WGET_G...
   ^
../include/wget/wget.h:165:49: note: expanded from macro 'WGET_GCC_ALLOC_SIZE'
#   define WGET_GCC_ALLOC_SIZE(a) __attribute__ ((__alloc_size__(a)))
  ^
../include/wget/wget.h:459:44: warning: unknown attribute '__alloc_size__'
  ignored [-Wattributes]
RETURNS_NONNULL LIBWGET_WARN_UNUSED_RESULT WGET_GCC_ALLOC_SIZE2(1,2) WGE...
   ^
../include/wget/wget.h:166:53: note: expanded from macro 'WGET_GCC_ALLOC_SIZE2'
#   define WGET_GCC_ALLOC_SIZE2(a, b) __attribute__ ((__alloc_size__(a, b)))
  ^
../include/wget/wget.h:465:44: warning: unknown attribute '__alloc_size__'
  ignored [-Wattributes]
RETURNS_NONNULL LIBWGET_WARN_UNUSED_RESULT WGET_GCC_ALLOC_SIZE(2)
   ^
../include/wget/wget.h:165:49: note: expanded from macro 'WGET_GCC_ALLOC_SIZE'
#   define WGET_GCC_ALLOC_SIZE(a) __attribute__ ((__alloc_size__(a)))
  ^
../include/wget/wget.h:477:28: warning: unknown attribute '__alloc_size__'
  ignored [-Wattributes]
LIBWGET_WARN_UNUSED_RESULT WGET_GCC_ALLOC_SIZE(2)
   ^
../include/wget/wget.h:165:49: note: expanded from macro 'WGET_GCC_ALLOC_SIZE'
#   define WGET_GCC_ALLOC_SIZE(a) __attribute__ ((__alloc_size__(a)))
  ^
../include/wget/wget.h:485:28: warning: unknown attribute '__alloc_size__'
  ignored [-Wattributes]
LIBWGET_WARN_UNUSED_RESULT WGET_GCC_ALLOC_SIZE(2)
   ^
../include/wget/wget.h:165:49: note: expanded from macro 'WGET_GCC_ALLOC_SIZE'
#   define WGET_GCC_ALLOC_SIZE(a) __attribute__ ((__alloc_size__(a)))
  ^
../include/wget/wget.h:558:49: warning: unknown attribute '__alloc_size__'
  ignored [-Wattributes]

Re: Wget2 and Error: 'gettext' version == 0.16.1 is too old

[Jeffrey Walton]

On Mon, Mar 30, 2020 at 1:13 AM  wrote:
>
> Hi Jeffrey,
>
> For fixing your issue with gettext, you want to set the PATH variable, not 
> PKG_CONFIG_PATH.
>
> Try this:
>
> export PATH=$HOME/tmp/ok_to_delete:$PATH
> ./bootstrap
>
> You might need to set PKG_CONFIG_PATH for libraries, but the gettext binary 
> will be found via the PATH variable.
>
> For the failure due to lzip, you could simply remove the dependency by 
> deleting the relevant line in bootstrap.conf. Lzip is only needed for 
> building a distribution tarball so it's not really a dependency.

Thanks. Most distros provide a Zip program, including OS X and Solaris:

$ command -v zip
/usr/bin/zip

I guess there's something else wrong with it?

Is it possible to relax the Autotools requirement:

running: AUTOPOINT=true LIBTOOLIZE=true autoreconf --verbose --install
--force -I m4  --no-recursive
autoreconf: Entering directory `.'
autoreconf: running: true --force
autoreconf: running: aclocal -I m4 --force -I m4 ${ACLOCAL_FLAGS}
/usr/share/aclocal/aalib.m4:12: warning: underquoted definition of AM_PATH_AALIB
/usr/share/aclocal/aalib.m4:12:   run info '(automake)Extending aclocal'
/usr/share/aclocal/aalib.m4:12:   or see
http://www.gnu.org/software/automake/manual/automake.html#Extending-aclocal
configure.ac:22: error: Autoconf version 2.69 or higher is required
configure.ac:22: the top level
autom4te: /usr/bin/gm4 failed with exit status: 63
aclocal: /usr/bin/autom4te failed with exit status: 63
autoreconf: aclocal failed with exit status: 63
./bootstrap: autoreconf failed

Solaris provides:

$ autoconf --version
autoconf (GNU Autoconf) 2.68

Re: Wget2 and Error: 'gettext' version == 0.16.1 is too old

[Jeffrey Walton]

On Mon, Mar 30, 2020 at 3:02 AM Tim Rühsen  wrote:
>
> Hi Jeff,
>
> On 3/30/20 8:37 AM, Jeffrey Walton wrote:
> > ...
> > Is it possible to relax the Autotools requirement:
> >
> > running: AUTOPOINT=true LIBTOOLIZE=true autoreconf --verbose --install
> > --force -I m4  --no-recursive
> > autoreconf: Entering directory `.'
> > autoreconf: running: true --force
> > autoreconf: running: aclocal -I m4 --force -I m4 ${ACLOCAL_FLAGS}
> > /usr/share/aclocal/aalib.m4:12: warning: underquoted definition of 
> > AM_PATH_AALIB
> > /usr/share/aclocal/aalib.m4:12:   run info '(automake)Extending aclocal'
> > /usr/share/aclocal/aalib.m4:12:   or see
> > http://www.gnu.org/software/automake/manual/automake.html#Extending-aclocal
> > configure.ac:22: error: Autoconf version 2.69 or higher is required
> > configure.ac:22: the top level
> > autom4te: /usr/bin/gm4 failed with exit status: 63
> > aclocal: /usr/bin/autom4te failed with exit status: 63
> > autoreconf: aclocal failed with exit status: 63
> > ./bootstrap: autoreconf failed
> >
> > Solaris provides:
> >
> >  $ autoconf --version
> >  autoconf (GNU Autoconf) 2.68
>
> AFAICS, Solaris doesn't come with any autotools by default. Someone
> installed them on your Solaris machine. Autotools is a pure developer
> software and should be kept up-to-date.

Yeah, it is not installed by default.

pkg install autoconf automake libtool

gets you the older version on Solaris. They want folks to buy a
service contract to get the latest tools. Ransomware...

That's one of the reasons I try to support Solaris so well. I want to
poke Oracle in the eye for ransoming free software . Every time we
make something available for free, we take money they earn on our hard
work out of their pockets.

Jeff

Doxygen docs

[Jeffrey Walton]

Hi Everyone/Tim,

This is from Wget2 master on Fedora 31. It looks like Doxygen.

If you are using Doxygen, then a nice option to detect Doxygen issues
is Clang's -Wdocumentation. Set CC=clang, and
CFLAGS="-Wdocumentation". It does a nice job of catching mistakes,
like missing parameters or undocumented parameters. I found it is a
little nicer then the doxygen tool.

Generating code for file wget.h...
/home/wget2/include/wget/wget.h:1499: warning: unable to resolve
reference to `wget_hpkp_db_free' for \ref command
/home/wget2/include/wget/wget.h:1503: warning: unable to resolve
reference to `wget_hpkp_db_free' for \ref command
/home/wget2/include/wget/wget.h:1505: warning: unable to resolve
reference to `wget_hpkp_db_check_pubkey' for \ref command
/home/wget2/include/wget/wget.h:1501: warning: unable to resolve
reference to `wget_hpkp_db_free' for \ref command
/home/wget2/include/wget/wget.h:1509: warning: unable to resolve
reference to `wget_hpkp_db_load' for \ref command
/home/wget2/include/wget/wget.h:1507: warning: unable to resolve
reference to `wget_hpkp_db_add' for \ref command
/home/wget2/include/wget/wget.h:1511: warning: unable to resolve
reference to `wget_hpkp_db_save' for \ref command

Wget2 and one failed self test

[Jeffrey Walton]

Hi Everyone/Tim,

I'm catching one failed self test building Wget2 master on Fedora 32:

PASS: libwget_tlssess_fuzzer
PASS: libwget_utils_fuzzer
PASS: libwget_xml_parse_buffer_fuzzer
../build-aux/test-driver: line 107: 613669
Aborted (core dumped) "$@" > $log_file 2>&1
FAIL: wget_options_fuzzer

And:

FAIL: wget_options_fuzzer
=

testing 12 bytes from
'/home/jwalton/Build-Scripts/wget2/fuzz/wget_options_fuzzer.in/25bfdf3efa19822ed9f2174c083d882f5b737e43'
double free or corruption (out)
FAIL wget_options_fuzzer (exit status: 134)

Jeff

Re: [bug #58354] Wget doesn't parse URIs starting with http:/

[Jeffrey Walton]

On Tue, May 12, 2020 at 6:45 AM Luca Bernardi  wrote:
>
> URL:
>   
>
>  Summary: Wget doesn't parse URIs starting with http:/
>  Project: GNU Wget
> Submitted by: f0ff
> Submitted on: Tue 12 May 2020 10:45:17 AM UTC
> Category: None
> Severity: 3 - Normal
> Priority: 5 - Normal
>   Status: None
>  Privacy: Public
>  Assigned to: None
>  Originator Name:
> Originator Email:
>  Open/Closed: Open
>  Release: 1.14
>  Discussion Lock: Any
> Operating System: GNU/Linux
>  Reproducibility: Every Time
>Fixed Release: None
>  Planned Release: None
>   Regression: None
>Work Required: None
>   Patch Included: No
>
> ___
>
> Details:
>
> Hi,
> Wget refuses to parse URIs that start with http:/ (note single slash), e.g.
> http:/wp-includes/css/dist/block-library/style.min.css?ver=5.4.1. These are
> widely accepted by browsers.
>
> Command that I've used: `wget --user-agent=Mozilla --content-disposition
> --page-requisites --adjust-extension --restrict-file-names=windows -d -e
> robots=off -m -k -E -r -l 10 -p -N -F -P crawl  -nH $IP`

You may as well make the slashes optional in the protocol string.
Berners Lee does not like them anyway,
https://www.mentalfloss.com/uk/history/27802/10-inventors-who-came-to-regret-their-creations.

Jeff

Re: Undefined reference to gnutls_protocol_set_priority() when compiling latest wget version

[Jeffrey Walton]

On Fri, May 15, 2020 at 1:09 PM Stephen Kirby  wrote:
>  ...
> on the emulated phone, I am trying:
>
> wget -O filename http://###.##.###.## (i.e., here I use the IP address
> found via nslookup on the named URL)
>
> Then, I get:
> HTTP request sent, awaiting response... 302 object moved
> Location: https://(here it lists the correctly named URL)
> Resolving (named URL)... Failed: Name or Server not known
> wget: unable to resolve host address "named URL"

Verbose output is On by default. Maybe try Debug output with -d or
--debug. Also see
https://www.gnu.org/software/wget/manual/html_node/Logging-and-Input-File-Options.html.

Jeff

Wget2 and '...+digest -https -ssl +ipv6...' on OS X

[Jeffrey Walton]

Hi Everyone/Tim,

I noticed something is sideways on my Wget2 recipe on OS X 10.12. I
configure with --with-openssl=yes and --with-ssl=openssl, but Wget2
lacks TLS support:

$ ~/tmp/ok2delete/bin/wget2 --version
GNU Wget2 1.99.2 - multithreaded metalink/file/website downloader
+digest -https -ssl +ipv6 +iri +large-file +nls -ntlm -opie +psl -hsts
+iconv +idn2 +zlib +lzma -brotlidec -zstd +bzip2 -lzip -http2 -gpgme

config.log shows positive test results for the OpenSSL gear. The build
log shows linking to OpenSSL (though it seems to be through Gnulib).

Wget2 and HTTPS seem to work as expected:

$ ~/tmp/ok2delete/bin/wget2 https://www.google.com
OCSP URI not given and not found in certificate. Skipping OCSP check for cert 0.
OCSP URI not given and not found in certificate. Skipping OCSP check for cert 1.
[0] Downloading 'https://www.google.com' ...
Saving 'index.html'
HTTP response 200 OK [https://www.google.com]

Any ideas why I'm having trouble with TLS?

==

Here are the summary options.

Summary of build options:

  Version:1.99.2
  Host OS:darwin16.7.0
  Install prefix: /Users/jwalton/tmp/ok2delete
  Compiler:   clang
  CFlags: -I/Users/jwalton/tmp/ok2delete/include
-I/Users/jwalton/tmp/ok2delete/include
-I/Users/jwalton/tmp/ok2delete/include
-I/Users/jwalton/tmp/ok2delete/include
-I/Users/jwalton/tmp/ok2delete/include
-I/Users/jwalton/tmp/ok2delete/include -DNDEBUG -g2 -O2 -march=native
-fPIC -pthread -I/Users/jwalton/tmp/ok2delete/include -DNDEBUG
  LDFlags:-L/Users/jwalton/tmp/ok2delete/lib
-Wl,-rpath,@loader_path/../lib
-Wl,-rpath,/Users/jwalton/tmp/ok2delete/lib
  Libs:   -L/Users/jwalton/tmp/ok2delete/lib -lpcre2-8
-L/Users/jwalton/tmp/ok2delete/lib -lidn2
-L/Users/jwalton/tmp/ok2delete/lib -llzma
-L/Users/jwalton/tmp/ok2delete/lib -lz -lbz2
-L/Users/jwalton/tmp/ok2delete/lib -lpsl
-L/Users/jwalton/tmp/ok2delete/lib -lssl -lcrypto -ldl -lpthread
  Library types:  shared=yes, static=yes
  Small libraries:yes
  SSL/TLS support:openssl
  GZIP compression:   yes
  BZIP2 compression:  yes
  LZMA compression:   yes
  Brotli compression: no
  Zstd compression:   no
  Lzip compression:   no
  IDNA support:   IDNA 2008 (libidn2)
  PSL support:yes
  HSTS support:   no
  HTTP/2.0 support:   no
  Documentation:  no (neither Doxygen nor Pandoc found)
  Wget2 docs:
  Libwget docs:
  PCRE support:   yes, via libpcre2
  Tests:  Valgrind testing not enabled
  Assertions: no
  POSIX xattr:yes
  Microhttpd support: no (CFLAGS:
-I/Users/jwalton/tmp/ok2delete/include -I/usr/local/include
-I/usr/local/include/p11-kit-1. LIBS:  -L/usr/local/lib -lgnutls)
  Fuzzing build:  no,
  GPGME:  no
<>

Re: Wget fails inside a docker container

[Jeffrey Walton]

On Mon, Aug 10, 2020 at 11:13 AM Matti Kamarainen  wrote:
>
> not sure if this email address is right for asking this kind of questions,
> but I hope so.
>
> I'm trying to download data inside an Ubuntu 18.04 docker container using
> wget. Mostly it works fine, but one specific file can't be downloaded:
> "wget
> ftp://ftp.cpc.ncep.noaa.gov/precip/PEOPLE/wd52ws/global_temp/CPC_GLOBAL_T_V0.x_0.5deg.lnx.2020;
> fails. The download starts, but the progress bar does not advance.
>
> Outside the docker the download of this particular file works fine.
>
> What might be the issue?

Use debug mode. --debug,
https://www.gnu.org/software/wget/manual/html_node/Logging-and-Input-File-Options.html

Wget2 bootstrap and "Unknown option: --"

[Jeffrey Walton]

Hi Everyone/Tim,

Are you guys interested in this? It's from a Ubuntu 4 system. Python
is there, but it is antique.

What I have found is, the best way to check Python's version is
'python -V'. It works on old and new Python.

Jeff



Unknown option: --
usage: python [option] ... [-c cmd | file | -] [arg] ...
Try `python -h' for more information.
./bootstrap: Error: 'python' not found

./bootstrap: Please install the prerequisite programs



And:

$ python --version
Unknown option: --

$ python -V
Python 2.3.4

Re: Wget2 bootstrap and "Unknown option: --"

[Jeffrey Walton]

On Mon, Aug 10, 2020 at 1:02 PM Darshit Shah  wrote:
>
> Interesting.. Thanks for the report.
>
> However, it may make sense to keep this behavior since I don't think
> that the script is compatible with Python 2. I'll have to check it
> again. I will likely change the behaviour to explicitly check for Python
> 3 and fall back to the bourne shell script instead.

This patch will get to the "Python 2.3.4 is too old" message.

* https://github.com/noloader/Build-Scripts/blob/master/patch/wget2.patch

Jeff

Wget2 and FAIL: wget_options_fuzzer

[Jeffrey Walton]

Hi Everyone/Tim,

I knocked the dust off the Wget2 script. I'm catching one failed self test:

PASS: libwget_atom_url_fuzzer
PASS: libwget_bar_fuzzer
PASS: libwget_base64_fuzzer
PASS: libwget_cookie_fuzzer
PASS: libwget_css_url_fuzzer
PASS: libwget_hpkp_fuzzer
PASS: libwget_hsts_fuzzer
PASS: libwget_html_url_fuzzer
PASS: libwget_http_client_fuzzer
PASS: libwget_http_parse_fuzzer
PASS: libwget_iri_fuzzer
PASS: libwget_metalink_parse_fuzzer
PASS: libwget_netrc_fuzzer
PASS: libwget_ocsp_fuzzer
PASS: libwget_robots_parse_fuzzer
PASS: libwget_sitemap_url_fuzzer
PASS: libwget_tlssess_fuzzer
PASS: libwget_utils_fuzzer
PASS: libwget_xml_parse_buffer_fuzzer
../build-aux/test-driver: line 107: 20822 Aborted
(core dumped) "$@" > $log_file 2>&1
FAIL: wget_options_fuzzer

Attached in test-suite.log.

Jeff
<>

Wget2 fuzzer crash on ODROID XU4

[Jeffrey Walton]

Hi Everyone/Tim,

Here's another crash on the fuzzer. This came from an ODROID XU4.

Here's the text from the log file in case I screw up the attachment again.

FAIL: wget_options_fuzzer
=

testing 7 bytes from
'/home/jwalton/wget2/fuzz/wget_options_fuzzer.in/c692273deb2772da307ffe37041fef77bf4baa97'
GNU Wget2 1.99.2 - multithreaded metalink/file/website downloader

+digest -https -ssl +ipv6 +iri +large-file +nls -ntlm -opie +psl -hsts
+iconv +idn2 +zlib +lzma -brotlidec -zstd +bzip2 -lzip -http2 -gpgme

Jeff
<>
<>

Re: Wget2 fuzzer crash on ODROID XU4

[Jeffrey Walton]

On Mon, Jun 22, 2020 at 2:10 PM Jeffrey Walton  wrote:
>
> Hi Everyone/Tim,
>
> Here's another crash on the fuzzer. This came from an ODROID XU4.
>
> Here's the text from the log file in case I screw up the attachment again.
>
> FAIL: wget_options_fuzzer
> =
>
> testing 7 bytes from
> '/home/jwalton/wget2/fuzz/wget_options_fuzzer.in/c692273deb2772da307ffe37041fef77bf4baa97'
> GNU Wget2 1.99.2 - multithreaded metalink/file/website downloader
>
> +digest -https -ssl +ipv6 +iri +large-file +nls -ntlm -opie +psl -hsts
> +iconv +idn2 +zlib +lzma -brotlidec -zstd +bzip2 -lzip -http2 -gpgme

I think I managed to get a backtrace out of it, but I am not sure how
good it is.

$ ../libtool --mode=execute gdb wget_options_fuzzer
GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from
/home/jwalton/Build-Scripts/wget2/fuzz/.libs/wget_options_fuzzer...done.
(gdb) r
Starting program:
/home/jwalton/Build-Scripts/wget2/fuzz/.libs/wget_options_fuzzer
Cannot parse expression `.L1207 4@r4'.
warning: Probes-based dynamic linker interface failed.
Reverting to original interface.

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".

Program received signal SIGILL, Illegal instruction.
_armv7_tick () at crypto/armv4cpuid.S:136
136 crypto/armv4cpuid.S: No such file or directory.
(gdb) c
Continuing.
testing 7 bytes from
'/home/jwalton/Build-Scripts/wget2/fuzz/wget_options_fuzzer.in/c692273deb2772da307ffe37041fef77bf4baa97'
GNU Wget2 1.99.2 - multithreaded metalink/file/website downloader

+digest -https -ssl +ipv6 +iri +large-file +nls -ntlm -opie +psl -hsts
+iconv +idn2 +zlib +lzma -brotlidec -zstd +bzip2 -lzip -http2 -gpgme

Copyright (C) 2012-2015 Tim Ruehsen
Copyright (C) 2015-2020 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later
<http://www.gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please send bug reports and questions to .
free(): invalid pointer

Program received signal SIGABRT, Aborted.
__libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:47
47  ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S: No such file
or directory.
(gdb) bt full
#0  __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:47
No locals.
#1  0xb6e4cb32 in __libc_signal_restore_set (set=0xbeffef84)
at ../sysdeps/unix/sysv/linux/nptl-signals.h:80
_a2tmp = -1090523260
_a2 = -1090523260
_nametmp = 175
_a3tmp = 0
_a3 = 0
_a1 = 0
_a4tmp = 8
_a1tmp = 2
_a4 = 8
_name = 175
#2  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:48
set = {__val = {0, 0, 0, 4241216, 320252, 3070228848, 320132,
320140, 3070088761, 320140, 3070229196, 5, 0, 0,
3070228848, 0, 3070228848, 3070229312, 3070229312, 3070204448,
320188, 0, 3070088761, 320196, 4294967295, 5, 3068334024,
3070205888, 0, 32, 3068447921, 3070204888}}
pid = 
tid = 
ret = 
---Type  to continue, or q  to quit---
#3  0xb6e4d82e in __GI_abort () at abort.c:79
save_stage = 1
act = {__sigaction_handler = {sa_handler = 0x1c4,
sa_sigaction = 0x1c4}, sa_mask = {__val = {3069747704, 3070202984,
  320540, 320536, 3069747704, 3070202984, 0, 2275345624,
  3069747704, 3070202984, 3069734728, 71104550, 3069757083,
  3069741960, 320644, 3070224752, 3070226432, 2863311531,
  320536, 320540, 3070198028, 0, 0, 3069751837,
  2275345624, 0, 0, 3069757083, 320740, 3070202984,
  320644, 320652}}, sa_flags = -1090522616,
  sa_restorer = 0xb6ebe057 <__GI___mmap+22>}
sigs = {__val = {32, 0 }}
#4  0xb6e75460 in __libc_message (action=action@entry=do_abort,
fmt=) at ../sysdeps/posix/libc_fatal.c:181
ap = {__ap = 0xbefff244}
fd = 2
list = 
 

Wget2 built ok on DragonFly

[Jeffrey Walton]

Hi Everyone/Tim,

Wget2 built ok on DragonFly. The triplet is x86_64-unknown-dragonfly5.6.

I can't seem to figure out how to install HTTP::Daemon and
HTTP::Request on DragonFly, so the self tests did not run. Sorry about
that.

Jeff

Re: Wget2 and FAIL: wget_options_fuzzer

[Jeffrey Walton]

On Sun, Jun 21, 2020 at 5:16 AM Tim Rühsen  wrote:
>
> Hi Jeff,
>
> you included a GMP config.log - possibly you can add config.log from
> wget2 !? It includes mostly anything to reproduce your system
> environment and CFLAGS.

Damn, sorry about that. It sucks getting old. I miss my mind the most.

Try this one.

Jeff
<>
<>

Wget2 and ignoring return value of 'freopen', declared with attribute warn_unused_result

[Jeffrey Walton]

Hi Everyone/Tim,

I see this on occasion:

libwget_xml_parse_buffer_fuzzer.c: In function 'LLVMFuzzerTestOneInput':
libwget_xml_parse_buffer_fuzzer.c:65:2: warning: ignoring return value
of 'freopen', declared with attribute warn_unused_result
[-Wunused-result]
  freopen("/dev/null", "r", stdin);
  ^~~~

I _think_ the way to handle that nowadays in a portable way is to:

#define WGET_UNUSED(x) ((void)(x))
int ret = freopen("/dev/null", "r", stdin);
WGET_UNUSED(ret);

In the past a cast to void used to work. But it has not been working
lately, with GCC9 or GCC10, and it now generates a warning, too:

(void)freopen("/dev/null", "r", stdin);

The WGET_UNUSED macro is really portable. You can also use it for
function parameters on all the compilers I test. (Unlike, say, a GCC
attribute).

Jeff

Re: Wget2 fuzzer crash on ODROID XU4

[Jeffrey Walton]

On Tue, Jun 23, 2020 at 4:03 PM Tim Rühsen  wrote:
>
> Hi Jeff,
>
> thank you, the backtrace made it clear. The issue was a free() on a
> string literal. A fix has been pushed.
>
> Can you test with your setup from 17.06. ? Hopefully it was the same issue.

Yes, master tested OK.

Jeff

Wget2 and untranslated messages

[Jeffrey Walton]

Hi Everyone/Tim,

On the ODROID XU4 I'm seeing a lot of these:

rm -f et.gmo && /home/jwalton/tmp/ok2delete/bin/msgfmt -c --statistics
--verbose -o et.gmo et.po
rm -f fr.gmo && /home/jwalton/tmp/ok2delete/bin/msgfmt -c --statistics
--verbose -o fr.gmo fr.po
rm -f fi.gmo && /home/jwalton/tmp/ok2delete/bin/msgfmt -c --statistics
--verbose -o fi.gmo fi.po
et.po: 51 translated messages, 15 fuzzy translations, 386 untranslated messages.
fi.po: 49 translated messages, 15 fuzzy translations, 388 untranslated messages.
fr.po: 338 translated messages, 55 fuzzy translations, 59 untranslated messages.

I'm not sure if it matters. They don't really bother me.

Jeff

Re: How to download and archive big website with a wiki and a forum thoroughly and efficiently?

[Jeffrey Walton]

On Sun, Dec 6, 2020 at 9:01 PM Kulagin  wrote:
>
> I'm on Windows. I would like to download http://www.tekkenzaibatsu.com/ which
> has different subdirectories like http://www.tekkenzaibatsu.com/wiki/ and
> http://www.tekkenzaibatsu.com/forums/. The website will be closed later
> this month, as the owner has announced a few days ago.
>
> The http://www.tekkenzaibatsu.com/forums/ will need login support. I
> already have working cookies.txt ready to feed into programs.
>
> I googled this issue and there are 2 main programs that people suggest:
> httrack and wget. I managed to login in both and to start downloading with
> both. The problem with httrack is that it downloads with 20 kb/s speed.
>
> Then I tried wget. After few hours of googling and reading manual I finally
> managed to generate a big-big console command to finally start downloading
> at adequate speed and what I need:
>
> > wget -4 --recursive --page-requisites --adjust-extension --no-clobber
> > --convert-links --random-wait -e robots=off --force-directories
> > --load-cookies tekkenzaibatsu.com_cookies.txt --user-agent='Mozilla/5.0
> > (X11; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0'
> > http://www.tekkenzaibatsu.com/ -R
> > "/editpost.php?action=editpost=,/newreply.php?action=newreply=,/showthread.php?postid=#post,/online.php/,/search.php?s==showresults=,/search.php?s=,/private.php?action=newmessage=,/member2.php?action=addlist=buddy=,/newreply.php?action=newreply=,/newthread.php?action=newthread=*"
>
>
>  The problem with wget is even, though, it will reject all the editpost,
> create new thread, new reply, reply, profile, search and all other
> repeating pages on the forum:
> [image: image.png]
>
> It will still first follow them, download them and only reject them at the
> last stage, it will physically delete all these pages only afterwards(from
> what I read and tested myself by trying to download pages in the -R list).
> The problem with this is that there are 36,095 topics on the forum at the
> moment, with many posts in these topics, which will result in around 50
> pages total, if not many more.
>
> I would also like to preserve all the appended files to the forums posts,
> on wiki pages, in articles and others, as well as all the images that are
> displayed on the website and the forum, which aren't necessarily on the
> same domain as the website(those [img]https://imguuur.com/nMi44[/img] and
> all other). So the whole website, wiki and forums stay pretty much fully
> functional.
>
>
> I read how to mirror a phpDB forum on the archive.org:
> https://www.archiveteam.org/index.php?title=PhpBB
>
>
> And it suggests to use wget, which has the problem I described above, it'll
> take me weeks or months do download these hundreds of thousands of unneeded
> pages, and it could just might be infinite of them using this approach.
>
>
> Is there a way to not follow all these unneded links in wget? Or maybe is
> there some other tool which will allow me to download the website at
> adequate speed(1 page a second should be just fine resulting in only 13
> hours of work(assuming 5 pages total) needed to download the whole
> website) and not download all the other pages that are not needed?
>
>
> Please help and thank you for your attention.

I think the easiest way to grab a copy of the wiki in your case is to
ask the administrator to back it up and then make the download
available to users.

Here is the script we use to backup our wiki. The database is MySQL
and the backup software is Duplicity. The script performs a full
backup every 3 months, and incremental backups otherwise. We send the
backup to a remote server over SCP.

Once you have the backup you can restore select parts into a new
installation you create, like in a VM.

# cat /etc/cron.daily/daily-backup
#!/usr/bin/env bash

# Cleanup the database
mysqlcheck my_wiki --auto-repair --user=mwuser --password=X

# We need to do this because duplicity can't backup running MySQL database
mysqldump --single-transaction --routines --events --triggers
--add-drop-table --extended-insert -u mwuser -h 127.0.0.1
-pfeswecewrukahach my_wiki > /backup/wiki.sql

export PASSPHRASE=Y

duplicity --full-if-older-than 3M --allow-source-mismatch --exclude
/root/.cache --exclude /mnt --exclude /tmp --exclude /proc / --exclude
/var/lib/mysql --exclude=/lost+found --exclude=/dev --exclude=/sys
--exclude=/boot/grub --exclude=/etc/fstab
--exclude=/etc/sysconfig/network-scripts/
--exclude=/etc/udev/rules.d/70-persistent-net.rules
sftp:///wiki_backup

Jeff

Wget2 warnings on 32-bit machine

[Jeffrey Walton]

Hi Everyone/Tim,

I'm testing on 32-bit hardware. It's an old VIA Padlock machine
running Peppermint, which is based on Ubuntu. There are a few type
related warnings that might be of interest.

Jeff

==
http.c: In function 'wget_http_get_response_cb':
http.c:1180:20: warning: format '%lX' expects argument of type 'long
unsigned int', but argument 2 has type 'size_t {aka unsigned int}'
[-Wformat=]
 error_printf(_("Chunk size overflow: %lX\n"), chunk_size);
^
private.h:42:28: note: in definition of macro '_'
 # define _(STRING) gettext(STRING)

==
ssl_openssl.c: In function 'ssl_resume_session':
ssl_openssl.c:1248:11: warning: passing argument 4 of
'wget_tls_session_get' from incompatible pointer type
[-Wincompatible-pointer-types]
, ) == 0
   ^
In file included from ssl_openssl.c:56:0:
../include/wget/wget.h:1569:2: note: expected 'size_t * {aka unsigned
int *}' but argument is of type 'long unsigned int *'
  wget_tls_session_get(const wget_tls_session_db *tls_session_db,
const char *host, void **data, size_t *size);

==
stats_site.c: In function 'print_csv_entry':
stats_site.c:206:71: warning: format '%ld' expects argument of type
'long int', but argument 15 has type 'int64_t {aka long long int}'
[-Wformat=]
  wget_fprintf(_fp, "%llu,%llu,%s,%d,%d,%d,%lld,%lld,%lld,%lld,%d,%d,%ld,%s\n",
 ~~^
 %lld
stats_site.c:209:73:
   doc->initial_response_duration, doc->encoding,
doc->signature_status, doc->last_modified, doc->mime_type);

Re: Timeout never happens with https://www.tesco.com

[Jeffrey Walton]

On Mon, Dec 7, 2020 at 9:29 PM Tom Crane  wrote:
>
> On Mon, 7 Dec 2020, Petr Pisar wrote:
>
> > V??Mon, Dec 07, 2020 at 03:44:32AM +,??Tom Crane napsal(a):
> ...
> >
> > I suspect you use OpenSSL for TLS and wget is missing the fcntl() call from
> > src/openssl.c.
>
> That was it.  Checking the Slackware distro's build script
> http://slackware.osuosl.org/slackware64-current/source/n/wget/wget.SlackBuild
> shows it specifies,
>
> ./configure --with-ssl=openssl

There's also --with-libssl-prefix when OpenSSL is not in a standard
location. I believe the same option is needed when using an updated
GnuTLS.

(I build against OpenSSL because OpenSSL is easier to update. OpenSSL
has fewer dependencies than GnuTLS. I don't know how
--with-libssl-prefix behaves with GnuTLS).

You might need --with-libssl-prefix on a system like OS X. OS X
provides an antique version OpenSSL 0.9.x, so you usually need to
build and install OpenSSL 1.1.x.

For example:

# OS X 10.5 (PowerMac)
$ /usr/bin/openssl version
OpenSSL 0.9.7l 28 Sep 2006

# OS X 10.9
$ /usr/bin/openssl version
OpenSSL 0.9.8zg 14 July 2015

# OS X 10.12
$ /usr/bin/openssl version
OpenSSL 0.9.8zh 14 Jan 2016

Jeff

Bad arg length for Socket::inet_ntoa

[Jeffrey Walton]

Hi Everyone/Tim,

The "Bad arg length for Socket::inet_ntoa" is still going on with
Ubuntu 18.04. It is causing 6 self test failures for Wget 1.20.3. The
count seems to be creeping up.

How many years do they need to fix it?

https://bugs.launchpad.net/ubuntu/+source/libhttp-daemon-perl/+bug/1904907

Jeff

How to enable Fuzz?

[Jeffrey Walton]

Hi Tim/Darshit,

I think I would like to enable the fuzz tests. They appear to be
disabled at the moment:

Making all in fuzz
make[2]: Entering directory '/home/jwalton/Build-Scripts/wget-1.21.1/fuzz'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/jwalton/Build-Scripts/wget-1.21.1/fuzz'

How do I enable fuzz? What needs to be installed?

Jeff

Re: How to enable Fuzz?

[Jeffrey Walton]

On Sat, Jan 9, 2021 at 6:44 AM Tim Rühsen  wrote:
>
> On 09.01.21 12:24, Jeffrey Walton wrote:
> > Hi Tim/Darshit,
> >
> > I think I would like to enable the fuzz tests. They appear to be
> > disabled at the moment:
> >
> >  Making all in fuzz
> >  make[2]: Entering directory 
> > '/home/jwalton/Build-Scripts/wget-1.21.1/fuzz'
> >  make[2]: Nothing to be done for 'all'.
> >  make[2]: Leaving directory 
> > '/home/jwalton/Build-Scripts/wget-1.21.1/fuzz'
> >
> > How do I enable fuzz? What needs to be installed?
>
> I think you want to run the regression tests in fuzz/. `make check`
> should do this.

Do you mean cd into fuzz/ and 'make check', separate from the top
level 'cmake check'? Or should it be included in the top level 'make
check'?

Are there any prereqs to running the fuzzing gear?

Jeff

Re: Wget 1.21 and Alpine Linix 3.12

[Jeffrey Walton]

On Thu, Jan 14, 2021 at 12:57 PM Jeffrey Walton  wrote:
>
> Hi Tim/Darshit,
>
> Alpine Linx 3.12 has a BusyBox implementation of Wget. It is very
> crippled, and it needs a real Wget installed.
>
> I'm trying to build Wget 1.21/OpenSSL 1.0.2. Wget configure is failing at:
>
> checking for MD5 in -lcrypto... no
> configure: error: openssl development library not found for MD5
> Failed to configure Wget
> Bootstrap failed for Wget
>
> Looking at the OpenSSL archive:
>
> $ nm ~/.build-scripts/wget/lib/libcrypto.a | grep MD5 2>/dev/null
> 01f0 T MD5_Final
> 0340 T MD5_Init
> 01e0 T MD5_Transform
>  T MD5_Update
>  R MD5_version
>  T MD5
>
> It looks like MD5 is there.
>
> According to config.log:
>
> OPENSSL_CFLAGS=''
> OPENSSL_LIBS='/home/jwalton/.build-scripts/wget/lib/libssl.a
> /home/jwalton/.build-scripts/wget/lib/libcrypto.a'
>
> But then the libs are not used in the conf test:
>
> configure:30720: checking for MD5 in -lcrypto
> configure:30743: gcc -o conftest -DNDEBUG
> -I/home/jwalton/.build-scripts/wget/include  conftest.c -lcrypto
> -ldl >&5
> /usr/lib/gcc/x86_64-alpine-linux-musl/9.3.0/../../../../x86_64-alpine-linux-musl/bin/ld:
> cannot find -lcrypto
>
> In addition, my LDFLAGS that include -L
> /home/jwalton/.build-scripts/wget/lib are not used.

This problem appears to be LDFLAGS related. Configure is not finding
-lcrypto because OpenSSL is not installed for the system, but it is
installed in $HOME/.../lib.

Jeff

Wget 1.21 and Alpine Linix 3.12

[Jeffrey Walton]

Hi Tim/Darshit,

Alpine Linx 3.12 has a BusyBox implementation of Wget. It is very
crippled, and it needs a real Wget installed.

I'm trying to build Wget 1.21/OpenSSL 1.0.2. Wget configure is failing at:

checking for MD5 in -lcrypto... no
configure: error: openssl development library not found for MD5
Failed to configure Wget
Bootstrap failed for Wget

Looking at the OpenSSL archive:

$ nm ~/.build-scripts/wget/lib/libcrypto.a | grep MD5 2>/dev/null
01f0 T MD5_Final
0340 T MD5_Init
01e0 T MD5_Transform
 T MD5_Update
 R MD5_version
 T MD5

It looks like MD5 is there.

According to config.log:

OPENSSL_CFLAGS=''
OPENSSL_LIBS='/home/jwalton/.build-scripts/wget/lib/libssl.a
/home/jwalton/.build-scripts/wget/lib/libcrypto.a'

But then the libs are not used in the conf test:

configure:30720: checking for MD5 in -lcrypto
configure:30743: gcc -o conftest -DNDEBUG
-I/home/jwalton/.build-scripts/wget/include  conftest.c -lcrypto
-ldl >&5
/usr/lib/gcc/x86_64-alpine-linux-musl/9.3.0/../../../../x86_64-alpine-linux-musl/bin/ld:
cannot find -lcrypto

In addition, my LDFLAGS that include -L
/home/jwalton/.build-scripts/wget/lib are not used.

Jeff

Re: Does wget support SOCKS proxy?

[Jeffrey Walton]

On Wed, Jan 13, 2021 at 5:05 PM Peng Yu  wrote:
>
> Hi,
>
> I don't see where wget support socks proxy or not in the manpage. Does
> wget support SOCKS proxy? If it supports SOCKS, could anybody let me
> know how to use it the command (not by .wgetrc)? Thanks.

Check the Wget manual at gnu.org:
https://www.gnu.org/software/wget/manual/wget.html.

To answer the question, yes Wget supports HTTP proxies. Users seem to
have trouble with lame proxies.

Jeff

File downloads OK, but Wget 1.21 exits failure?

[Jeffrey Walton]

Hi Everyone,

I'm using Wget 1.21 to download the HSTS data file from
https://gitlab.com/rockdaboot/libhsts. It looks like:

if ! wget --debug -O hsts.json --ca-certificate=cacert.pem \
  
'https://raw.github.com/chromium/chromium/master/net/http/transport_security_state_static.json'
then
  echo "Failed to download hsts.json"
  exit 1
fi

The file downloads, but Wget returns 1. I'm not sure what is failing.
ls shows data in hsts.json, and --debug does not show a problem.

Any ideas what might be wrong?

Thanks in advance.

Jeff

==

Here is output from wget --debug:

Setting --output-document (outputdocument) to hsts.json
Setting --ca-certificate (cacertificate) to
/home/jwalton/.build-scripts/cacert/github-ca-zoo.pem
DEBUG output created by Wget 1.21.1 on linux-gnu.

Reading HSTS entries from /home/jwalton/.wget-hsts
--2021-01-17 04:57:40--  http://libhsts-0.1.0.tar.gz/
Resolving libhsts-0.1.0.tar.gz... failed: Name or service not known.
wget: unable to resolve host address 'libhsts-0.1.0.tar.gz'
--2021-01-17 04:57:40--
https://raw.github.com/chromium/chromium/master/net/http/transport_security_state_static.json
Resolving raw.github.com... 151.101.208.133
Caching raw.github.com => 151.101.208.133
Connecting to raw.github.com|151.101.208.133|:443... connected.
Created socket 4.
Releasing 0x563a444c3530 (new refcount 1).
Initiating SSL handshake.
Handshake successful; connected socket 4 to SSL handle 0x563a444c10c0
certificate:
  subject: CN=www.github.com,O=GitHub\\, Inc.,L=San Francisco,ST=California,C=US
  issuer:  CN=DigiCert SHA2 High Assurance Server
CA,OU=www.digicert.com,O=DigiCert Inc,C=US
X509 certificate successfully verified and matches host raw.github.com

---request begin---
GET /chromium/chromium/master/net/http/transport_security_state_static.json
HTTP/1.1
User-Agent: Wget/1.21.1
Accept: */*
Accept-Encoding: identity
Host: raw.github.com
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 301 Moved Permanently
Connection: keep-alive
Content-Length: 0
Location: 
https://raw.githubusercontent.com/chromium/chromium/master/net/http/transport_security_state_static.json
Accept-Ranges: bytes
Date: Sun, 17 Jan 2021 09:57:40 GMT
Via: 1.1 varnish
Age: 0
X-Served-By: cache-ewr18122-EWR
X-Cache: MISS
X-Cache-Hits: 0
Vary: Accept-Encoding
X-Fastly-Request-ID: b7d16eac0d4d01064ed3d77024c35c3cc43c8d4b

---response end---
301 Moved Permanently
Registered socket 4 for persistent reuse.
Location: 
https://raw.githubusercontent.com/chromium/chromium/master/net/http/transport_security_state_static.json
[following]
] done.
--2021-01-17 04:57:40--
https://raw.githubusercontent.com/chromium/chromium/master/net/http/transport_security_state_static.json
Resolving raw.githubusercontent.com... 151.101.208.133
Caching raw.githubusercontent.com => 151.101.208.133
Connecting to raw.githubusercontent.com|151.101.208.133|:443... connected.
Created socket 5.
Releasing 0x563a444d3f30 (new refcount 1).
Initiating SSL handshake.
Handshake successful; connected socket 5 to SSL handle 0x563a444d1fc0
certificate:
  subject: CN=www.github.com,O=GitHub\\, Inc.,L=San Francisco,ST=California,C=US
  issuer:  CN=DigiCert SHA2 High Assurance Server
CA,OU=www.digicert.com,O=DigiCert Inc,C=US
X509 certificate successfully verified and matches host
raw.githubusercontent.com

---request begin---
GET /chromium/chromium/master/net/http/transport_security_state_static.json
HTTP/1.1
User-Agent: Wget/1.21.1
Accept: */*
Accept-Encoding: identity
Host: raw.githubusercontent.com
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 14124436
Cache-Control: max-age=300
Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'; sandbox
Content-Type: text/plain; charset=utf-8
ETag: "979ae28b92411e9ffeaad0d801873fe4d362f5d85eb761f13f11ea63d1de614d"
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Frame-Options: deny
X-XSS-Protection: 1; mode=block
via: 1.1 varnish (Varnish/6.0), 1.1 varnish
X-GitHub-Request-Id: 1192:785A:66AF94:706910:600409DC
Accept-Ranges: bytes
Date: Sun, 17 Jan 2021 09:57:41 GMT
X-Served-By: cache-ewr18151-EWR
X-Cache: MISS, HIT
X-Cache-Hits: 0, 0
X-Timer: S1610877461.043534,VS0,VE0
Vary: Authorization,Accept-Encoding
Access-Control-Allow-Origin: *
X-Fastly-Request-ID: 2ecb3bbf010212d286e50be43ddd38a7c8383b2d
Expires: Sun, 17 Jan 2021 10:02:41 GMT
Source-Age: 57

---response end---
200 OK
Disabling further reuse of socket 4.
Closed 4/SSL 0x563a444c10c0
Registered socket 5 for persistent reuse.
Parsed Strict-Transport-Security max-age = 31536000, includeSubDomains = false
Updated HSTS host: raw.githubusercontent.com:443 (max-age: 31536000,
includeSubdomains: false)
Length: 14124436 (13M) [text/plain]
Saving to: 'hsts.json'

hsts.json   100%[===>]  13.47M  11.8MB/sin 1.1s