Re: [Bug-wget] Regression wget-1.19 .netrc authentication not working

[Tim Rühsen]

On Montag, 6. Februar 2017 16:11:26 CET Axel Reinhold wrote:
> Dear Giuseppe,
> 
> no more .netrc auth for http with 1.19 - this patch worked for me:
> 
> --- src/http.c.orig 2017-02-06 16:03:45.0 +0100
> +++ src/http.c  2017-02-06 16:03:45.0 +0100
> @@ -1900,7 +1900,7 @@
>   *passwd = NULL;
> 
> /* Check for ~/.netrc if none of the above match */
> -  if (opt.netrc && (!user || !passwd))
> +  if (opt.netrc && (!*user || !*passwd))
>   search_netrc (u->host, (const char **) user, (const char **)
> passwd, 0);
> 
> /* We only do "site-wide" authentication with "global" user/password

Thanks Axel,

fix pushed to repo.

Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Tilde issue with recursive download when IRI is enabled and a page uses Shift JIS

[Tim Rühsen]

On Montag, 6. Februar 2017 05:02:57 CET William Prescott wrote:
> Hello,
> 
> I'm encountering a problem when recursively downloading from a website when
> the URL contains a tilde and the page encoding claims to be Shift JIS.
> 
> I've tried both Wget 1.17.1 (from Ubuntu 16.04) and 1.19 (from source,
> with Libidn2 0.16).
> I believe my local character encoding is UTF-8.
> 
> The first page will download okay, but then most pages after it will get the
> tilde converted to "%E2%80%BE" ("‾"), which, as one would expect, doesn't
> work.

Hi William,

reproducable by:

$echo '~'|iconv -f SHIFT-JIS -t utf-8
‾

$echo -n '~'|iconv -f SHIFT-JIS -t utf-8|od -t x1
000 e2 80 be

So this seems not be a Wget issue, but maybe a general character conversion 
issue. Not sure what Wget could do...

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] [PATCH] Add support for --retry-on-http-error

[Tim Rühsen]

Hi Tom,

thanks for your work and for signing the FSF copyright assignment !

Just some smallish things:

- Please amend the commit message to be in GNU style (see 'git log' for 
examples). The easiest for us maintainers is when you create the patch with 
'git format-patch -1' and add it as attachment.

- Please add a space between function name and (

- Please use xfree instead of free

- Could you amend check_retry_on_http_error so that no memory allocation is 
used ? E.g. using a strchrnul () loop to separate the status codes.

- There is a function 'cleanup' in src/init.c to free allocated stuff (if 
DEBUG_MALLOC is defined). Please free opt.retry_on_http_error there.

Regards, Tim


On Montag, 6. Februar 2017 20:42:29 CET Tom Szilagyi wrote:
> Consider given HTTP response codes as non-fatal, transient errors.
> Supply a comma-separated list of 3-digit HTTP response codes as
> argument. Useful to work around special circumstances where retries
> are required, but the server responds with an error code normally not
> retried by Wget. Such errors might be 503 (Service Unavailable) and
> 429 (Too Many Requests). Retries enabled by this option are performed
> subject to the normal retry timing and retry count limitations of
> Wget.
> 
> Using this option is intended to support special use cases only and is
> generally not recommended, as it can force retries even in cases where
> the server is actually trying to decrease its load. Please use it
> wisely and only if you know what you are doing.
> 
> Example use and a starting point for manual testing:
>   wget --retry-on-http-error=429,503 http://httpbin.org/status/503
> ---
>  doc/wget.texi | 15 +++
>  src/http.c| 29 +
>  src/init.c|  1 +
>  src/main.c|  1 +
>  src/options.h |  1 +
>  5 files changed, 47 insertions(+)
> 
> diff --git a/doc/wget.texi b/doc/wget.texi
> index 8e57aaa..00a8d4b 100644
> --- a/doc/wget.texi
> +++ b/doc/wget.texi
> @@ -1718,6 +1718,21 @@ some few obscure servers, which never send HTTP
> authentication challenges, but accept unsolicited auth info, say, in
> addition to form-based authentication.
> 
> +@item --retry-on-http-error=@var{code[,code,...]}
> +Consider given HTTP response codes as non-fatal, transient errors.
> +Supply a comma-separated list of 3-digit HTTP response codes as
> +argument. Useful to work around special circumstances where retries
> +are required, but the server responds with an error code normally not
> +retried by Wget. Such errors might be 503 (Service Unavailable) and
> +429 (Too Many Requests). Retries enabled by this option are performed
> +subject to the normal retry timing and retry count limitations of
> +Wget.
> +
> +Using this option is intended to support special use cases only and is
> +generally not recommended, as it can force retries even in cases where
> +the server is actually trying to decrease its load. Please use it
> +wisely and only if you know what you are doing.
> +
>  @end table
> 
>  @node HTTPS (SSL/TLS) Options, FTP Options, HTTP Options, Invoking
> diff --git a/src/http.c b/src/http.c
> index 3c3c8b2..6822bee 100644
> --- a/src/http.c
> +++ b/src/http.c
> @@ -3982,6 +3982,30 @@ gethttp (const struct url *u, struct url
> *original_url, struct http_stat *hs, return retval;
>  }
> 
> +/* Check whether the supplied HTTP status code is among those
> +   listed for the --retry-on-http-error option. */
> +static bool
> +check_retry_on_http_error (const int statcode)
> +{
> +  if (!opt.retry_on_http_error)
> +return false;
> +
> +  bool ret = false;
> +  char * retry_conf = strdup(opt.retry_on_http_error);
> +  char * tok = strtok(retry_conf, ",");
> +  while (tok)
> +{
> +  if (atoi(tok) == statcode)
> +{
> +  ret = true;
> +  break;
> +}
> +  tok = strtok(NULL, ",");
> +}
> +  free(retry_conf);
> +  return ret;
> +}
> +
>  /* The genuine HTTP loop!  This is the part where the retrieval is
> retried, and retried, and retried, and...  */
>  uerr_t
> @@ -4319,6 +4343,11 @@ http_loop (const struct url *u, struct url
> *original_url, char **newloc, logprintf (LOG_NOTQUIET, _("\
>  Remote file does not exist -- broken link!!!\n"));
>  }
> +  else if (check_retry_on_http_error(hstat.statcode))
> +{
> +  printwhat (count, opt.ntry);
> +  continue;
> +}
>else
>  {
>logprintf (LOG_NOTQUIET, _("%s ERROR %d: %s.\n"),
> diff --git a/src/init.c b/src/init.c
> index 623ef9d..01dcb06 100644
> --- a/src/init.c
> +++ b/src/init.c
> @@ -304,6 +304,7 @@ static const struct {
>{ "restrictfilenames", NULL, 
> cmd_spec_restrict_file_names }, { "retrsymlinks", _symlinks,  
>   cmd_boolean },
>{ "retryconnrefused", _connrefused, cmd_boolean },
> +  { "retryonhttperror", _on_http_error, cmd_string },
>{ "robots",   _robots,cmd_boolean },
>{ 

[Bug-wget] Wget 1.19.1 released

[Tim Rühsen]

Hello,

we are pleased to announce the new version of GNU wget 1.19.1.

This is mostly a bug fix release for 1.19.
One new option --retry-on-http-error has been added.

Many thanks go to all the contributors and list activists !

Contributors (from the git log):
Axel Reinhold
Dagobert Michelsen
Tim Rühsen
Tom Szilagyi
Yousong Zhou
Zhiming Wang


The new version is available for download here:

https://ftp.gnu.org/gnu/wget/wget-1.19.1.tar.gz
https://ftp.gnu.org/gnu/wget/wget-1.19.1.tar.xz

and the GPG detached signatures using the key 0x08302DB6A2670428:

https://ftp.gnu.org/gnu/wget/wget-1.19.1.tar.gz.sig
https://ftp.gnu.org/gnu/wget/wget-1.19.1.tar.xz.sig

To reduce load on the main server, you can use this redirector service
which automatically redirects you to a mirror:

http://ftpmirror.gnu.org/wget/wget-1.19.1.tar.gz
http://ftpmirror.gnu.org/wget/wget-1.19.1.tar.xz


Noteworthy changes:

* Fix bugs, a regression, portability/build issues

* Add new option --retry-on-http-error


Please report any problem you may experience to the bug-wget@gnu.org
mailing list.

For the maintainers of Wget,
Tim

signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] [PATCH] Add support for --retry-on-http-error

[Tim Rühsen]

Hi Tom,

On Freitag, 10. Februar 2017 22:04:31 CET Tom Szilagyi wrote:
> Hi Tim,
> 
> Thank you for the review comments. Please find attached a new version
> of the patch prepared according to your guidance. I found it easier to
> use plain old strchr instead of strchrnul, I hope that is OK.

Applied your patch. Thanks for your contribution !

Regards, Tim

> 
> Thanks,
> Tom
> 
> On Thu, Feb 09, 2017 at 09:38:26PM +0100, Tim Rühsen wrote:
> > Hi Tom,
> > 
> > thanks for your work and for signing the FSF copyright assignment !
> > 
> > Just some smallish things:
> > 
> > - Please amend the commit message to be in GNU style (see 'git log' for
> > examples). The easiest for us maintainers is when you create the patch
> > with
> > 'git format-patch -1' and add it as attachment.
> > 
> > - Please add a space between function name and (
> > 
> > - Please use xfree instead of free
> > 
> > - Could you amend check_retry_on_http_error so that no memory allocation
> > is
> > used ? E.g. using a strchrnul () loop to separate the status codes.
> > 
> > - There is a function 'cleanup' in src/init.c to free allocated stuff (if
> > DEBUG_MALLOC is defined). Please free opt.retry_on_http_error there.
> > 
> > Regards, Tim
> > 
> > On Montag, 6. Februar 2017 20:42:29 CET Tom Szilagyi wrote:
> > > Consider given HTTP response codes as non-fatal, transient errors.
> > > Supply a comma-separated list of 3-digit HTTP response codes as
> > > argument. Useful to work around special circumstances where retries
> > > are required, but the server responds with an error code normally not
> > > retried by Wget. Such errors might be 503 (Service Unavailable) and
> > > 429 (Too Many Requests). Retries enabled by this option are performed
> > > subject to the normal retry timing and retry count limitations of
> > > Wget.
> > > 
> > > Using this option is intended to support special use cases only and is
> > > generally not recommended, as it can force retries even in cases where
> > > the server is actually trying to decrease its load. Please use it
> > > wisely and only if you know what you are doing.
> > > 
> > > Example use and a starting point for manual testing:
> > >   wget --retry-on-http-error=429,503 http://httpbin.org/status/503
> > > 
> > > ---
> > > 
> > >  doc/wget.texi | 15 +++
> > >  src/http.c| 29 +
> > >  src/init.c|  1 +
> > >  src/main.c|  1 +
> > >  src/options.h |  1 +
> > >  5 files changed, 47 insertions(+)
> > > 
> > > diff --git a/doc/wget.texi b/doc/wget.texi
> > > index 8e57aaa..00a8d4b 100644
> > > --- a/doc/wget.texi
> > > +++ b/doc/wget.texi
> > > @@ -1718,6 +1718,21 @@ some few obscure servers, which never send HTTP
> > > authentication challenges, but accept unsolicited auth info, say, in
> > > addition to form-based authentication.
> > > 
> > > +@item --retry-on-http-error=@var{code[,code,...]}
> > > +Consider given HTTP response codes as non-fatal, transient errors.
> > > +Supply a comma-separated list of 3-digit HTTP response codes as
> > > +argument. Useful to work around special circumstances where retries
> > > +are required, but the server responds with an error code normally not
> > > +retried by Wget. Such errors might be 503 (Service Unavailable) and
> > > +429 (Too Many Requests). Retries enabled by this option are performed
> > > +subject to the normal retry timing and retry count limitations of
> > > +Wget.
> > > +
> > > +Using this option is intended to support special use cases only and is
> > > +generally not recommended, as it can force retries even in cases where
> > > +the server is actually trying to decrease its load. Please use it
> > > +wisely and only if you know what you are doing.
> > > +
> > > 
> > >  @end table
> > >  
> > >  @node HTTPS (SSL/TLS) Options, FTP Options, HTTP Options, Invoking
> > > 
> > > diff --git a/src/http.c b/src/http.c
> > > index 3c3c8b2..6822bee 100644
> > > --- a/src/http.c
> > > +++ b/src/http.c
> > > @@ -3982,6 +3982,30 @@ gethttp (const struct url *u, struct url
> > > *original_url, struct http_stat *hs, return retval;
> > > 
> > >  }
> > > 
> > > +/* Check whether the supplied HTTP status code is among those
> > > +   listed for the -

Re: [Bug-wget] Test-504.py sometimes fails on slow machines

[Tim Rühsen]

On Dienstag, 14. Februar 2017 16:20:26 CET Adam Sampson wrote:
> On Tue, Feb 14, 2017 at 04:03:13PM +, Adam Sampson wrote:
> > Your patch compiles, but doesn't fix the broken test. tcpdump traces
> > attached of the test succeeding and failing with the patch applied (it's
> > exactly the same behaviour as before).
> 
> If it helps: here's a patch to the test server to add a delay after
> sending the headers -- this makes it always fail (on the second request)
> for me, even on a fast AMD64 machine.

That helps indeed :-)

I don't love to debug unreadable python code...
from what I see, there is a general problem in the python http server code.

Adding the check below make Test-504 survive. But than the next stop is Test-
cookie-401.py, where we have a similar problem.
except ServerError as se:
print(se.__str__())
+if rule_name == "Response" and 
self.rules[rule_name].response_code == 504:
+return(None, None)
return(content, None)

The solution should be to have a Content-Length + body or no body at all for 
error codes. My python knowledge is not good enough to do this work properly. 
Hope that someone else can jump in.

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Wget 1.19 was't build on MerSDK.

[Tim Rühsen]

On Freitag, 10. Februar 2017 03:41:36 CET lour...@openmailbox.org wrote:
> Hello! I tryed to build a fresh wget release, but got the following
> error:
> 
> http://paste.fedoraproject.org/552005/48668697/
> 
> But when i making package on armv7hl architechture, i don't have
> anything errors:
> 
> https://paste.fedoraproject.org/552004/68684914/
> 
> Could you tell me, which flag i must disable for succes build?

Hi,

your iconv setup seems to be broken.

iconv.h and libiconv are found, but they seem to not work properly.

checking for iconv.h... yes
checking for iconv... yes
checking for working iconv... no

Regards, Tim

signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Wget 1.19 was't build on MerSDK.

[Tim Rühsen]

On Freitag, 10. Februar 2017 08:57:18 CET Tim Rühsen wrote:
> On Freitag, 10. Februar 2017 03:41:36 CET lour...@openmailbox.org wrote:
> > Hello! I tryed to build a fresh wget release, but got the following
> > error:
> > 
> > http://paste.fedoraproject.org/552005/48668697/
> > 
> > But when i making package on armv7hl architechture, i don't have
> > anything errors:
> > 
> > https://paste.fedoraproject.org/552004/68684914/
> > 
> > Could you tell me, which flag i must disable for succes build?
> 
> Hi,
> 
> your iconv setup seems to be broken.
> 
> iconv.h and libiconv are found, but they seem to not work properly.
> 
> checking for iconv.h... yes
> checking for iconv... yes
> checking for working iconv... no

You could have a look into config.log (search for 'working iconv and look why 
the test program failed). If unsure, send config.log here and I'll try to 
figure 
out what is going on.

Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] [PATCH] Implement --keep-badhash to keep Metalink's files with a bad hash

[Tim Rühsen]

On Freitag, 19. August 2016 07:34:42 CEST Matthew White wrote:
> On Thu, 18 Aug 2016 12:50:48 +0200
> 
> Tim Rühsen <tim.rueh...@gmx.de> wrote:
> > On Mittwoch, 17. August 2016 17:55:47 CEST Matthew White wrote:
> > > On Thu, 04 Aug 2016 17:08:20 +0200
> > > 
> > > Tim Ruehsen <tim.rueh...@gmx.de> wrote:
> > > > On Thursday, August 4, 2016 12:40:49 PM CEST Matthew White wrote:
> > > > > On Thu, 04 Aug 2016 12:08:50 +0200
> > > > > 
> > > > > Tim Ruehsen <tim.rueh...@gmx.de> wrote:
> > > > > > On Wednesday, August 3, 2016 2:40:14 PM CEST Matthew White wrote:
> > > > > > > On Wed, 03 Aug 2016 14:05:06 +0200
> > > > > > > 
> > > > > > > Tim Ruehsen <tim.rueh...@gmx.de> wrote:
> > > > > > > > On Tuesday, August 2, 2016 11:27:28 AM CEST Matthew White 
wrote:
> > > > > > > > > On Mon, 01 Aug 2016 16:32:30 +0200
> > > > > > > > > 
> > > > > > > > > Tim Ruehsen <tim.rueh...@gmx.de> wrote:
> > > > > > > > > > On Saturday, July 30, 2016 9:41:48 PM CEST Matthew White
> > 
> > wrote:
> > > > > > > > > > > Hello!
> > > > > > > > > > > 
> > > > > > > > > > > I think that sometimes it could help to keep downloaded
> > > > > > > > > > > Metalink's
> > > > > > > > > > > files
> > > > > > > > > > > which have a bad hash.
> > > > > > > > > > > 
> > > > > > > > > > > The default wget behaviour is to delete such files.
> > > > > > > > > > > 
> > > > > > > > > > > This patch provides a way to keep files which have a bad
> > > > > > > > > > > hash
> > > > > > > > > > > through
> > > > > > > > > > > the
> > > > > > > > > > > option --keep-badhash. It appends the suffix .badhash to
> > > > > > > > > > > the
> > > > > > > > > > > file
> > > > > > > > > > > name,
> > > > > > > > > > > except without overwriting existing files. In the latter
> > > > > > > > > > > case,
> > > > > > > > > > > an
> > > > > > > > > > > unique
> > > > > > > > > > > suffix is appended after .badhash.
> > > > > > > > > > > 
> > > > > > > > > > > I made this patch working on the following branch:
> > > > > > > > > > > master (latest 20cac2c5ab3d63aacfba35fb10878a2d490e2377)
> > > > > > > > > > > git://git.savannah.gnu.org/wget.git
> > > > > > > > > > > 
> > > > > > > > > > > What do you think?
> > > > > > > > > > 
> > > > > > > > > > Hi Matthew,
> > > > > > > > > > 
> > > > > > > > > > good work !
> > > > > > > > > > 
> > > > > > > > > > While your FSF assignment is underway (PM), we can
> > > > > > > > > > continue
> > > > > > > > > > polishing.
> > > > > > > > > > 
> > > > > > > > > > Didn't test your code yet,  but from what I see, there are
> > > > > > > > > > just
> > > > > > > > > > these
> > > > > > > > > > peanuts:
> > > > > > > > > > 
> > > > > > > > > > 1.
> > > > > > > > > > +  bhash = malloc (strlen (name) + strlen (".badhash") +
> > > > > > > > > > 1);
> > > > > > > > > > +  strcat (strcpy (bhash, name), ".badhash");
> > > > > > > > > > 
> > > > > > > > > > Please consider using concat_strings() from util.h.
> > > > > > > > > > And please leave an empty line between var declaration and
> > >

Re: [Bug-wget] Wget tests

[Tim Rühsen]

On Dienstag, 16. August 2016 11:23:04 CEST Dale R. Worley wrote:
> Tim Rühsen <tim.rueh...@gmx.de> writes:
> > We use standard automake tests (search the internet if you are interested
> > in details).
> > 
> > We have the 'legacy' tests/ directory with tests written in Perl. And the
> > 'current' testenv/ directory with tests written in Python.
> > 
> > See tests/Makefile.am resp. testenv/Makefile.am for the list of executed
> > tests.
> That points to part of the problem:  I don't have those files in the Git
> repository I downloaded from savannah.gnu.edu.  ("git clone
> git://git.savannah.gnu.org/wget.git")

Somethin went wrong... try again:

tim@debian:/tmp$ git clone git://git.savannah.gnu.org/wget.git
Klone nach 'wget' ...
remote: Counting objects: 29988, done.
remote: Compressing objects: 100% (5909/5909), done.
remote: Total 29988 (delta 23999), reused 29934 (delta 23968)
Empfange Objekte: 100% (29988/29988), 12.20 MiB | 237.00 KiB/s, Fertig. 

Löse Unterschiede auf: 100% (23999/23999), Fertig.  

Prüfe Konnektivität ... Fertig. 


tim@debian:/tmp$ cd wget

tim@debian:/tmp/wget$ ls -la
 
insgesamt 852   

drwxr-xr-x 15 tim  tim 660 Aug 16 18:34 .
drwxrwxrwt 27 root root580 Aug 16 18:33 ..
-rw-r--r--  1 tim  tim2182 Aug 16 18:34 AUTHORS
-rwxr-xr-x  1 tim  tim   31606 Aug 16 18:34 bootstrap
-rw-r--r--  1 tim  tim2040 Aug 16 18:34 bootstrap.conf
drwxr-xr-x  2 tim  tim  80 Aug 16 18:34 build-aux
-rw-r--r--  1 tim  tim 814 Aug 16 18:34 cfg.mk
-rw-r--r--  1 tim  tim   0 Aug 16 18:34 ChangeLog
-rw-r--r--  1 tim  tim  682941 Aug 16 18:34 ChangeLog-2014-12-10
-rw-r--r--  1 tim  tim   24259 Aug 16 18:34 configure.ac
drwxr-xr-x  2 tim  tim 100 Aug 16 18:34 contrib
-rw-r--r--  1 tim  tim   35147 Aug 16 18:34 COPYING
drwxr-xr-x  2 tim  tim 140 Aug 16 18:34 doc
drwxr-xr-x  8 tim  tim 260 Aug 16 18:34 .git
-rw-r--r--  1 tim  tim1434 Aug 16 18:34 .gitignore
-rw-r--r--  1 tim  tim  75 Aug 16 18:34 .gitmodules
drwxr-xr-x  2 tim  tim  40 Aug 16 18:34 gnulib
drwxr-xr-x  2 tim  tim 140 Aug 16 18:34 m4
-rw-r--r--  1 tim  tim1609 Aug 16 18:34 MAILING-LIST
-rw-r--r--  1 tim  tim2909 Aug 16 18:34 Makefile.am
drwxr-xr-x  2 tim  tim 120 Aug 16 18:34 msdos
-rw-r--r--  1 tim  tim   35669 Aug 16 18:34 NEWS
drwxr-xr-x  2 tim  tim 140 Aug 16 18:34 po
-rw-r--r--  1 tim  tim4064 Aug 16 18:34 README
-rw-r--r--  1 tim  tim   10220 Aug 16 18:34 README.checkout
drwxr-xr-x  2 tim  tim1640 Aug 16 18:34 src
drwxr-xr-x  8 tim  tim1000 Aug 16 18:34 testenv
drwxr-xr-x  3 tim  tim1940 Aug 16 18:34 tests
-rw-r--r--  1 tim  tim 959 Aug 16 18:34 .travis.yml
drwxr-xr-x  2 tim  tim 160 Aug 16 18:34 util
drwxr-xr-x  2 tim  tim 380 Aug 16 18:34 vms
-rw-r--r--  1 tim  tim  11 Aug 16 18:34 .x-sc_po_check
-rw-r--r--  1 tim  tim 133 Aug 16 18:34 .x-sc_trailing_blank

tim@debian:/tmp/wget$ ls -la */Makefile.am
-rw-r--r-- 1 tim tim 3386 Aug 16 18:34 doc/Makefile.am
-rw-r--r-- 1 tim tim 4076 Aug 16 18:34 src/Makefile.am
-rw-r--r-- 1 tim tim 3749 Aug 16 18:34 testenv/Makefile.am
-rw-r--r-- 1 tim tim 5671 Aug 16 18:34 tests/Makefile.am
-rw-r--r-- 1 tim tim 1392 Aug 16 18:34 util/Makefile.am

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] [PATCH v4] (resend) Add --use-askpass=COMMAND support

[Tim Rühsen]

Hi Liam,

thanks, we received the the FSF copyright assignment for Wget.

Can you give me an example of an external program to use with --use-askpass 
(maybe a 'standard' one available on common Linux distros) ?

I still have trouble with:
+  /* Set the end byte to \0, and decrement bytes */
+  tmp[bytes--] = '\0';
+
+  /* Remove a possible new line */
+  while (bytes >= 0 &&
+(tmp[bytes] == '\0' || tmp[bytes] == '\n' || tmp[bytes] == '\r'))
+tmp[bytes--] = '\0';
+
+  *answer = xmemdup (tmp, bytes + 2);

You introduce a buffer overflow by one byte here.

You could write instead e.g.:
while (bytes > 0 &&
(tmp[bytes - 1] == '\0' || tmp[bytes - 1] == '\n' || tmp[bytes - 1] == 
'\r'))
  bytes--;

*answer = xmemdup0(tmp, bytes);

(needs to add xmemdup0 gnulib module to bootstrap.conf)

And if you do that, you can also:
  bytes = read (com[0], tmp, sizeof (tmp));
instead of
  bytes = read (com[0], tmp, sizeof (tmp) - 1);

The patch looks fine otherwise ! Just have to test it with a program - do you 
have something in mind ?

Regards, Tim


On Donnerstag, 1. September 2016 11:22:32 CEST Liam R. Howlett wrote:
> This adds the --use-askpass option which is disabled by default.
> 
> --use-askpass=COMMAND will request the username and password for a given
> URL by executing the external program COMMAND.  If COMMAND is left
> blank, then the external program in the environment variable
> WGET_ASKPASS will be used.  If WGET_ASKPASS is not set then the
> environment variable SSH_ASKPASS is used.  If there is no value set, an
> error is returned.  If an error occurs requesting the username or
> password, wget will exit.
> 
> 
> I am resending this patch because the FSF Contribution Agreement has been
> signed by our legal team.
> 
> Liam R. Howlett (1):
>   Add --use-askpass=COMMAND support
> 
>  bootstrap.conf |   1 +
>  doc/wget.texi  |  17 ++---
>  src/init.c |  44 +++
>  src/main.c | 112
> + src/options.h  | 
>  1 +
>  src/url.c  |   6 
>  src/url.h  |   1 +
>  7 files changed, 178 insertions(+), 4 deletions(-)



signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] [PATCH 09/25] Enforce Metalink file name verification, strip directory if necessary

[Tim Rühsen]

On Montag, 12. September 2016 20:18:30 CEST Eli Zaretskii wrote:
> > From: Tim Ruehsen 
> > Date: Mon, 12 Sep 2016 13:00:32 +0200
> > 
> > > +  char *basename = name;
> > > +
> > > +  while ((name = strstr (basename, "/")))
> > > +basename = name + 1;
> > 
> > Could you use strrchr() ? something like
> > 
> > char *basename = strrchr (name, '/');
> > 
> > if (basename)
> > 
> >   basename += 1;
> > 
> > else
> > 
> >   basename = name;
> 
> I think we want to use ISSEP, no?  Otherwise Windows file names with
> backslashes will misfire.

Good point. What about device names ?

So maybe base_name() from Gnulib module 'dirname' is the right choice !?
See https://www.gnu.org/software/gnulib/manual/html_node/basename.html

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] [PATCH] Patch to change behavior with redirects under --recurse.

[Tim Rühsen]

On Freitag, 7. Oktober 2016 15:40:55 CEST Dale R. Worley wrote:
> Tim Ruehsen  writes:
> > the changes in recur.c are not acceptable. They circumvent too many checks
> > like host-spanning, excludes and even --https-only.
> 
> I suppose it depends on what you consider the semantics to be.
> Generally, I look at it if I've specified to download http://x/y/z and
> http://x/y/z redirects to http://a/b/c, if http://x/y/z passes the tests
> I've specified, then the page should be downloaded; the fact that it's
> redirected to http://a/b/c is incidental.  Most checks *should* be
> circumvented.
> 
> I guess I'd make exceptions for --https-only, which is presumably
> placing a requirement on *how* the pages should be fetched, and probably
> the robots check, as that's a policy statement by the server.

If you become redirected to another host/domain, it is wget policy not to do 
so except the user explicitly states it (--span-host or --domains).

Your case is a redirection within the same domain - which my patch considers 
to be ok (even if that redirection contains an explicitly unwanted path/
component). Even that might be dangerous as a default behavior- that is why I 
want to see some more opinions.

We could add another cli option for fine-tuning here.

Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] [WARNING - NOT VIRUS SCANNED] Re: [PATCH] Patch to change behavior with redirects under --recurse.

[Tim Rühsen]

On Freitag, 7. Oktober 2016 10:57:38 CEST Dale R. Worley wrote:
> Tim Ruehsen  writes:
> > Here is a less invasive patch for review & discussion.
> > 
> > WDYT ?
> 
> It looks OK to me, but I'm not very familiar with the code.  I assume
> that it passes all the tests we've written, which covers the cases that
> I care about.

Yes, it does.

I accidentally pushed the patch today, but if anyone has concerns, it can be 
reverted at any time. But still, please think about possible side effects.

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] [PATCH] Implement --keep-badhash to keep Metalink's files with a bad hash

[Tim Rühsen]

On Mittwoch, 17. August 2016 17:55:47 CEST Matthew White wrote:
> On Thu, 04 Aug 2016 17:08:20 +0200
> 
> Tim Ruehsen  wrote:
> > On Thursday, August 4, 2016 12:40:49 PM CEST Matthew White wrote:
> > > On Thu, 04 Aug 2016 12:08:50 +0200
> > > 
> > > Tim Ruehsen  wrote:
> > > > On Wednesday, August 3, 2016 2:40:14 PM CEST Matthew White wrote:
> > > > > On Wed, 03 Aug 2016 14:05:06 +0200
> > > > > 
> > > > > Tim Ruehsen  wrote:
> > > > > > On Tuesday, August 2, 2016 11:27:28 AM CEST Matthew White wrote:
> > > > > > > On Mon, 01 Aug 2016 16:32:30 +0200
> > > > > > > 
> > > > > > > Tim Ruehsen  wrote:
> > > > > > > > On Saturday, July 30, 2016 9:41:48 PM CEST Matthew White 
wrote:
> > > > > > > > > Hello!
> > > > > > > > > 
> > > > > > > > > I think that sometimes it could help to keep downloaded
> > > > > > > > > Metalink's
> > > > > > > > > files
> > > > > > > > > which have a bad hash.
> > > > > > > > > 
> > > > > > > > > The default wget behaviour is to delete such files.
> > > > > > > > > 
> > > > > > > > > This patch provides a way to keep files which have a bad
> > > > > > > > > hash
> > > > > > > > > through
> > > > > > > > > the
> > > > > > > > > option --keep-badhash. It appends the suffix .badhash to the
> > > > > > > > > file
> > > > > > > > > name,
> > > > > > > > > except without overwriting existing files. In the latter
> > > > > > > > > case,
> > > > > > > > > an
> > > > > > > > > unique
> > > > > > > > > suffix is appended after .badhash.
> > > > > > > > > 
> > > > > > > > > I made this patch working on the following branch:
> > > > > > > > > master (latest 20cac2c5ab3d63aacfba35fb10878a2d490e2377)
> > > > > > > > > git://git.savannah.gnu.org/wget.git
> > > > > > > > > 
> > > > > > > > > What do you think?
> > > > > > > > 
> > > > > > > > Hi Matthew,
> > > > > > > > 
> > > > > > > > good work !
> > > > > > > > 
> > > > > > > > While your FSF assignment is underway (PM), we can continue
> > > > > > > > polishing.
> > > > > > > > 
> > > > > > > > Didn't test your code yet,  but from what I see, there are
> > > > > > > > just
> > > > > > > > these
> > > > > > > > peanuts:
> > > > > > > > 
> > > > > > > > 1.
> > > > > > > > +  bhash = malloc (strlen (name) + strlen (".badhash") + 1);
> > > > > > > > +  strcat (strcpy (bhash, name), ".badhash");
> > > > > > > > 
> > > > > > > > Please consider using concat_strings() from util.h.
> > > > > > > > And please leave an empty line between var declaration and
> > > > > > > > code -
> > > > > > > > just
> > > > > > > > for
> > > > > > > > readability.
> > > > > > > > 
> > > > > > > > 2.
> > > > > > > > Please add 'link' and 'unlink' to bootstrap.conf. Gnulib will
> > > > > > > > emulate
> > > > > > > > these on platforms where they are not available (or have a
> > > > > > > > slightly
> > > > > > > > different behavior). I guess we simply forgot 'unlink' when we
> > > > > > > > switched
> > > > > > > > to gnulib.
> > > > > > > > 
> > > > > > > > 3.
> > > > > > > > The (GNU) commit messages should ideally look like:
> > > > > > > > 
> > > > > > > > one line of short description
> > > > > > > > 
> > > > > > > > file changes
> > > > > > > > 
> > > > > > > > long description
> > > > > > > > 
> > > > > > > > For example:
> > > > > > > > Add new option --keep-badhash
> > > > > > > > 
> > > > > > > > * src/init.c: Add keepbadhash
> > > > > > > > * src/main.c: Add keep-badhash
> > > > > > > > * src/options.h: Add keep_badhash
> > > > > > > > * doc/wget.texi: Add docs for --keep-badhash
> > > > > > > > * src/metalink.h: Add prototypes badhash_suffix(),
> > > > > > > > badhash_or_remove()
> > > > > > > > * src/metalink.c: New functions badhash_suffix(),
> > > > > > > > badhash_or_remove().
> > > > > > > > 
> > > > > > > >   (retrieve_from_metalink): Call append .badhash()
> > > > > > > > 
> > > > > > > > With --keep-badhash, append .badhash to Metalink's files with
> > > > > > > > checksum
> > > > > > > > mismatch, except without overwriting existing files.
> > > > > > > > 
> > > > > > > > Without --keep-badhash, remove downloaded files with checksum
> > > > > > > > mismatch
> > > > > > > > (this conforms to the old behaviour).
> > > > > > > > 
> > > > > > > > [This also applies to to your other patches]
> > > > > > > > 
> > > > > > > > 4.
> > > > > > > > Please not only write 'keepbadhash', but also what you did
> > > > > > > > (e.g.
> > > > > > > > remove,
> > > > > > > > rename, add, ...), see my example above.
> > > > > > > > 
> > > > > > > > Those ChangeLog entries should allow finding changes / faults
> > > > > > > > /
> > > > > > > > regressions
> > > > > > > > etc. even when a versioning system is not at hand, e.g. when
> > > > > > > > unpacking
> > > > > > > > the
> > > > > > > > sources from a tarball. (Not debatable that consulting commit
> > > > > > > > messages
> > > > > > > > directly is much more powerful.)
> > > > > > > > 
> > > > > > > > 5.
> > > > > > > > You write 

Re: [Bug-wget] Wget - acess list bypass / race condition PoC

[Tim Rühsen]

On Donnerstag, 18. August 2016 15:34:12 CEST Giuseppe Scrivano wrote:
> Hi,
> 
> Tim Rühsen <tim.rueh...@gmx.de> writes:
> > Please review / test this patch.
> > 
> > Please check the 'Reported-by' in the commit message and if you got a CVE
> > number, please report for inclusion into the commit message (and/or the
> > code).
> > 
> > Regards, Tim
> > 
> > On Mittwoch, 17. August 2016 10:40:35 CEST Dawid Golunski wrote:
> >> Random file name + .part extension on temporary files would already be
> >> good improvement (even if still stored within the same directory) and
> >> help prevent the exploitation.
> 
> I still think we should used a fixed extension, not a random file name.

You mean *only* a fixed extension without random part ?
With very long file names (length >= max length of supported by filesystem) we 
need a variant of the current file name shorten algorithm. IMO, this is 
unneeded code inflation.

> If wget crashes or the process is terminated for any reason, these files
> will be left around. 

Yes, but easy to see / find using 'wget_*.part' pattern.

> With a deterministic name, at least we can recover
> from what was left.

With -A/-R and --spider only (HTML | CSS) files are affected. Which are very 
seldom large enough that downloading again would hurt. 
With --recursive, all those files have to be redownloaded anyways.

We could use a simple 6 or 8 hex digit hash of the original file instead of the 
random part to have a deterministic file name. Is something like that what you 
think of ?

> IMO, it is enough to open these files with rw only for the user and not
> add any extra complexity.  It is not wget responsibility to take care of
> a misconfigured server that allows to execute random files fetched from
> http/ftp.

To cite myself :)
"But there is also non-obvious wget behavior in creating those (temp) files in 
the filesystem."

The Wget docs just don't make clear that these files come into existence for a 
while. Of course we could amend the docs and lean back... but it still is not 
an intuitive behavior and I fear people might trap into that pit. And we could 
easily prevent it with some lines of code...

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] TEST_NAME in Python tests?

[Tim Rühsen]

On Mittwoch, 24. August 2016 20:08:17 CEST Dale R. Worley wrote:
> In the file testenv/README is:
> 
> Next, is the const variable, TEST_NAME that defines the name of the
> Test.
> 
> Both, the HTTPTest and FTPTest modules have the same prototype:
> {
> name,
> pre_hook,
> test_options,
> post_hook,
> protocols
> }
> name should be a string, and is usually passed to the TEST_NAME
> variable,
> 
> Remember to always name the Test correctly using the TEST_NAME variable.
> This is essential since a directory with the Test Name is created and this
> can cause synchronization problems when the Parallel Test Harness is used.
> One can use the following command on Unix systems to check for TEST_NAME
> clashes:
> $ grep -r -h "TEST_NAME =" | cut -c13- | uniq -c -d
> 
> However, *none* of the tests in testenv has a TEST_NAME parameter.
> 
> In addition, the HTTPTest class does not have a "name" argument for its
> constructor.  (See testenv/test/http_test.py, line 15.)
> 
> What should be done about this?

Very good finding.
name/Test_NAME are remnants. We (well, I :-) removed that some while ago 
(using the file name of the test as unique identifier) and just forgot to amend 
the README.
See commit 926e42d4678689195a0bbed210c6d027db7cc390

A patch to correct this would be welcome !

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] wget for windows - current build?

[Tim Rühsen]

On Freitag, 30. September 2016 19:17:42 CEST Eli Zaretskii wrote:
> > Date: Fri, 30 Sep 2016 16:52:55 +0200 (SAST)
> > From: "ge...@mweb.co.za" 
> > 
> > So, is there a "secret" new place hosting a newer version for Windows? Or
> > is the 1.11 on sourceforge actually okay? And - while I am already asking
> > all these stupid questions - would that version actually handle larger
> > file sizes already?
> You can find a 32-bit Windows port of 1.16.1 here:
> 
>   https://sourceforge.net/projects/ezwinports/files/?source=navbar

It shouldn't be too hard to write a script that cross-compiles wget and some 
dependencies via mingw. But would such an .exe really work on a real Windows 
machine ?

Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] wget for windows - current build?

[Tim Rühsen]

On Samstag, 1. Oktober 2016 18:29:30 CEST Eli Zaretskii wrote:
> > From: Tim Rühsen <tim.rueh...@gmx.de>
> > Cc: "ge...@mweb.co.za" <ge...@mweb.co.za>
> > Date: Sat, 01 Oct 2016 13:04:25 +0200
> > 
> > It shouldn't be too hard to write a script that cross-compiles wget and
> > some dependencies via mingw. But would such an .exe really work on a real
> > Windows machine ?
> 
> I'm not sure I understand the question.  If cross-compiling works,
> then why won't the result run as expected?

Well, some years ago I copied cross compiled executables (32bit) onto a WinXP 
machine. Executing these didn't error, but they immediately returned without 
doing anything. Even the first printf() line didn't do anything.
While executing the same executables with wine on the machine that I used for 
compilation, they worked fine.

I didn't have any time to investigate - but since then I remember cross-
compilation as a theoretical thing without any practical use :-)

While it seems pretty easy to generate a wget.exe on Linux and even run it 
through wine, it seems not to work out that easily on a real Windows. At least 
these questions for a recent Windows executable are pretty common - and the 
Windows affine users here do not have a easy solution as it seems.

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] wget for windows - current build?

[Tim Rühsen]

On Sonntag, 2. Oktober 2016 10:11:33 CEST Eli Zaretskii wrote:
> > From: Tim Rühsen <tim.rueh...@gmx.de>
> > Cc: ge...@mweb.co.za
> > Date: Sat, 01 Oct 2016 20:12:28 +0200
> > 
> > If you like to create a README.windows maybe with (basic) explanations on
> > how to build wget on Windows plus pointers to your port(s), we include it
> > into the project.
> 
> That's okay (will do when I have time), but I think it would be more
> useful to have a link on the Wget Web page to the places where Windows
> binaries can be found.

Basically we only provide source code plus recipes how to build binaries.
IMO, it is ok to link to places where one can download ready-to-go binaries - 
as long there is also a recipe for users to (re-)generate these binaries on 
their own.

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] wget for windows - current build?

[Tim Rühsen]

On Samstag, 1. Oktober 2016 20:27:47 CEST Eli Zaretskii wrote:
> > From: Tim Rühsen <tim.rueh...@gmx.de>
> > Cc: ge...@mweb.co.za
> > Date: Sat, 01 Oct 2016 18:10:26 +0200
> > 
> > > > It shouldn't be too hard to write a script that cross-compiles wget
> > > > and
> > > > some dependencies via mingw. But would such an .exe really work on a
> > > > real
> > > > Windows machine ?
> > > 
> > > I'm not sure I understand the question.  If cross-compiling works,
> > > then why won't the result run as expected?
> > 
> > Well, some years ago I copied cross compiled executables (32bit) onto a
> > WinXP machine. Executing these didn't error, but they immediately
> > returned without doing anything. Even the first printf() line didn't do
> > anything.
> > While executing the same executables with wine on the machine that I used
> > for compilation, they worked fine.
> 
> Sounds like some incompatibility between the import libraries you had
> in that cross-environment and the corresponding DLLs on the target
> Windows XP machine.
> 
> > While it seems pretty easy to generate a wget.exe on Linux and even run it
> > through wine, it seems not to work out that easily on a real Windows. At
> > least these questions for a recent Windows executable are pretty common -
> > and the Windows affine users here do not have a easy solution as it
> > seems.
> Building Wget on Windows is easy if you have an operational
> development environment.  What's not easy is running the test suite
> and figuring out what each failure means, then fixing the sources as
> needed.
> 
> Anyway, in addition to my site, which offers a 32-bit build, the MSYS2
> project offers both 32-bit and 64-bit builds (although I cannot vouch
> for their thoroughness in running the test suite -- not that I know
> they didn't, mind you).  People who ask about that must be doing that
> out of ignorance; perhaps we should include pointers to those places
> in the distribution.

Sounds good :-)

If you like to create a README.windows maybe with (basic) explanations on how 
to build wget on Windows plus pointers to your port(s), we include it into the 
project.

BTW, meanwhile libidn fixed several security issues, as well as gnutls, libpsl 
and wget itself ;-)

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Does -o work correctly now?

[Tim Rühsen]

On Dienstag, 8. November 2016 15:26:14 CET Dale R. Worley wrote:
> I've been getting a script working, and as part of it, it appears that
> -o does not work correctly.  Specifically -o is supposed to send all
> "messages" to a specified log file rather than stderr.  But what I see
> is that no messages are sent to the log file.
> 
> The command in question is:
> 
> wget --no-verbose \
> -o ~/temp/log-file \
> --mirror --trust-server-names --convert-links --page-requisites \
> --include-directories=/assignments \
> --limit-rate=20k \
> http://www.iana.org/assignments/index.html
> 
> With the (now obsolete) wget distributed with my OS, GNU Wget 1.16.1, -o
> behaves as documented.  With wget from commit 00ae9b4 (which I think is
> the latest), which reports itself as GNU Wget 1.18.88-00ae-dirty, -o
> seems to have no effect.
> 
> There are any number of mistakes I could have made in this test, but
> since the symptom is so simple, I figured I'd ask if anyone else has
> noticed whether -o works or does not in the latest commits.

Hi Dale, you are right.

Looks like commit dd5c549f6af8e1143e1a6ef66725eea4bcd9ad50 broke it.

Sadly, the test suite doesn't catch it.

@Piotr Could you take a look ?

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] contributing in development

[Tim Rühsen]

On Sonntag, 6. November 2016 01:16:43 CET Oleg Akchurin wrote:
> Hello all. I'd like to get participate in development of wget.
> 
> BR, Oleg.

Welcome, Oleg.

A good starting point might be involving into the discussions here. Try to 
answer questions, review ideas and/or patches, come up with your own 
questions.

And we need help for wget2... 
git clone git://git.savannah.gnu.org/wget/wget2.git

We currently use the issue tracker and wiki on Gitlab or Github.
https://gitlab.com/rockdaboot/wget2
https://github.com/rockdaboot/wget2

The repos are all synced.

Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] [PATCH] Respect -o parameter

[Tim Rühsen]

On Mittwoch, 9. November 2016 10:57:01 CET Piotr Wajda wrote:
>  * log.c: don't choose log output dynamically when opt.lfilename is set
> ---
>  src/log.c | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/src/log.c b/src/log.c
> index e068acf..51f30c4 100644
> --- a/src/log.c
> +++ b/src/log.c
> @@ -956,11 +956,11 @@ static void
>  check_redirect_output (void)
>  {
>  #ifndef WINDOWS
> -  /* If it was redirected already to log file by SIGHUP or SIGUSR1,
> -   * it was permanent and since that redirect_request_signal_name is set.
> +  /* If it was redirected already to log file by SIGHUP, SIGUSR1 or -o
> parameter, +   * it was permanent.
> * If there was no SIGHUP or SIGUSR1 and shell is interactive
> * we check if process is fg or bg before every line is printed.*/
> -  if (!redirect_request_signal_name && shell_is_interactive)
> +  if (!redirect_request_signal_name && shell_is_interactive &&
> !opt.lfilename) {
>if (tcgetpgrp (STDIN_FILENO) != getpgrp ())
>  {

Thanks, Piotr ! Commit has been pushed.

Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Misspelling of "succeeded"

[Tim Rühsen]

On Samstag, 22. Oktober 2016 16:54:08 CEST Göran Uddeborg wrote:
> While reviewing my Swedish translation of wget, Anders Jonsson
> discovered a misspelling in one of the msgids, that is, in the source
> code:
> 
> #: src/metalink.c:351
> msgid "Signature validation suceeded.\n"

Thanks, fixed and pushed.

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] [PATCH v5] bug #45790: wget prints it's progress even when background

[Tim Rühsen]

Thanks, just pushed.

On Freitag, 21. Oktober 2016 17:12:58 CEST losgrandes wrote:
> Fixed trailing whitespace errors. Sorry for sending dirty patch. Should be
> good now.
> 
> * src/log.c: Use tcgetpgrp(STDIN_FILENO) != getpgrp() to determine when to
> print to STD* or logfile. Deprecate log_request_redirect_output function.
>   Use different file handles for STD* and logfile, to easily switch between
> them when changing fg/bg. * src/log.h: Make redirect_output function
> externally linked.
> * src/main.c: Don't use deprecated log_request_redirect_output function. Use
> redirect_output instead. * src/mswindows.c: Don't use deprecated
> log_request_redirect_output function. Use redirect_output instead. ---
>  src/log.c   | 125
> +--- src/log.h   | 
>  1 +
>  src/main.c  |   2 +-
>  src/mswindows.c |   5 +--
>  4 files changed, 85 insertions(+), 48 deletions(-)
> 
> diff --git a/src/log.c b/src/log.c
> index a1338ca..bcd4d2e 100644
> --- a/src/log.c
> +++ b/src/log.c
> @@ -80,6 +80,18 @@ as that of the covered work.  */
> logging is inhibited, logfp is set back to NULL. */
>  static FILE *logfp;
> 
> +/* Descriptor of the stdout|stderr */
> +static FILE *stdlogfp;
> +
> +/* Descriptor of the wget.log* file (if created) */
> +static FILE *filelogfp;
> +
> +/* Name of log file */
> +static char *logfile;
> +
> +/* Is interactive shell ? */
> +static int shell_is_interactive;
> +
>  /* A second file descriptor pointing to the temporary log file for the
> WARC writer.  If WARC writing is disabled, this is NULL.  */
>  static FILE *warclogfp;
> @@ -611,16 +623,18 @@ log_init (const char *file, bool appendp)
>  {
>if (HYPHENP (file))
>  {
> -  logfp = stdout;
> +  stdlogfp = stdout;
> +  logfp = stdlogfp;
>  }
>else
>  {
> -  logfp = fopen (file, appendp ? "a" : "w");
> -  if (!logfp)
> +  filelogfp = fopen (file, appendp ? "a" : "w");
> +  if (!filelogfp)
>  {
>fprintf (stderr, "%s: %s: %s\n", exec_name, file, strerror
> (errno)); exit (WGET_EXIT_GENERIC_ERROR);
>  }
> +  logfp = filelogfp;
>  }
>  }
>else
> @@ -631,7 +645,8 @@ log_init (const char *file, bool appendp)
>   stderr only if the user actually specifies `-O -'.  He says
>   this inconsistency is harder to document, but is overall
>   easier on the user.  */
> -  logfp = stderr;
> +  stdlogfp = stderr;
> +  logfp = stdlogfp;
> 
>if (1
>  #ifdef HAVE_ISATTY
> @@ -646,6 +661,11 @@ log_init (const char *file, bool appendp)
>save_context_p = true;
>  }
>  }
> +
> +#ifndef WINDOWS
> +  /* Initialize this values so we don't have to ask every time we print
> line */ +  shell_is_interactive = isatty (STDIN_FILENO);
> +#endif
>  }
> 
>  /* Close LOGFP (only if we opened it, not if it's stderr), inhibit
> @@ -880,59 +900,78 @@ log_cleanup (void)
> 
>  /* When SIGHUP or SIGUSR1 are received, the output is redirected
> elsewhere.  Such redirection is only allowed once. */
> -static enum { RR_NONE, RR_REQUESTED, RR_DONE } redirect_request = RR_NONE;
>  static const char *redirect_request_signal_name;
> 
> -/* Redirect output to `wget-log'.  */
> +/* Redirect output to `wget-log' or back to stdout/stderr.  */
> 
> -static void
> -redirect_output (void)
> +void
> +redirect_output (bool to_file, const char *signal_name)
>  {
> -  char *logfile;
> -  logfp = unique_create (DEFAULT_LOGFILE, false, );
> -  if (logfp)
> +  if (to_file && logfp != filelogfp)
>  {
> -  fprintf (stderr, _("\n%s received, redirecting output to %s.\n"),
> -   redirect_request_signal_name, quote (logfile));
> -  xfree (logfile);
> -  /* Dump the context output to the newly opened log.  */
> -  log_dump_context ();
> +  if (signal_name)
> +{
> +  fprintf (stderr, "\n%s received.", signal_name);
> +}
> +  if (!filelogfp)
> +{
> +  filelogfp = unique_create (DEFAULT_LOGFILE, false, );
> +  if (filelogfp)
> +{
> +  fprintf (stderr, _("\nRedirecting output to %s.\n"),
> +  quote (logfile));
> +  /* Store signal name to tell wget it's permanent redirect to
> log file */ +  redirect_request_signal_name = signal_name;
> +  logfp = filelogfp;
> +  /* Dump the context output to the newly opened log.  */
> +  log_dump_context ();
> +}
> +  else
> +{
> +  /* Eek!  Opening the alternate log file has failed.  Nothing
> we +can do but disable printing completely. */
> +  fprintf (stderr, _("%s: %s; disabling logging.\n"),
> +  (logfile) ? logfile : DEFAULT_LOGFILE, strerror
> (errno)); +  inhibit_logging = true;
> +   

Re: [Bug-wget] [PATCH v3] bug #45790: wget prints it's progress even when background

[Tim Rühsen]

On Dienstag, 18. Oktober 2016 08:30:34 CEST Wajda, Piotr wrote:
> Can you please review my 3 patches?

Sorry, I had the impression there was some discussion going on !?

But apart from that:
- please add a space after if statements.
- please add a space between function name and (.
- please make local commits with appropriate commit messages (see git log for 
examples), generate your patches with 'git format-patch -3' (-3 for your last 
3 commits) and send them as attachments.

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] [PATCH v2] bug #48811: netrc password wins over interactive --ask-password

[Tim Rühsen]

Hi Piotr,

please include netrc.h in utils.c.

In getftp(): 
  struct net_credentials *ftp_cred = malloc (sizeof *ftp_cred);
...
  ftp_cred = pick_credentials()

This looks like a memleak. Also, where do you free ftp_cred ?
Is it really necessary  to allocate ftp_cred on each call to getftp (just a 
question, maybe it is) ?

Basically the same in gethttp()...

Try 'make check-valgrind' to test for some kinds of memleaks.

Regards, Tim

On Mittwoch, 19. Oktober 2016 11:53:22 CEST losgrandes wrote:
> * src/ftp.c: Leverage new struct net_credentials and function
> pick_credentials. pick_credentials is responsible for taking proper order
> when selecting source of credentials. * src/http.c: Leverage new struct
> net_credentials and function pick_credentials. * src/utils.c: New function
> pick_credentials.
> * src/utils.h: New struct net_credentials.
> 
> ---
>  src/ftp.c   | 19 +++
>  src/http.c  | 18 +++---
>  src/utils.c | 19 +++
>  src/utils.h | 11 +++
>  4 files changed, 44 insertions(+), 23 deletions(-)
> 
> diff --git a/src/ftp.c b/src/ftp.c
> index 39f20fa..cc98ca3 100644
> --- a/src/ftp.c
> +++ b/src/ftp.c
> @@ -327,7 +327,8 @@ getftp (struct url *u, struct url *original_url,
>uerr_t err = RETROK;  /* appease the compiler */
>FILE *fp = NULL;
>char *respline, *tms;
> -  const char *user, *passwd, *tmrate;
> +  const char *tmrate;
> +  struct net_credentials *ftp_cred = malloc (sizeof *ftp_cred);
>int cmd = con->cmd;
>wgint expected_bytes = 0;
>bool got_expected_bytes = false;
> @@ -359,13 +360,7 @@ getftp (struct url *u, struct url *original_url,
> 
>*qtyread = restval;
> 
> -  user = u->user;
> -  passwd = u->passwd;
> -  search_netrc (u->host, (const char **), (const char **), 1);
> -  user = user ? user : (opt.ftp_user ? opt.ftp_user : opt.user);
> -  if (!user) user = "anonymous";
> -  passwd = passwd ? passwd : (opt.ftp_passwd ? opt.ftp_passwd :
> opt.passwd); -  if (!passwd) passwd = "-wget@";
> +  ftp_cred = pick_credentials (u, opt.ftp_user, opt.ftp_passwd, opt.user,
> opt.passwd, 1);
> 
>dtsock = -1;
>local_sock = -1;
> @@ -461,18 +456,18 @@ getftp (struct url *u, struct url *original_url,
> 
>/* Second: Login with proper USER/PASS sequence.  */
>logprintf (LOG_VERBOSE, _("Logging in as %s ... "),
> - quotearg_style (escape_quoting_style, user));
> + quotearg_style (escape_quoting_style, ftp_cred->user));
>if (opt.server_response)
>  logputs (LOG_ALWAYS, "\n");
>if (con->proxy)
>  {
>/* If proxy is in use, log in as username@target-site. */
> -  char *logname = concat_strings (user, "@", u->host, (char *) 0);
> -  err = ftp_login (csock, logname, passwd);
> +  char *logname = concat_strings (ftp_cred->user, "@", u->host,
> (char *) 0); +  err = ftp_login (csock, logname, ftp_cred->passwd);
>xfree (logname);
>  }
>else
> -err = ftp_login (csock, user, passwd);
> +err = ftp_login (csock, ftp_cred->user, ftp_cred->passwd);
> 
>/* FTPRERR, FTPSRVERR, WRITEFAILED, FTPLOGREFUSED, FTPLOGINC */
>switch (err)
> diff --git a/src/http.c b/src/http.c
> index 7e2c4ec..41eaa42 100644
> --- a/src/http.c
> +++ b/src/http.c
> @@ -1813,7 +1813,7 @@ time_to_rfc1123 (time_t time, char *buf, size_t
> bufsize) static struct request *
>  initialize_request (const struct url *u, struct http_stat *hs, int *dt,
> struct url *proxy, bool inhibit_keep_alive, bool *basic_auth_finished, -   
> wgint *body_data_size, char **user, char **passwd, uerr_t
> *ret) +wgint *body_data_size, struct net_credentials
> **http_cred, uerr_t *ret) {
>bool head_only = !!(*dt & HEAD_ONLY);
>struct request *req;
> @@ -1876,20 +1876,16 @@ initialize_request (const struct url *u, struct
> http_stat *hs, int *dt, struct u request_set_header (req,
> "Accept-Encoding", "identity", rel_none);
> 
>/* Find the username and password for authentication. */
> -  *user = u->user;
> -  *passwd = u->passwd;
> -  search_netrc (u->host, (const char **)user, (const char **)passwd, 0);
> -  *user = *user ? *user : (opt.http_user ? opt.http_user : opt.user);
> -  *passwd = *passwd ? *passwd : (opt.http_passwd ? opt.http_passwd :
> opt.passwd); +  *http_cred = pick_credentials (u, opt.http_user,
> opt.http_passwd, opt.user, opt.passwd, 0);
> 
>/* We only do "site-wide" authentication with "global" user/password
> * values unless --auth-no-challange has been requested; URL
> user/password * info overrides. */
> -  if (*user && *passwd && (!u->user || opt.auth_without_challenge))
> +  if ((*http_cred)->user && (*http_cred)->passwd && (!u->user ||
> opt.auth_without_challenge)) {
>/* If this is a host for which we've already received a Basic
> * challenge, we'll go ahead and send Basic authentication creds. */
> - 

Re: [Bug-wget] [PATCH v4] bug #45790: wget prints it's progress even when background

[Tim Rühsen]

Could you fix these, please...

Applying: bug #45790: wget prints it's progress even when background
.git/rebase-apply/patch:104: trailing whitespace.
  if (!filelogfp) 
.git/rebase-apply/patch:126: trailing whitespace.
  else 
.git/rebase-apply/patch:158: trailing whitespace.
  /* If it was redirected already to log file by SIGHUP or SIGUSR1, 
.git/rebase-apply/patch:159: trailing whitespace.
   * it was permanent and since that redirect_request_signal_name is set. 
.git/rebase-apply/patch:160: trailing whitespace.
   * If there was no SIGHUP or SIGUSR1 and shell is interactive 
warning: squelched 3 whitespace errors
warning: 8 lines add whitespace errors.

On Mittwoch, 19. Oktober 2016 11:53:21 CEST losgrandes wrote:
> * src/log.c: Use tcgetpgrp(STDIN_FILENO) != getpgrp() to determine when to
> print to STD* or logfile. Deprecate log_request_redirect_output function.
>   Use different file handles for STD* and logfile, to easily switch between
> them when changing fg/bg. * src/log.h: Make redirect_output function
> externally linked.
> * src/main.c: Don't use deprecated log_request_redirect_output function. Use
> redirect_output instead. * src/mswindows.c: Don't use deprecated
> log_request_redirect_output function. Use redirect_output instead. ---
>  src/log.c   | 125
> +--- src/log.h   | 
>  1 +
>  src/main.c  |   2 +-
>  src/mswindows.c |   5 +--
>  4 files changed, 85 insertions(+), 48 deletions(-)
> 
> diff --git a/src/log.c b/src/log.c
> index a1338ca..bcd4d2e 100644
> --- a/src/log.c
> +++ b/src/log.c
> @@ -80,6 +80,18 @@ as that of the covered work.  */
> logging is inhibited, logfp is set back to NULL. */
>  static FILE *logfp;
> 
> +/* Descriptor of the stdout|stderr */
> +static FILE *stdlogfp;
> +
> +/* Descriptor of the wget.log* file (if created) */
> +static FILE *filelogfp;
> +
> +/* Name of log file */
> +static char *logfile;
> +
> +/* Is interactive shell ? */
> +static int shell_is_interactive;
> +
>  /* A second file descriptor pointing to the temporary log file for the
> WARC writer.  If WARC writing is disabled, this is NULL.  */
>  static FILE *warclogfp;
> @@ -611,16 +623,18 @@ log_init (const char *file, bool appendp)
>  {
>if (HYPHENP (file))
>  {
> -  logfp = stdout;
> +  stdlogfp = stdout;
> +  logfp = stdlogfp;
>  }
>else
>  {
> -  logfp = fopen (file, appendp ? "a" : "w");
> -  if (!logfp)
> +  filelogfp = fopen (file, appendp ? "a" : "w");
> +  if (!filelogfp)
>  {
>fprintf (stderr, "%s: %s: %s\n", exec_name, file, strerror
> (errno)); exit (WGET_EXIT_GENERIC_ERROR);
>  }
> +  logfp = filelogfp;
>  }
>  }
>else
> @@ -631,7 +645,8 @@ log_init (const char *file, bool appendp)
>   stderr only if the user actually specifies `-O -'.  He says
>   this inconsistency is harder to document, but is overall
>   easier on the user.  */
> -  logfp = stderr;
> +  stdlogfp = stderr;
> +  logfp = stdlogfp;
> 
>if (1
>  #ifdef HAVE_ISATTY
> @@ -646,6 +661,11 @@ log_init (const char *file, bool appendp)
>save_context_p = true;
>  }
>  }
> +
> +#ifndef WINDOWS
> +  /* Initialize this values so we don't have to ask every time we print
> line */ +  shell_is_interactive = isatty (STDIN_FILENO);
> +#endif
>  }
> 
>  /* Close LOGFP (only if we opened it, not if it's stderr), inhibit
> @@ -880,59 +900,78 @@ log_cleanup (void)
> 
>  /* When SIGHUP or SIGUSR1 are received, the output is redirected
> elsewhere.  Such redirection is only allowed once. */
> -static enum { RR_NONE, RR_REQUESTED, RR_DONE } redirect_request = RR_NONE;
>  static const char *redirect_request_signal_name;
> 
> -/* Redirect output to `wget-log'.  */
> +/* Redirect output to `wget-log' or back to stdout/stderr.  */
> 
> -static void
> -redirect_output (void)
> +void
> +redirect_output (bool to_file, const char *signal_name)
>  {
> -  char *logfile;
> -  logfp = unique_create (DEFAULT_LOGFILE, false, );
> -  if (logfp)
> +  if (to_file && logfp != filelogfp)
>  {
> -  fprintf (stderr, _("\n%s received, redirecting output to %s.\n"),
> -   redirect_request_signal_name, quote (logfile));
> -  xfree (logfile);
> -  /* Dump the context output to the newly opened log.  */
> -  log_dump_context ();
> +  if (signal_name)
> +{
> +  fprintf (stderr, "\n%s received.", signal_name);
> +}
> +  if (!filelogfp)
> +{
> +  filelogfp = unique_create (DEFAULT_LOGFILE, false, );
> +  if (filelogfp)
> +{
> +  fprintf (stderr, _("\nRedirecting output to %s.\n"),
> +  quote (logfile));
> +  /* Store signal name to tell wget it's permanent redirect to
> log file */ +  

Re: [Bug-wget] [PATCH v2] bug #46584: wget --spider always returns zero exit status

[Tim Rühsen]

Thanks, pushed.

On Mittwoch, 19. Oktober 2016 11:53:23 CEST losgrandes wrote:
> * src/ftp.c: Return error as exit value if even one file doesn't exist
> 
> ---
>  src/ftp.c | 9 -
>  1 file changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/src/ftp.c b/src/ftp.c
> index cc98ca3..77dc9cd 100644
> --- a/src/ftp.c
> +++ b/src/ftp.c
> @@ -1186,6 +1186,7 @@ Error in server response, closing control
> connection.\n")); if (opt.spider)
>  {
>bool exists = false;
> +  bool all_exist = true;
>struct fileinfo *f;
>uerr_t _res = ftp_get_listing (u, original_url, con, );
>/* Set the DO_RETR command flag again, because it gets unset when
> @@ -1201,6 +1202,8 @@ Error in server response, closing control
> connection.\n")); {
>exists = true;
>break;
> +} else {
> +  all_exist = false;
>  }
>f = f->next;
>  }
> @@ -1221,7 +1224,11 @@ Error in server response, closing control
> connection.\n")); con->csock = -1;
>fd_close (dtsock);
>fd_close (local_sock);
> -  return RETRFINISHED;
> +  if (all_exist) {
> +  return RETRFINISHED;
> +  } else {
> +  return FTPNSFOD;
> +  }
>  }
> 
>if (opt.verbose)



signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Wget error after install

[Tim Rühsen]

On Montag, 28. November 2016 09:34:58 CET Mukram Khan wrote:
> Hi Wget Team,
> 
> Could you please assist me on the below error I am getting after installing
> Wget 1.9.1 version on AIX servers.
> 
>  # wget --version
> exec(): 0509-036 Cannot load program wget because of the following errors:
> 0509-130 Symbol resolution failed for wget because:
> 0509-136   Symbol SSLv2_client_method (number 134) is not exported
> from
>dependent module /usr/lib/libssl.a(libssl.so.0.9.8).
> 0509-192 Examine .loader section symbols with the
>  'dump -Tv' command.
> 
> I would appreciate if you could assist on this.
> 
> 
> Thanks,
> Mukram

Did you consult the internet ?
A quick search reveals: https://forum.xorux.com/index.php?p=/discussion/43/
connect-pl-issue-after-install-openssl-ifix-on-aix

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Wget C developer

[Tim Rühsen]

On Freitag, 9. Dezember 2016 22:39:55 CET Vinicius Arruda wrote:
> Hi,
> 
> Do you still need a C developer ? I am from Brazil and I am undergraduate
> student in computer science and want to help.

What exactly would you like to do for wget ? Are you motivated enough to 
discuss and realize it ? If so, tell us about it your idea... and we'll be on 
your side to give you assistance...

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Wget C developer

[Tim Rühsen]

On Samstag, 10. Dezember 2016 13:00:00 CET Vinicius Arruda wrote:
> I am an undergraduate student in computer science and never participate in
> a larger software development. I want get some experience. If has some
> little project inside de Wget project that I can work on will be great.

Please consider:

1. Do not answer via PM, always via mailing list  (see https://daniel.haxx.se/
blog/2013/10/08/dont-email-me/)
2. Do not top post (search the internet for the reasons)

You found the mailing list, so you also found the bug tracker !?
(If you go there, you'll see that it is normally not about having C skills. 
much more of having knowledge/experience with internet protocols, debugging, 
API/library ecosystem and the like.)
Maybe you'll find something you like to work on.

First clone the project and build it on your own. By this you learn your daily 
tools (git, build tools, auto* tools, make, etc).
Now try to reproduce one of the open bugs with your freshly built wget binary.
 If you can't reproduce, add this info to the bug tracker, so we can possibly 
close the bug as 'not reproducible' or 'fixed'.
If you can reproduce it, go ahead with reading the source code and try to find 
the part that you want to fix. Fix it, post your fix/patch on the mailing list 
for discussion/review. If you come to this point, you are already done.

If you struggle because of missing knowledge regarding building and/or 
debugging, using git or whatever: First try to consult the internet (it's all 
there), second ask on the mailing list for assistance.

Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] bug: wget crashes at certain zoom level

[Tim Rühsen]

Thanks,

AFAICS, that issue has already been fixed in wget 1.18 (commit 
7cb9efa668f80ab5ca4d25133c3133e10473d1ef).

Regards, Tim

On Mittwoch, 30. November 2016 11:53:30 CET Amr Alaa wrote:
> I noticed that wget crashes whenever I try to increase gnome-terminal font
> size using (ctrl+shift+PLUS) to a certain size.
> 
> $ wget --header='Host: download-cf.jetbrains.com' --header='User-Agent:
> Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:49.0) Gecko/20100101
> Firefox/49.0' --header='Accept:
> text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
> --header='Accept-Language:
> en,en-US;q=0.8,ar;q=0.7,ar-EG;q=0.5,de;q=0.3,de-DE;q=0.2'
> --header='Referer:
> https://www.jetbrains.com/idea/download/download-thanks.html?platform=window
> sZip' --header='Cookie: _ga=GA1.2.1842554626.1477170940;
> _mkto_trk=id:426-QVD-114:_mch-jetbrains.com-1477170939574-30637;
> _ceg.s=ohg71e; _ceg.u=ohg71e; _gat_UA-47631155-3=1;
> _dc_gtm_UA-47631155-3=1' --header='Connection: keep-alive'
> --header='Upgrade-Insecure-Requests: 1' '
> https://download-cf.jetbrains.com/idea/ideaIU-2016.3.win.zip' -O
> 'ideaIU-2016.3.win.zip' -c
> 
> --2016-11-30 11:41:25--
> https://download-cf.jetbrains.com/idea/ideaIU-2016.3.win.zip
> Resolving download-cf.jetbrains.com (download-cf.jetbrains.com)...
> 54.192.185.122, 54.192.185.183, 54.192.185.82, ...
> Connecting to download-cf.jetbrains.com
> (download-cf.jetbrains.com)|54.192.185.122|:443...
> connected.
> HTTP request sent, awaiting response... 206 Partial Content
> Length: 612177072 (584M), 611693447 (583M) remaining [binary/octet-stream]
> Saving to: ‘ideaIU-2016.3.win.zip’
> 
> ideaIU-2016.3.win.zip
> 0%[ ] 669.55K
> 155KB/s   ideaIU-2016.3.win.zip
> 0%[  ideaIU-2016.3.win.zip
> 0%[ideaIU-2016.3.win.z   0%[
> ideaIU-2016.-2016.3.win.z   0% 942.58K   142KB/seta 70m 5s [1]
> 13932 *segmentation fault (core dumped)*  wget -c --header='Host:
> download-cf.jetbrains.com' -O  -c
> 
> 
> $ uname -a
> Linux ubuntu 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016
> x86_64 x86_64 x86_64 GNU/Linux
> 
> 
> $ gnome-terminal --version
> GNOME Terminal 3.18.3
> 
> 
> $ wget --version
> GNU Wget 1.17.1 built on linux-gnu.
> 
> +digest -gpgme +https +ipv6 +iri +large-file -metalink +nls +ntlm
> +opie -psl +ssl/openssl
> 
> Wgetrc:
> /etc/wgetrc (system)
> Locale:
> /usr/share/locale
> Compile:
> gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/etc/wgetrc"
> -DLOCALEDIR="/usr/share/locale" -I. -I../../src -I../lib
> -I../../lib -Wdate-time -D_FORTIFY_SOURCE=2 -I/usr/include
> -DHAVE_LIBSSL -DNDEBUG -g -O2 -fPIE -fstack-protector-strong
> -Wformat -Werror=format-security -DNO_SSLv2 -D_FILE_OFFSET_BITS=64
> -g -Wall
> Link:
> gcc -DHAVE_LIBSSL -DNDEBUG -g -O2 -fPIE -fstack-protector-strong
> -Wformat -Werror=format-security -DNO_SSLv2 -D_FILE_OFFSET_BITS=64
> -g -Wall -Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro
> -Wl,-z,now -L/usr/lib -lpcre -luuid -lssl -lcrypto -lz -lidn
> ftp-opie.o openssl.o http-ntlm.o ../lib/libgnu.a
> 
> Copyright (C) 2015 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> .
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> 
> Originally written by Hrvoje Niksic .
> Please send bug reports and questions to .



signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Favicon is not downloaded (Suggestion for improvement)

[Tim Rühsen]

On Donnerstag, 5. Januar 2017 10:40:48 CET Dale R. Worley wrote:
> Павел Серегов  writes:
> > Often not exist code for favicon (in index.html), but site have.
> > 
> > My suggestion:
> > If use wget -m, need make download  http://example.com/favicon.ico
> > 
> > How do you like the idea?
> 
> The documentation for -m is:
> 
>-m
>--mirror
>Turn on options suitable for mirroring.  This option turns on
>recursion and time-stamping, sets infinite recursion depth and
>keeps FTP directory listings.  It is currently equivalent to -r
> -N -l inf --no-remove-listing.
> 
> I suggest defining "--favicon" specifically to download
> http(s):///favicon.ico, and then add --favicon to the specification
> of --mirror.

With default 'on'. Dale, do you mind to open an issue for that at https://
github.com/rockdaboot/wget2 ?
IMO, it should go there first.

Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] ot: clicking email links advice

[Tim Rühsen]

On Donnerstag, 5. Januar 2017 09:34:39 CET Ander Juaristi wrote:
> Hi,
> 
> On 28/12/16 05:57, voy...@sbt.net.au wrote:
> > is there a way to run wget with that url and, tell it to 'press' one of
> > the buttons?
> 
> Not directly as you describe. Wget does not submit web forms.
> 
> You would need to write an external application to parse the HTML,
> generate the target link and then feed that link to wget.
> 
> Or,
> 
> You could use wget2, which we're designing as a library, although it's
> still in pre-alpha. It has functions to extract links from an HTML
> document (see the example in [1]), although I don't know if it can
> extract URLs from  fields as well, which is what I
> guess you need. Maybe @Tim can give more details on this.

The 'action' URL will not be followed automatically by wget, it is filtered out 
by purpose. Libwget should parse it, maybe wget2 does not filter it out right 
now - but this should be fixed.

Basically you need to parse HTML for more infos - the library contains a 
scanner and example code to use it. If you want to go that way, you need some 
knowledge/background about the HTML form mechanism (as Dale pointed out).

If you just have one use case, take a browser's 'development' tools (Firefox 
and Chromium have those), fill in your form as usual and just see what is 
transferred to the server. You can then set header lines and content via wget.
Cookies normally play a role when it comes to login/authentication.

Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] buildbot failure in OpenCSW Buildbot on wget-solaris10-i386

[Tim Rühsen]

Not sure why, but some of my modifications didn't make it into the commit :-(

Tim


On Freitag, 6. Januar 2017 16:32:30 CET build...@opencsw.org wrote:
> The Buildbot has detected a new failure on builder wget-solaris10-i386 while
> building wget. Full details are available at:
> https://buildfarm.opencsw.org/buildbot/builders/wget-solaris10-i386/builds/
> 195
> 
> Buildbot URL: https://buildfarm.opencsw.org/buildbot/
> 
> Buildslave for this Build: unstable10x
> 
> Build Reason: The SingleBranchScheduler scheduler named
> 'schedule-wget-solaris10-i386' triggered this build Build Source Stamp:
> [branch master] 2427ca4ac090dc6cc0c527f4ac4fc5a4bd468eb1 Blamelist:
> vijeth-aradhya 
> 
> BUILD FAILED: failed shell_2 shell_3
> 
> Sincerely,
>  -The Buildbot



signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Unable to establish SSL connection error

[Tim Rühsen]

As Raitis wrote privately:
It has been fixed by updating to a newer version.

@Raitis Please consider https://daniel.haxx.se/blog/2013/10/08/dont-email-me/

Tim


On Freitag, 6. Januar 2017 14:41:05 CET Dale R. Worley wrote:
> Raitis Misa  writes:
> > I'm trying to download APOD with line - wget.exe -x -r -k -E -nc -e
> > robots=off --page-requisites --tries=2 --level=2 --timeout=20
> > --user-agent="Mozilla 1.5" --secure-protocol=TLSv1
> > --no-check-certificate http://apod.nasa.gov/apod/archivepix.html
> > 
> > other --secure-protocol= options gives the same result as well as not
> > using --no-check-certificate .
> > 
> > GNU Wget 1.11.4
> > Microsoft Windows [Version 10.0.10586]
> 
> You have not told us what the results were and any error messages that
> were output.
> 
> When I execute the similar command line (on Linux), I get this error:
> 
> Connecting to apod.nasa.gov
> (apod.nasa.gov)|2001:4d0:2310:150::22|:443... connected. OpenSSL:
> error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
> Unable to establish SSL connection.
> 
> If I leave off the --secure-protocol option, it works.  Similarly, if I
> specify --secure-protocol=TLSv1_1 or --secure-protocol=TLSv1_2, it
> works.  So I suspect that TLSv1 isn't supported by that server.
> 
> Dale



signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] cookies.c:(.text+0xd6c): undefined reference to `psl_builtin_outdated'

[Tim Rühsen]

On Dienstag, 27. Dezember 2016 17:46:05 CET Thomas Schweikle wrote:
> Hi!
> 
> I am trying to compile wget 1.18 on CentOS 7.3 and receive following error:
> 
>   CC   exits.o
> if test -n ""; then cp "./build_info.c.in" .; fi
> /bin/perl "../build-aux/build_info.pl" \
> "../src/build_info.c"
> if test -n ""; then rm -f build_info.c.in; fi
>   CC   build_info.o
>   CC   metalink.o
> echo '/* version.c */' > version.c
> echo '/* Autogenerated by Makefile - DO NOT EDIT */' >> version.c
> echo '' >> version.c
> echo '#include "version.h"' >> version.c
> echo 'const char *version_string = "1.18.93-b9ed";' >> version.c
> echo 'const char *compilation_string = "'gcc -DHAVE_CONFIG_H
> -DSYSTEM_WGETRC=\"/etc/wgetrc\" -DLOCALEDIR=\"/usr/share/locale\" -I.
> -I../lib -I../lib   -DHAVE_LIBSSL -DNDEBUG '";' \
> 
> | sed -e 's/[\\"]/\\&/g' -e 's/\\"/"/' -e 's/\\";$/";/' >> version.c
> 
> echo 'const char *link_string = "'gcc  -DHAVE_LIBSSL -DNDEBUG  \
>   -lcares   -lpcre   -lmetalink   -lssl -lcrypto   -lz   -lpsl
> ftp-opie.o openssl.o http-ntlm.o ../lib/libgnu.a  -lcrypto
> '";' \
> 
> | sed -e 's/[\\"]/\\&/g' -e 's/\\"/"/' -e 's/\\";$/";/' >> version.c
> 
>   CC   version.o
>   CC   ftp-opie.o
>   CC   openssl.o
>   CC   http-ntlm.o
>   CCLD wget
> cookies.o: In function `check_domain_match':
> cookies.c:(.text+0xd6c): undefined reference to `psl_builtin_outdated'
> collect2: error: ld returned 1 exit status
> make[3]: *** [wget] Error 1
> make[3]: Leaving directory `/var/lib/jenkins/sharedspace/wget/build/src'
> make[2]: *** [all] Error 2
> make[2]: Leaving directory `/var/lib/jenkins/sharedspace/wget/build/src'
> make[1]: *** [all-recursive] Error 1
> make[1]: Leaving directory `/var/lib/jenkins/sharedspace/wget/build'
> make: *** [all] Fehler 2
> Build step 'Shell ausführen' marked build as failure

Hi,

either get a newer libpsl (recommended) or disable libpsl with './configure --
without-libpsl' (not recommended, but ok if you are not going to use cookies).

Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Bug with SSL Connection

[Tim Rühsen]

On Dienstag, 27. Dezember 2016 23:22:17 CET Алексей Павлов wrote:
> OS: Linux Mint 18.1, wget from official repositories
> 
> blastbox@blastbox-G750JH ~ $ wget --secure-protocol=auto
> https://web.archive.org
> --2016-12-27 23:19:06--  https://web.archive.org/
> Resolving web.archive.org (web.archive.org)... 92.255.241.100,
> 2a02:2698:a000::64
> Connecting to web.archive.org (web.archive.org)|92.255.241.100|:443...
> connected.
> OpenSSL: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
> Unable to establish SSL connection.
> 
> blastbox@blastbox-G750JH ~ $ wget --secure-protocol=SSLv2
> https://web.archive.org
> --2016-12-27 23:19:13--  https://web.archive.org/
> OpenSSL: unimplemented 'secure-protocol' option value 1
> Please report this issue to bug-wget@gnu.org
> Aborted
> 
> blastbox@blastbox-G750JH ~ $ wget --secure-protocol=SSLv3
> https://web.archive.org
> --2016-12-27 23:19:18--  https://web.archive.org/
> OpenSSL: unimplemented 'secure-protocol' option value 2
> Please report this issue to bug-wget@gnu.org
> Aborted

The underlying TLS layer (OpenSSL) is likely not built with SSL enabled.
This has been done widely due to general SSL vulnerabilities (e.g. POODLE).
You should ask the Lint OpenSSL packagers about details and/or search the web 
for more details.

Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] [GSoC] Add wget_robots_parse unit test

[Tim Rühsen]

On 03/28/2017 09:44 AM, Didik Setiawan wrote:
> I have made a pull request about robots API unit test on github [1] as my GSoC
> 2017 microproject attempt.
> Any feedback will be appreciated.

Thanks, Didek.

I'll review soon on Github.

BTW, anyone interested in work related to Wget2 should 'watch' the repo
on https://github.com/rockdaboot/wget2.

Due to the number of messages it might be overwhelming if every
change/commit/comment pops up in this mailing list. Of course, we can
change that if there is a larger general interest here.

Regards, Tim



signature.asc
Description: OpenPGP digital signature

Re: [Bug-wget] `make check` fails 4 tests on fresh wget2 clone

[Tim Rühsen]

On Donnerstag, 23. März 2017 11:21:37 CEST Darshit Shah wrote:
> There are two points here, we should keep the mandatory dependencies at
> a bare minimum. That is one of the features that made Wget 1.x more
> popular with the embedded crowd. If someone does not care about
> non-ascii domain names, it should be possible to build Wget2 without
> IDNA.
> 
> However, when building with IDNA, we may make IDNA 2008 (libidn2) the
> minimum requirement. For this, the important thing to look at is which
> platforms / systems will suffer. Far as I am aware, most distros and
> platforms have libidn2 available as a package, but if there are systems
> where this is not the case, we should evaluate whether we would like to
> drop full compatibility for such systems.
> 
> Anyways, this boils down to:
> 
> 1. IDNA is not a hard build dependency. Hence, we should detect for its
>absence and ignore running tests that depend on it.

Just made a commit to address this issue.

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] PATCH: Fix FTBFS on GNU/Hurd

[Tim Rühsen]

On 03/29/2017 01:33 PM, Svante Signell wrote:
> On Sat, 2017-03-11 at 12:48 +0100, Tim Rühsen wrote:
>> Hi Svante,
>>
>> On Freitag, 10. März 2017 14:20:56 CET Svante Signell wrote:
>>> Hello,
> ...
>>> Thank you for your attention.
>>
>> You should address the gnulib project directly. You patch gnulib files, 
>> which 
>> are imported/generated during the Wget build.
>>
>> So please write to bug-gnu...@gnu.org. I am sure, your patches are welcome. 
>> And with that all projects using gnulib will benefit in the future.
> 
> Hi again.
> 
> Doing some more searching I found that the problem in gnulib is already fixed 
> by
> the upstream patch: gnulib.git-4084b3a1094372b960ce4a97634e08f4538c8bdd.patch
> dated 21 Feb 2017.
> 
> Since the latest version of wget is 1.19.1 was released 11 Feb 2017 this patch
> is not yet in upstream wget. Maybe a new bug-fix release 1.19.2 would be 
> useful?

Thanks for reporting back !

The next release will include the latest gnulib.

Regards, Tim



signature.asc
Description: OpenPGP digital signature

[Bug-wget] (unencrypted) Article explaining Wget in German computer magazin c't

[Tim Rühsen]

Ups, sorry for message encryption

On Freitag, 31. März 2017 14:49:45 CEST Tim Rühsen wrote:
> On 03/31/2017 02:47 PM, Giuseppe Scrivano wrote:
> > Hi Tim,
> > 
> > Tim Rühsen <tim.rueh...@gmx.de> writes:
> >> FYI,
> >> 
> >> c't 8/2017 has a 4 paged article explaining Wget and several kinds of
> >> usages (German language only).
> > 
> > cool!  Is there any online version of it?
> 
> Yes, but not for free, it is 1,49 €.
> 
> https://shop.heise.de/katalog/kleiner-datensauger
> 
> Regards, Tim



signature.asc
Description: This is a digitally signed message part.

[Bug-wget] Article explaining Wget in German computer magazin c't

[Tim Rühsen]

FYI,

c't 8/2017 has a 4 paged article explaining Wget and several kinds of
usages (German language only).

Regards, Tim




signature.asc
Description: OpenPGP digital signature

Re: [Bug-wget] Article explaining Wget in German computer magazin c't

[Tim Rühsen]

binIXjEbOxIW2.bin
Description: PGP/MIME version identification


encrypted.asc
Description: OpenPGP encrypted message

Re: [Bug-wget] [PATCH v4] (resend) Add --use-askpass=COMMAND support

[Tim Rühsen]

On Mittwoch, 7. September 2016 12:15:26 CEST Liam R. Howlett wrote:
> * Tim Ruehsen  [160907 04:49]:
> > Hi Liam,
> > 
> > I made a mistake and accidentally pushed your v4 patch to master a few
> > days
> > ago.
> > 
> > Here is a fix for several issues (warnings, C89 incompatibility, using
> > strpbrk to find end-of-line, append newline to error messages).
> > 
> > I removed your extra checks, so that a user can just work with executables
> > in> 
> > the PATH (posix_spawnp searches the path):
> >   wget --use-askpass=ssh-askpass 
> > 
> > If the executable does not exists (or something goes wrong when
> > executing),
> > wget will error & stop at that point anyways.
> > 
> > Here is the fix, WDYT ?
> 
> I like the change to use strpbrk.  This patch works with my testing.
> 
> Just a few comments:
> 
> 
> 1. Since you are using xmemdup0, can't you also drop these lines?
> (1101-1102):
>   /* Make sure there is a trailing 0 */
> tmp[bytes] = '\0';

This is needed to have a trailing 0, else string functions - in this case 
strpbrk - would crash (not knowing where to stop).

> I think you can change the read line to remove the -1 from the size of
> the read (line 1092) if you drop the above:
> 
> - bytes = read (com[0], tmp, sizeof (tmp) - 1);
> + bytes = read (com[0], tmp, sizeof (tmp));

Need that -1 for the trailing 0 :-)

> 2. Is there a whitespace issue here?
> +  if ((p = strpbrk(tmp, "\r\n")))
>^- Whitespace missing?

Yes, thank you !

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] [GSoC] Proposal: “Design and implementation of a framework for plugins.”

[Tim Rühsen]

On 04/02/2017 09:40 PM, Akash Rawal wrote:
> 
> 
> On 04/02/2017 10:41 PM, Darshit Shah wrote:
>> * Akash Rawal  [170331 23:34]:
>>> Hello,
>>>
>>> First of all, thanks for the initial review of my proposal!
>>>
>>> I have made some changes in response to the comments as well as last
>>> mail on the mailing list by Darshit Shah. Kindly consider reviewing it.
>>> For reference here is the link.
>>> https://docs.google.com/document/d/1SBM-jNG5dJL85C7iK9aid2eDlByrxOsYUNDrk4049vQ/edit?usp=sharing
>>>
>>>
>>>
>>> The proposal is still not final, in the sense that I am analysing the
>>> code for further ideas of what can be customized using plugins.
>>>
>>> I was asked to elaborate on rewriting of command line option processing
>>> code. So here are my thoughts on it.
>>>
>>> The current code in src/options.c has following drawbacks:
>>>  -  Built around order-insensitive and idempotent model (Supplying
>>> an option modifies a variable). This might be adequete for wget,
>>> but might not be the case for plugins. Repeated options or
>>> order of options can matter for plugins.
>> Breaking the order-insensitive and idempotent model of command line
>> processing is an absolute no-go. It will definitely break many scripts
>> out there. We do wish to make it easy to transition from Wget to Wget2
>> and this is a huge change in how options are processed.  Also, I see no
>> clear advantages to why this should be required.
> Backward compatibility will only be broken if options are changed,
> which I didn't plan to do. wget2 will support all options it
> supports now. Existing options shall remain order-insensitive
> and idempotent, only new options and plugin-side options may
> take advantage of greater flexibility.
> 
> Many programs have order-sensitive (e.g. ffmpeg, options
> apply to input or output file based on placement) and non-idempotent
> (e.g. -v (verbose) option in many program, -vv for higher verbosity)
>>>  -  The need for keeping the list of options sorted. I don't think
>>> we can expect plugin writers to do the same.
>> No. Plugins should not be allowed to tinker with certain core
>> functionalities and that includes how options are processed.
>>>  -  And most importantly, options cannot be added at runtime.
>> Which is a very good thing. Keeping everything at compile time means we
>> can have most of that code on the stack without the need for costly
>> dynamic memory allocations on each invokation. Options processing code
>> should definitely not be dynamic.
> Okay if that is your requirement I'll remove it from my proposal.
>>>
>>> So far I think rewriting the option processing code would be easier
>>> than trying to patch existing code to support custom options.
>>> Some parts of the original code could be reused but structure
>>> would undergo a drastic change anyway.
>>
>> Easier? Have you every tried writing one? The number of use cases and
>> corner cases you'd have to handle is massive. That alone could be an
>> entire GSoC project. You can't expect to write it as a byproduct of your
>> project.
> I have written simpler ones. But even in wget's code option processor is
> not so large (~400 LOC, starting from src/option.c:853)
> Cases for long options:
> --option
> --option=value
> --option value
> --no-option
> Cases for short options:
> -x
> -x value
> -xvalue
> -xyzw
> -xyzvalue
> -xyz value
>>
>>>
>>> The idea I have is to expose an interface for plugins to describe
>>> command line options they want to receive. I have attached a
>>> prototype header. From this interface, wget will have a list of
>>> command line options to support, along with plugin function to
>>> be called for handling plugin specific options. In addition to
>>> using this to parse command line, wget can Use the descriptions
>>> and category information provided to generate --help output.
>>
>> First let me ask you, how exactly do you envision a user using plugins
>> with Wget2? How are plugins selected? How does the user even know if a
>> plugin exists?
> How about:
> $ wget_plugins="foo" wget2 --foo-option http://example.com/file.html
> Or:
> $ wget2 --plugin=foo --foo-option http://example.com/file.html
> Hypothetical example: music site parser that filters by rating
> $ wget2 --plugin=music_rating_filter --music_min_rating=4 \
>   --recursive --level=20 \
>   http://example_music_site.com/music/1
> Multiple plugins:
> $ wget_plugins="foo:bar" wget2 --plugin=foo --plugin=bar \
>   --foo-option --bar-option=value http://example.com/file.html
> Or:
> $ wget2 --plugin=foo --plugin=bar --foo-option --bar-option=value \
>   http://example.com/file.html
> $ #The above is also an example of non-idempotent option --plugin
> Loading plugins which are not installed:
> $ wget2 --local-plugin=/path/to/libfoo.so --foo-option \
>   http://example.com/file.html
> 
> And regarding user 

Re: [Bug-wget] Misuse of idn2_free()

[Tim Rühsen]

On Samstag, 8. April 2017 18:55:12 CEST Gisle Vanem wrote:
> > Ups, I was sure that we already reallocate the idn2 memory in iri.c/
> > idn_encode(), but we don't do that yet.
> 
> You're confusing Wget with Wget2 :-)

Yes, I guess so too :-)

> > Could you please try this patch (on top of the current master):
> > 
> > diff --git a/src/iri.c b/src/iri.c
> > index 2a4da1de..75eaeea6 100644
> 
> Work fine!

Thanks for feedback !

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] [GSoC] Git & Github workflow

[Tim Rühsen]

On Sonntag, 2. April 2017 19:01:04 CEST Avinash Sonawane wrote:
> On Thu, Mar 23, 2017 at 9:08 PM, Avinash Sonawane  wrote:
> > If we get helpful responses, I will add this "contributing guide for
> > GSoC students" to our wiki.
> 
> As promised earlier, I just added a "contributing guide for GSoC
> students" to our wiki.[0]
> 
> I hope this helps GSoC aspiring students to not hold back just because
> they weren't comfortable with git/Github. :)
> 
> It's still far from being perfect. So I invite your contributions.
> 
> Finally, I would like to thank Tim for his inputs. Thank you!

Thank you - That is really useful contribution !

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Problems with (not) building wget against libiconv

[Tim Rühsen]

On Samstag, 15. April 2017 22:36:33 CEST Mojca Miklavec wrote:
> Hi,
> 
> I'm experiencing a recent regression in wget builds. It worked fine in
> version 1.17.1 and it fails now with 1.19.1.
> 
> I can do some bisection, but I guess I could blame at least the
> following commit:
> 
> http://git.savannah.gnu.org/cgit/wget.git/commit/src/url.c?id=7e585fe23dd69c
> adfc3f44df0c5f8bc47ab37cbb
> 
> that has not been properly restored in
> 
> http://git.savannah.gnu.org/cgit/wget.git/commit/src/url.c?id=517d799b6f94e6
> 0e302806a2daa14c7a4ac3fbbd
> 
> but more importantly: m4/gnulib-comp.m4 seems to opportunistically
> define HAVE_ICONV even if I try to do my best not to.
> 
> 
> We need portable wget binaries just for the sake of downloading a few
> packages. This is why we try to exclude as many libraries as possible
> (the only other alternative would be to build them statically). The
> biggest problem arises when the build machine has a certain package
> that cannot be found on the target machine.
> 
> We configure wget with
> 
> --enable-ipv6 --disable-iri --disable-nls --disable-ntlm
> --disable-pcre --without-libiconv-prefix --without-libintl-prefix
> --without-libuuid --without-libpsl --without-ssl --without-zlib
> 
> and the build fails both on Solaris 10 (on the same buildfarm as you
> are using) and on Mac OS X 10.6.
> 
> Both say:
> checking for iconv.h... yes
> 
> And then fail in the last step (with url.o containing iconv symbols).
> 
> Solaris:
> 
>   CCLD wget
> Undefined first referenced
>  symbol  in file
> libiconv_close  url.o  (symbol belongs to implicit
> dependency
> /opt/csw/lib/gcc/i386-pc-solaris2.10/5.2.0/../../../libiconv.so.2)
> libiconv_open   url.o  (symbol belongs to implicit
> dependency
> /opt/csw/lib/gcc/i386-pc-solaris2.10/5.2.0/../../../libiconv.so.2) libiconv
>url.o  (symbol belongs to implicit dependency
> /opt/csw/lib/gcc/i386-pc-solaris2.10/5.2.0/../../../libiconv.so.2) ld:
> fatal: symbol referencing errors. No output written to wget
> collect2: error: ld returned 1 exit status
> 
> 
> Mac:
> 
>   CCLD wget
> Undefined symbols:
>   "_iconv_close", referenced from:
>   _convert_fname in url.o
>   _convert_fname in url.o
>   "_iconv", referenced from:
>   _convert_fname in url.o
>   _convert_fname in url.o
>   "_iconv_open", referenced from:
>   _convert_fname in url.o
> ld: symbol(s) not found
> collect2: ld returned 1 exit status
> 
> 
> I would be grateful if this could be fixed.

Hi Mojca,

of course we try to fix this.

The first step is to reproduce the problem. I just tried on this build farm on 
'unstable10x' (Solaris 10) with exactly the configure flags from above.

All compiled well. What am I doing wrong (or what are you doing wrong ?)

Configure summary:
  Version:   1.19.1.23-b2c38
  Host OS:   solaris2.10
  Install prefix:/usr/local
  Compiler:  gcc
  CFlags:-I/opt/csw/include -I/opt/csw/include   -DNDEBUG   -
D_REENTRANT
  LDFlags:   
  Libs:  -L/opt/csw/lib -lgpgme -lassuan -lsocket -lgpg-error -L/
opt/csw/lib -lmetalink   
  SSL:   no
  Zlib:  no
  PSL:   no
  Digest:yes
  NTLM:  no
  OPIE:  yes
  POSIX xattr:   no
  Debugging: yes
  Assertions:no
  Valgrind:  Valgrind testing not enabled
  Metalink:  yes
  Resolver:  libc, --bind-dns-address and --dns-servers not available
  GPGME: yes
  IRI:   no

Also, you might add --disable-metalink to avoid libgpgme.

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Problems with (not) building wget against libiconv

[Tim Rühsen]

On Sonntag, 16. April 2017 00:24:22 CEST Mojca Miklavec wrote:
> On 16 April 2017 at 00:07, Tim Rühsen wrote:
> > On Samstag, 15. April 2017 22:36:33 CEST Mojca Miklavec wrote:
> >> Hi,
> >> 
> >> I'm experiencing a recent regression in wget builds. It worked fine in
> >> version 1.17.1 and it fails now with 1.19.1.
> >> 
> >> We configure wget with
> >> 
> >> --enable-ipv6 --disable-iri --disable-nls --disable-ntlm
> >> --disable-pcre --without-libiconv-prefix --without-libintl-prefix
> >> --without-libuuid --without-libpsl --without-ssl --without-zlib
> >> 
> >> and the build fails both on Solaris 10 (on the same buildfarm as you
> >> are using) and on Mac OS X 10.6.
> >> 
> >> Both say:
> >> checking for iconv.h... yes
> >> 
> >> And then fail in the last step (with url.o containing iconv symbols).
> > 
> > of course we try to fix this.
> > 
> > The first step is to reproduce the problem. I just tried on this build
> > farm on 'unstable10x' (Solaris 10) with exactly the configure flags from
> > above.
> > 
> > All compiled well. What am I doing wrong (or what are you doing wrong ?)
> 
> I didn't add any flags like I see them in your summary (I want to
> avoid linking against any of those libraries anyway):
> 
> -I/opt/csw/include -I/opt/csw/include
> -L/opt/csw/lib -lgpgme -lassuan -lsocket -lgpg-error
> -L/opt/csw/lib -lmetalink
> 
> Another difference is that I don't have /opt/csw/gnu in PATH, but I
> don't think that makes any difference in this case. I can send full
> logs for comparison if needed.
> 
> > Configure summary:
> >   Version:   1.19.1.23-b2c38
> >   Host OS:   solaris2.10
> >   Install prefix:/usr/local
> >   Compiler:  gcc
> >   CFlags:-I/opt/csw/include -I/opt/csw/include   -DNDEBUG   -
> > 
> > D_REENTRANT
> > 
> >   LDFlags:
> >   Libs:  -L/opt/csw/lib -lgpgme -lassuan -lsocket -lgpg-error
> >   -L/
> > 
> > opt/csw/lib -lmetalink
> > 
> >   SSL:   no
> >   Zlib:  no
> >   PSL:   no
> >   Digest:yes
> >   NTLM:  no
> >   OPIE:  yes
> >   POSIX xattr:   no
> >   Debugging: yes
> >   Assertions:no
> >   Valgrind:  Valgrind testing not enabled
> >   Metalink:  yes
> >   Resolver:  libc, --bind-dns-address and --dns-servers not
> >   available
> >   GPGME: yes
> >   IRI:   no
> 
>   Version:   1.19.1
>   Host OS:   solaris2.10
>   Install prefix:/usr/local
>   Compiler:  /opt/csw/bin/gcc-5.2
>   CFlags:-DNDEBUG   -D_REENTRANT
>   LDFlags:
>   Libs:
>   SSL:   no
>   Zlib:  no
>   PSL:   no
>   Digest:yes
>   NTLM:  no
>   OPIE:  yes
>   POSIX xattr:   no
>   Debugging: yes
>   Assertions:no
>   Valgrind:  Valgrind testing not enabled
>   Metalink:  no
>   Resolver:  libc, --bind-dns-address and --dns-servers not
> available GPGME: no
>   IRI:   no
> 
> Mojca

The question was more on which OpenCWS machine did you test (you said you are 
using the same build platform !? on 'unstable10x' was my test, libraries have 
been automatically detected).

My configure call:
./configure --enable-ipv6 --disable-iri --disable-nls --disable-ntlm --disable-
pcre --without-libiconv-prefix --without-libintl-prefix --without-libuuid --
without-libpsl --without-ssl --without-zlib

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Problems with (not) building wget against libiconv

[Tim Rühsen]

On Sonntag, 16. April 2017 10:00:05 CEST Mojca Miklavec wrote:
> On 16 April 2017 at 09:27, Tim Rühsen wrote:
> > The question was more on which OpenCWS machine did you test (you said you
> > are using the same build platform !? on 'unstable10x' was my test,
> > libraries have been automatically detected).
> 
> Yes, unstable10x. I suspect unstable10s would have the same problem, I
> can try. (But perhaps with slightly different environmental variables?
> Because your configure report shows additional libraries that mine
> doesn't.)
> 
> I didn't use any map files this time (I should use them for the final
> build).
> 
> The problem is reproducible at least on Mac and FreeBSD. Another Mac
> user reported that the problem went away after removing --disable-iri,
> but I guess that means linking against libiconv (which isn't available
> on Solaris by default and thus the binary would fail to work when
> there's no package management available).

First, let's find out why it works on my setup (I didn't setup anything except 
2 aliases, but maybe Dagobert Michelsen did that for me).

Maybe you can compare with your settings

> cat .bash_profile 
alias l='ls -la --color'
alias make='make -j10'
set +o ignoreeof
export RSYNC_PROXY=proxy:3128
PKG_CONFIG_PATH=/opt/csw/lib/pkgconfig
PATH=/opt/csw/gnu:/opt/csw/bin:$PATH:~/bin
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
export LANGUAGE=en_US.UTF-8

> set
BASH=/opt/csw/bin/bash
BASHOPTS=cmdhist:complete_fullquote:expand_aliases:extquote:force_fignore:hostcomplete:interactive_comments:login_shell:progcomp:promptvars:sourcepath
BASH_ALIASES=()
BASH_ARGC=()
BASH_ARGV=()
BASH_CMDS=()
BASH_LINENO=()
BASH_SOURCE=()
BASH_VERSINFO=([0]="4" [1]="3" [2]="33" [3]="1" [4]="release" [5]="i386-pc-
solaris2.10")
BASH_VERSION='4.3.33(1)-release'
BASICSETTINGS=/etc/environment/default-settings
COLUMNS=142
DIRSTACK=()
DISPLAY=localhost:10.0
EDITOR=vi
EUID=11058
FCEDIT=vi
FIGNORE='~:.o:.orig:.ori:.old:.backup'
GROUPS=()
HISTCONTROL=ignoreboth
HISTFILE=/home/rockdaboot/.bash_history
HISTFILESIZE=500
HISTSIZE=500
HOME=/home/rockdaboot
HOSTNAME=unstable10x
HOSTTYPE=i386
IFS=$' \t\n'
LANG=en_US.UTF-8
  
LANGUAGE=en_US.UTF-8
  
LC_ALL=en_US.UTF-8  
  
LESS=MQi
  
LINES=46
  
LOGNAME=rockdaboot  
  
MACHTYPE=i386-pc-solaris2.10
MAIL=/var/mail//rockdaboot
MAILCHECK=60
MANPATH=/usr/man:/usr/dt/man:/usr/openwin/man:/opt/SUNWspro/man:/opt/csw/man:/
usr/sfw/man
MLIBHOME=/opt/SUNWmlib
OPTERR=1
OPTIND=1
OSTYPE=solaris2.10
PAGER=less
PATH=/opt/csw/gnu:/opt/csw/bin:/usr/bin:/usr/sbin:/sbin:/usr/ccs/bin:/usr/dt/
bin:/usr/openwin/bin:/opt/SUNWspro/bin:/opt/bop/bin:/opt/csw/bin:/usr/sfw/
bin:/usr/sfw/sbin:/home/rockdaboot/bin
PIPESTATUS=([0]="0")
PKG_CONFIG_PATH=/opt/csw/lib/pkgconfig
PPID=6886
PS1='\[\e[1m\]\u\[\e[0m\]@\H [\[\e[1m\]global\[\e[0m\]]:\[\e[1m\]\w\[\e[0m\] > 
'
PS2='continued> '
PS3='select> '
PS4='+ '
PWD=/home/rockdaboot
RSYNC_PROXY=proxy:3128
SEDEBUG=0
SHELL=/opt/csw/bin/bash
SHELLOPTS=braceexpand:emacs:hashall:histexpand:history:interactive-
comments:monitor
SHLVL=1
SSHAGENT=/usr/bin/ssh-agent
SSH_CLIENT='192.168.1.2 37275 22'
SSH_CONNECTION='192.168.1.2 37275 192.168.1.33 22'
SSH_TTY=/dev/pts/2
TERM=xterm-256color
TZ=Europe/Berlin
UID=11058
USER=rockdaboot
VISUAL=vi
ZONENAME='[\[\e[1m\]global\[\e[0m\]]'
_=LANGUAGE
rockdaboot@unstable10x [global]:~ > setenv
-bash: setenv: command not found
rockdaboot@unstable10x [global]:~ > cat .bash_
.bash_history  .bash_profile  
rockdaboot@unstable10x [global]:~ > cat .bash_profile 
alias l='ls -la --color'
alias make='make -j10'
set +o ignoreeof
export RSYNC_PROXY=proxy:3128
PKG_CONFIG_PATH=/opt/csw/lib/pkgconfig
PATH=/opt/csw/gnu:/opt/csw/bin:$PATH:~/bin
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
export LANGUAGE=en_US.UTF-8


From what I saw in your config.log, both HAVE_ICONV_H and HAVE_ICONV are 
defined. Obviously -liconv does work with your settings. So I am still confused 
and the best things would be to have those settings to reproduce the problem.

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Problems with (not) building wget against libiconv

[Tim Rühsen]

On Sonntag, 16. April 2017 10:00:05 CEST Mojca Miklavec wrote:
> On 16 April 2017 at 09:27, Tim Rühsen wrote:
> > The question was more on which OpenCWS machine did you test (you said you
> > are using the same build platform !? on 'unstable10x' was my test,
> > libraries have been automatically detected).
> 
> Yes, unstable10x. I suspect unstable10s would have the same problem, I
> can try. (But perhaps with slightly different environmental variables?
> Because your configure report shows additional libraries that mine
> doesn't.)
> 
> I didn't use any map files this time (I should use them for the final
> build).
> 
> The problem is reproducible at least on Mac and FreeBSD.

Also I can't reproduce the issue on FreeBSD 11.

./configure summary:
  Version:   1.19.1.23-b2c38-dirty
  Host OS:   freebsd11.0
  Install prefix:/usr/local
  Compiler:  cc
  CFlags:-DNDEBUG   -D_THREAD_SAFE
  LDFlags:   
  Libs:  
  SSL:   no
  Zlib:  no
  PSL:   no
  Digest:yes
  NTLM:  no
  OPIE:  yes
  POSIX xattr:   yes
  Debugging: yes
  Assertions:no
  Valgrind:  Valgrind testing not enabled
  Metalink:  no
  Resolver:  libc, --bind-dns-address and --dns-servers not available
  GPGME: no
  IRI:   no

Please make sure, you test with latest wget git.

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Too verbose version information

[Tim Rühsen]

On 04/18/2017 04:40 AM, Mojca Miklavec wrote:
> Hello,
> 
> I find it a bit annoying that I end up with all of the following
> information after building wget:
> 
>> ./src/wget --version
> GNU Wget 1.19.1.25-92bf-dirty built on darwin10.8.0.
> 
> -cares +digest -gpgme -https +ipv6 -iri +large-file -metalink -nls
> -ntlm +opie -psl -ssl
> 
> Wgetrc:
> /usr/local/etc/wgetrc (system)
> Compile:
> /usr/bin/gcc -DHAVE_CONFIG_H
> -DSYSTEM_WGETRC="/usr/local/etc/wgetrc"
> -DLOCALEDIR="/usr/local/share/locale" -I. -I../../wget-from-git/src
> -I../lib -I../../wget-from-git/lib
> -I/Developer/SDKs/MacOSX10.5.sdk/usr/include -DNDEBUG -arch ppc
> -isysroot /Developer/SDKs/MacOSX10.5.sdk -mmacosx-version-min=10.5
> Link:
> /usr/bin/gcc -DNDEBUG -arch ppc -isysroot
> /Developer/SDKs/MacOSX10.5.sdk -mmacosx-version-min=10.5 -arch ppc
> -L/Developer/SDKs/MacOSX10.5.sdk/usr/lib -isysroot
> /Developer/SDKs/MacOSX10.5.sdk -mmacosx-version-min=10.5 ftp-opie.o
> ../lib/libgnu.a -liconv
> 
> In particular: why should the user be aware of the exact path where I
> had wget stored on my hard drive (-I../../wget-from-git/src), of the
> fact that I didn't configure in-place and why should the user bother
> with so many environmental variables and the fact that the binary was
> cross-compiled?
> 
> Is there any way to remove all that information?
> 
> 
> The first part is reasonable (even if I don't like the fact that the
> exact OS version is hardcoded; after security updates the binary will
> change only due to the increased minor version, so reproducibility is
> lost):
> 
> GNU Wget 1.19.1.25-92bf-dirty built on darwin10.8.0.
> 
> -cares +digest -gpgme -https +ipv6 -iri +large-file -metalink -nls
> -ntlm +opie -psl -ssl
> 
> Wgetrc:
> /usr/local/etc/wgetrc (system)
> 
> but I would like to have a way to avoid storing so many details in the
> second part.

This is basically not for the user but for us developers.
If someone opens a bug report, (s)he should give us this output, so we
can try to reproduce the problem with as less effort as possible.

We don't want you to remove this information :-). The users of your
package will address us (and not you) with problems.

But you are free to privately keep a set of patches to remove this
information (and maybe add your own, e.g. your email address for bug
reports).

Regards, Tim



signature.asc
Description: OpenPGP digital signature

Re: [Bug-wget] Problems with (not) building wget against libiconv

[Tim Rühsen]

On 04/18/2017 12:57 PM, Tim Rühsen wrote:
> On 04/18/2017 04:27 AM, Mojca Miklavec wrote:
>> I copiled a folder with bootstrapped wget to the Solaris box, but as
>> expected I ended up with:
>>
>>> ldd src/wget
>> libsocket.so.1 =>/lib/libsocket.so.1
>> libnsl.so.1 =>   /lib/libnsl.so.1
>> librt.so.1 =>/lib/librt.so.1
>> libiconv.so.2 => /opt/csw/lib/libiconv.so.2
>> libunistring.so.2 => /opt/csw/lib/libunistring.so.2
>> ...
>>
>> And having libraries from /opt/csw there is simply not acceptable
>> since the binaries won't work on machines without OpenCSW installed.
>>
>> So my next question would be: how can I get rid of dependency on libiconv?
>>
>> I didn't have this problem in version 1.17.1.
>>
>> Repeating the configure string:
>>
>> --enable-ipv6 --disable-iri --disable-nls --disable-ntlm \
>> --disable-pcre --without-libiconv-prefix --without-libintl-prefix \
>> --without-libuuid --without-libpsl --without-ssl --without-zlib
> 
> The OpenCSW Solaris box "unstable10x" and likely all others are
> configured to use a GNU environment for building. That includes a
> separate 'libiconv.so' as well as system headers being loaded from the
> GNU environment.
> 
> ./configure correctly finds and uses the 'iconv.h' and 'libiconv.so'
> within the GNU environment. That's not what you want - you basically
> want a build on a plain Solaris box so that the 'wget' executable does
> not reference any GNU libraries.
> 
> Solaris has iconv functionality built into the libc - so there is no
> reason for you not to use it. The question is, how you can do do that.
> 
> You have to do some manual changes (you could set up a shell script
> doing that with 'sed'):
> 
> 1. Change include of iconv.h in src/url.h:
> 
> diff --git a/src/url.c b/src/url.c
> index 4aaef63..35b8a49 100644
> --- a/src/url.c
> +++ b/src/url.c
> @@ -44,7 +44,7 @@ as that of the covered work.  */
>  #include "c-strcase.h"
> 
>  #ifdef HAVE_ICONV
> -# include 
> +# include "/usr/include/iconv.h"
>  #endif
>  #include 
> 
> 2. edit src/Makefile.am to remove -liconv and -lunistring (each has two
> occurrences)
> 
> 
> Now cd into src and 'make clean && make' and your 'wget' executable is
> clean:
> 
> ldd wget:
> libsocket.so.1 =>/lib/libsocket.so.1
> libnsl.so.1 =>   /lib/libnsl.so.1
> librt.so.1 =>/lib/librt.so.1
> libc.so.1 => /lib/libc.so.1
> libmp.so.2 =>/lib/libmp.so.2
> libmd.so.1 =>/lib/libmd.so.1
> libscf.so.1 =>   /lib/libscf.so.1
> libaio.so.1 =>   /lib/libaio.so.1
> libdoor.so.1 =>  /lib/libdoor.so.1
> libuutil.so.1 => /lib/libuutil.so.1
> libgen.so.1 =>   /lib/libgen.so.1
> libm.so.2 => /lib/libm.so.2
> 
> 
> Maybe there is an easier way on OpenCSW. If you find out, let us know.
> 
> 
> The problem with using the Solaris compiler and/or iconv() is that there
> is a known bug in the implementation that ./configure checks for.
> If it finds this bug, HAVE_ICONV will not be defined and compilation of
> src/url.c then fails.

Just pushed some commits. That should enable you to
$ CC=cc ./configure ... (all your options)

to get a clean wget executable (with just Solaris system libraries).

Regards, Tim



signature.asc
Description: OpenPGP digital signature

Re: [Bug-wget] PATCH: tests for SSL

[Tim Rühsen]

Hi Vijo,

On 04/18/2017 06:56 PM, Vijo Cherian wrote:
> Added a framework for perl based SSL tests, and some tests to start with.
> In case this is of interest, I will add more tests for SSL: client
> certificates, CRLs, negative tests etc.
> Also not included : making these tests a part of "make check".
> 
> TESTING :  only on ubuntu 16

thank you for this contribution !

Your commit has been slightly amended (trailing white space removed,
commit message changed to GNU style) and pushed to master.

Maybe you are interested to inspect Wget2 testing to see if your tests
are already covered there. If not we would be pleased if you could add
them there as well.

Just 'git clone https://github.com/rockdaboot/wget2' and jump in !

Regards, Tim



signature.asc
Description: OpenPGP digital signature

Re: [Bug-wget] Patch: Always surround the "WARC-Target-URI" value with angle brackets

[Tim Rühsen]

Thanks, Bejamin,

your patch is applied (trivial, no FSF copyright assignment required).

Regards, Tim

On Freitag, 3. März 2017 09:00:57 CET Benjamin Esham wrote:
> Hello,
> 
> When producing WARC files, Wget records the requested URI in the
> "WARC-Target-URI" field. I noticed that Wget encloses the value of this URI
> within  in blocks with "WARC-Type: request", but not those
> with types of "response", "resource", "revisit", or "metadata". Enclosing
> URIs within angle brackets is required by the spec [1]. I'm attaching a
> patch that adds the angle brackets for all block types.
> 
> (Doing this for "request" blocks was the subject of bug 47281 [2], which was
> fixed almost exactly a year ago. My patch simply extends the use of the
> warc_write_header_uri function to the other appropriate places.)
> 
> Here is a truncated example of the output from Wget 1.19.1:
> 
> WARC/1.0
> WARC-Type: response
> WARC-Record-ID: 
> WARC-Warcinfo-ID: 
> WARC-Concurrent-To: 
> WARC-Target-URI: https://www.gnu.org/software/wget/
> 
> And from the patched version:
> 
> WARC/1.0
> WARC-Type: response
> WARC-Record-ID: 
> WARC-Warcinfo-ID: 
> WARC-Concurrent-To: 
> WARC-Target-URI: 
> 
> Best regards,
> 
> Benjamin
> 
> 
> [1] http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf
> 
> [2] http://savannah.gnu.org/bugs/?47281



signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] recursive_retrieve()

[Tim Rühsen]

On Freitag, 3. März 2017 13:18:38 CET João Bernardo Sousa wrote:
> Hello.
> 
> I'm getting this error when debugging "Removing file due to recursive
> rejection criteria in recursive_retrieve()".
> 
> I'll be glad if you could help me:
> 
> http://stackoverflow.com/questions/42578544/downloading-pdf-files-with-wget-> 
> characters-after-file-extension

Did you try wildcard matching ? (-A "*.pdf*")

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] PATCH: Fix FTBFS on GNU/Hurd

[Tim Rühsen]

Hi Svante,

On Freitag, 10. März 2017 14:20:56 CET Svante Signell wrote:
> Hello,
> 
> wget currently does not build from source on GNU/Hurd since Debian version
> 1.18- 4. This is due to that HAVE_PTHREAD_RWLOCK_RDLOCK_PREFER_WRITER is
> not defined by configure and then assumes that the function
> pthread_rwlockattr_setkind_np() is available. On GNU/Hurd it is not. The
> Hurd libpthread is built from the sources in glibc/libpthread/*, not in
> glibc/nptl.
> 
> The attached patch fixes the build problems by conditioning on __GNU__ which
> is unique for GNU/Hurd. However, a better solution would probably be to
> detect if the function pthread_rwlockattr_setkind_np() is available in
> configure.ac or m4/*.m4 or check if TPS+SCHED_FIFO/SCHED_RR is supported by
> pthread_rwlock_rdlock(), see below.
> 
> As written in the comments of m4/pthread_rwlock_rdlock.m4 POSIX-2008 only
> requires this for specific implementations:
> 
> dnl POSIX:2008 makes this requirement only for implementations that support
> TPS dnl (Thread Priority Scheduling) and only for the scheduling policies
> SCHED_FIFO dnl and SCHED_RR, see
> dnl
> http://pubs.opengroup.org/onlinepubs/9699919799/functions/pthread_rwlock_rd
> l ock.html
> dnl but test verifies the guarantee regardless of TPS and regardless of
> dnl scheduling policy.
> 
> Thank you for your attention.

You should address the gnulib project directly. You patch gnulib files, which 
are imported/generated during the Wget build.

So please write to bug-gnu...@gnu.org. I am sure, your patches are welcome. 
And with that all projects using gnulib will benefit in the future.

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Vulnerability Report - CRLF Injection in Wget Host Part

[Tim Rühsen]

On Dienstag, 7. März 2017 02:01:06 CET Orange Tsai wrote:
> I am surprise that `http://u...@evil.com:8...@good.com` will connect to `
> evil.com`, not `good.com`.
> Most of URL parser will recognize `good.com` is host part. Like this
> advisory, https://curl.haxx.se/docs/adv_20161102J.html
> It seem more dangerous if a developer still rely on the result of parse URL
> than my original report.
> 
> Some testing:
> $ python try.py 'http://user@127.3.3.3:80@127.2.2.2/x'
> 
> Python scheme=http, host=user@127.3.3.3:80@127.2.2.2, port=
> PHP scheme=http, host=127.2.2.2, port=
> Perl scheme=http, host=127.2.2.2, port=80
> Ruby2 scheme=http, host=127.2.2.2, port=
> GO scheme=http, host=127.2.2.2, port=
> Java scheme=http, host=, port=-1
> JS scheme=http, host=127.2.2.2, port=null
> 
> 
> 
> But it seems also the same root cause and fixed at this patch. :)
> By the way, would you mind that allocating a CVE-ID to address this?

I'd appreciate that. But I never did that, so who does allocate a CVE how and 
where ? I am willing to learn :-)

Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] inet_ntop() in mswindows.c

[Tim Rühsen]

On Montag, 6. März 2017 16:45:57 CET Gisle Vanem wrote:
> Just a detail, but in src/mswindows.c, there is:
> 
>   #ifdef ENABLE_IPV6
>   /* An inet_ntop implementation that uses WSAAddressToString.
>  Prototype complies with POSIX 1003.1-2004.  This is only used under
>  IPv6 because Wget prints IPv4 addresses using inet_ntoa.  */
> 
> This is wrong since 1) inet_ntoa() is no longer used. And 2) since
> inet_ntop() is used for IPv4 too, 'ENABLE_IPV6' should then become
> '!defined(HAVE_INET_NTOP)'. Thus:
> 
> @@ -572,10 +572,10 @@
>  }
> 
> 
> -#ifdef ENABLE_IPV6
> +#if !defined(HAVE_INET_NTOP)
>  /* An inet_ntop implementation that uses WSAAddressToString.
> -   Prototype complies with POSIX 1003.1-2004.  This is only used under
> -   IPv6 because Wget prints IPv4 addresses using inet_ntoa.  */
> +   Prototype complies with POSIX 1003.1-2004.  This is used
> +   for both IPv4 and IPv6.  */
> 
>  const char *
>  inet_ntop (int af, const void *src, char *dst, socklen_t cnt)

inet_ntop should be covered by gnulib, also for mswindows. What about removing 
inet_ntop from mswindows.c ? Does it compile and work ?

Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Misuse of idn2_free()

[Tim Rühsen]

Thanks, Gisle.

pushed with several additional fixes/cleanups regarding idn2.

Regards, Tim

On Samstag, 8. April 2017 10:09:22 CEST Gisle Vanem wrote:
> The 'idn_decode()' function now simply uses 'xstrdup()'.
> And in host.c + connect.c there are calls to 'idn2_free()'
> on this pointer:
>   if (opt.enable_iri && (name = idn_decode ((char *) print)) != NULL)
>{
> int len = strlen (print) + strlen (name) + 4;
> str = xmalloc (len);
> snprintf (str, len, "%s (%s)", name, print);
> str[len-1] = '\0';
> idn2_free (name);   << !
>}
> 
> Since the above 'name' is NOT from libidn, I get a crash when
> mixing a MinGW built libidn2.dll with a MSVC built Wget.exe.
> 
> I think someone forgot to change the above code when 'idn_decode()'
> got simplified. This patch works for me:
> 
> --- a/connect.c 2017-01-19 21:37:55
> +++ b/connect.c 2017-04-08 09:57:24
> @@ -284,7 +284,7 @@
> str = xmalloc (len);
> snprintf (str, len, "%s (%s)", name, print);
> str[len-1] = '\0';
> -  idn2_free (name);
> +  xfree (name);
>   }
> 
> logprintf (LOG_VERBOSE, _("Connecting to %s|%s|:%d... "),
> 
> --- a/host.c 2017-01-19 21:37:55
> +++ b/host.c 2017-04-08 10:02:42
> @@ -850,7 +850,7 @@
> str = xmalloc (len);
> snprintf (str, len, "%s (%s)", name, host);
> str[len-1] = '\0';
> -  idn2_free (name);
> +  xfree (name);
>   }
> 
> logprintf (LOG_VERBOSE, _("Resolving %s... "),
> 
> 
> 
> --gv



signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Misuse of idn2_free()

[Tim Rühsen]

Ups, I was sure that we already reallocate the idn2 memory in iri.c/
idn_encode(), but we don't do that yet.

Could you please try this patch (on top of the current master):

diff --git a/src/iri.c b/src/iri.c
index 2a4da1de..75eaeea6 100644
--- a/src/iri.c
+++ b/src/iri.c
@@ -290,6 +290,13 @@ idn_encode (const struct iri *i, const char *host)
 
   xfree (utf8_encoded);
 
+  if (ret == IDN2_OK && ascii_encoded)
+{
+  char *tmp = xstrdup (ascii_encoded);
+  idn2_free (ascii_encoded);
+  ascii_encoded = tmp;
+}
+
   return ret == IDN2_OK ? ascii_encoded : NULL;
 }

I can also see that idn_decode() can be resurrected with the new libidn2 2.0.0 
functionality. I'll do that within the next days.

Regards, Tim

On Samstag, 8. April 2017 11:52:59 CEST Gisle Vanem wrote:
> Tim Rühsen wrote:
> > Thanks, Gisle.
> > 
> > pushed with several additional fixes/cleanups regarding idn2.
> 
> Too much cleanups I guess since it's crashing because no
> 'idn2_free()' called when needed. This works here:
> 
> --- a/url.c 2017-04-08 11:24:21
> +++ b/url.c 2017-04-08 11:38:37
> @@ -944,6 +944,7 @@
>   {
> xfree (u->host);
> u->host = new;
> +  u->idn_allocated = true;
> host_modified = true;
>   }
>   }
> @@ -1222,6 +1223,9 @@
>   {
> if (url)
>   {
> +  if (url->idn_allocated)
> +idn2_free (url->host);
> +  else
> xfree (url->host);
> 
> xfree (url->path);
> 
> --- a/url.h 2017-04-08 11:24:21
> +++ b/url.h 2017-04-08 11:35:26
> @@ -84,6 +84,7 @@
> enum url_scheme scheme;   /* URL scheme */
> 
> char *host;   /* Extracted hostname */
> +  bool idn_allocated;   /* 'host' allocated by libidn2 */
> int port; /* Port number */
> 
> ---



signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Problems with (not) building wget against libiconv

[Tim Rühsen]

Hi Mojca,

On 04/18/2017 04:27 AM, Mojca Miklavec wrote:
> On 16 April 2017 at 18:54, Tim Rühsen wrote:
>>
>> Please make sure, you test with latest wget git.
> 
> I have some problems with bootstrapping.
> 
> On Solaris it's:
> 
>> ./bootstrap
> ./bootstrap: syntax error at line 91: `me_=$' unexpected

I experience this as well and use 'bash ./bootstrap'.
Just pushed a commit to mention this in README.checkout.

> On Mac it's:
> 
> sed: 1: "lib/unicase/special-cas ...": extra characters at the end of l 
> command
> ./bootstrap: bootstrap_post_import_hook failed

The 'sed' command used in 'bootstrap.conf' seems to be incompatible to
GNU sed. Just pushed a commit to make this more portable. Please test
and report.

> After that the build succeeds, but it's linking against libiconv
> (contrary to --without-libiconv-prefix):
> 
>> otool -L wget
> wget:
> /usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 125.2.11)
> 
> On Mac that should be OK (I hope), but on Solaris there is no libiconv
> by default.
> 
> I copiled a folder with bootstrapped wget to the Solaris box, but as
> expected I ended up with:
> 
>> ldd src/wget
> libsocket.so.1 =>/lib/libsocket.so.1
> libnsl.so.1 =>   /lib/libnsl.so.1
> librt.so.1 =>/lib/librt.so.1
> libiconv.so.2 => /opt/csw/lib/libiconv.so.2
> libunistring.so.2 => /opt/csw/lib/libunistring.so.2
> ...
> 
> And having libraries from /opt/csw there is simply not acceptable
> since the binaries won't work on machines without OpenCSW installed.
> 
> So my next question would be: how can I get rid of dependency on libiconv?
> 
> I didn't have this problem in version 1.17.1.

I'll have a look into this.

Regards, Tim



signature.asc
Description: OpenPGP digital signature

Re: [Bug-wget] wget Tutorial

[Tim Rühsen]

Hi Frederick,

please always write to the mailing list, private mail only if it is
about private things ;-)

I read a bit the ISO/IEC 23009-1:2014 spec (Information technology --
Dynamic adaptive streaming over HTTP (DASH) -- Part 1: Media
presentation description and segment formats).


In short: The worst way of streaming data I have ever seen (totally
blown up) !

Anyways, it's all about (real) timing. The file names can be
'calculated' by some very obscure formula using some details from the
client_manifest.mpd and the current time. If any of those values in the
mpd file ever change, the file names will be totally different.

On *nix you can use the following script to download/record the stream:


#!/bin/bash

outfile="bbc2017-07-30.mp4"

rm -f $outfile
wget -q -O-
http://vs-dash-ww-rd-live.bbcfmt.hs.llnwd.net:80/al/lossless/A1/IS.mp4
>$outfile

lastseq=$(expr $(date +%s) '*' 100 / 384 - 32)
seq=$lastseq

while true; do
  # wait for seq to change - then we have 3.84s to download it
  while test $seq = $lastseq; do
sleep 0.5
seq=$(expr $(date +%s) '*' 100 / 384 - 32)
  done
  lastseq=$seq
  wget -q -O-
http://vs-dash-ww-rd-live.bbcfmt.hs.llnwd.net:80/al/lossless/A1/${seq}.m4s
>>$outfile
  ls -la $outfile
done



This won't work if you have a clock skew or a slow bandwidth.

Have fun with it !

With Best Regards, Tim



On 08/02/2017 02:52 AM, Frederick George Wilson wrote:
> Dear Tim,
> 
> I wrote a tutorial on listening to the BBC Proms 2017 via wget:
> https://archive.org/download/70075/2017-07-31-updated-tutorial-BBC-MPEG-DASH-FLAC.txt
> which is listed on the Internet Archive item details page:
> http://archive.org/details/70075 .
> 
> The -w 3 and --random-wait command line elements work most of the time --
> however, during particularly slow or quiet stretches of music or spoken
> word, the 404 Not Found errors pile up as wget runs ahead of the stream, or
> conversely, dense stretches of music cause wget to lag behind.  I've tried
> with -w 2 and -w 4, but neither of those ever captures more than a few
> minutes.
> 
> Any suggestions?
> 
> 
> Many Thanks,
> FW
> 



signature.asc
Description: OpenPGP digital signature

Re: [Bug-wget] Gzip Content-Encoding Patches

[Tim Rühsen]

On Montag, 31. Juli 2017 23:17:42 CEST Tim Schlueter wrote:
> Hi,
> 
> Please see the attached patches which add automatic gzip decompression
> for HTTP files with the Content-Encoding response header set correctly.
> 
> It also adjusts a downloaded file's extension for br, compress, and
> deflate Content-Encodings.
> 
> Since the first patch set:
> * doc/wget.texi has been updated to reflect the changes in the patches.
> * Commit messages have been changed to be in the GNU change log style.
> * The patches are attached to this email instead of being in the body.

Very good work !

I pushed your commit to branch 'gzip' at https://gitlab.com/gnuwget/wget for 
further review and CI testing (just had time to glance at it and made a few 
gzipped downloads :-))

> I have not yet had a chance to look at what would be involved to add
> automated tests for this patch set.

IMO, the python test server should realize the offered encoding and compress 
the response body appropriately. This might be trivial to a python coder.

Anyways, if you have resources to help out at wget2 (https://gitlab.com/
gnuwget/wget), that would be awesome.

Will test your path(es) tomorrow and give you feedback.

With Best Regards, Tim

signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Errors-only mode

[Tim Rühsen]

On Donnerstag, 10. August 2017 22:26:44 CEST Dale R. Worley wrote:
> Tim Rühsen <tim.rueh...@gmx.de> writes:
> > wget -o/dev/null URL
> > 
> > You can check for errors via the $? (on Linux, there should be something
> > comparable on other systems).
> 
> Yes, but since I'm running it in crontab, I really do want to have
> output if an error occurs.  Of course, I can rig that by testing $?, but
> I'd rather see something like grep's behavior, where there is output to
> stderr if and only if an error was detected.

There are many different types of errors, some of them might be expected (e.g. 
404 is an error for you, but in a different scenario it is expected and 200 is 
an error).

Anyways, one practical approach would be to use e.g. '-o/tmp/log' and print 
the last X lines on return status != 0.

Example:
wget -o/tmp/log URL || tail -3 /tmp/log

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Errors-only mode

[Tim Rühsen]

On Mittwoch, 9. August 2017 22:08:12 CEST Dale R. Worley wrote:
> Is there a way of invoking wget that produces no output if the operation
> is successful (let's assuming that I'm fetching exactly one URL) and
> produces appropriate error messages if it is not?
> 
> I would have thought this was easy, but I can't figure out how to do
> it.  Even redirecting stdout to null doesn't work, since wget sends a
> final success message to stderr:

wget -o/dev/null URL

You can check for errors via the $? (on Linux, there should be something 
comparable on other systems).

$ wget -o/dev/null www.google.de
$ echo $?
0

$ wget -o/dev/null www.google.de/failure
$ echo $?
8

Regards, Tim

signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Code installation issues

[Tim Rühsen]

Hi Michael,


On 07/11/2017 03:37 PM, Michael wrote:
> Hello there,
> 
> I have installed the wget2 code on a Raspberry pi machine.
> 
> The make check report error on the unit tests: "
> test_buffer_printf: Failed with format ('%hhd','-1'): '255' != '-1'
> test_buffer_printf: Failed with format ('%hhd','-10'): '246' != '-10'
> test_buffer_printf: Failed with format ('%hhd','-1'): '255' != '-1'
> test_buffer_printf: Failed with format ('%hhi','-1'): '255' != '-1'
> test_buffer_printf: Failed with format ('%hhi','-10'): '246' != '-10'
> test_buffer_printf: Failed with format ('%hhi','-1'): '255' != '-1'
> .
> .
> .
> test_buffer_printf: Failed with format ('%-011.11hhi','-1'): '255'
> != '-001'
> test_buffer_printf: Failed with format ('%-011.11hhi','-10'): '246'
> != '-010'
> test_buffer_printf: Failed with format ('%-011.11hhi','-1'): '255'
> != '-001'
> ERROR: 4056 out of 195769 basic tests failed
> This may completely break Wget functionality !!!"

I'll check that... looks like a 'short int' just has 8 bits in your
environment !?

> 
> After sudo make install I run /usr/local/bin/wget2 and get:
> "wget2: error while loading shared libraries: libwget.so.0: cannot open
> shared object file: No such file or directory"
> 
> I managed to operate src/wget2 properly.
> 
> Where should I install the libwget.so.0?

The library should go to /usr/local/lib/.
After 'make  install' it needs a 'ldconfig' to find the library (assumed
that /usr/lib/local is configured somewhere in /etc/ld.so.conf.d/*).

With Best Regards, Tim




signature.asc
Description: OpenPGP digital signature

Re: [Bug-wget] Test certificate host name verification fails with GnuTLS 3.5.12+

[Tim Rühsen]

On Samstag, 8. Juli 2017 15:32:44 CEST Ludovic Courtès wrote:
> Hello,
> 
> I experienced the test failure reported at
>  for
> ‘testenv/Test--https.py’ and related tests with:
> 
>   The certificate's owner does not match hostname
> 
> There’s no problem when wget is built against GnuTLS 3.5.9; the test
> failure shows up when wget is built against GnuTLS 3.5.13.
> 
> After digging a bit, I found this change in GnuTLS 3.5.12 ‘NEWS’:
> 
> --8<---cut here---start->8---
> ** libgnutls: gnutls_x509_crt_check_hostname2() no longer matches IP
> addresses against DNS fields of certificate (CN or DNSname). The previous
> behavior was to tolerate some misconfigured servers, but that was
> non-standard and skipped any IP constraints present in higher level
> certificates. --8<---cut
> here---end--->8---
> 
> I think the fix is (1) to explicitly regenerate test certificates that
> use “localhost” as their ‘DNSname’ (when replying to certtool’s “Enter a
> dnsName of the subject of the certificate”), and (2) to use “localhost”
> instead of “127.0.0.1” in test URIs.
> 
> Thoughts?

Thanks again, fixed now by

- hard-coding the server domain to 'localhost'
- replacing 127.0.0.1 by localhost in several tests
- regenerating the server cert and crl files

> 
> Ludo’.

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Adoption to content management systems

[Tim Rühsen]

Hello Michael,

On Freitag, 7. Juli 2017 12:17:18 CEST Michael wrote:
> Hello,
> 
> I intend to do some changes to the wget program so it will fit the WordPress
> and other content management systems. I will send you the revised code when
> I am done.
> 
> These are very important changes as it will enable us to keep the content
> management system in the backend and generate static html pages out of it:
> 1.Security - the place of the content management system remains hidden
> from crackers and virus developers.
> 2.Speed - static html pages loads much faster than program generated
> program (such as php).
> 3.Site integrity - we can see the generated site structure and take
> actions to optimize it.
> 4.File names will be normal file name rather than "index?p=41.html".
> 
> I have created requirement document for it. I want to get your feedback on
> it before I start coding:
> https://docs.google.com/document/d/1vo-gUg_rn7H6whpYgFq9K5TkP7fEi3DvMbVNwOhs
> WE8/edit?usp=sharing

Thanks for your writing and your thoughts !

You cover a few points that already have been asked for and I would appreciate 
it if you are going to work on the points in your document.

We are currently working on Wget2[1], so please consider to put new features/
code in here as well. The basic steps are
- sign up at Gitlab
- fork [1]
- add the new option (maybe --generate-static is better than --wordpress !?)
- add the first piece (script code)
- add one or more test cases (in tests/)
- create a MR (Merge Request) and we review your code

Let us know if you have questions.
Feel free to open issues at [1].

Happy coding ! :-)

[1] https://gitlab.com/gnuwget/wget2

> 
> Thank you,
> 
> Michael

With Best Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] PATCH: tests for SSL

[Tim Rühsen]

Hi Vijo,

great work !

There is just a bit of polishing to be done... :-)


- some whitespace warnings (maybe you set your editor to remove trailing
whitespace when saving !?):

.git/rebase-apply/patch:516: trailing whitespace.
my $revokecmd = "openssl ca -config $caconf -revoke $servercrt &&
.git/rebase-apply/patch:795: trailing whitespace.
issuerAltName=issuer:copy
.git/rebase-apply/patch:802: trailing whitespace.
 certificate = $dir/interca.crt
.git/rebase-apply/patch:865: trailing whitespace.
issuerAltName=issuer:copy
.git/rebase-apply/patch:872: trailing whitespace.
 certificate = $dir/test-ca-cert.pem
warning: 5 lines add whitespace errors.


- you use the 'openssl' tool from the tests.
Is it possible to create the required files apart from the tests and
store them in tests/certs ? You can provide a script and/or a README how
to create these files (see testenv/certs/README).
If that is not possible, the tests should see if the 'openssl' command
is available. If unavailable the test(s) should SKIP.


- some tests fail here with 'make check -j4' but succeed with -j1
(failures are a bit random):
FAIL: Test-https-tlsv1.px
FAIL: Test-https-tlsv1x.px
FAIL: Test-https-clientcert.px


- this test always fails (wget is build with GnuTLS):
FAIL: Test-https-badcerts.px

Releasing 0x559c258a74d0 (new refcount 1).
GnuTLS: ASN1 parser: Error in DER parsing.
Closed fd 3
Unable to establish SSL connection.
Can't use an undefined value as a symbol reference at SSLServer.pm line 131.
Test failed: wrong code returned (was: 4, expected: 5)
FAIL Test-https-badcerts.px (exit status: 255)



With Best Regards, Tim



On 04/27/2017 06:19 AM, Vijo Cherian wrote:
> Thank you Tim.
> 
> Attached is the last set of patches from me for SSL testing.
> I will apply these to wget2 and start working on wget2 from now.
> 
> Best,
> Vijo.
> 
> 
> On Wed, Apr 19, 2017 at 4:12 AM, Tim Rühsen <tim.rueh...@gmx.de> wrote:
> 
>> Hi Vijo,
>>
>> On 04/18/2017 06:56 PM, Vijo Cherian wrote:
>>> Added a framework for perl based SSL tests, and some tests to start with.
>>> In case this is of interest, I will add more tests for SSL: client
>>> certificates, CRLs, negative tests etc.
>>> Also not included : making these tests a part of "make check".
>>>
>>> TESTING :  only on ubuntu 16
>>
>> thank you for this contribution !
>>
>> Your commit has been slightly amended (trailing white space removed,
>> commit message changed to GNU style) and pushed to master.
>>
>> Maybe you are interested to inspect Wget2 testing to see if your tests
>> are already covered there. If not we would be pleased if you could add
>> them there as well.
>>
>> Just 'git clone https://github.com/rockdaboot/wget2' and jump in !
>>
>> Regards, Tim
>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [Bug-wget] PATCH: tests for SSL

[Tim Rühsen]

Hi Vijo,

maybe I miss something ?

$ cat tests/Test-https-pfs.log 
Running test Test-https-pfs
IO::Socket::IP configuration failed at SSLServer.pm line 71.
Failed to get ssl sock at SSLServer.pm line 168.
Can't call method "accept" on an undefined value at SSLServer.pm line 86.

Regards, Tim

On Samstag, 29. April 2017 13:21:04 CEST Vijo Cherian wrote:
> Thank you for your review and comments, Tim.
> 
> Attached is the revised set of patches for the same tests.
> 
> 
> Best,
> Vijo.
> 
> On Fri, Apr 28, 2017 at 3:42 AM, Tim Rühsen <tim.rueh...@gmx.de> wrote:
> > Hi Vijo,
> > 
> > great work !
> > 
> > There is just a bit of polishing to be done... :-)
> > 
> > 
> > - some whitespace warnings (maybe you set your editor to remove trailing
> > whitespace when saving !?):
> > 
> > .git/rebase-apply/patch:516: trailing whitespace.
> > my $revokecmd = "openssl ca -config $caconf -revoke $servercrt &&
> > .git/rebase-apply/patch:795: trailing whitespace.
> > issuerAltName=issuer:copy
> > .git/rebase-apply/patch:802: trailing whitespace.
> > 
> >  certificate = $dir/interca.crt
> > 
> > .git/rebase-apply/patch:865: trailing whitespace.
> > issuerAltName=issuer:copy
> > .git/rebase-apply/patch:872: trailing whitespace.
> > 
> >  certificate = $dir/test-ca-cert.pem
> > 
> > warning: 5 lines add whitespace errors.
> > 
> > 
> > - you use the 'openssl' tool from the tests.
> > Is it possible to create the required files apart from the tests and
> > store them in tests/certs ? You can provide a script and/or a README how
> > to create these files (see testenv/certs/README).
> > If that is not possible, the tests should see if the 'openssl' command
> > is available. If unavailable the test(s) should SKIP.
> > 
> > 
> > - some tests fail here with 'make check -j4' but succeed with -j1
> > (failures are a bit random):
> > FAIL: Test-https-tlsv1.px
> > FAIL: Test-https-tlsv1x.px
> > FAIL: Test-https-clientcert.px
> > 
> > 
> > - this test always fails (wget is build with GnuTLS):
> > FAIL: Test-https-badcerts.px
> > 
> > Releasing 0x559c258a74d0 (new refcount 1).
> > GnuTLS: ASN1 parser: Error in DER parsing.
> > Closed fd 3
> > Unable to establish SSL connection.
> > Can't use an undefined value as a symbol reference at SSLServer.pm line
> > 131.
> > Test failed: wrong code returned (was: 4, expected: 5)
> > FAIL Test-https-badcerts.px (exit status: 255)
> > 
> > 
> > 
> > With Best Regards, Tim
> > 
> > On 04/27/2017 06:19 AM, Vijo Cherian wrote:
> > > Thank you Tim.
> > > 
> > > Attached is the last set of patches from me for SSL testing.
> > > I will apply these to wget2 and start working on wget2 from now.
> > > 
> > > Best,
> > > Vijo.
> > > 
> > > On Wed, Apr 19, 2017 at 4:12 AM, Tim Rühsen <tim.rueh...@gmx.de> wrote:
> > >> Hi Vijo,
> > >> 
> > >> On 04/18/2017 06:56 PM, Vijo Cherian wrote:
> > >>> Added a framework for perl based SSL tests, and some tests to start
> > 
> > with.
> > 
> > >>> In case this is of interest, I will add more tests for SSL: client
> > >>> certificates, CRLs, negative tests etc.
> > >>> Also not included : making these tests a part of "make check".
> > >>> 
> > >>> TESTING :  only on ubuntu 16
> > >> 
> > >> thank you for this contribution !
> > >> 
> > >> Your commit has been slightly amended (trailing white space removed,
> > >> commit message changed to GNU style) and pushed to master.
> > >> 
> > >> Maybe you are interested to inspect Wget2 testing to see if your tests
> > >> are already covered there. If not we would be pleased if you could add
> > >> them there as well.
> > >> 
> > >> Just 'git clone https://github.com/rockdaboot/wget2' and jump in !
> > >> 
> > >> Regards, Tim



signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Gzip Content-Encoding Patches

[Tim Rühsen]

Hi Tim,

sorry for the delay.


I just pushed your patches. Thanks gain for your work !


With Best Regards, Tim



signature.asc
Description: OpenPGP digital signature

Re: [Bug-wget] [bug #51666] Please hash the hostname in ~/.wget-hsts files

[Tim Rühsen]

On Freitag, 18. August 2017 14:51:12 CEST Ander Juaristi wrote:
> Follow-up Comment #2, bug #51666 (project wget):
> 
> I'm not generally against these kind of small tweaks that don't harm and
> slightly improve user's privacy.
> 
> If Firefox doesn't do it, we don't care: it's their business and they will
> end up doing it if users request that feature (maybe because they saw it in
> wget).
> 
> Private SSH keys can be protected with a password if you want to.

As long as it is optional...

It would be nice being file compatible with Firefox (at least reading Firefox 
HSTS db).
Maybe the sqlite backend that has been mentioned earlier should then work with 
the same settings (hashed/not hashed).

> We can do both, hash and still keep the readable to the user only. If the
> overhead is not much I would go for it. That is the basis of every security
> framework out there: if the benefits of having 2 security mechanisms instead
> of only 1 outweigh the drawbacks, then implement 2 instead of 1.

Absolutely, but in this special case you open up a can of worms. From a 
security standpoint, the average home directory is a nightmare. Once someone 
gets access to it (read or write)...

Regards, Tim

signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Last modified dates from archive.org's Wayback Machine

[Tim Rühsen]

Hi Greg,

was a pleasure to implement this. The change has been pushed to our git
repository and will be part of the next release.

Also opened an issue for Wget2 as a task for our GSOC students :-)


With Best Regards, Tim



On 05/03/2017 03:25 AM, Gregory R Fellow wrote:
> Hi. I greatly appreciate that wget is able to store the last modified
> date and time when sites make that information available. For the
> Wayback Machine on archive.org the last modified information is provided
> in the "X-Archive-Orig-last-modified" field instead of the usual
> "Last-Modified" field. If you can consider adding support for this in
> the future that would be very helpful.
> 
> Thank you very much for your time and for making wget.
> 
> Greg Fellow
> 
> 



signature.asc
Description: OpenPGP digital signature

Re: [Bug-wget] PATCH: tests for SSL

[Tim Rühsen]

Hi Vijo,

maybe a misunderstanding... all your new tests fail here reproducible (always 
!), so maybe I miss some Perl module or have a different version than you.

FAIL: Test-https-selfsigned.px
FAIL: Test-https-tlsv1.px
FAIL: Test-https-weboftrust.px
FAIL: Test-https-clientcert.px
FAIL: Test-https-crl.px
FAIL: Test-https-pfs.px
FAIL: Test-https-tlsv1x.px
FAIL: Test-https-badcerts.px

I see at least two errors in tests/Test-https-selfsigned.log that should not 
be there:


Running test Test-https-selfsigned
IO::Socket::IP configuration failed at SSLServer.pm line 71.
Failed to get ssl sock at SSLServer.pm line 168.
Can't call method "accept" on an undefined value at SSLServer.pm line 86.

This concerns me the most, since I have no clue what to do.


Resolving wgettestingserver (wgettestingserver)... failed: Name or service not 
known.
wget: unable to resolve host address 'wgettestingserver'
Test failed: wrong code returned (was: 4, expected: 5)

Maybe you have 'wgettestingserver' in your /etc/hosts !?

Let me know what i can do or if I should test anything.

Regards, Tim


On Dienstag, 2. Mai 2017 08:07:31 CEST Vijo Cherian wrote:
> Attached now.
> 
> On Tue, May 2, 2017 at 8:06 AM, Vijo Cherian <coderv...@gmail.com> wrote:
> > Sorry about that Tim.
> > I reworked the patches again.
> > This time, I ran 'make check -j4" in a loop overnight.
> > 
> > Best,
> > Vijo.
> > 
> > On Sun, Apr 30, 2017 at 5:49 AM, Tim Rühsen <tim.rueh...@gmx.de> wrote:
> >> Hi Vijo,
> >> 
> >> maybe I miss something ?
> >> 
> >> $ cat tests/Test-https-pfs.log
> >> Running test Test-https-pfs
> >> IO::Socket::IP configuration failed at SSLServer.pm line 71.
> >> Failed to get ssl sock at SSLServer.pm line 168.
> >> Can't call method "accept" on an undefined value at SSLServer.pm line 86.
> >> 
> >> Regards, Tim
> >> 
> >> On Samstag, 29. April 2017 13:21:04 CEST Vijo Cherian wrote:
> >> > Thank you for your review and comments, Tim.
> >> > 
> >> > Attached is the revised set of patches for the same tests.
> >> > 
> >> > 
> >> > Best,
> >> > Vijo.
> >> > 
> >> > On Fri, Apr 28, 2017 at 3:42 AM, Tim Rühsen <tim.rueh...@gmx.de> wrote:
> >> > > Hi Vijo,
> >> > > 
> >> > > great work !
> >> > > 
> >> > > There is just a bit of polishing to be done... :-)
> >> > > 
> >> > > 
> >> > > - some whitespace warnings (maybe you set your editor to remove
> >> 
> >> trailing
> >> 
> >> > > whitespace when saving !?):
> >> > > 
> >> > > .git/rebase-apply/patch:516: trailing whitespace.
> >> > > my $revokecmd = "openssl ca -config $caconf -revoke $servercrt &&
> >> > > .git/rebase-apply/patch:795: trailing whitespace.
> >> > > issuerAltName=issuer:copy
> >> > > .git/rebase-apply/patch:802: trailing whitespace.
> >> > > 
> >> > >  certificate = $dir/interca.crt
> >> > > 
> >> > > .git/rebase-apply/patch:865: trailing whitespace.
> >> > > issuerAltName=issuer:copy
> >> > > .git/rebase-apply/patch:872: trailing whitespace.
> >> > > 
> >> > >  certificate = $dir/test-ca-cert.pem
> >> > > 
> >> > > warning: 5 lines add whitespace errors.
> >> > > 
> >> > > 
> >> > > - you use the 'openssl' tool from the tests.
> >> > > Is it possible to create the required files apart from the tests and
> >> > > store them in tests/certs ? You can provide a script and/or a README
> >> 
> >> how
> >> 
> >> > > to create these files (see testenv/certs/README).
> >> > > If that is not possible, the tests should see if the 'openssl'
> >> > > command
> >> > > is available. If unavailable the test(s) should SKIP.
> >> > > 
> >> > > 
> >> > > - some tests fail here with 'make check -j4' but succeed with -j1
> >> > > (failures are a bit random):
> >> > > FAIL: Test-https-tlsv1.px
> >> > > FAIL: Test-https-tlsv1x.px
> >> > > FAIL: Test-https-clientcert.px
> >> > > 
> >> > > 
> >> > > - this test always fails (wget is build with GnuTLS):
> >> > > FAIL: Test-https-badcerts.px
> >> > > 
> >> > &

Re: [Bug-wget] PATCH: tests for SSL

[Tim Rühsen]

Hi Vijo,

just found out that HOSTALIASES doesn't work on any of my machines, at least 
the name to IP resolution. You must have a special environment :-(

Name to name alias works.
There is a special service, where you can do what you want (but maybe you have 
better idea):

$ cat /home/tim/src/wget1.x/tests/wgethosts 
bar 127.0.0.1
foo 127.0.0.1.xip.io

$ LC_ALL=C HOSTALIASES=/home/tim/src/wget1.x/tests/wgethosts wget bar
--2017-05-03 21:41:45--  http://bar/
Resolving bar (bar)... failed: Name or service not known.
wget: unable to resolve host address 'bar'

$ LC_ALL=C HOSTALIASES=/home/tim/src/wget1.x/tests/wgethosts wget foo
--2017-05-03 21:42:39--  http://foo/
Resolving foo (foo)... 127.0.0.1
Connecting to foo (foo)|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2417 (2.4K) [text/html]
Saving to: 'index.html.5'

index.html.5100%
[>]   2.36K  --.-KB/s
in 0s  

2017-05-03 21:42:39 (324 MB/s) - 'index.html.5' saved [2417/2417]


With Best Regards, Tim


On Dienstag, 2. Mai 2017 08:07:31 CEST Vijo Cherian wrote:
> Attached now.
> 
> On Tue, May 2, 2017 at 8:06 AM, Vijo Cherian <coderv...@gmail.com> wrote:
> > Sorry about that Tim.
> > I reworked the patches again.
> > This time, I ran 'make check -j4" in a loop overnight.
> > 
> > Best,
> > Vijo.
> > 
> > On Sun, Apr 30, 2017 at 5:49 AM, Tim Rühsen <tim.rueh...@gmx.de> wrote:
> >> Hi Vijo,
> >> 
> >> maybe I miss something ?
> >> 
> >> $ cat tests/Test-https-pfs.log
> >> Running test Test-https-pfs
> >> IO::Socket::IP configuration failed at SSLServer.pm line 71.
> >> Failed to get ssl sock at SSLServer.pm line 168.
> >> Can't call method "accept" on an undefined value at SSLServer.pm line 86.
> >> 
> >> Regards, Tim
> >> 
> >> On Samstag, 29. April 2017 13:21:04 CEST Vijo Cherian wrote:
> >> > Thank you for your review and comments, Tim.
> >> > 
> >> > Attached is the revised set of patches for the same tests.
> >> > 
> >> > 
> >> > Best,
> >> > Vijo.
> >> > 
> >> > On Fri, Apr 28, 2017 at 3:42 AM, Tim Rühsen <tim.rueh...@gmx.de> wrote:
> >> > > Hi Vijo,
> >> > > 
> >> > > great work !
> >> > > 
> >> > > There is just a bit of polishing to be done... :-)
> >> > > 
> >> > > 
> >> > > - some whitespace warnings (maybe you set your editor to remove
> >> 
> >> trailing
> >> 
> >> > > whitespace when saving !?):
> >> > > 
> >> > > .git/rebase-apply/patch:516: trailing whitespace.
> >> > > my $revokecmd = "openssl ca -config $caconf -revoke $servercrt &&
> >> > > .git/rebase-apply/patch:795: trailing whitespace.
> >> > > issuerAltName=issuer:copy
> >> > > .git/rebase-apply/patch:802: trailing whitespace.
> >> > > 
> >> > >  certificate = $dir/interca.crt
> >> > > 
> >> > > .git/rebase-apply/patch:865: trailing whitespace.
> >> > > issuerAltName=issuer:copy
> >> > > .git/rebase-apply/patch:872: trailing whitespace.
> >> > > 
> >> > >  certificate = $dir/test-ca-cert.pem
> >> > > 
> >> > > warning: 5 lines add whitespace errors.
> >> > > 
> >> > > 
> >> > > - you use the 'openssl' tool from the tests.
> >> > > Is it possible to create the required files apart from the tests and
> >> > > store them in tests/certs ? You can provide a script and/or a README
> >> 
> >> how
> >> 
> >> > > to create these files (see testenv/certs/README).
> >> > > If that is not possible, the tests should see if the 'openssl'
> >> > > command
> >> > > is available. If unavailable the test(s) should SKIP.
> >> > > 
> >> > > 
> >> > > - some tests fail here with 'make check -j4' but succeed with -j1
> >> > > (failures are a bit random):
> >> > > FAIL: Test-https-tlsv1.px
> >> > > FAIL: Test-https-tlsv1x.px
> >> > > FAIL: Test-https-clientcert.px
> >> > > 
> >> > > 
> >> > > - this test always fails (wget is build with GnuTLS):
> >> > > FAIL: Test-https-badcerts.px
> >> > > 
> >> > > Rel

Re: [Bug-wget] Wget keeps crashing on me

[Tim Rühsen]

On Sonntag, 14. Mai 2017 21:34:22 CEST Eli Zaretskii wrote:
> > From: Tim Rühsen <tim.rueh...@gmx.de>
> > Cc: William Higgs <whiggs.it...@gmail.com>, 'Darshit Shah'
> > <dar...@gmail.com>, Eli Zaretskii <e...@gnu.org> Date: Sun, 14 May 2017
> > 20:17:29 +0200
> > 
> > @Eli Mybe you could test latest master without a debugger as well !?
> 
> You mean Jernej, not Eli, right?

Oh, sorry. Yeah, I meant Jernej.

Regards, Tim

signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] [PATCH] Script to create certs & keys

[Tim Rühsen]

On Samstag, 13. Mai 2017 22:29:42 CEST Vijo Cherian wrote:
> Script to re-generate all certs & keys for testing

That is simply great !

I just used it to re-generated those certs with invalid date fields and that 
made it possible to remove the OpenSSL requirement from Test-https-badcerts.px.

We surely can use that script for Wget2 as well.
If you don't mind, I would adapt it to use GnuTLS tools instead OpenSSL (at 
least I put it on my todo list).

With Best Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] [PATCH 1/3] Added tests for HTTP authentication using credentials from .netrc

[Tim Rühsen]

Hi Tomas,

thank you very much for the fixes and the new tests !

Currently, patch 1/3 breaks two Metalink tests:

FAIL: Test-metalink-http.py
FAIL: Test-metalink-http-quoted.py

I am still investigating, but it has something to do with the
environment that you reset in testenv/test/base_test.py:

ret_code = call(params, env={"HOME": os.getcwd()})

Maybe a bug that just comes out with a different locale/charset encoding.


With Best Regards, Tim



On 05/12/2017 07:17 PM, Tomas Hozza wrote:
> Getting credentials from .netrc has been broken from time to time, thus
> adding a test coverage to prevent regressions.
> 
> Also added setting of "HOME" environment variable when executing wget,
> to make sure LocalFiles like .netrc, which are created just for the
> test, are actually used.
> 
> Signed-off-by: Tomas Hozza 
> ---
>  testenv/Makefile.am |  3 ++
>  testenv/Test-auth-basic-netrc-pass-given.py | 68 
> +
>  testenv/Test-auth-basic-netrc-user-given.py | 68 
> +
>  testenv/Test-auth-basic-netrc.py| 66 
>  testenv/test/base_test.py   |  2 +-
>  5 files changed, 206 insertions(+), 1 deletion(-)
>  create mode 100755 testenv/Test-auth-basic-netrc-pass-given.py
>  create mode 100755 testenv/Test-auth-basic-netrc-user-given.py
>  create mode 100755 testenv/Test-auth-basic-netrc.py
> 
> diff --git a/testenv/Makefile.am b/testenv/Makefile.am
> index 3febec7..7104314 100644
> --- a/testenv/Makefile.am
> +++ b/testenv/Makefile.am
> @@ -75,6 +75,9 @@ if HAVE_PYTHON3
>TESTS = Test-504.py   \
>  Test-auth-basic-fail.py \
>  Test-auth-basic.py  \
> +Test-auth-basic-netrc.py\
> +Test-auth-basic-netrc-user-given.py \
> +Test-auth-basic-netrc-pass-given.py \
>  Test-auth-both.py   \
>  Test-auth-digest.py \
>  Test-auth-no-challenge.py   \
> diff --git a/testenv/Test-auth-basic-netrc-pass-given.py 
> b/testenv/Test-auth-basic-netrc-pass-given.py
> new file mode 100755
> index 000..43dfe34
> --- /dev/null
> +++ b/testenv/Test-auth-basic-netrc-pass-given.py
> @@ -0,0 +1,68 @@
> +#!/usr/bin/env python3
> +from sys import exit
> +from test.http_test import HTTPTest
> +from misc.wget_file import WgetFile
> +
> +"""
> +This test ensures Wget uses credentials from .netrc for Basic 
> Authorization Negotiation.
> +In this case we test that .netrc credentials are used in case only
> +password is given on the command line.
> +Also, we ensure that Wget saves the host after a successful auth and
> +doesn't wait for a challenge the second time.
> +"""
> +# File Definitions 
> ###
> +File1 = "I am an invisble man."
> +File2 = "I too am an invisible man."
> +
> +User = "Sauron"
> +Password = "TheEye"
> +
> +File1_rules = {
> +"Authentication": {
> +"Type"  : "Basic",
> +"User"  : User,
> +"Pass"  : Password
> +}
> +}
> +File2_rules = {
> +"ExpectHeader"  : {
> +"Authorization" : "Basic U2F1cm9uOlRoZUV5ZQ=="
> +}
> +}
> +
> +Netrc = "machine 127.0.0.1\n\tlogin {0}".format(User)
> +
> +A_File = WgetFile ("File1", File1, rules=File1_rules)
> +B_File = WgetFile ("File2", File2, rules=File2_rules)
> +Netrc_File = WgetFile (".netrc", Netrc)
> +
> +WGET_OPTIONS = "--password={0}".format(Password)
> +WGET_URLS = [["File1", "File2"]]
> +
> +Files = [[A_File, B_File]]
> +LocalFiles = [Netrc_File]
> +
> +ExpectedReturnCode = 0
> +ExpectedDownloadedFiles = [A_File, B_File, Netrc_File]
> +
> + Pre and Post Test Hooks 
> #
> +pre_test = {
> +"ServerFiles"   : Files,
> +"LocalFiles": LocalFiles
> +}
> +test_options = {
> +"WgetCommands"  : WGET_OPTIONS,
> +"Urls"  : WGET_URLS
> +}
> +post_test = {
> +"ExpectedFiles" : ExpectedDownloadedFiles,
> +"ExpectedRetcode"   : ExpectedReturnCode
> +}
> +
> +err = HTTPTest (
> +pre_hook=pre_test,
> +test_params=test_options,
> +post_hook=post_test
> +).begin ()
> +
> +exit (err)
> diff --git a/testenv/Test-auth-basic-netrc-user-given.py 
> b/testenv/Test-auth-basic-netrc-user-given.py
> new file mode 100755
> index 000..57b6148
> --- /dev/null
> +++ b/testenv/Test-auth-basic-netrc-user-given.py
> @@ -0,0 +1,68 @@
> +#!/usr/bin/env python3
> +from sys import exit
> +from test.http_test import HTTPTest
> +from misc.wget_file import WgetFile
> +
> +"""
> +This test ensures Wget uses credentials from .netrc for Basic 
> Authorization Negotiation.
> +In this case we test that .netrc credentials are 

Re: [Bug-wget] [PATCH 1/3] Added tests for HTTP authentication using credentials from .netrc

[Tim Rühsen]

Hi Tomas,

fixed the problem by skipping the processing of 'pubring.kbx'. This file
is created by libgpgme if we change $HOME.

Your changes have been pushed now. Thanks for your work.

The charset/locale issue is a general problem when the test suite reads
and compares binary data. Currently, we don't have to fix that.



With Best Regards, Tim



On 05/15/2017 04:53 PM, Tomas Hozza wrote:
> Hello Tim.
> 
> I run all the tests before sending patches and they all passed. If you can 
> send me more info about the failures, I could have a look at it.
> 
> My locale is:
> $ locale
> LANG=en_US.UTF-8
> LC_CTYPE="en_US.UTF-8"
> LC_NUMERIC=sk_SK.UTF-8
> LC_TIME=sk_SK.UTF-8
> LC_COLLATE="en_US.UTF-8"
> LC_MONETARY=sk_SK.UTF-8
> LC_MESSAGES="en_US.UTF-8"
> LC_PAPER=sk_SK.UTF-8
> LC_NAME="en_US.UTF-8"
> LC_ADDRESS="en_US.UTF-8"
> LC_TELEPHONE="en_US.UTF-8"
> LC_MEASUREMENT=sk_SK.UTF-8
> LC_IDENTIFICATION="en_US.UTF-8"
> LC_ALL=
> 
> Thanks.
> 
> Regards,
> Tomas
> 
> On 15.05.2017 15:52, Tim Rühsen wrote:
>> Hi Tomas,
>>
>> thank you very much for the fixes and the new tests !
>>
>> Currently, patch 1/3 breaks two Metalink tests:
>>
>> FAIL: Test-metalink-http.py
>> FAIL: Test-metalink-http-quoted.py
>>
>> I am still investigating, but it has something to do with the
>> environment that you reset in testenv/test/base_test.py:
>>
>> ret_code = call(params, env={"HOME": os.getcwd()})
>>
>> Maybe a bug that just comes out with a different locale/charset encoding.
>>
>>
>> With Best Regards, Tim
>>
>>
>>
>> On 05/12/2017 07:17 PM, Tomas Hozza wrote:
>>> Getting credentials from .netrc has been broken from time to time, thus
>>> adding a test coverage to prevent regressions.
>>>
>>> Also added setting of "HOME" environment variable when executing wget,
>>> to make sure LocalFiles like .netrc, which are created just for the
>>> test, are actually used.
>>>
>>> Signed-off-by: Tomas Hozza <tho...@redhat.com>
>>> ---
>>>  testenv/Makefile.am |  3 ++
>>>  testenv/Test-auth-basic-netrc-pass-given.py | 68 
>>> +
>>>  testenv/Test-auth-basic-netrc-user-given.py | 68 
>>> +
>>>  testenv/Test-auth-basic-netrc.py| 66 
>>> 
>>>  testenv/test/base_test.py   |  2 +-
>>>  5 files changed, 206 insertions(+), 1 deletion(-)
>>>  create mode 100755 testenv/Test-auth-basic-netrc-pass-given.py
>>>  create mode 100755 testenv/Test-auth-basic-netrc-user-given.py
>>>  create mode 100755 testenv/Test-auth-basic-netrc.py
>>>
>>> diff --git a/testenv/Makefile.am b/testenv/Makefile.am
>>> index 3febec7..7104314 100644
>>> --- a/testenv/Makefile.am
>>> +++ b/testenv/Makefile.am
>>> @@ -75,6 +75,9 @@ if HAVE_PYTHON3
>>>TESTS = Test-504.py   \
>>>  Test-auth-basic-fail.py \
>>>  Test-auth-basic.py  \
>>> +Test-auth-basic-netrc.py\
>>> +Test-auth-basic-netrc-user-given.py \
>>> +Test-auth-basic-netrc-pass-given.py \
>>>  Test-auth-both.py   \
>>>  Test-auth-digest.py \
>>>  Test-auth-no-challenge.py   \
>>> diff --git a/testenv/Test-auth-basic-netrc-pass-given.py 
>>> b/testenv/Test-auth-basic-netrc-pass-given.py
>>> new file mode 100755
>>> index 000..43dfe34
>>> --- /dev/null
>>> +++ b/testenv/Test-auth-basic-netrc-pass-given.py
>>> @@ -0,0 +1,68 @@
>>> +#!/usr/bin/env python3
>>> +from sys import exit
>>> +from test.http_test import HTTPTest
>>> +from misc.wget_file import WgetFile
>>> +
>>> +"""
>>> +This test ensures Wget uses credentials from .netrc for Basic 
>>> Authorization Negotiation.
>>> +In this case we test that .netrc credentials are used in case only
>>> +password is given on the command line.
>>> +Also, we ensure that Wget saves the host after a successful auth and
>>> +doesn't wait for a challenge the second time.
>>> +"""
>>> +# File Definitions 
>>> #

Re: [Bug-wget] [PATCH 3/3] Add command line option to disable use of .netrc

[Tim Rühsen]

On 05/16/2017 01:42 PM, Giuseppe Scrivano wrote:
> Tomas Hozza  writes:
> 
>> The option to turn off reading .netrc by configuration in .wgetrc is
>> still there. This command line option was supposed to be just
>> something extra. Anyway Tim already merged the change. Do you want to
>> revert it?
> 
> Tim, are you in favor of this change?

Definitely. I don't see much gain in .wgetrc commands that are 'hidden'
from the users by these kinds of 'obscurity' measurements. It's also a
waste of time to document commands and options separately.

> Well, not worth a revert probably, we are already exposing many options
> this way.
> 
> Giuseppe
> 
> 

With Best Regards, Tim



signature.asc
Description: OpenPGP digital signature

Re: [Bug-wget] [PATCH 1/3] Added tests for HTTP authentication using credentials from .netrc

[Tim Rühsen]

P.S.:

Just saw that the byte seems a bit random, on a second run I got:

'utf-8' codec can't decode byte 0xc3 in position 18: invalid
continuation byte


With Best Regards, Tim



On 05/15/2017 05:04 PM, Tim Rühsen wrote:
> Hi Tomas,
> 
> my locale looks similar:
> 
> $ locale
> LANG=en_US.UTF-8
> LANGUAGE=en_US.UTF-8
> LC_CTYPE="en_US.UTF-8"
> LC_NUMERIC="en_US.UTF-8"
> LC_TIME="en_US.UTF-8"
> LC_COLLATE="en_US.UTF-8"
> LC_MONETARY="en_US.UTF-8"
> LC_MESSAGES="en_US.UTF-8"
> LC_PAPER="en_US.UTF-8"
> LC_NAME="en_US.UTF-8"
> LC_ADDRESS="en_US.UTF-8"
> LC_TELEPHONE="en_US.UTF-8"
> LC_MEASUREMENT="en_US.UTF-8"
> LC_IDENTIFICATION="en_US.UTF-8"
> LC_ALL=en_US.UTF-8
> 
> 
> Reproducable here *without* your patches:
> 
> $ TESTS_ENVIRONMENT="env -i; " make check
> 
> 
> The system is Debian unstable amd64.
> I asked Darshit, the author of the python script, if he could take a
> look... maybe this is something trivial for a Pythonista.
> 
> The traceback is:
> 
> Unhandled exception caught.
> 'ascii' codec can't decode byte 0xc1 in position 18: ordinal not in
> range(128)
> Traceback (most recent call last):
>   File "./Test-metalink-http.py", line 124, in 
> err = http_test.begin ()
>   File "/usr/oms/src/wget1.x/testenv/test/http_test.py", line 41, in begin
> self.do_test()
>   File "/usr/oms/src/wget1.x/testenv/test/base_test.py", line 187, in
> do_test
> self.post_hook_call()
>   File "/usr/oms/src/wget1.x/testenv/test/base_test.py", line 206, in
> post_hook_call
> self.hook_call(self.post_configs, 'Post Test Function')
>   File "/usr/oms/src/wget1.x/testenv/test/base_test.py", line 196, in
> hook_call
> conf.find_conf(conf_name)(conf_arg)(self)
>   File "/usr/oms/src/wget1.x/testenv/conf/expected_files.py", line 36,
> in __call__
> local_fs = self.gen_local_fs_snapshot()
>   File "/usr/oms/src/wget1.x/testenv/conf/expected_files.py", line 30,
> in gen_local_fs_snapshot
> f['content'] = fp.read()
>   File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
> return codecs.ascii_decode(input, self.errors)[0]
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc1 in position 18:
> ordinal not in range(128)
> 
> 
> With Best Regards, Tim
> 
> 
> 
> On 05/15/2017 04:53 PM, Tomas Hozza wrote:
>> Hello Tim.
>>
>> I run all the tests before sending patches and they all passed. If you can 
>> send me more info about the failures, I could have a look at it.
>>
>> My locale is:
>> $ locale
>> LANG=en_US.UTF-8
>> LC_CTYPE="en_US.UTF-8"
>> LC_NUMERIC=sk_SK.UTF-8
>> LC_TIME=sk_SK.UTF-8
>> LC_COLLATE="en_US.UTF-8"
>> LC_MONETARY=sk_SK.UTF-8
>> LC_MESSAGES="en_US.UTF-8"
>> LC_PAPER=sk_SK.UTF-8
>> LC_NAME="en_US.UTF-8"
>> LC_ADDRESS="en_US.UTF-8"
>> LC_TELEPHONE="en_US.UTF-8"
>> LC_MEASUREMENT=sk_SK.UTF-8
>> LC_IDENTIFICATION="en_US.UTF-8"
>> LC_ALL=
>>
>> Thanks.
>>
>> Regards,
>> Tomas
>>
>> On 15.05.2017 15:52, Tim Rühsen wrote:
>>> Hi Tomas,
>>>
>>> thank you very much for the fixes and the new tests !
>>>
>>> Currently, patch 1/3 breaks two Metalink tests:
>>>
>>> FAIL: Test-metalink-http.py
>>> FAIL: Test-metalink-http-quoted.py
>>>
>>> I am still investigating, but it has something to do with the
>>> environment that you reset in testenv/test/base_test.py:
>>>
>>> ret_code = call(params, env={"HOME": os.getcwd()})
>>>
>>> Maybe a bug that just comes out with a different locale/charset encoding.
>>>
>>>
>>> With Best Regards, Tim
>>>
>>>
>>>
>>> On 05/12/2017 07:17 PM, Tomas Hozza wrote:
>>>> Getting credentials from .netrc has been broken from time to time, thus
>>>> adding a test coverage to prevent regressions.
>>>>
>>>> Also added setting of "HOME" environment variable when executing wget,
>>>> to make sure LocalFiles like .netrc, which are created just for the
>>>> test, are actually used.
>>>>
>>>> Signed-off-by: Tomas Hozza <tho...@redhat.com>
>>>> ---
>>>>  testenv/Makefile.am |  3 ++
>>>>  testenv/Test-auth-basic-netrc-pass-given.py | 68 
>>

Re: [Bug-wget] [PATCH 1/3] Added tests for HTTP authentication using credentials from .netrc

[Tim Rühsen]

Hi Tomas,

my locale looks similar:

$ locale
LANG=en_US.UTF-8
LANGUAGE=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8


Reproducable here *without* your patches:

$ TESTS_ENVIRONMENT="env -i; " make check


The system is Debian unstable amd64.
I asked Darshit, the author of the python script, if he could take a
look... maybe this is something trivial for a Pythonista.

The traceback is:

Unhandled exception caught.
'ascii' codec can't decode byte 0xc1 in position 18: ordinal not in
range(128)
Traceback (most recent call last):
  File "./Test-metalink-http.py", line 124, in 
err = http_test.begin ()
  File "/usr/oms/src/wget1.x/testenv/test/http_test.py", line 41, in begin
self.do_test()
  File "/usr/oms/src/wget1.x/testenv/test/base_test.py", line 187, in
do_test
self.post_hook_call()
  File "/usr/oms/src/wget1.x/testenv/test/base_test.py", line 206, in
post_hook_call
self.hook_call(self.post_configs, 'Post Test Function')
  File "/usr/oms/src/wget1.x/testenv/test/base_test.py", line 196, in
hook_call
conf.find_conf(conf_name)(conf_arg)(self)
  File "/usr/oms/src/wget1.x/testenv/conf/expected_files.py", line 36,
in __call__
local_fs = self.gen_local_fs_snapshot()
  File "/usr/oms/src/wget1.x/testenv/conf/expected_files.py", line 30,
in gen_local_fs_snapshot
f['content'] = fp.read()
  File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc1 in position 18:
ordinal not in range(128)


With Best Regards, Tim



On 05/15/2017 04:53 PM, Tomas Hozza wrote:
> Hello Tim.
> 
> I run all the tests before sending patches and they all passed. If you can 
> send me more info about the failures, I could have a look at it.
> 
> My locale is:
> $ locale
> LANG=en_US.UTF-8
> LC_CTYPE="en_US.UTF-8"
> LC_NUMERIC=sk_SK.UTF-8
> LC_TIME=sk_SK.UTF-8
> LC_COLLATE="en_US.UTF-8"
> LC_MONETARY=sk_SK.UTF-8
> LC_MESSAGES="en_US.UTF-8"
> LC_PAPER=sk_SK.UTF-8
> LC_NAME="en_US.UTF-8"
> LC_ADDRESS="en_US.UTF-8"
> LC_TELEPHONE="en_US.UTF-8"
> LC_MEASUREMENT=sk_SK.UTF-8
> LC_IDENTIFICATION="en_US.UTF-8"
> LC_ALL=
> 
> Thanks.
> 
> Regards,
> Tomas
> 
> On 15.05.2017 15:52, Tim Rühsen wrote:
>> Hi Tomas,
>>
>> thank you very much for the fixes and the new tests !
>>
>> Currently, patch 1/3 breaks two Metalink tests:
>>
>> FAIL: Test-metalink-http.py
>> FAIL: Test-metalink-http-quoted.py
>>
>> I am still investigating, but it has something to do with the
>> environment that you reset in testenv/test/base_test.py:
>>
>> ret_code = call(params, env={"HOME": os.getcwd()})
>>
>> Maybe a bug that just comes out with a different locale/charset encoding.
>>
>>
>> With Best Regards, Tim
>>
>>
>>
>> On 05/12/2017 07:17 PM, Tomas Hozza wrote:
>>> Getting credentials from .netrc has been broken from time to time, thus
>>> adding a test coverage to prevent regressions.
>>>
>>> Also added setting of "HOME" environment variable when executing wget,
>>> to make sure LocalFiles like .netrc, which are created just for the
>>> test, are actually used.
>>>
>>> Signed-off-by: Tomas Hozza <tho...@redhat.com>
>>> ---
>>>  testenv/Makefile.am |  3 ++
>>>  testenv/Test-auth-basic-netrc-pass-given.py | 68 
>>> +
>>>  testenv/Test-auth-basic-netrc-user-given.py | 68 
>>> +
>>>  testenv/Test-auth-basic-netrc.py| 66 
>>> 
>>>  testenv/test/base_test.py   |  2 +-
>>>  5 files changed, 206 insertions(+), 1 deletion(-)
>>>  create mode 100755 testenv/Test-auth-basic-netrc-pass-given.py
>>>  create mode 100755 testenv/Test-auth-basic-netrc-user-given.py
>>>  create mode 100755 testenv/Test-auth-basic-netrc.py
>>>
>>> diff --git a/testenv/Makefile.am b/testenv/Makefile.am
>>> index 3febec7..7104314 100644
>>> --- a/testenv/Makefile.am
>>> +++ b/testenv/Makefi

Re: [Bug-wget] PATCH: tests for SSL

[Tim Rühsen]

Hi Vijo,


thanks so much !


I had to manually merge the changes from your patch since I also made
some changes in the meantime. Please continue from branch
'tmp-gitlab-ci' on https://gitlab.com/gnuwget/wget.


I made several changes to get the new Tests going ('make distcheck'
VPATH issues, HOSTALIAS issue, fixed WgetFeature.pm to allow multiple
required features, auto-generate certs/*.conf, cleanup on exit, ...)

There are still some issue to let the tests run in parallel (maybe your
devel system has an old version of automake - so the tests are always
run in serial !?).

Especially we need a own certs/root.conf for each test, else the tests
write to the same file names at the same time (random failures here).

Also, there are remnants like '1122.pem' in the tests/ directory.
Something that a recent 'make distcheck' doesn't allow any more.
I guess they are from 'openssl' tool ... will check tomorrow.

Brrr, Perl is not my favorite, hope I will forget everything I learned
today within the next weeks ;-)


With Best Regards, Tim



On 05/09/2017 04:04 AM, Vijo Cherian wrote:
> Hi Tim,
>   Another revision of the patch is attached.
> 
>  What I changed from last patch :
>- Skip if the name cannot be resolved
>- skip if not using openssl as a temp work around till I figure out
> gnutls-openssl cert issues.
> 
> Best,
> Vijo.
> 
> On Thu, May 4, 2017 at 8:01 AM, Tim Rühsen <tim.rueh...@gmx.de> wrote:
> 
>> Hi Vijo,
>>
>> I fixed what I could fix (as totally Perl ignorant).
>>
>> There is still one issue left that I can't figure out.
>>
>> Test-https-badcerts.px fails here with this in the log:
>>
>>
>> Connecting to wgettestingserver (wgettestingserver)|127.0.0.1|:60443...
>> connected.
>> Created socket 3.
>> Releasing 0x5571c84304e0 (new refcount 1).
>> GnuTLS: ASN1 parser: Error in DER parsing.
>> Closed fd 3
>> Unable to establish SSL connection.
>> Test failed: wrong code returned (was: 4, expected: 5)
>>
>> $ openssl version
>> OpenSSL 1.1.0e  16 Feb 2017
>>
>>
>>
>> With Best Regards, Tim
>>
>>
>>
>> On 05/02/2017 05:07 PM, Vijo Cherian wrote:
>>> Attached now.
>>>
>>> On Tue, May 2, 2017 at 8:06 AM, Vijo Cherian <coderv...@gmail.com>
>> wrote:
>>>
>>>> Sorry about that Tim.
>>>> I reworked the patches again.
>>>> This time, I ran 'make check -j4" in a loop overnight.
>>>>
>>>> Best,
>>>> Vijo.
>>>>
>>>> On Sun, Apr 30, 2017 at 5:49 AM, Tim Rühsen <tim.rueh...@gmx.de> wrote:
>>>>
>>>>> Hi Vijo,
>>>>>
>>>>> maybe I miss something ?
>>>>>
>>>>> $ cat tests/Test-https-pfs.log
>>>>> Running test Test-https-pfs
>>>>> IO::Socket::IP configuration failed at SSLServer.pm line 71.
>>>>> Failed to get ssl sock at SSLServer.pm line 168.
>>>>> Can't call method "accept" on an undefined value at SSLServer.pm line
>> 86.
>>>>>
>>>>> Regards, Tim
>>>>>
>>>>> On Samstag, 29. April 2017 13:21:04 CEST Vijo Cherian wrote:
>>>>>> Thank you for your review and comments, Tim.
>>>>>>
>>>>>> Attached is the revised set of patches for the same tests.
>>>>>>
>>>>>>
>>>>>> Best,
>>>>>> Vijo.
>>>>>>
>>>>>> On Fri, Apr 28, 2017 at 3:42 AM, Tim Rühsen <tim.rueh...@gmx.de>
>> wrote:
>>>>>>> Hi Vijo,
>>>>>>>
>>>>>>> great work !
>>>>>>>
>>>>>>> There is just a bit of polishing to be done... :-)
>>>>>>>
>>>>>>>
>>>>>>> - some whitespace warnings (maybe you set your editor to remove
>>>>> trailing
>>>>>>> whitespace when saving !?):
>>>>>>>
>>>>>>> .git/rebase-apply/patch:516: trailing whitespace.
>>>>>>> my $revokecmd = "openssl ca -config $caconf -revoke $servercrt &&
>>>>>>> .git/rebase-apply/patch:795: trailing whitespace.
>>>>>>> issuerAltName=issuer:copy
>>>>>>> .git/rebase-apply/patch:802: trailing whitespace.
>>>>>>>
>>>>>>>  certificate = $dir/interca.crt
>>>>>>>
>>>>>>> .git/rebase-apply/patch:865: trailing whitespace.
>>

Re: [Bug-wget] wget and srcset tag

[Tim Rühsen]

On Montag, 12. Juni 2017 17:07:30 CEST Chris wrote:
> Hi Tim,
> 
> I just created a test page at -
> https://www.anfractuosity.com/files/test2.html
> were I still get the issue.
> 
> The version is 'GNU Wget 1.19.1 built on linux-gnu.'

Thanks, Chris.

The issue is reproducible with latest git, thanks to your test page. 
I'll create a test case tomorrow and then we'll fix it.
It has something to do with If-Modified-Since. If you use 
--no-if-modified-since 
the links are converted correctly.

The good news is: Wget2 (https://gitlab.com/gnuwget/wget2) does it correctly 
:-)

With Best Regards, Tim

> 
> cheers
> Chris
> 
> On 12 June 2017 at 15:35, Tim Rühsen <tim.rueh...@gmx.de> wrote:
> > On 06/12/2017 10:27 AM, chris wrote:
> > > Hi Tim,
> > > 
> > > Thanks for your reply, I notice the following in the debug logs:
> > > 
> > > """
> > > will convert url
> > > http://www.anfractuosity.com/wp-content/uploads/2014/02/fsk.png to local
> > > site_output/fsk.png
> > > will convert url
> > > https://www.anfractuosity.com/wp-content/uploads/2014/02/fsk.png to
> > 
> > local
> > 
> > > site_output/fsk.png.html
> > > """
> > > 
> > > The difference between those URLs seems to be one is https and one
> > > isn't.
> > > When I wget those URLs though, both seem to return a .png, with 'Length:
> > > 51068 (50K) [image/png]'.
> > > 
> > > So I'm a bit confused why I get the fsk.png.html URL.
> > 
> > What version of wget are you using ? (1.19.1 here)
> > 
> > I tried some combinations of srcset (with https and http) and your
> > original options. I thought of an issue with redirection (because that's
> > an answer with text/html Content-Type).
> > 
> > Could you create a small reproducer page ? e.g. like
> > 
> >  > srcset="https://www.anfractuosity.com/wp-content/uploads/2014/02/fsk.png
> > 533w,
> > http://www.anfractuosity.com/wp-content/uploads/2014/02/fsk-266x300.png
> > 266w">
> > 
> > 
> > With whatever paths you are using for the .png files.
> > I don't want to download tons of files (limited bandwidth here).
> > 
> > > cheers
> > > Chris
> > > 
> > > On Mon, Jun 12, 2017 at 9:08 AM, Tim Rühsen <tim.rueh...@gmx.de> wrote:
> > >> Hi Chris,
> > >> 
> > >> On 06/11/2017 05:24 PM, chris wrote:
> > >>> Hi,
> > >>> 
> > >>> I'm just wondering if I've possibly found a bug, unless I'm just doing
> > >>> something incorrectly (which I assume is more likely).
> > >>> 
> > >>> I grab my webpage using 'wget -T1 -t1 -E -k -H -nd -N -p -P
> > >>> site_output
> > >>> https://www.anfractuosity.com/projects/ultrasound-networking/ > note1
> > 
> > 2>
> > 
> > >>> note2'
> > >>> 
> > >>> But i notice the srcset tags in the resulting downloaded files produce
> > >> 
> > >>> 'srcset="fsk.png.html 533w, fsk-266x300.png 266w" sizes="(max-width:
> > >> 533px)
> > >> 
> > >>> 100vw, 533px" />' in the output index.html.
> > >>> 
> > >>> On the actual webpage it looks like "srcset="
> > >>> https://www.anfractuosity.com/wp-content/uploads/2014/02/fft.png
> > >> 
> > >> 762w,"
> > >> 
> > >>> no .html extension on the .png.
> > >> 
> > >> You requested -E (--adjust-extension) and -k (--convert-links).
> > >> That would change the file name when the server tags the file as
> > >> content-type 'text/html'. You could see that in the debug output
> > >> (options -d or --debug).
> > >> 
> > >>> Cheers
> > >>> Chris
> > >> 
> > >> With Best Regards, Tim



signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Output no longer going to stderr

[Tim Rühsen]

On 05/22/2017 05:19 AM, Barry Kauler wrote:
> On 5/21/17, Tim Rühsen <tim.rueh...@gmx.de> wrote:
>> On Sonntag, 21. Mai 2017 10:06:28 CEST Barry Kauler wrote:
>>> Hi guys,
>>> I recently upgraded wget to 1.19.1, but have discovered that it has
>>> broken some of my scripts.
>>>
>>> These are scripts that expect output on stderr. For example, have
>>> "wget ... 2> logfile" or "wget . 2>&1 |  "
>>>
>>> Instead, I just get text message "Redirecting output to ‘wget-log.1’",
>>> and get a logfile at ~/wget-log.1
>>
>> I can't reproduce that with 1.19.1, also not with the command posted at the
>>
>> link below.
>> What system are you on  and what is your complete command line ?
>>
>> With Best Regards, Tim
>>
>>> Is there a good reason for this change?
>>>
>>> If not, I would like the old behaviour back please.
>>>
>>> I see someone else has posted about this:
>>> https://www.makemkv.com/forum2/viewtopic.php?f=3=15885
>>>
>>> Regards,
>>> Barry Kauler
>>
> 
> This is the line in a shell script, that has worked for years, now broken:
> 
>  LANG=C wget -4 -t 2 -T 20 --waitretry=20 --spider -S "${URLSPEC}" >
> /tmp/download_file_spider.log1 2>&1
> 
> I am running an experimental distribution, compiled in OpenEmbedded
> (that Yocto uses).
> 
> I notice commits late in 2016 that look like they might be the cause.
> HOWEVER, the problem has disappeared.

Which commits are these ?

Git blame doesn't find a code change for fork_to_background() since it
was introduced in 1999.

If you use and explicit logfile (-o / --output-file), that code for
creating 'wget-log' isn't triggered.

'man wget':
"   -b
   --background
   Go to background immediately after startup.  If no output
file is specified via the -o, output is redirected to wget-log.
"

With Best Regards, Tim



signature.asc
Description: OpenPGP digital signature

Re: [Bug-wget] Wget reports : This version does not have support for IRIs

[Tim Rühsen]

On 05/31/2017 12:42 PM, Darshit Shah wrote:
> On 31 May 2017 at 10:15, Tim Rühsen <tim.rueh...@gmx.de> wrote:
> 
>> On 05/31/2017 09:48 AM, Cos Chan wrote:
>>> On Tue, May 30, 2017 at 4:03 PM, Darshit Shah <dar...@gmail.com> wrote:
>>>
>>>> This occurs because you are running a version of Wget that was compiled
>>>> without IRI support.
>>>>
>>>> See. the output of `wget --version`, it shows "-iri". However you tried
>> to
>>>> explicitly set the encodings which requires IRI support. Either remove
>> the
>>>> --remote-encoding and --local-encoding switches, or use a version of
>> Wget
>>>> with IRI support compiled in.
>>>>
>>>
>>>
>>> Thanks, May I know how to build wget with iri supporting?
>>
>> We do not directly support Windows builds.
>>
>>
> But this is a BSD Build. The version string says:
> "GNU Wget 1.19 built on freebsd11.0. "

Oh yes, sorry :-)

Then there is a general description in README.checkout.

E.g. if you install from tarball:
./configure
make

Of course you have to install the prerequisites before, the details are
distribution dependent.

With Best Regards, Tim



signature.asc
Description: OpenPGP digital signature

Re: [Bug-wget] wget and srcset tag

[Tim Rühsen]

Fixed in current master (fix release will be 1.19.2).

Thanks for your report and help !


With Best Regards, Tim



On 06/12/2017 06:07 PM, Chris wrote:
> Hi Tim,
> 
> I just created a test page at -
> https://www.anfractuosity.com/files/test2.html
> were I still get the issue.
> 
> The version is 'GNU Wget 1.19.1 built on linux-gnu.'
> 
> cheers
> Chris
> 
> 
> On 12 June 2017 at 15:35, Tim Rühsen <tim.rueh...@gmx.de> wrote:
> 
>> On 06/12/2017 10:27 AM, chris wrote:
>>> Hi Tim,
>>>
>>> Thanks for your reply, I notice the following in the debug logs:
>>>
>>> """
>>> will convert url
>>> http://www.anfractuosity.com/wp-content/uploads/2014/02/fsk.png to local
>>> site_output/fsk.png
>>> will convert url
>>> https://www.anfractuosity.com/wp-content/uploads/2014/02/fsk.png to
>> local
>>> site_output/fsk.png.html
>>> """
>>>
>>> The difference between those URLs seems to be one is https and one isn't.
>>> When I wget those URLs though, both seem to return a .png, with 'Length:
>>> 51068 (50K) [image/png]'.
>>>
>>> So I'm a bit confused why I get the fsk.png.html URL.
>>
>> What version of wget are you using ? (1.19.1 here)
>>
>> I tried some combinations of srcset (with https and http) and your
>> original options. I thought of an issue with redirection (because that's
>> an answer with text/html Content-Type).
>>
>> Could you create a small reproducer page ? e.g. like
>> 
>> > srcset="https://www.anfractuosity.com/wp-content/uploads/2014/02/fsk.png
>> 533w,
>> http://www.anfractuosity.com/wp-content/uploads/2014/02/fsk-266x300.png
>> 266w">
>> 
>>
>> With whatever paths you are using for the .png files.
>> I don't want to download tons of files (limited bandwidth here).
>>
>>> cheers
>>> Chris
>>>
>>> On Mon, Jun 12, 2017 at 9:08 AM, Tim Rühsen <tim.rueh...@gmx.de> wrote:
>>>
>>>> Hi Chris,
>>>>
>>>>
>>>> On 06/11/2017 05:24 PM, chris wrote:
>>>>> Hi,
>>>>>
>>>>> I'm just wondering if I've possibly found a bug, unless I'm just doing
>>>>> something incorrectly (which I assume is more likely).
>>>>>
>>>>> I grab my webpage using 'wget -T1 -t1 -E -k -H -nd -N -p -P site_output
>>>>> https://www.anfractuosity.com/projects/ultrasound-networking/ > note1
>> 2>
>>>>> note2'
>>>>>
>>>>> But i notice the srcset tags in the resulting downloaded files produce
>>>>> 'srcset="fsk.png.html 533w, fsk-266x300.png 266w" sizes="(max-width:
>>>> 533px)
>>>>> 100vw, 533px" />' in the output index.html.
>>>>>
>>>>> On the actual webpage it looks like "srcset="
>>>>> https://www.anfractuosity.com/wp-content/uploads/2014/02/fft.png
>>>> 762w,"
>>>>> no .html extension on the .png.
>>>>
>>>> You requested -E (--adjust-extension) and -k (--convert-links).
>>>> That would change the file name when the server tags the file as
>>>> content-type 'text/html'. You could see that in the debug output
>>>> (options -d or --debug).
>>>>
>>>>>
>>>>> Cheers
>>>>> Chris
>>>>>
>>>>
>>>> With Best Regards, Tim
>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [Bug-wget] [GSoC Update] Week 2

[Tim Rühsen]

On Samstag, 17. Juni 2017 13:53:06 CEST Didik Setiawan wrote:
> On Tue, Jun 13, 2017 at 06:37:27PM +0200, Darshit Shah wrote:
> > >   - http_server_port still hardcoded.
> > 
> > This is important. The port should be a random number. Usually, passing 0
> > in the port number makes the kernel choose an open one for you.
> > 
> > Having a randomised port is important to ensure that multiple runs don't
> > step on each other.
> 
> I still don't know how to accomplish this. Maybe, it just my understanding
> that when I pass 0 in MHD port number, the result is still 0.
> Another approach, when I look into the old code, it generate port number by
> calling wget_tcp_get_local_port(). But, I need to call wget_tcp_init() and
> wget_tcp_listen() respectively in order to get proper result.
> Conclusion, do I need to use existing wget_tcp_get_local_port() to get the
> port, or maybe there is a function in MHD to do that?

All you need is the socket descriptor. How to call getsockname() + 
getnameinfo() to retrieve the port number you see in libwget/net.c/
wget_tcp_get_local_port().

If MHD doesn't have such a function, either try to get the socket descriptor 
or extend MHD with a small function (similar code as in  
wget_tcp_get_local_port).

Regards, Tim

signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] What are the tests testing?

[Tim Rühsen]

Hi Josef,


On 06/12/2017 09:23 AM, Josef Moellers wrote:
> Hello Tim,
> 
> Thanks for the reply.
> 
> On 10.06.2017 13:36, Tim Rühsen wrote:
>> On Freitag, 9. Juni 2017 17:02:15 CEST Josef Moellers wrote:
>>> Hi,
>>
>> Hi Josef,
>>
>>> I'm currently trying to build test suites for openQA.
>>> One of the candidates is wget and, luckily, it already provides quite an
>>> extensive test suite.
>>> I have successfully built an RPM which has all that is needed for the tests.
>>> One test, Test-ftp-iri-fallback.px, fails on SLES12-SP2 and I can't see
>>> why. 
>>
>> Look at tests/Test-ftp-iri-fallback.log, if you can't interpret the content 
>> send it here.
> 
> I cannot find any such file, no *.log" anywhere in the vicinity of the
> tests.

Ok, the .log files just contain the output of each single test when
tested with 'make check'. If you use run-px, copy & pasting from the
console is the right thing to do.

> Ah ... maybe I should have addede that I'm working on a slightly older
> version of wget: 1.14, which we ship with SLES12.

So I compiled 1.14 (git tag v1.14) and used run-px to run the test suite
- but still can't reproduce the problem (Debian unstable here, `locale`
shows all set to 'en_US.UTF-8').

> 
> 227 Entering Passive Mode (127,0,0,1,155,189)
> --> RETR français.txt^M
> 
> 150 Opening ASCII mode data connection.
> Length: 12 (unauthoritative)
> 
>  0K   100% 2.95M=0s
> 
> 226 File retrieval complete. Data connection has been closed.
> 2017-06-09 16:42:53 (2.95 MB/s) - â<80><98>français.txtâ<80><99> saved [12]
> 
> Test failed: file français.txt not downloaded

My out put looks identical except the these last lines:

227 Entering Passive Mode (127,0,0,1,175,123)
--> RETR français.txt

150 Opening ASCII mode data connection.
Length: 12 (unauthoritative)

 0K   100% 2.05M=0s

226 File retrieval complete. Data connection has been closed.
2017-06-12 09:39:27 (2.05 MB/s) - ‘fran\347ais.txt’ saved [12]

Test successful.


I just can guess:
- something with your locale (what does the 'locale' command output ?)
- something with iconv() function

Does the same test fail if you use Wget 1.19.1 ?


And from your 'update 2':
> If I add the option "-O fran${ccedilla_l1}ais.txt" to the cmdline,
> then the test succeeds:

Of course it does ;-) You simply created the expected output file...
(circumventing the real test.)

With Best Regards, Tim



signature.asc
Description: OpenPGP digital signature

Re: [Bug-wget] Shouldn't wget strip leading spaces from a URL?

[Tim Rühsen]

On Mittwoch, 14. Juni 2017 11:49:59 CEST L A Walsh wrote:
> Dale R. Worley wrote:
> >  But of course, no [RFC3986-conforming] URL
> >  contains an embedded space because that's what it
> >  says in RFC 3986, which is "what *defines* what a
> >  URL *is*"[sic; should read "is one definition of
> 
> a URL.
> ---
> Right, just like speed limit signs define
> what the maximum speed is.
> 
> There is the "model" and there is reality.  To believe that
> the model replaces and/or dictates reality is not
> realistic and bordering on some mental pathology.
> 
> I understand what you are saying Dale.  My dad was a lawyer,
> and life would be so much easier if specs, RFCs or other
> models of reality were the only thing we had to pay attention
> to.  But... to do so generally creates various levels of
> discomfort and/or headaches.
> 
> >  Now, someone can provide a string that contains spaces and claim
> >  it's a URL, but it isn't. The question is, What to do with it?  My
> >  preference is to barf and tell the user that what they provided
> >  wasn't a proper URL.
> 
> ---
> I.e.: not doing what you can to give them some output
> that is your _best_ _attempt_ to give them what they wanted
> (excluding dangerous interpretations).
> 
> A friendly user-interface attempts to help the user get
> what they want despite their not asking for it according to
> regulation or with poor syntax or spelling.
> 
> >  Beyond that, one might do some simple tidying up, such as removing
> >  leading and trailing spaces.  That fix, by the way, is known to be
> >  safe, *because a URL can't contain a space*, and so any trailing
> >  space can't actually be part of the URL.
> 
> 
> One might argue that leading and trailing space, since they
> are not "internal" to the URL, aren't really a part of the URL.
> 
> >  It gets uglier when there are invalid characters in the middle of
> >  the URL, because simply deleting them is unlikely to produce the
> >  results the user expected.
> 
> ---
> Yup.  Thus my original post thinking that they should be
> removed since they can't really be part of a URL and as "characters
> non gratis", should be removed before sending them to a remote
> website.

Just in short, there are two 'realities' here
1. The RFC which defines a (part of a) protocol between client and server. 
Clients and servers have to follow this standard, if they deviate they are 
out. This is 'reality' one.

2. User input... well, every (web) client does interpret user input 
differently. But every client tries hard to 'WYGIWYM' (What You Get Is What 
You Mean).
Basically, the problem is solved (or should be) by browsers, so why not do as 
they do ? Well, we can do it similarly but should not forget that 'wget' is a 
'power user' tool while a browser is used by everyone.
People use 'wget' also for very special tasks, e.g. downloading a file which 
name consists of a simple space. Wget would become useless for these people 
(count myself in here) if they couldn't -comfortable- enter a URL with a 
trailing space (wget knows how to escape that, following the RFC).

Example:
  wget 'https://example.com/ '
Should wget download download this space named file or (silently) strip the 
space and download index.html ?
Two answers here, which one has more weight ? Maybe the one that pertains 
disturb backward compatibility !?

> 
> -linda

With Best Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] overwrite

[Tim Rühsen]

Hi Ansgar,


On 06/13/2017 09:40 AM, taschenberggr...@streber24.de wrote:
>Hi,
> 
>a common question online is how to properly force wget to make an
>overwrite of an existing file name.
> 
>The existing options are quite confusing and I am under the impression
>that even taking what works does not imply users understand what they
>do.
> 
>The background of my question is
>https://bugs.winehq.org/show_bug.cgi?id=43100
>and your section in the manual
> "When running Wget without ‘-N’, ‘-nc’, ‘-r’, or ‘-p’, downloading the same 
> file
>  in the same directory will result in the original copy of file being 
> preserved
> and the second copy being named ‘file.1’. If that file is downloaded yet 
> again,
> the third copy will be named ‘file.2’, and so on."

I admit this is pretty confusing. It's historically grown and we won't
change it to not break existing scripts etc.

>If I want to disable that default behaviour what option do I take? I go
>with -nc but I have no clue why I have to take n and c, and online
>recommendations vary from recommending n, c, and nc.
> 
>I am looking for something intuitive like:
>wget --overwrite[1] https://dl.winehq.org/wine-builds/Release.key

There are several possibilities. I personally prefer to make a backup of
existing files before downloading (just in case download stops in the
middle and leaves me with a broken file). That can be used to move an
existing file out of the way:

mv -f Release.key Release.key.bak
wget https://dl.winehq.org/wine-builds/Release.key

Wget can do this without an additional command, even rotates up to an
arbitrary number (see 'man wget'):

wget --backup=3 https://dl.winehq.org/wine-builds/Release.key


But if you still want to replace a file in place (not recommended), you can

wget -O Release.key https://dl.winehq.org/wine-builds/Release.key

or (basically the same)

wget -O- https://dl.winehq.org/wine-builds/Release.key > Release.key


The -nc just switches off saving multiple versions of the file (.1, .2,
...).

Wget is designed not to easily overwrite files resp. to prevent
accidentally overwrites.

>Best,
>Ansgar

With Best Regards, Tim



signature.asc
Description: OpenPGP digital signature

Re: [Bug-wget] HTTPS Python tests fail if localhost resolves to ::1

[Tim Rühsen]

Hi Tomas,

On Freitag, 2. Juni 2017 11:38:42 CEST Tomas Hozza wrote:
> Hi.
> 
> In Fedora 26+ the /etc/hosts lists "localhost" as a domain for both
> 127.0.0.1 and ::1. This makes wget's testsuite to fail during build.
> 
> Failing tests:
> Test--https.py
> Test-pinnedpubkey-der-https.py
> Test-pinnedpubkey-hash-https.py
> Test-pinnedpubkey-pem-https.py
> 
> 
> [root@b28dfb71db2e sources]# cat testenv/Test--https.log
> Python runtime initialized with LC_CTYPE=C (a locale with default ASCII
> encoding), which may cause Unicode compatibility problems. Using C.UTF-8,
> C.utf8, or UTF-8 (if available) as alternative Unicode-compatible locales
> is recommended. Setting --no-config (noconfig) to 1
> Setting --ca-certificate (cacertificate) to
> /sources/testenv/certs/ca-cert.pem DEBUG output created by Wget
> 1.19.1.68-5d4ad on linux-gnu.
> 
> Reading HSTS entries from /sources/testenv/Test--https.py-test/.wget-hsts
> URI encoding = 'ANSI_X3.4-1968'
> converted 'https://127.0.0.1:4/File1' (ANSI_X3.4-1968) ->
> 'https://127.0.0.1:4/File1' (UTF-8) Converted file name 'File1' (UTF-8)
> -> 'File1' (ANSI_X3.4-1968)
> --2017-06-02 09:28:34--  https://127.0.0.1:4/File1
> Loaded CA certificate '/sources/testenv/certs/ca-cert.pem'
> Certificates loaded: 1
> Connecting to 127.0.0.1:4... connected.
> Created socket 3.
> Releasing 0x024ee050 (new refcount 0).
> Deleting unused 0x024ee050.
> The certificate's owner does not match hostname '127.0.0.1'

Are you near latest git ?
testenv/certs/server-cert.pem should be made for '127.0.0.1' and 'localhost' 
(see README) . There is also a recipe for certtool to (re-)generate server-
cert.pem.

Check the cert with 'certtool -i --infile server-cert.pem' (see Subject 
Alternative Name).

> URI encoding = 'ANSI_X3.4-1968'
> converted 'https://127.0.0.1:4/File2' (ANSI_X3.4-1968) ->
> 'https://127.0.0.1:4/File2' (UTF-8) Converted file name 'File2' (UTF-8)
> -> 'File2' (ANSI_X3.4-1968)
> --2017-06-02 09:28:34--  https://127.0.0.1:4/File2
> Connecting to 127.0.0.1:4... connected.
> Created socket 3.
> Releasing 0x0278f1d0 (new refcount 0).
> Deleting unused 0x0278f1d0.
> The certificate's owner does not match hostname '127.0.0.1'
> Running Test Test--https.py
> /sources/src/wget --debug --no-config
> --ca-certificate=/sources/testenv/certs/ca-cert.pem
> https://127.0.0.1:4/File1 https://127.0.0.1:4/File2
> ['/sources/src/wget', '--debug', '--no-config',
> '--ca-certificate=/sources/testenv/certs/ca-cert.pem',
> 'https://127.0.0.1:4/File1', 'https://127.0.0.1:4/File2'] Error:
> Expected file File1 not found..
> Traceback (most recent call last):
>   File "./Test--https.py", line 53, in 
> protocols=Servers
>   File "/sources/testenv/test/http_test.py", line 41, in begin
> self.do_test()
>   File "/sources/testenv/test/base_test.py", line 187, in do_test
> self.post_hook_call()
>   File "/sources/testenv/test/base_test.py", line 206, in post_hook_call
> self.hook_call(self.post_configs, 'Post Test Function')
>   File "/sources/testenv/test/base_test.py", line 196, in hook_call
> conf.find_conf(conf_name)(conf_arg)(self)
>   File "/sources/testenv/conf/expected_files.py", line 54, in __call__
> raise TestFailed('Expected file %s not found.' % file.name)
> exc.test_failed.TestFailed: Expected file File1 not found.
> FAIL Test--https.py (exit status: 1)
> 
> I didn't have time to investigate this thoroughly yet, but I thought I'll
> let you know in case the issue will be obvious to anyone. I suspect that
> there will be a mismatch between the address on which the HTTPS server runs
> and the data in the certificate it uses.
> 
> Regards,
> Tomas

With Best Regards, Tim

signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] http server responding with 416 but file was not transferred completely

[Tim Rühsen]

On 09/14/2017 12:11 PM, Josef Moellers wrote:
> On 14.09.2017 10:12, Tim Rühsen wrote:
>> On 09/14/2017 09:53 AM, Josef Moellers wrote:
>>> Hi,
>>>
>>> We have seen (at least) one server who has
>>> Accept-Ranges: bytes
>>> in his header but, upon receiving a request with
>>> Range: bytes=23068672-
>>> responds with
>>> HTTP/1.1 416 Requested Range Not Satisfiable
>>> although the file was not transferred completely!
>>>
>>> wget then claims that
>>> The file is already fully retrieved; nothing to do.
>>>
>>> E.g.
>>> run
>>>   wget https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO
>>> the, after a couple of MB, abort the transfer and then continue the
>>> download:
>>>   wget --continue
>>> https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO
>>>
>>> Maybe the check in src/http.c:
>>> 3821   if (statcode == HTTP_STATUS_RANGE_NOT_SATISFIABLE
>>> 3822   || (!opt.timestamping && hs->restval > 0 && statcode ==
>>> HTTP_STATUS_OK
>>> 3823   && contrange == 0 && contlen >= 0 && hs->restval >= contlen))
>>>
>>> should be changed and any HTTP_STATUS_RANGE_NOT_SATISFIABLE with an
>>> incomplete file should show something like
>>>
>>> "download continue failed, file incomplete"
>>
>> Well, that would be ok for this special server.
>>
>> Normally 416 together with a server timestamp matching the file's
>> timestamp means that the file is complete (as far as the client can
>> judge from HTTP).
>>
>> IMO, if the server is broken (or misbehaves) then the server should be
>> repaired. Except it is a very common misbehavior. In which case we could
>> consider a work-around on the client side.
>>
> 
> So I humbly propose the attached patch.
> I tried to create a pull request, but got a 403.

Thanks for the patch - I'll test it in the next days.
BTW, we currently work on Wget2 where we have a related issue, if you
like to take a look at it: https://gitlab.com/gnuwget/wget2/issues/278

With Best Regards, Tim



signature.asc
Description: OpenPGP digital signature

Re: [Bug-wget] http server responding with 416 but file was not transferred completely

[Tim Rühsen]

Hi Josef,

pushed your patch - thanks for your contribution.

BTW, the example from your original post doesn't show that wrong server
behavior any more. Maybe the server got fixed meanwhile !?


With Best Regards, Tim



On 09/14/2017 07:47 PM, Josef Moellers wrote:
> On 14.09.2017 17:06, Tim Rühsen wrote:
>> On 09/14/2017 12:11 PM, Josef Moellers wrote:
>>> On 14.09.2017 10:12, Tim Rühsen wrote:
>>>> On 09/14/2017 09:53 AM, Josef Moellers wrote:
>>>>> Hi,
>>>>>
>>>>> We have seen (at least) one server who has
>>>>> Accept-Ranges: bytes
>>>>> in his header but, upon receiving a request with
>>>>> Range: bytes=23068672-
>>>>> responds with
>>>>> HTTP/1.1 416 Requested Range Not Satisfiable
>>>>> although the file was not transferred completely!
>>>>>
>>>>> wget then claims that
>>>>> The file is already fully retrieved; nothing to do.
>>>>>
>>>>> E.g.
>>>>> run
>>>>>   wget https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO
>>>>> the, after a couple of MB, abort the transfer and then continue the
>>>>> download:
>>>>>   wget --continue
>>>>> https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO
>>>>>
>>>>> Maybe the check in src/http.c:
>>>>> 3821   if (statcode == HTTP_STATUS_RANGE_NOT_SATISFIABLE
>>>>> 3822   || (!opt.timestamping && hs->restval > 0 && statcode ==
>>>>> HTTP_STATUS_OK
>>>>> 3823   && contrange == 0 && contlen >= 0 && hs->restval >= 
>>>>> contlen))
>>>>>
>>>>> should be changed and any HTTP_STATUS_RANGE_NOT_SATISFIABLE with an
>>>>> incomplete file should show something like
>>>>>
>>>>> "download continue failed, file incomplete"
>>>>
>>>> Well, that would be ok for this special server.
>>>>
>>>> Normally 416 together with a server timestamp matching the file's
>>>> timestamp means that the file is complete (as far as the client can
>>>> judge from HTTP).
>>>>
>>>> IMO, if the server is broken (or misbehaves) then the server should be
>>>> repaired. Except it is a very common misbehavior. In which case we could
>>>> consider a work-around on the client side.
>>>>
>>>
>>> So I humbly propose the attached patch.
>>> I tried to create a pull request, but got a 403.
>>
>> Thanks for the patch - I'll test it in the next days.
> 
> I have attached a simple webserver that simulates the error:
> when a request with a Range comes in, it replies with 416 and also
> returns an unsanely huge Content-Length. You'll need glib2 and
> microhttpd for it to build.
> 
> I was able to reproduce the issue with this server and check that the
> patch fixes it by causing wget to retry (until --retries is exhausted).
> 
>> BTW, we currently work on Wget2 where we have a related issue, if you
>> like to take a look at it: https://gitlab.com/gnuwget/wget2/issues/278
> 
> I'll do that.
> 
> Josef
> 



signature.asc
Description: OpenPGP digital signature

Re: [Bug-wget] Wget cannot validate https://ftp.gnu.org?

[Tim Rühsen]

Hi Jeffrey,


I can't reproduce your issue on the first try (Debian unstable here).


That means the issuers cert (DST Root CA X3,O=Digital Signature Trust
Co.) is part of the systems's CA cert store.

$ ls -la /etc/ssl/certs/*X3*
lrwxrwxrwx 1 root root 53 27-10-11 09:39:52
/etc/ssl/certs/DST_Root_CA_X3.pem ->
/usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt

But now let's change the the CA directory to a place where no CAs are
stored *and* then add that X3 CA cert from
https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem.txt
(saved as x3.pem).

$ wget --ca-directory=/ --ca-certificate=x3.pem
https://ftp.gnu.org/gnu/libunistring/libunistring-0.9.7.tar.gz
(Download OK)

As a test that we really only load x3.pem:
$ wget --ca-directory=/ --ca-certificate=x3.pem https://google.com
ERROR: The certificate of ‘google.de’ is not trusted.
ERROR: The certificate of ‘google.de’ hasn't got a known issuer.
ERROR: The certificate of ‘google.de’ was signed using an insecure
algorithm.

Caveat: wget has been build with GnuTLS (3.5.15). The OpenSSL (1.1.0f)
code seems not to support --ca-directory !? It succeeds with both the
above tests. While we only actively support GnuTLS, we accept OpenSSL
code patches (if you like to provide one).

With Best Regards, Tim



On 10/15/2017 05:36 AM, Jeffrey Walton wrote:
> So it looks like the behavior below is inherited from OpenSSL:
> 
> $ openssl s_client -connect ftp.gnu.org:443 -servername ftp.gnu.org
> -CAfile ~/.cacert/lets-encrypt-root-x3.pem
> CONNECTED(0003)
> ...
> Verify return code: 2 (unable to get issuer certificate)
> 
> However, OpenSSL also has -partial-chain (thanks to Dave Thompson) so
> we can pin trust at the cross-certified Let's Encrypt X3 root:
> 
> $ openssl s_client -connect ftp.gnu.org:443 -servername ftp.gnu.org
> -CAfile ~/.cacert/lets-encrypt-root-x3.pem -partial_chain
> CONNECTED(0003)
> depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
> verify return:1
> depth=0 CN = ftp.gnu.org
> verify return:1
> ---
> Certificate chain
>  0 s:/CN=ftp.gnu.org
>i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
>  1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
>i:/O=Digital Signature Trust Co./CN=DST Root CA X3
> ---
>...
>Verify return code: 0 (ok)
> ---
> read:errno=0
> 
> Loog thorugh the Wget 1.18 manual
> (https://www.gnu.org/software/wget/manual/wget.html) I don't see a
> similar option.
> 
> So my question is, does Wget allow us to do the same? If so, then how
> do we do it?
> 
> Jeff
> 
> On Sat, Oct 14, 2017 at 6:53 PM, Jeffrey Walton  wrote:
>> I'm having trouble downloading tarballs from ftp.gnu.org using wget.
>>
>> wget --ca-certificate="$HOME/.cacert/lets-encrypt-root-x3.pem"
>> "https://ftp.gnu.org/gnu/libunistring/libunistring-0.9.7.tar.gz; -O
>> libunistring-0.9.7.tar.gz
>> --2017-10-14 17:59:40--
>> https://ftp.gnu.org/gnu/libunistring/libunistring-0.9.7.tar.gz
>> Resolving ftp.gnu.org (ftp.gnu.org)... 208.118.235.20, 2001:4830:134:3::b
>> Connecting to ftp.gnu.org (ftp.gnu.org)|208.118.235.20|:443... connected.
>> ERROR: cannot verify ftp.gnu.org's certificate, issued by 'CN=Let\'s
>> Encrypt Authority X3,O=Let\'s Encrypt,C=US':
>>   unable to get issuer certificate
>> To connect to ftp.gnu.org insecurely, use `--no-check-certificate'.
>>
>> The CA file lets-encrypt-root-x3.pem is provided at
>> https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem.txt,
>> and it is shown below. It has the CA bit set, basic constraints are
>> present, and proper key usage are present. It appears to be a valid ca
>> cert.
>>
>> The thing that looks unusual to me is, the addition of characters in
>> the distinguished name. For example, it appears Wget add a slash to
>> escape the single apostrophe in the common name.
>>
>> Does anyone have an idea what I might be doing wrong? Or if things are
>> working as expected, then how do I use the certificate to download the
>> file using Wget?
>>
>> **
>>
>> $ wget -V
>> GNU Wget 1.19.1 built on solaris2.11.
>>
>> -cares +digest -gpgme +https +ipv6 +iri +large-file -metalink +nls
>> +ntlm +opie -psl +ssl/openssl
>>
>> Wgetrc:
>> /usr/local/etc/wgetrc (system)
>> Locale:
>> /usr/local/share/locale
>> Compile:
>> gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/usr/local/etc/wgetrc"
>> -DLOCALEDIR="/usr/local/share/locale" -I. -I../lib -I../lib
>> -I/usr/local/include -DNDEBUG -D_REENTRANT -I/usr/include/pcre
>> -DNDEBUG -m64
>> Link:
>> gcc -I/usr/include/pcre -DNDEBUG -m64 -m64
>> -Wl,-rpath,/usr/local/lib64 -L/usr/local/lib64 -lpcre -luuid -lidn2
>> /usr/local/lib64/libssl.so /usr/local/lib64/libcrypto.so
>> -R/usr/local/lib64 -ldl -lz -lssl -lcrypto -ldl -lpthread
>> ftp-opie.o openssl.o http-ntlm.o ../lib/libgnu.a -lsocket -lnsl
>> -lnsl -lnsl -lsocket -lsocket /usr/local/lib64/libiconv.so
>> -R/usr/local/lib64 /usr/local/lib64/libunistring.so
>> /usr/local/lib64/libiconv.so 

Re: [Bug-wget] lex compile problem on AIX 7.1

[Tim Rühsen]

On 10/02/2017 10:00 AM, l...@langs.se wrote:
> Hi!
> 
> I get the following error when compiling wget 1.19.1 on AIX 7.1:
> 
> make all-am
> CC connect.o
> CC convert.o
> CC cookies.o
> CC ftp.o
> lex -ocss.c
> 0: Warning: 1285-300 The o flag is not valid.
> 0: Warning: 1285-300 The s flag is not valid.
> 0: Warning: 1285-300 The s flag is not valid.
> 0: Warning: 1285-300 The . flag is not valid.
> 
> Seems the LEX arguments are not valid?
> 
> Any suggestions?

Hi,

some 'lex' versions *must have* a space after -o, some others *must not
have* a space there. We decided not to use a space since this covers
most build environments.

So either add a space in src/Makefile.am (and start over with the
./configure step and following) or use 'flex' what almost everybody does.

If you know a convenient rule for configure.ac, let us know.

Regards, Tim



signature.asc
Description: OpenPGP digital signature