Re: [Bug-wget] [PATCH 21/27] New option --metalink-index to process Metalink application/metalink4+xml

2016-09-30 Thread Matthew White 
Hi,

On Thu, 29 Sep 2016 06:03:01 +0200
Matthew White  wrote:

> +  while ((n = FILE_SYSTEM_PREFIX_LEN (p)))
> +p += n;

Fix attached to silence Coverity's false positive.

Thanks to Tim Ruehsen and Darshit Shah:
http://lists.gnu.org/archive/html/bug-wget/2016-09/msg00170.html
http://lists.gnu.org/archive/html/bug-wget/2016-09/msg00171.html
http://lists.gnu.org/archive/html/bug-wget/2016-09/msg00172.html

Regards,
Matthew

-- 
Matthew White 
>From 0538e791fb5ded49775e3ffb67001108d4e27b0e Mon Sep 17 00:00:00 2001
From: Matthew White 
Date: Fri, 26 Aug 2016 11:19:24 +0200
Subject: [PATCH 21/27] New option --metalink-index to process Metalink
 application/metalink4+xml

* NEWS: Mention the effect of --metalink-index over Metalink
* src/init.c: Add new option metalinkindex (opt.metalink_index),
  initialize to -1
* src/main.c: Add new option metalink-index (--metalink-index=NUMBER)
* src/options.h: Add new option metalink_index (int)
* src/metalink.h: Add declaration of functions fetch_metalink_file(),
  replace_metalink_basename()
* src/metalink.c: Add functions fetch_metalink_file() simple file
  fetch, replace_metalink_basename() replace file basename
* src/metalink.c (retrieve_from_metalink): New. Process Metalink
  application/metalink4+xml of opt.metalink_index ordinal number
* doc/wget.texi: Add new option metalink-index (--metalink-index)
  documentation
* doc/metalink-standard.txt: Updated doc. Add documentation about
  Metalink application/metalink4+xml metaurls download naming system
* doc/metalink-standard.txt: Update Metalink/XML and HTTP examples
* testenv/Makefile.am: Add new files
* testenv/Test-metalink-http-xml.py: New file. Metalink/HTTP automated
  Metalink/XML "application/metalink4+xml" --metalink-index tests
* testenv/Test-metalink-http-xml-trust.py: New file. Metalink/HTTP
  automated Metalink/XML "application/metalink4+xml" --metalink-index
  retrieval with --trust-server-names tests

WARNING: Do not use lib/dirname.c (dir_name) to get the directory
name, it may append a dot '.' character to the directory name.
---
 NEWS|   3 +
 doc/metalink-standard.txt   |  36 +++-
 doc/wget.texi   |   9 +
 src/init.c  |   5 +
 src/main.c  |   3 +
 src/metalink.c  | 348 
 src/metalink.h  |   4 +
 src/options.h   |   1 +
 testenv/Makefile.am |   2 +
 testenv/Test-metalink-http-xml-trust.py | 272 +
 testenv/Test-metalink-http-xml.py   | 272 +
 11 files changed, 947 insertions(+), 8 deletions(-)
 create mode 100755 testenv/Test-metalink-http-xml-trust.py
 create mode 100755 testenv/Test-metalink-http-xml.py

diff --git a/NEWS b/NEWS
index 2153d9a..ca7eaba 100644
--- a/NEWS
+++ b/NEWS
@@ -9,6 +9,9 @@ Please send GNU Wget bug reports to .
 
 * Changes in Wget X.Y.Z
 
+* When processing a Metalink header, --metalink-index= allows
+  to process the header's application/metalink4+xml files.
+
 * When processing a Metalink file, --trust-server-names enables the
   use of the destination file names specified in the Metalink file,
   otherwise a safe destination file name is computed.
diff --git a/doc/metalink-standard.txt b/doc/metalink-standard.txt
index 18acaaa..78709fb 100644
--- a/doc/metalink-standard.txt
+++ b/doc/metalink-standard.txt
@@ -65,10 +65,16 @@ ignored, see '1. Security features'.
 
 
 When --trust-server-names is off, the basename of the --input-metalink
-file, if available, or of the mother URL is trusted.
+file, if available, or of the mother URL is trusted. This trusted name
+is the radix of any subsequent file name.
+
+When a Metalink/HTTP in encountered, any fetched Metalink/XML file has
+its own ordinal number appended as suffix to the trusted name. In this
+case scenario, an unique Metalink/XML file is saved each time applying
+an additional suffix to the currently computed name when necessary.
 
 The files described by a Metalink/XML file will be named sequentially
-applying a suffix to the trusted name.
+applying an additional suffix to the currently trusted/computed name.
 
 3.1.1.2 With --trust-server-names
 =
@@ -91,6 +97,8 @@ found unsafe too, the file is not downloaded.
 4.1 Example files
 =
 
+See [1 #section-1.1].
+
 cat > bugus.meta4 << EOF
 
 
@@ -134,8 +142,9 @@ be informed to the caller of libmetalink's metalink_parse_file().
 Fetched metalink:file elements shall be wrote using the unique "name"
 field as file name [1 #section-4.1.2.1].
 
-A metalink:file url's file name shall not substitute the "name" field,
-see '3. Download file name'.
+A metalink:file url's file name shall not

Re: [Bug-wget] [PATCH 21/27] New option --metalink-index to process Metalink application/metalink4+xml

2016-09-29 Thread Matthew White 
Hi Giuseppe,

On Thu, 29 Sep 2016 22:31:21 +0200
Giuseppe Scrivano  wrote:

> Hi Matthew,
> 
> Matthew White  writes:
> 
> > diff --git a/src/init.c b/src/init.c
> > index 6729c5a..26f3886 100644
> > --- a/src/init.c
> > +++ b/src/init.c
> > @@ -248,6 +248,7 @@ static const struct {
> >{ "login",_user,  cmd_string },/* 
> > deprecated*/
> >{ "maxredirect",  _redirect,  cmd_number },
> >  #ifdef HAVE_METALINK
> > +  { "metalinkindex",_index, cmd_number_inf },
> >{ "metalinkoverhttp", _over_http, cmd_boolean },
> >  #endif
> >{ "method",   ,cmd_string_uppercase },
> > @@ -385,6 +386,10 @@ defaults (void)
> >   bit pattern will be the least of the implementors' worries.  */
> >xzero (opt);
> >  
> > +#ifdef HAVE_METALINK
> > +  opt.metalink_index = -1;
> > +#endif
> > +
> >opt.cookies = true;
> >opt.verbose = -1;
> >opt.ntry = 20;
> > diff --git a/src/main.c b/src/main.c
> > index ac6ee2c..d48e3b2 100644
> > --- a/src/main.c
> > +++ b/src/main.c
> > @@ -354,6 +354,7 @@ static struct cmdline_option option_data[] =
> >  { "rejected-log", 0, OPT_VALUE, "rejectedlog", -1 },
> >  { "max-redirect", 0, OPT_VALUE, "maxredirect", -1 },
> >  #ifdef HAVE_METALINK
> > +{ "metalink-index", 0, OPT_VALUE, "metalinkindex", -1 },
> >  { "metalink-over-http", 0, OPT_BOOLEAN, "metalinkoverhttp", -1 },
> >  #endif
> >  { "method", 0, OPT_VALUE, "method", -1 },
> > @@ -714,6 +715,8 @@ Download:\n"),
> >  N_("\
> > --keep-badhash  keep files with checksum mismatch 
> > (append .badhash)\n"),
> >  N_("\
> > +   --metalink-index=NUMBER Metalink application/metalink4+xml 
> > metaurl ordinal NUMBER\n"),
> 
> "Metalink" should be lower case.

With "Metalink" I meant the standard/protocol Metalink/HTTP, the HTTP should be 
implied by application/metalink4+xml, hence it is missing.

Once the Patch v2 26/27 
http://lists.gnu.org/archive/html/bug-wget/2016-09/msg00157.html is applied, 
"Metalink" may refer to a plain HTTP header too of application/metalink4+xml 
type.

This is why I prefer to use "Metalink" and not "Metalink/HTTP".

> 
> Giuseppe

Regards,
Matthew

-- 
Matthew White 


pgpRM24rXOSHZ.pgp
Description: PGP signature

Re: [Bug-wget] [PATCH 21/27] New option --metalink-index to process Metalink application/metalink4+xml

2016-09-29 Thread Giuseppe Scrivano 
Hi Matthew,

Matthew White  writes:

> diff --git a/src/init.c b/src/init.c
> index 6729c5a..26f3886 100644
> --- a/src/init.c
> +++ b/src/init.c
> @@ -248,6 +248,7 @@ static const struct {
>{ "login",_user,  cmd_string },/* deprecated*/
>{ "maxredirect",  _redirect,  cmd_number },
>  #ifdef HAVE_METALINK
> +  { "metalinkindex",_index, cmd_number_inf },
>{ "metalinkoverhttp", _over_http, cmd_boolean },
>  #endif
>{ "method",   ,cmd_string_uppercase },
> @@ -385,6 +386,10 @@ defaults (void)
>   bit pattern will be the least of the implementors' worries.  */
>xzero (opt);
>  
> +#ifdef HAVE_METALINK
> +  opt.metalink_index = -1;
> +#endif
> +
>opt.cookies = true;
>opt.verbose = -1;
>opt.ntry = 20;
> diff --git a/src/main.c b/src/main.c
> index ac6ee2c..d48e3b2 100644
> --- a/src/main.c
> +++ b/src/main.c
> @@ -354,6 +354,7 @@ static struct cmdline_option option_data[] =
>  { "rejected-log", 0, OPT_VALUE, "rejectedlog", -1 },
>  { "max-redirect", 0, OPT_VALUE, "maxredirect", -1 },
>  #ifdef HAVE_METALINK
> +{ "metalink-index", 0, OPT_VALUE, "metalinkindex", -1 },
>  { "metalink-over-http", 0, OPT_BOOLEAN, "metalinkoverhttp", -1 },
>  #endif
>  { "method", 0, OPT_VALUE, "method", -1 },
> @@ -714,6 +715,8 @@ Download:\n"),
>  N_("\
> --keep-badhash  keep files with checksum mismatch (append 
> .badhash)\n"),
>  N_("\
> +   --metalink-index=NUMBER Metalink application/metalink4+xml 
> metaurl ordinal NUMBER\n"),

"Metalink" should be lower case.

Giuseppe

[Bug-wget] [PATCH 21/27] New option --metalink-index to process Metalink application/metalink4+xml

2016-09-28 Thread Matthew White 
* NEWS: Mention the effect of --metalink-index over Metalink
* src/init.c: Add new option metalinkindex (opt.metalink_index),
  initialize to -1
* src/main.c: Add new option metalink-index (--metalink-index=NUMBER)
* src/options.h: Add new option metalink_index (int)
* src/metalink.h: Add declaration of functions fetch_metalink_file(),
  replace_metalink_basename()
* src/metalink.c: Add functions fetch_metalink_file() simple file
  fetch, replace_metalink_basename() replace file basename
* src/metalink.c (retrieve_from_metalink): New. Process Metalink
  application/metalink4+xml of opt.metalink_index ordinal number
* doc/wget.texi: Add new option metalink-index (--metalink-index)
  documentation
* doc/metalink-standard.txt: Updated doc. Add documentation about
  Metalink application/metalink4+xml metaurls download naming system
* doc/metalink-standard.txt: Update Metalink/XML and HTTP examples
* testenv/Makefile.am: Add new files
* testenv/Test-metalink-http-xml.py: New file. Metalink/HTTP automated
  Metalink/XML "application/metalink4+xml" --metalink-index tests
* testenv/Test-metalink-http-xml-trust.py: New file. Metalink/HTTP
  automated Metalink/XML "application/metalink4+xml" --metalink-index
  retrieval with --trust-server-names tests

WARNING: Do not use lib/dirname.c (dir_name) to get the directory
name, it may append a dot '.' character to the directory name.
---
 NEWS|   3 +
 doc/metalink-standard.txt   |  36 +++-
 doc/wget.texi   |   9 +
 src/init.c  |   5 +
 src/main.c  |   3 +
 src/metalink.c  | 348 
 src/metalink.h  |   4 +
 src/options.h   |   1 +
 testenv/Makefile.am |   2 +
 testenv/Test-metalink-http-xml-trust.py | 272 +
 testenv/Test-metalink-http-xml.py   | 272 +
 11 files changed, 947 insertions(+), 8 deletions(-)
 create mode 100755 testenv/Test-metalink-http-xml-trust.py
 create mode 100755 testenv/Test-metalink-http-xml.py

diff --git a/NEWS b/NEWS
index 2153d9a..ca7eaba 100644
--- a/NEWS
+++ b/NEWS
@@ -9,6 +9,9 @@ Please send GNU Wget bug reports to .
 
 * Changes in Wget X.Y.Z
 
+* When processing a Metalink header, --metalink-index= allows
+  to process the header's application/metalink4+xml files.
+
 * When processing a Metalink file, --trust-server-names enables the
   use of the destination file names specified in the Metalink file,
   otherwise a safe destination file name is computed.
diff --git a/doc/metalink-standard.txt b/doc/metalink-standard.txt
index 18acaaa..78709fb 100644
--- a/doc/metalink-standard.txt
+++ b/doc/metalink-standard.txt
@@ -65,10 +65,16 @@ ignored, see '1. Security features'.
 
 
 When --trust-server-names is off, the basename of the --input-metalink
-file, if available, or of the mother URL is trusted.
+file, if available, or of the mother URL is trusted. This trusted name
+is the radix of any subsequent file name.
+
+When a Metalink/HTTP in encountered, any fetched Metalink/XML file has
+its own ordinal number appended as suffix to the trusted name. In this
+case scenario, an unique Metalink/XML file is saved each time applying
+an additional suffix to the currently computed name when necessary.
 
 The files described by a Metalink/XML file will be named sequentially
-applying a suffix to the trusted name.
+applying an additional suffix to the currently trusted/computed name.
 
 3.1.1.2 With --trust-server-names
 =
@@ -91,6 +97,8 @@ found unsafe too, the file is not downloaded.
 4.1 Example files
 =
 
+See [1 #section-1.1].
+
 cat > bugus.meta4 << EOF
 
 
@@ -134,8 +142,9 @@ be informed to the caller of libmetalink's 
metalink_parse_file().
 Fetched metalink:file elements shall be wrote using the unique "name"
 field as file name [1 #section-4.1.2.1].
 
-A metalink:file url's file name shall not substitute the "name" field,
-see '3. Download file name'.
+A metalink:file url's file name shall not substitute the "name" field.
+
+Security exceptions are explained in '3. Download file name'.
 
 4.5 Multi-Source download
 =
@@ -160,9 +169,19 @@ $ wget --metalink-over-http http://127.0.0.1/dir/file.ext
 5.3 Metalink/HTTP header answer
 ===
 
-Link: http://ftpmirror.gnu.org/bash/bash-4.3-patches/bash43-001; 
rel=duplicate; pref; pri=2
-Link: http://another.url/common_name; rel=duplicate; pref; pri=1
-Digest: SHA-256=7LPf8mSGZ1E+MVVLOtBUzNifzjjjM2fJRZrDooUVN0I=
+See [2 #section-1.1].
+
+Etag: "thvDyvhfIqlvFe+A9MYgxAfm1q5="
+Link: ; rel=duplicate
+Link: ; rel=duplicate
+Link: ; rel=describedby;