Re: [Bug-wget] /usr/bin/env: invalid option -- 'S'

2019-05-30 Thread Tim Rühsen 


On 30.05.19 15:56, Jeffrey Walton wrote:
> On Thu, May 30, 2019 at 9:50 AM Tim Rühsen  wrote:
>>
>> On 30.05.19 12:49, Jeffrey Walton wrote:
>>> On Thu, May 30, 2019 at 6:30 AM Tim Rühsen  wrote:

 Reverted 79be99aff (passes CI). Could you give it try, please ?
>>>
>>> Where can we get that?
>>>
>>> It is not on the mirror (yet?), and https://www.gnu.org/software/wget/
>>> does not list Git or Savannah access.
>>
>> It's on Savannah and Gitlab, branch master,
>> commit b3f86f90cc5b3d3d1c2d6f720818fda9486108ec
> 
> OK, thanks.
> 
> I used PERL5LIB to put teests/ on path for Perl. It looks like at
> least one Debian machine I have is back to the Socket::inet_ntoa
> problems.

Well, using older (Debian) distributions will see that issue. The IPv6
patch has been added to Debian not long ago.

> I'm calling it good.
> 
> The Perl people need to fix Socket::inet_ntoa, and the Debian people
> need to make it available. I'm guessing Debian is the holdup. They
> will leave things broke rather than supplying an update. It is a waste
> of time to file a Debian bug report.

They don't add new functionality to Debian stable, just fixing bugs. And
that IPv6 patch can be considered as 'new feature'. Debian is just not a
rolling release - live with it or change to another distro. All has it's
pros and cons. I live pretty well with Debian unstable (since ~ 2002).
Fix and new features come in pretty fast - except as in times of freeze
(we are currently in (deep ?) freeze).

BTW, the perl people never added the missing IPv6 code. Maybe they
abandoned the module we currently use - and we should use a different
module !?. But I am not into perl on purpose - it's one of the most
stupid programming languages I have ever seen (in short: write once,
read never - hard to maintain). It's simply not 'future save'.

Regards, Tim



signature.asc
Description: OpenPGP digital signature

Re: [Bug-wget] /usr/bin/env: invalid option -- 'S'

2019-05-30 Thread Jeffrey Walton 
On Thu, May 30, 2019 at 9:50 AM Tim Rühsen  wrote:
>
> On 30.05.19 12:49, Jeffrey Walton wrote:
> > On Thu, May 30, 2019 at 6:30 AM Tim Rühsen  wrote:
> >>
> >> Reverted 79be99aff (passes CI). Could you give it try, please ?
> >
> > Where can we get that?
> >
> > It is not on the mirror (yet?), and https://www.gnu.org/software/wget/
> > does not list Git or Savannah access.
>
> It's on Savannah and Gitlab, branch master,
> commit b3f86f90cc5b3d3d1c2d6f720818fda9486108ec

OK, thanks.

I used PERL5LIB to put teests/ on path for Perl. It looks like at
least one Debian machine I have is back to the Socket::inet_ntoa
problems.

I'm calling it good.

The Perl people need to fix Socket::inet_ntoa, and the Debian people
need to make it available. I'm guessing Debian is the holdup. They
will leave things broke rather than supplying an update. It is a waste
of time to file a Debian bug report.

Jeff

Re: [Bug-wget] /usr/bin/env: invalid option -- 'S'

2019-05-30 Thread Tim Rühsen 
On 30.05.19 12:49, Jeffrey Walton wrote:
> On Thu, May 30, 2019 at 6:30 AM Tim Rühsen  wrote:
>>
>> Reverted 79be99aff (passes CI). Could you give it try, please ?
> 
> Where can we get that?
> 
> It is not on the mirror (yet?), and https://www.gnu.org/software/wget/
> does not list Git or Savannah access.

It's on Savannah and Gitlab, branch master,
commit b3f86f90cc5b3d3d1c2d6f720818fda9486108ec

Regards, Tim



signature.asc
Description: OpenPGP digital signature

Re: [Bug-wget] /usr/bin/env: invalid option -- 'S'

2019-05-30 Thread Jeffrey Walton 
On Thu, May 30, 2019 at 7:21 AM Jeffrey Walton  wrote:
>
> On Thu, May 30, 2019 at 6:49 AM Jeffrey Walton  wrote:
> >
> > On Thu, May 30, 2019 at 6:30 AM Tim Rühsen  wrote:
> > >
> > > Reverted 79be99aff (passes CI). Could you give it try, please ?
> >
> > Where can we get that?
> >
> > It is not on the mirror (yet?), and https://www.gnu.org/software/wget/
> > does not list Git or Savannah access.
>
> I performed a sed on all the *.px files. (If this was wrong let me
> know what you need).
>
> for file in $(find "$PWD" -name '*.px')
> do
> sed -e 's|env -S perl -I .|env perl|g' "$file" > "$file.fixed"
> chmod +w "$file"
> mv "$file.fixed" "$file"
> chmod +x "$file"
> chmod -w "$file"
> done
>
> The +x was needed for a few of the files (most were OK). Otherwise got
> permission denied.
>
> After the change, most tests pass. There were some failures that remained:
>
> FAIL: Test-https-pfs.px
> FAIL: Test-https-tlsv1.px
> FAIL: Test-https-tlsv1x.px
> FAIL: Test-https-selfsigned.px
> SKIP: Test-https-weboftrust.px
> FAIL: Test-https-clientcert.px
> FAIL: Test-https-crl.px
> PASS: Test-https-badcerts.px
>
> $ cd ./wget-1.20.3/tests
> $ ./Test-https-pfs.px
> Can't locate WgetFeature.pm in @INC (you may need to install the
> WgetFeature module) (@INC contains: /etc/perl
> /usr/local/lib/x86_64-linux-gnu/perl/5.24.1
> /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24
> /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24
> /usr/share/perl/5.24 /usr/local/lib/site_perl
> /usr/lib/x86_64-linux-gnu/perl-base) at ./Test-https-pfs.px line 6.
> BEGIN failed--compilation aborted at ./Test-https-pfs.px line 6.

I don't know Perl, but it looks like this is the problem:
https://perlmaven.com/cant-locate-in-inc . And this may be the
solution: https://perlmaven.com/how-to-add-a-relative-directory-to-inc
.

I can't help but laugh at the solutions. It never ceases to amazes me
at the f**k'd up convoluted solutions engineers come up with.
Apparently 'use ./WgetFeature.pm;' would have been way too easy.

And apparently including PWD by default would have been way too easy.
We have already accepted the risk of compiling and running
Test-https-pfs.px in /some/user/directory. Using WgetFeature.pm, which
is side-by-side and needed by Test-https-pfs.px, adds no additional
risk.

Instead we have to suffer the shit the Perl people came up with. More
Darwin awards...

Jeff

Re: [Bug-wget] /usr/bin/env: invalid option -- 'S'

2019-05-30 Thread Jeffrey Walton 
On Thu, May 30, 2019 at 6:49 AM Jeffrey Walton  wrote:
>
> On Thu, May 30, 2019 at 6:30 AM Tim Rühsen  wrote:
> >
> > Reverted 79be99aff (passes CI). Could you give it try, please ?
>
> Where can we get that?
>
> It is not on the mirror (yet?), and https://www.gnu.org/software/wget/
> does not list Git or Savannah access.

I performed a sed on all the *.px files. (If this was wrong let me
know what you need).

for file in $(find "$PWD" -name '*.px')
do
sed -e 's|env -S perl -I .|env perl|g' "$file" > "$file.fixed"
chmod +w "$file"
mv "$file.fixed" "$file"
chmod +x "$file"
chmod -w "$file"
done

The +x was needed for a few of the files (most were OK). Otherwise got
permission denied.

After the change, most tests pass. There were some failures that remained:

FAIL: Test-https-pfs.px
FAIL: Test-https-tlsv1.px
FAIL: Test-https-tlsv1x.px
FAIL: Test-https-selfsigned.px
SKIP: Test-https-weboftrust.px
FAIL: Test-https-clientcert.px
FAIL: Test-https-crl.px
PASS: Test-https-badcerts.px

$ cd ./wget-1.20.3/tests
$ ./Test-https-pfs.px
Can't locate WgetFeature.pm in @INC (you may need to install the
WgetFeature module) (@INC contains: /etc/perl
/usr/local/lib/x86_64-linux-gnu/perl/5.24.1
/usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24
/usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24
/usr/share/perl/5.24 /usr/local/lib/site_perl
/usr/lib/x86_64-linux-gnu/perl-base) at ./Test-https-pfs.px line 6.
BEGIN failed--compilation aborted at ./Test-https-pfs.px line 6.

And

$ find . -name WgetFeature.pm
./WgetFeature.pm

Jeff

Re: [Bug-wget] /usr/bin/env: invalid option -- 'S'

2019-05-30 Thread Jeffrey Walton 
On Thu, May 30, 2019 at 6:30 AM Tim Rühsen  wrote:
>
> Reverted 79be99aff (passes CI). Could you give it try, please ?

Where can we get that?

It is not on the mirror (yet?), and https://www.gnu.org/software/wget/
does not list Git or Savannah access.

Jeff

Re: [Bug-wget] /usr/bin/env: invalid option -- 'S'

2019-05-30 Thread Tim Rühsen 
Reverted 79be99aff (passes CI). Could you give it try, please ?

Regards, Tim

On 30.05.19 11:57, Jeffrey Walton wrote:
> On Thu, May 30, 2019 at 5:40 AM Tim Rühsen  wrote:
>>
>> The shebang line here reads
>>
>> #!/usr/bin/env -S perl -I .
>>
>> /usr/bin/env is part of coreutils ans should know '-S', even on Debian
>> stretch.
>>
>> $ /usr/bin/env --version
>> env (GNU coreutils) 8.30
>>
>> @Jeff, what does '/usr/bin/env --help' show ? Or 'man env' ?
> 
> $ /usr/bin/env --version
> env (GNU coreutils) 8.26
> Copyright (C) 2016 Free Software Foundation, Inc.
> ...
> 
> $ /usr/bin/env --help
> Usage: /usr/bin/env [OPTION]... [-] [NAME=VALUE]... [COMMAND [ARG]...]
> Set each NAME to VALUE in the environment and run COMMAND.
> 
> Mandatory arguments to long options are mandatory for short options too.
>   -i, --ignore-environment  start with an empty environment
>   -0, --null   end each output line with NUL, not newline
>   -u, --unset=NAME remove variable from the environment
>   --help display this help and exit
>   --version  output version information and exit
> 
> A mere - implies -i.  If no COMMAND, print the resulting environment.
> 
> GNU coreutils online help: 
> Full documentation at: 
> or available locally via: info '(coreutils) env invocation'
> jwalton@tinkerboard:~/cryptopp$
> 



signature.asc
Description: OpenPGP digital signature

Re: [Bug-wget] /usr/bin/env: invalid option -- 'S'

2019-05-30 Thread Jeffrey Walton 
On Thu, May 30, 2019 at 5:40 AM Tim Rühsen  wrote:
>
> The shebang line here reads
>
> #!/usr/bin/env -S perl -I .
>
> /usr/bin/env is part of coreutils ans should know '-S', even on Debian
> stretch.
>
> $ /usr/bin/env --version
> env (GNU coreutils) 8.30
>
> @Jeff, what does '/usr/bin/env --help' show ? Or 'man env' ?

$ /usr/bin/env --version
env (GNU coreutils) 8.26
Copyright (C) 2016 Free Software Foundation, Inc.
...

$ /usr/bin/env --help
Usage: /usr/bin/env [OPTION]... [-] [NAME=VALUE]... [COMMAND [ARG]...]
Set each NAME to VALUE in the environment and run COMMAND.

Mandatory arguments to long options are mandatory for short options too.
  -i, --ignore-environment  start with an empty environment
  -0, --null   end each output line with NUL, not newline
  -u, --unset=NAME remove variable from the environment
  --help display this help and exit
  --version  output version information and exit

A mere - implies -i.  If no COMMAND, print the resulting environment.

GNU coreutils online help: 
Full documentation at: 
or available locally via: info '(coreutils) env invocation'
jwalton@tinkerboard:~/cryptopp$

Re: [Bug-wget] /usr/bin/env: invalid option -- 'S'

2019-05-30 Thread Tim Rühsen 
OK, just got a PM showing that '-S' is new. Not even known by coreutils
8.28 (used on Ubuntu 18.04). So this is a no-go and needs to be addressed.

Thanks for pointing out.

Regards, Tim

On 30.05.19 11:40, Tim Rühsen wrote:
> The shebang line here reads
> 
> #!/usr/bin/env -S perl -I .
> 
> /usr/bin/env is part of coreutils ans should know '-S', even on Debian
> stretch.
> 
> $ /usr/bin/env --version
> env (GNU coreutils) 8.30
> 
> @Jeff, what does '/usr/bin/env --help' show ? Or 'man env' ?
> 
> Regards, Tim
> 
> On 29.05.19 14:35, Darshit Shah wrote:
>> That's very weird, the shebang line in that file reads:
>>
>> ```
>> #!/usr/bin/env perl
>> ```
>>
>> No options are being passed to env there. I'm going to have to take another
>> look at this later
>>
>> * Jeffrey Walton  [190529 14:21]:
>>> Hi Everyone/Tim,
>>>
>>> Debian 9.9:
>>>
>>> $ lsb_release -a
>>> No LSB modules are available.
>>> Distributor ID: Debian
>>> Description:Debian GNU/Linux 9.9 (stretch)
>>> Release:9.9
>>> Codename:   stretch
>>>
>>> $ make check
>>> ...
>>>
>>> PASS: Test-ftp-pasv-not-supported.px
>>> FAIL: Test-https-pfs.px
>>> FAIL: Test-https-tlsv1.px
>>> FAIL: Test-https-tlsv1x.px
>>> FAIL: Test-https-selfsigned.px
>>> SKIP: Test-https-weboftrust.px
>>> FAIL: Test-https-clientcert.px
>>> FAIL: Test-https-crl.px
>>> PASS: Test-https-badcerts.px
>>>
>>> Trying to run manually:
>>>
>>> $ ./wget-1.20.3/tests/Test-https-pfs.px
>>> /usr/bin/env: invalid option -- 'S'
>>> Try '/usr/bin/env --help' for more information.
>>>
>>> And
>>>
>>> $ /usr/bin/env --version
>>> env (GNU coreutils) 8.26
>>>
>>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [Bug-wget] /usr/bin/env: invalid option -- 'S'

2019-05-30 Thread Tim Rühsen 
The shebang line here reads

#!/usr/bin/env -S perl -I .

/usr/bin/env is part of coreutils ans should know '-S', even on Debian
stretch.

$ /usr/bin/env --version
env (GNU coreutils) 8.30

@Jeff, what does '/usr/bin/env --help' show ? Or 'man env' ?

Regards, Tim

On 29.05.19 14:35, Darshit Shah wrote:
> That's very weird, the shebang line in that file reads:
> 
> ```
> #!/usr/bin/env perl
> ```
> 
> No options are being passed to env there. I'm going to have to take another
> look at this later
> 
> * Jeffrey Walton  [190529 14:21]:
>> Hi Everyone/Tim,
>>
>> Debian 9.9:
>>
>> $ lsb_release -a
>> No LSB modules are available.
>> Distributor ID: Debian
>> Description:Debian GNU/Linux 9.9 (stretch)
>> Release:9.9
>> Codename:   stretch
>>
>> $ make check
>> ...
>>
>> PASS: Test-ftp-pasv-not-supported.px
>> FAIL: Test-https-pfs.px
>> FAIL: Test-https-tlsv1.px
>> FAIL: Test-https-tlsv1x.px
>> FAIL: Test-https-selfsigned.px
>> SKIP: Test-https-weboftrust.px
>> FAIL: Test-https-clientcert.px
>> FAIL: Test-https-crl.px
>> PASS: Test-https-badcerts.px
>>
>> Trying to run manually:
>>
>> $ ./wget-1.20.3/tests/Test-https-pfs.px
>> /usr/bin/env: invalid option -- 'S'
>> Try '/usr/bin/env --help' for more information.
>>
>> And
>>
>> $ /usr/bin/env --version
>> env (GNU coreutils) 8.26
>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [Bug-wget] [PATCH] Disable automatic wget headers.

2019-05-30 Thread Darshit Shah 
Hi,

Sorry for the delay, I've been a little too busy with other things.

I've attached a testcase file containing two failing tests:

1. I disable the "Host" header, then set it manually, then I disable an
   arbitrary header, the "Host" header gets disabled. This is incorrect.

2. I disabled the "Cookie" header, but it was still sent when the server sent a
   Cookie. I assume something similar will happen if you disable the
   "Authorization" header, but the server sends a auth request.

* adham elkarn  [190506 11:26]:
> From: sulfastor 
> 
>   Hi, Thank you again Darshit for your response. The RejectHeaderField 
> rule rejects ANY header
>   of the header field while RejectHeader rejects ONLY the specified full 
> header.
>   Since we wanted to be sure a header field is not sent to the server we 
> wrote this rule.
> 
>   * doc/wget.texi: Added --disable-header documentation.
> * fuzz/wget_options_fuzzer.dict: Update with --disable-header inputs.
> * src/http.c (disabled_header): Checks for disabled headers
>   (request_set_header): Doesn't let header to be set if disabled
>   (gethttp): frees disabled header to let overriding
> * src/init.c (cmd_dis_header), (check_user_disabled_header) added new 
> option disabled_headers.
> * src/main.c: added new option --disable-header, added help 
> description
> * src/options.h: added new option --disable-header
>   * src/utils.h (vec_remove_header)
>   * src/utils.c (vec_remove_header) removes all header instances from 
> vector
> * testenv/Makefile.am: Added new test files
> * testenv/server/http/http_server.py: Added new rule RejectHeaderField
> * testenv/conf/reject_header_field.py: Added new rule 
> RejectHeaderField
> * testenv/README: Added help description for new rule
> * testenv/Test-disable-default-headers.py: Test without using --header
> * testenv/Test-disable-headers-after.py: Test using --header before 
> --disable-header
> * testenv/Test-disable-headers-before.py: Test using --header after 
> --disable-header
> 
> Signed-off-by: sulfastor , adham elkarn 
> 
> ---
>  doc/wget.texi   | 15 +
>  fuzz/wget_options_fuzzer.dict   | 17 ++
>  src/http.c  | 28 -
>  src/init.c  | 48 ---
>  src/main.c  |  3 +
>  src/options.h   |  1 +
>  src/utils.c | 43 +
>  src/utils.h |  1 +
>  testenv/Makefile.am |  3 +
>  testenv/README  |  4 ++
>  testenv/Test-disable-default-headers.py | 74 +++
>  testenv/Test-disable-headers-after.py   | 80 +
>  testenv/Test-disable-headers-before.py  | 78 
>  testenv/conf/reject_header_field.py | 12 
>  testenv/server/http/http_server.py  |  8 +++
>  15 files changed, 405 insertions(+), 10 deletions(-)
>  create mode 100644 testenv/Test-disable-default-headers.py
>  create mode 100644 testenv/Test-disable-headers-after.py
>  create mode 100644 testenv/Test-disable-headers-before.py
>  create mode 100644 testenv/conf/reject_header_field.py
> 
> diff --git a/doc/wget.texi b/doc/wget.texi
> index 7eada2dd..6301cbf6 100644
> --- a/doc/wget.texi
> +++ b/doc/wget.texi
> @@ -1542,6 +1542,21 @@ wget --header="Host: foo.bar" http://localhost/
>  In versions of Wget prior to 1.10 such use of @samp{--header} caused
>  sending of duplicate headers.
>  
> +@cindex disable header, choose
> +@item --disable-header=@var{list}
> +Specify comma-separated header fields to remove from the @sc{http}
> +request.
> +
> +@example
> +@group
> +wget --disable-header='Accept,User-Agent,Authorization' \
> +https://example.com/
> +@end group
> +@end example
> +
> +Specifying a header field with @samp{--header} after disabling it
> +will override it and include it in the @sc{http} request headers.
> +
>  @cindex Content-Encoding, choose
>  @item --compression=@var{type}
>  Choose the type of compression to be used.  Legal values are
> diff --git a/fuzz/wget_options_fuzzer.dict b/fuzz/wget_options_fuzzer.dict
> index 9a2dbd8e..12d54d60 100644
> --- a/fuzz/wget_options_fuzzer.dict
> +++ b/fuzz/wget_options_fuzzer.dict
> @@ -30,6 +30,22 @@
>  "human"
>  "csv"
>  "json"
> +"Authorization"
> +"User-Agent"
> +"Referer"
> +"Cache-Control"
> +"Pragma"
> +"If-Modified-Since"
> +"Range"
> +"Accept"
> +"Accept-Encoding"
> +"Host"
> +"Connection"
> +"Proxy-Connection"
> +"Content-Type"
> +"Content-Length"
> +"Proxy-Authorization"
> +"Cookie"
>  "accept="
>  "accept-regex="
>  "adjust-extension="
> @@ -66,6 +82,7 @@
>  "delete-after="
>  "directories="
>  "directory-prefix="
> +"disable-header="
>  "dns-caching="
>  "dns-timeout="
>  "domains="
> diff --git a/src/http.c b/src/http.c
> inde